Evaluating Complex Initiatives

Srik Gopal, who co-leads social sector consulting company FSG’s Strategic Learning and Evaluation practice, recently blogged at SSIR about evaluating complex social initiatives. He discusses two funders of large-scale, complex initiatives and their attempts to change the way they evaluate success, because as he says:

They are building on the recognition that a traditional approach to evaluation—assessing specific effects of a defined program according to a set of pre-determined outcomes, often in a way that connects those outcomes back to the initiative—is increasingly falling short.

He continues, stating that because complex systems are always changing, the evaluation tools used need to be “adaptive, flexible and iterative.” I’m a big fan of the idea that we need to get beyond simplistic “cause and effect” models of evaluation, which often ignore context. This is particularly true when dealing with complex initiatives and initiatives launched in unstable/changing or multiple environments – it’s important to go beyond whether something works in each location and get into why it does or doesn’t work.

The blog post gives a good feel for the flavor of the work that FSG has done to try to recognize this complexity and deploy tools that can capture the information needed to  create evaluations that can adapt to changing circumstances and capture not just outcomes but relationships and system dynamics, including a chart that shows 3 of their 9 propositions for evaluating complexity alongside what those propositions mean in the real world and some existing tools that can be used in order to capture the needed info.

However, this blog post doesn’t go beyond giving some flavor. It’s worth clicking the link they offer and going through the free signup process to see the full 30+ page report they offer. This includes all 9 propositions, both in snapshot form (page 5) and with full descriptions and case studies. There is also a chart similar to the one in the blog post on pp 31-32 that shows each proposition alongside a brief description of the proposition and some helpful evaluation tools/methods. Beyond the charts and descriptions, there are also 3 case studies.

This looks to be a great resource for those looking to design a thoughtful evaluation process for complex initiatives, as well as a way to re-think what organizations may be hoping to learn and capture for even more simple evaluation practices.

One-Two Punch: limited edition and helps a good cause

I grew up collecting things. As a teenager, I worked in a comic book and baseball card store. In college, I became a DJ at the radio station, and began seriously collecting records/music. The “limited edition” has always held a great allure for me. Not only does this increase the likelihood of an item increasing in value, there is also a feeling of being special – in a world of mass production, I’m one of only a limited number of people to own a specific artifact.

Of course, the value proposition does not always hold up. Many of the “limited edition” comic  books I bought in the early 90s have little to no value today, as the comics boom that happened during those years meant that many limited editions were produced in quantities much greater than any of the popular mainstream editions historically or since. And even when editions truly are limited, there needs to be some demand for the items for them to rise in value.

There is also the tricky notion of the collector mentality. When I was around age 12 or 13, I wanted a GI Joe plane because I had heard that toys that stayed in their packaging were a great collectible investment. Plus, this COBRA plane looked really cool. I was thrilled when I unwrapped it and effusively thanked my aunt. However, I was horrified when she suggested we open it and put it together, explaining how it had to stay in its original shrinkwrap to retain its value. That was my mom’s turn to be horrified, and she told me in no uncertain terms that I was going to open it up and play with it and stop acting so ungrateful, that toy cost a lot of money and it was bought to be played with.

I didn’t understand my mom’s point of view at the time, but by my college years, when I was collecting records, I started to get it. I tried to rid myself of the attitude and habits of “collector scum” – those who would buy up copies of pricey limited edition stuff and then never listen to it, just so they could sell it at a markup at a record fair. These people didn’t seem like real fans – music was made to be listened to, not stocked away as some sort of investment. Also, I had become somewhat disillusioned with the comics world and savvy enough to realize that many of these limited editions were simply cash grabs by megacorporations hoping to exploit fans by making them buy 5 copies of the same thing instead of one, due to “limited edition variant covers.” CD releases with bonus/hidden tracks struck that same chord with me, and I vowed not to be “collector scum” but rather a *real* fan.

Over the years, I’ve gone back and forth on the whole collector mentality – things are never as black and white as they seem in high school and college. Over the last several years, I’ve made an effort to put my money where my values are, which has primarily meant spending money on new books and records rather than used, and supporting businesses that I would like to see thriving, but has also meant paying a little more attention (and sometimes paying a little more $$) for purchases that have some component of social good (or in some cases are the lesser of all evils).

I recently bought a couple things that I consider to be big-ticket splurges, and in each case I looked at the price, and decided that it wasn’t worth it – this was something that I didn’t *need*. But then I saw that there was a charitable component to the purchase, which along with each item being a limited edition pushed me over the edge and let me self-justify the purchase decision.

The first purchase was a pair of prescription sunglasses from Warby Parker. I’ve bought my glasses from Warby Parker for several years now. When I first needed glasses, I went with the cheapest, simplest pair at the mall (same place that I got my eye exam), but over the years I was convinced that glasses, like shoes, get daily wear and you can justify spending a little more to get something well-styled and somewhat unique. Initially I was a big fan of SEE eyewear, discovered on a trip to Nashville, but when it came time for a new pair of glasses it was difficult to order online and I wasn’t making a multi-state trip just for glasses. I looked at a couple other online options and could have saved more money with Zenni, but I liked the design aesthetic at Warby Parker better and knowing that my purchase would provide a free pair of glasses to people in need via VisionSpring pushed me over the top to become a loyal customer.

Still, I tend to wear glasses a little too long, letting my prescription expire at times. And the prescription sunglasses, while a necessity due to the need to wear them while driving (arguably the time when glasses are most important), are an extra $50 at Warby Parker. When they announced the limited edition Kidd sunglasses, which support 826 National / 826 Valencia (a literacy/writing nonprofit for children that I’ve long admired), I struggled with the purchase. The glasses were cool-looking, but I already had a pair of sunglasses that I liked even a little better. Though that pair had a scratch that might be causing headaches. In the end, the combination of knowing that I was helping a good cause and that there would only be 825 other people with this pair of sunglasses pushed me over the edge, and I bought a pair.

The other recent purchase was a limited edition copy of The David Foster Wallace Reader. David Foster Wallace has been one of my favorite authors since 1996ish, and when his archives were released and I saw how heavily he annotated his own books, it was freeing in a way I never imagined – I began underlining and dog-earing my own books realizing that to keep myself from doing so was keeping myself from fully enjoying them. My attempts to keep spines uncracked was a holdover from my collector days. And so when I first heard about the release of this book, I figured that it was a waste of money, I had copies of all the uncollected work anyway, I’d wait until it came out in paperback and then find a copy on sale. Hachette didn’t need my money – I’d spend it on other authors like Roxane Gay and Amelia Gray, who could use the boost in sales a bit more.

But then I heard about the limited edition. At $300, it was 10x as expensive as the version I considered way too pricey. But it’s limited to 0nly 250 copies. And the painting used for the cover of the book, by Karen Green, would be cut into 250 pieces with one piece each included in the limited edition copies. Then, to top it all off, $50 from each book would go to a youth poetry/literacy/creative writing nonprofit. Once I went to amazon and saw that it was being sold for $200 instead of $300, I barely hesitated before completing my purchase. Don’t get me wrong, the sale *definitely* helped, but the one-two punch of a limited edition and a significant portion of my purchase supporting a cause  that I believe in really led to my pulling the trigger.

I’m rarely any company’s target audience. I’m too cheap, and my tastes are a bit too particular in most cases. However, for those companies who are focusing on the long tail consumers, it is worth noting that the combination of a limited edition with a good cause can be really compelling. The limited edition appeals to the collector impulse, the desire to be unique and own something that very few others have. Combining it with a cause helps to ameliorate the guilt of spending on oneself, particularly for something that we clearly don’t *need*.

I’m sure there’s something more to say about the guilt collectors feel (at some point, the collection itself can become burdensome and akin to hoarding) and how charitable partnerships can help consumers overcome this guilt. For me, I’m just thrilled to have bought myself a present that brings me joy whenever I look at it, and to feel secure that I won’t ever have to take it out of the protective cellophane wrapper (and if I do, it will be my decision).

Link

Gallons per One Hundred Thousand Miles

Gallons per One Hundred Thousand Miles

Back in 2008, I blogged about Fuqua professor Rick Larrick’s research on flipping the way we talk about fuel efficiency from the familiar “miles per gallon” to “gallons used per 100 miles driven” and later that year I followed up with a post on some of the traction that idea had gained and the GPM calculator they had posted online (still up and running now).

“Giving the gas cost over the lifetime of the vehicle seems to give people a better understanding of its fuel efficiency,” Larrick said. “The current metrics used don’t help people to understand the true value of a fuel-efficient vehicle.”  -from the latest research

Last week I got a Fuqua email update that featured Larrick’s latest work on GPM, including the info that in 2013 the EPA had added “gallons per 100 miles driven” to its fuel economy labels. Larrick’s latest work shows that this label may soon warrant an update – apparently consumers prefer a fuel-efficient vehicle when they are shown the statistic of gallons per 100,000 miles driven, even when the expected cost savings in fuel efficiency does not make up for the higher cost of the fuel efficient vehicle.

“Consumers place a lot more weight on fuel efficiency when this information is given to them in terms of gas cost over 100,000 miles,” Camilleri said. “The amazing thing is that this greater weight persists even when the efficient vehicle doesn’t necessarily pay for itself in savings, which makes sense for the consumer who also cares about the environment.” 

I always find it fascinating when a change in language is able to impact cultural values and individual behavior, and the idea that this is so powerful that environmentalism could trump cost in a purchase decision is remarkable. However, I’m also a bit skeptical, as survey answers are not always a reliable indicator of actual consumer behavior. Still, even if this only works for cases when fuel efficiency is also cost efficient, this is a great way to make an impact by simply changing the language we use.

Evidence +/vs Innovation

Paul Carttar has an interesting post up over at the Bridgespan Group’s blog entitled Evidence and Innovation – Friend or Foe?

Carttar frames the discussion with an anecdote:

…during a recent discussion about what makes a nonprofit organization “high-performance.” One participant nominated innovation as a critical factor. To my astonishment, this stirred an impassioned dissent from another participant, a recognized and vocal proponent of evidence and accountability, who argued that in the nonprofit world the word “innovation” typically implies the generation of exciting new ideas, apparently free of any bothersome, killjoy demands for validation of merit.

Carttar talks about how this is nothing new – that during his time running the white house Social Innovation Fund, he often heard complaints that evaluation stifles innovation. And I’ve certainly seen numerous innovative approaches shut down or left un(der)funded because they’re not “evidence based” – but Carttar makes two important distinctions: 1) Innovation is less about “something new” and more about “something better,” and 2) ” hard evidence of relative performance is the most legitimate, productive way to determine what actually is better.”

Carttar then goes on to discuss the varying types of “hard evidence,” clearly stating that not all types are appropriate for all efforts. He makes the crucial distinction between startup and mid-stage enterprises, and what type of evaluation and “evidence” makes sense for each.

At its best, evidence serves as innovation’s good friend by stimulating continued improvement and providing potential beneficiaries, funders and other stakeholders an objective basis for determining whom to turn to and to support. In this way, evidence can not only “cull the herd” but actually propel the growth and scaling of the best innovations, enabling them over time to become the prevailing practice. In fact, that’s the hopeful theory underlying the SIF.

To be sure, there are plenty of opportunities for conflict between evidence and innovation, which must be diligently managed. Potential funders may demand unrealistically rigorous standards of evidence to assess relatively immature, still-evolving programs—potentially stifling the development of promising solutions. Ill-timed, poorly executed, or inaccurately interpreted evaluation studies can also prematurely choke off development. Or backers of a program with a robust empirical basis may hesitate to invest in further improvements (that is, continued innovation) for fear of undermining the program’s evidentiary support and perceived competitive advantage.

The discussion continues in the comments, and is worth reading for its thoughtfulness and appreciation of nuance.

John Kotter on organizational design

A review of John Kotter’s new book Accelerate: Building Strategic Agility for a Faster-Moving World in HBS Working Knowledge offers both a summary of the book’s key principles and an excerpt from the book itself. The concept builds on Kotter’s earlier work that focused on adding speed and agility to large businesses, and advocates for creating an organization that has “two operating systems” – one for everyday business and a smaller, agile system that “sits alongside to focus on the opportunities and demands of the future.”

Under a dual operating system, all processes and activities that involve what a company already knows how to do stay on the regular, hierarchical side of the company. High-stakes initiatives that involve change, speed, innovation or agility, go to the new agile network.

Reviewer Kim Girard continues, emphasizing that Kotter is not looking to abandon the traditional hierarchical model, but enhance it:

A dual operating system is a nod to what Kotter believes is some of the most interesting management thinking of the past few decades, from Michael Porter’s “wakeup call telling us that organizations need to pay attention to strategy much more explicitly and frequently,” to Clayton Christensen’s insights about how poorly companies handle the technological discontinuities inherent in a faster moving world. Kotter also credits recent work by Nobel Laureate Daniel Kahneman, who describes the brain as two coordinated systems, one more emotional, the other more rational.

 

In a typical organization—from the federal government to a pharmaceutical giant—a hierarchical operational structure meets daily demands through clear reporting relationships and responsibilities, Kotter writes. This structure minimizes risk, keeping people in boxes and silos, sorting work into departments, product divisions, and regions. Trouble is, managers in hierarchical organizations don’t promote or reward risk and innovation—they rely on routine, and turn to the same trusted people to run key initiatives.

Girard goes on to discuss Kotter’s 5 key principles for the dual operating systems, which ensure that the system works as envisioned (an “enhanced heirarcy” that focuses on leadership and innovation) and 8 accelerators that help managment tackle big opportunities for change. The accelerators are strikingly familiar, as many seem to have been adapted from Charles Duhigg’s The Power of Habit. Overall, an interesting approach to organizational design.

Jazz as business metaphor

As a fan of avant garde jazz and an MBA who reads a lot of professional develpment stuff, I’m often frustrated by articles that use jazz as a metaphor or analogy to impart some advice about the importance of improvisation in business. However, I liked several things in Michael Blanding’s recent review of Michael Wheeler’s book on negotiation: The Art of Negotiation.

Blanding opens the review with a quote from Eisenhower that I really like:

There’s a saying in the military: “Plans go out the window at the first contact with the enemy.” Even General Dwight Eisenhower—who oversaw the most ambitious military invasion in modern history—said, “Plans are worthless.” But he added an important caveat: “Planning is everything.”

The review covers many aspects of negotiation, from the aforementioned planning/preparedness, to dealing with uncertainty, to listening, to strategy, to mindfulness and more. Quotes from great negotiators are laid side by side with quotes from recognized artists/improvisers. Near the end of the review, Blanding says:

Being centered emotionally is essential to negotiation success. Wheeler says it requires being comfortable with seemingly contradictory feelings—for example, being simultaneously calm and alert—and approaching negotiation as an ongoing process of discovery about the situation, your counterpart, and perhaps even yourself.

And as long as I’m discussing this, I might as well provide a link to my friend Scott’s article a few years ago, talking about what jazz soloists know about creative collaboration.

 

The Green Prince

This past weekend I saw several films at the FullFrame Documentary Film Festival. The ones that I liked most were The Case of The Three Sided Dream, about Rahsaan Roland Kirk (who I’ve been a fan of for years) and The Chaperone, an animated short that was surprisingly funny.

The film that has stuck with me the most, though, was The Green Prince. It centers on the relationship between the son of one of the leaders of Hamas and his handler in the Shin Bet (Israeli secret service). I’ve seen several documentaries on Israel/Palestine, but this one really stood out.

Rather than focusing on the conflict and its history, or one particular series of events, or cause/effect, this was really a story about the relationship between two men who know that they are on different sides but have found some common ground. They’re both articulate and conflicted, and I was never quite sure where the story was going to go.

The film was really remarkable for its lack of moralizing. So many documentaries, particularly ones that deal with politics and war, keep hammering home a directorial viewpoint. The Green Prince seemed to keep trying to do the opposite – to avoid black and white depictions of right and wrong and instead to emphasize the shades of gray in between. It touched on issues of culture, personal identity, choice, responsibility, and more, but always with a very light touch.

I really appreciated the way this story was told, and since the film did well at Sundance, I’m guessing it will have decent distribution. I’d definitely recommend keeping an eye out for it.