Evaluating Complex Initiatives

Srik Gopal, who co-leads social sector consulting company FSG’s Strategic Learning and Evaluation practice, recently blogged at SSIR about evaluating complex social initiatives. He discusses two funders of large-scale, complex initiatives and their attempts to change the way they evaluate success, because as he says:

They are building on the recognition that a traditional approach to evaluation—assessing specific effects of a defined program according to a set of pre-determined outcomes, often in a way that connects those outcomes back to the initiative—is increasingly falling short.

He continues, stating that because complex systems are always changing, the evaluation tools used need to be “adaptive, flexible and iterative.” I’m a big fan of the idea that we need to get beyond simplistic “cause and effect” models of evaluation, which often ignore context. This is particularly true when dealing with complex initiatives and initiatives launched in unstable/changing or multiple environments – it’s important to go beyond whether something works in each location and get into why it does or doesn’t work.

The blog post gives a good feel for the flavor of the work that FSG has done to try to recognize this complexity and deploy tools that can capture the information needed to  create evaluations that can adapt to changing circumstances and capture not just outcomes but relationships and system dynamics, including a chart that shows 3 of their 9 propositions for evaluating complexity alongside what those propositions mean in the real world and some existing tools that can be used in order to capture the needed info.

However, this blog post doesn’t go beyond giving some flavor. It’s worth clicking the link they offer and going through the free signup process to see the full 30+ page report they offer. This includes all 9 propositions, both in snapshot form (page 5) and with full descriptions and case studies. There is also a chart similar to the one in the blog post on pp 31-32 that shows each proposition alongside a brief description of the proposition and some helpful evaluation tools/methods. Beyond the charts and descriptions, there are also 3 case studies.

This looks to be a great resource for those looking to design a thoughtful evaluation process for complex initiatives, as well as a way to re-think what organizations may be hoping to learn and capture for even more simple evaluation practices.

The evolution of strategic thinking in noprofits

Jed Emerson’s twitter feed (@blendedvalue) pointed me to this SSIR article by Barbara Kibbe entitled Five Things Strategy Isn’t, which Emerson describes in the comments as “just a really nice framing of how we got here and where we’re headed.”

The first half of the article is just that – a great summary of how “the dynamic duo of strategy and evaluation” has evolved from flip charts that simply record discussions, to logic models, to SROI and beyond. I’m oversimplifying what Kibbe herself calls an oversimplification, but that’s because it’s worth checking the article itself if you’re not familiar with the evolution of evaluation processes, tools and thinking from the 80s to now. My own nonprofit journey didn’t start until the 90s, but being in small local nonprofits, we certainly used 80s tools/thinking in the 90s (and sometimes still today).

The second half of the article opens by saying that the current debate over the value of strategic philanthropy is healthy, but in order to have that debate we should be careful in defining our terms. And Kibbe starts that definitional discussion by pointing out five times when what we call strategic thinking isn’t actually strategic:

• when it’s fixed – good strategy is never fixed, nor is it a single tool (or a pair of tools). Kibbe quotes Rosabeth Moss Kantor of HBS: “Strategy is a lot like improvisation—setting themes, destinations, directions, and then improvising around those themes.”
• when it’s insulated – context is key, and if strategy and evaluation are not considered together then both will suffer
• when it doesn’t consider people – strategy needs to be flexible enough to deal with the complexities of human beings
• when it’s old, hidden or boring – strategy needs to be current, compelling and shared in order for others to understand it and buy in
• when we are too attached to it – here Kibbe quotes The Independent Sector’s founder John Gardner saying “Philanthropy is the only source of truly flexible capital for the social good” and following up with how important it is for foundations to listen to and support good ideas from the field.

Kibbe closes with her hopes for the future, including this quote that I really liked:

When we look strategy (and evaluation) in the eye, we will see a useful and evolving suite of tools—no more, no less. Practiced well, and in tandem, they will continue to be powerful aids for decision-making but never substitutes for judgment.

 

Performance Measurement vs Impact Evaluation

Bridgespan’s e-newsletter pointed me to an article in the Nonprofit Times on performance measurement. It starts by saying that we often treat performance measurement like a math test, looking to see if we got the right or wrong answer, when we should be treating it like an essay, with multiple drafts each working toward improvement. As the authors say,

the primary question is not “‘are we doing it right?”, but rather “is this useful?”

The article goes on to discuss the difference between performance measurement (a managerial tool for continuous improvement) and impact evaluation (an effectiveness tool most often used by external funders):

Evaluation grows out of social science, an academic discipline with peer-reviewed methodologies. Specifically, impact evaluation often seeks to attribute causation (e.g., ‘did this specific program cause that outcome?’), and often is intended for an audience beyond the nonprofit program being evaluated (policy-makers, funders, academics, practitioners more broadly).  Impact evaluations tend to make more sense for well-established program models, not experiments and start-ups.

 

Performance Measurement, in contrast, is a management discipline, closely related to continuous improvement and organizational learning. It seeks rapid, incremental improvements in programs and their execution, and thereby outcomes for participants.

The article goes on to list some examples of the types of performance measurement, and some concrete steps for moving forward (including logic models development and a pilot evaluation process). The tools themselves were familiar to me, but the framing was not and I found it powerful. I particularly liked the closing sentence, “performance measurement implies a burden of action—not a burden of proof—to learn and improve.”

The article mentioned a couple of additional resources that look at performance measurement and impact evaluation:  “Measurement as Learning,” by Jeri Eckhart-Queenan and Matt Forti and Working Hard & Working Well by David Hunter. The latter is listed as a companion to Mario Morino’s Leap of Reason, which has been sitting on my office bookshelf unread for too long.

Staying pure after selling out

This week’s HBS Working Knowledge newsletter starts off with an interesting proposition: what happens when a well-known socially responsible business is acquired by a multinational?  Professors James E. Austin and Herman B. “Dutch” Leonard discuss their recent research, which examines such acquisitions as Ben & Jerry’s by Unilever, Tom’s of Maine by Colgate and Stonyfield Farms by Dannon.  Their work suggests that it is possible for a company to stay true to its social mission after acquisition, presented in a working paper asking “Can the Virtuous Mouse and the Wealthy Elephant Live Happily Ever After?”  

The discussion touches on some great questions, including why the elephants would want to acquire mice with a conscience and why it could be a good deal for the mouse (they’re not selling out – they’re scaling up).

An excerpt is below, but it’s totally worth your time to read the entire (brief) HBS interview with Austin and Leonard:

Q: How can elephants protect the mouse’s social value and brand integrity?

A: The more effective large companies have recognized that preserving the social icon’s distinctive culture and business approach is essential to preserving its key success factors. Consequently, they retain a large degree of organizational independence so as to prevent “contamination” of the social technology.

This stands in contrast to the common approach in acquisitions to integrate and rationalize the assets into the new owner’s systems, structure, and culture. Some of the specific mechanisms used in successful mouse-elephant agreements include governance structures and processes that give the “mice” review and even veto power over actions by the “elephants” that might jeopardize those elements that are deemed essential to the social values underlying the brand’s integrity.

Retaining the social entrepreneur in the joint venture is highly desirable…

Nonprofit Seeks Acquirer

I’ve been involved in a number of nonprofit mergers and acquisitions (and potential ones that did not happen) at the board and executive staff level. It’s always an interesting process, with a great deal of discussion around why the prospective parties would or would not fit together, potential synergies, etc. However, this is the first time that I’ve seen a call go out from a nonprofit/project seeking a sponsor. Usually, if it is a project seeking sponsorship, there are a few larger organizations that they have in mind, and they approach those folks individually and quietly. Is this more public approach by Social Actions indicative of the social web’s tendency towards openness, transparency and inclusiveness? A recognition that casting a wider net might bring unexpected partners to the table? Simply a commitment to using some of the tools that the organization helps others to utilize? Or just the ignorance of the founders as to “how things are normally done” allowing for fresh thinking on how to do them?

Whatever the answers (most likely a combination of all of the above), the slide show that they’ve put together (below) is great – clear, concise, and gives both the “what’s-in-it-for-me” for the future fiscal sponsor and the “what-we’re-looking-for” from the organization itself. I’ll be very interested to see how this plays out.

Social Actions Is Looking for Fiscal Sponsors in Canada and the United States

View SlideShare presentation (tags: socialactions)

Aspen Institute’s FIELD on Microfinance and Social Enterprise

Just learned of this nifty resource from the NP-Enterprise listserv: a new article called “Social Enterprise and MicroEnterprise: Understanding the Connection” posted at the Aspen Institute’s FIELD homepage (a program of the Aspen Institute focused on microenterprise as an anti-poverty strategy).  It’s a great brief on the basics of social enterprise, social purpose businesses, and micorenterprise.

Also on the homepage are links to a webinar on how microenterprises are using social enterprise to increase sustainability (free registration required, 90 min – unfortunately not able to be downloaded and saved, and I haven’t had time to listen to it in full yet) and a recent FIELD forum (their newsletter) focused on social enterprise.  Both of these resources include a case study of Mountain BizWorks, a project right here in North Carolina (up in Asheville) that I was previously unaware of.  Great reading!

McKinsey on Strategy – First Quarter Newsletter

I think I’ve mentioned before that I find the McKinsey Quarterly’s free resources available by signing up for their newsletter to be top notch.  This quarter’s entries for top strategy articles are no exception.  The descriptions below are taken directly from the e-mail newsletter.  I’ve deleted the ones that are only available to premium (paid) subscribers.  I’ve only read the climate change and Brad Bird/Pixar ones so far, but both were excellent.

Business strategies for climate change April 2008 The value at stake is huge. The winners will be companies that reposition themselves to take advantage of a low-carbon future.

Dissecting global trends: An example from Italy March 2008 Executives should examine the impact of trends on subindustries, segments, categories, and micromarkets before placing their bets.

The promise of prediction markets: A roundtable April 2008 Prediction markets draw together information dispersed across the company, but they face organizational and legal challenges.

Innovation lessons from Pixar: An interview with Oscar-winning director Brad Bird April 2008 His approach to fostering creativity among animators holds powerful lessons for any executive hoping to nurture innovation in teams and organizations.

McKinsey Interviews: Al Gore and More!

Consulting giant McKinsey & Co has a free subscription service offering a subscription to their business publication, the McKinsey Quarterly. While they do have some articles that are “premium” and require you to purchase full access, a startlingly large percentage of what they put out is free. It’s shocking, but they offer more free content than any of the other business knowledge services I subscribe to (HBS, SSIR, Net Impact) with the exception of Origo Inc’s cross-sector news which is totally free and published less frequently. N.B. All of these services require registration, which means giving out your name, e-mail address and company affiliation, but I find that the information I receive is worth far more than this small amount of personal data. Also, links to each of these services can be found at the right side of this page in the “Social Entrepreneurship Resources” list – no time to create individual links today – sorry!

Today I received an e-mail with the McKinsey Quarterly’s top interviews of 2007. I thought that folks might be interested in one with Al Gore and David Blood on investing in sustainability.

The others from the list that I found particularly helpful and/or interesting were:

Strategy’s strategist: An interview with Richard Rumelt
A giant in the field of strategy ruminates on strategic planning, diversification and focus, and the role of the CEO.

Crafting a message that sticks: An interview with Chip Heath
The key to effective communication: make it simple, make it concrete, and make it surprising.

Promoting growth and social progress: An interview with the president of Chile
Michelle Bachelet discusses her views on the roots of political upheaval in Latin America, and the link between economic development and the fight against poverty.

Leading change: An interview with the CEO of Deere & Company
Bob Lane details the steps his company took to engage the whole organization in an operational and cultural transformation.