NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

’Impact assessment’ has become a mantra across much of the social entrepreneurship and impact investing scene. Are we measuring enough? The right thing? In the right way? Or, maybe… are we measuring too much?

We all want ’impact’, and ’measurement’ sounds like a good way to get it. And there is certainly are a lot of good reasons for impact measurement, some of which led Ted London to postulate that “Managers of business ventures that work with the world’s poor need more than ?nancials and feel-good stories to measure success. They need to know exactly who’s bene?ting and how” (London)

As a consequence, conferences, courses and tools for impact measurement abound – and the topic has received favourable treatment here on NextBillion. But does it all add up? In a provocative roundup of USAID’s history, Andrew Natsios has coined the term ’Obsessive Measurement Disorder’ (OMD) for exactly that phenomenon – measuring too much. While he is mainly concerned with the wondrous world of “Official Development Aid” (ODA), his insights could hold implications for the alternative world of social entrepreneurship and inclusive market development.

Building on his ideas, this post won’t look ’why’ we should measure impact, but ’if’ we should do so. I will have a look at possible concerns about impact assessment, and ask for some alternatives.

1: Can we measure what matters?

A first concern stems from what’s measurable and what not. Following Natsios, the more transformational an outcome, the more difficult it is to measure. Transformation is driven by broad and lasting change – which takes time, involves a lot of soft issues, non-linear dynamics, arbitrary causal changes and often cannot be attributed to individual actor strategies. All these factors make measuring transformational change more difficult then non-transformational change.

Examples abound:

Aids treatment through external experts and imported drugs (low transformation) is easier to measure then aids prevention through a domestic, well-developed health system (high transformation), as a consequence, prevention tends to be underfunded.

Selling individual solar lamps (low transformation) is more easily measured then a roll-out of more complex to manage micro-grids or community-based solutions (higher transformation): And while Monitor Group judged the market for micro-grids at ten-times the size compared to the market for households solutions (for the Indian BoP), there’s a strong bias of solar lamp / solar home systems at conference and design exhibitions.

Last, giving stuff away to poor people for free on a limited scale (low transformation) is easier to measure then building up sound framework conditions for the development of a market economy (high transformation) – another factor that may explain why a “free” tactic comes out better in recent randomised control experiments?

This measurement paradox can have problematic implications: Funders with a strong orientation on measurable outcome will channel funds to highly visible, highly measurable activities – with transformational activities being underfunded. And entrepreneurs that want ’measureable’ impact will be drawn to ’measurable’ activities, not to the ones that drive real social change. In practice, this can enforce that concern that Ashoka’s Felix Oldenburg voiced about the funding of organisations rather the impact, as the effects of an investment are best measured at the organisational level.

2: It’s the (transaction) costs, stupid

Secondly, impact measurement has costs. These costs primarily occur for staff time and data management systems. But more importantly, they could occur in driving project complexity, and by setting adverse incentives that stem from measurement challenges (see point 1).

As Klaus Zimmerman, from responsAbility, stated succinctly on a panel during SOCAP Europe, it’s high transaction costs that lie behind many challenges in market development. Social innovators should work towards bringing costs down – with impact measurement being another ’poverty penalty’ (on top of the low transaction sums, lack of infrastructure, low-density of populations in rural areas, lacking judicial reliability, risk of corruption etc.). All these costs are not borne, or not to this degree, by the rich – will adding impact measurement on top really help to build inclusive business models?

3: Input, throughput, output, impact?

Impact measurement is concerned with what is normally the result of a long and complex intervention, involving multiple groups at different stages. And even if we would be able to measure that impact in a satisfactory manner – how would the results be interpreted by the actors involved in this process?

Research from ’mainstream’ management shows that a focus on measuring outcomes can be ineffective, or even detrimental, to achieving these outcomes. The same might be true in social entrepreneurship and inclusive market efforts. If an “impact measurement” identifies a challenge, the root of this problem might be the reason that prevents tackling it. For example, if the missing impact stems from a lack of sensibility or commitment of staff members, will these staff members that be willing and able to interpret these results, and take effective action? Or shouldn’t we focus more on input (selecting the right staff) and throughput (providing effective coaching and oversight) in the first place? Instead of measuring impact, which always happens ex-post?

Whether results from impact measurement that require action and change in the company might stem from lack of ability or conflicts of interest, there will often be a counter-activity to either change the results themselves or their interpretation, putting the stated goals behind impact measurement at risk.

4: Towards a neo-paternalist agenda?

Last, there’s a more fundamental concern behind the un-easiness with measurement. Entrepreneurship, BoP, inclusive markets – these concepts were meant to free the poor from well-meaning, paternalistic efforts, and allow them to bring in their own creativity. Give them a credit, and some training – and let them decide what they want to spend their money on! That’s a far step away from philanthropic, in which the donors’ perspective (including all sorts of errors and biases) determine to a large degree what’s happening.

Impact measurement can bring back this neo-paternalist agenda through the backdoor. When they get access to glasses – will they use them to watch soap operas (bad!), or to earn a living by an income-generating activity (good!)? When they get credit – will they spend it on consumption (bad!) or as an investment (good!)? When the poor get access to a fridge – will they store medicine (good!) or beer (bad!)*? While all of these are valid concerns, there’s a large amount of external judgement about what’s good and what’s bad in any given situation – something that might best be judged by the poor themselves.

This idea has implications for the mission businesses targeting the poor. Providing the poor with access to product and services is a very different idea then creating impact in their lives, and one that is more in line with granting the poor autonomy. Such an approach would also be more aligned with Amartya Sen’s capability approach: ’Access’ to is more akin to his focus on capabilities then realised changes in living conditions. On a more pragmatic level, access can often be translated to developing “markets”, for example for financial products, which allow the poor to have a choice among several competing products, that conform to certain minimum quality standards as defined by regulatory bodies.

Fixing impact measurement…

Responsible organisations that use impact measurement as an integrated part of their strategy might very well benefit from it. The concerns above, even if taken to the extreme, suggest that most organisations will find it difficult to do so, and that the benefits of impact measurement might not be those widely suggested in the literature.

However, depending on the funding sources used, creating accountability through impact measurement might seem unavoidable. Public bodies need to be “accountable” to taxpayers, private donors and foundations pursue a certain agenda and want to make sure that their money is well-spent. Still, if the above is true, these organisations would be better off by not pursuing an impact measurement approach, but some kind of alternative mechanisms.

… or going for alternatives?

What are the alternatives? Taking into account the problems above, here are some suggestions on how we can move from measurement to management, and from impact (whatever that may be) to sound processes:

– Putting the right people in charge: It’s people that make decisions that determine impact in the end, from front-end staff to an organisation’s leadership. Compare an impact investment fund started and is run by one committed woman and a development aid organisation run by seven politically selected men. What does that tell us about each organisations’ attention and sensibility to gender issues and empowerment?

– Wrap impact into the business model: If a company’s business model relies on their (formerly) poor sales agents earning a decent salary, the company should (and will) take care that the agents indeed earn a decent salary (risking high turnover otherwise). Wrapping impact into the business model in as many ways possible will help to make sure that impact measurement doesn’t become a detached exercise, but a core practice.

– Charging the poor: Some organisations rely on resources from low-income clients (that purchase their products), others on donations and funding from government bodies residing in far-away global cities. As organisations normally survive by focusing on their key customers, these two models might have strong implications for organisations depending on them.

– Learning from stories: Better stories can be a great source of learning, if organisations can resist the urge to solely tell “success stories”, and allow those stories from their staff and their customers that feature the disappointment, frustration and set-backs that occur in all inclusive market efforts, due to the size of complexity of the underlying challenges.

I’d guess there are a lot of ideas out there on how to fix, or move beyond impact measurement?

* Your humble blog author finds his fridge an incredibly useful device for cooling beer, a beverage that he holds in great esteem as a good German citizen and inhabitant. He would find it irritating to prevent others, the “poor”, from doing the same, and would resist any consultant conducting an impact assessment on his fridge.

Read more?

London, Ted. 2009. “Making Better Investments At the Base of the Pyramid.”. Harvard Business Review 87:106-13. http://hbr.org/2009/05/making-better-investments-at-the-base-of-the-pyramid/es

Natsios, Andrew. 2010. The Clash of the Counter-Bureaucracy and Development. Center for Global Development. http://www.cgdev.org/content/publications/detail/1424271

Please like NextBillion on Facebook and follow us on Twitter.