NextBillion - An Initiative of the William Davidson Institute at the University of Michigan

Many BoP ventures are beginning to think hard about metrics, more so than ever before. There has been much written recently about the importance of measuring impact and the large number of new tools and frameworks that have emerged, including many posts here on NextBillion which can be found by searching “metrics.” These posts, along with many other continuing conversations in the field are helping us all navigate the same questions around collecting meaningful metrics.

Measuring outputs and reiterating anecdotes isn’t sufficient in order to truly evaluate an intervention’s impact. Measuring outputs only indicates how well an organization implements what it has set out to do. This is certainly important for donors, who want to know how and where their funds have been used. But what if this implementation is wasting resources, or actually making the situation worse? It’s impossible to tell by only evaluating data on outputs.

Take the example of mosquito bed nets. By measuring number of nets distributed by an organization to a population (an output), you don’t know how or if the population is using the nets. The nets could still be in their packaging or being used as fishing net. I’m not saying ventures shouldn’t measure outputs, but there is no point measuring them unless they also consider the outcomes – the broader impact on their target population such as the influences of bed nets on locals income, education, health and relationships.

Most experts seem to agree that it makes intuitive sense to measure outcomes. Well why aren’t more organizations measuring outcomes? Based on observations and conversations we’ve had with numerous organizations, I’ve come up with three different reasons that highlight the underlying constraints:

1. Confusion – Many organizations don’t know where or how to start. This is mostly a result of lack of time dedicated to impact assessment at an organization. There is plenty of information out there on how to measure outcomes.
2. Uncertainty – Organizations don’t know what their outcomes will be, and they are nervous about hearing anything other than positive news. However, you can’t improve without understanding your weak points! The development community as a whole needs to emphasize and reward learning from mistakes. Ultimately, this will benefit everyone.
3. Apathy – Some program managers won’t do it until they are forced, and instead will continue using anecdotes and outputs in attempts to show their influence in alleviating poverty. As the attention around impact investing grows, soon these organizations will be “forced” by donors to measure outcomes. As more take the step to measuring outcomes, it will soon become the norm.

It will be interesting to see how these organizations evolve their thinking and action as the pressure to measure metrics increases both from funders and other organizations doing so. At the same time even some of the organizations currently measuring outcomes end up frustrated and without quality data due to poor methodology. At WDI we have developed a robust and straightforward data collection process for collecting impact data at the project level. My next post will focus on recapping outcomes from our upcoming BoP Impact Assessment Workshop which will share our framework and data collection process with participants.

I look forward to exploring the area of metrics, sharing our latest thinking at WDI, and continuing the conversation around metrics with you all.