Finally, an article speaking some truth about impact evaluation! 

Admittedly, the ability to articulate the impact of an intervention has held my rapt attention for the past couple of years.   Yet, contemplating the practicalities of conducting an impact evaluation makes me queasy. To produce reliable data requires resources, time, expertise and a very liberal tolerance for ambiguity.  And some prudent decision-making. 

Impact evaluation is not for everything, everyone, all the time, in every context, for every program or service.

My queasiness lurches towards nausea when I begin to think about how the data from impact evaluation could be hijacked for the benefit of briefing notes and sound bytes itemizing short-term successes and veiled failures. 

My nausea rushes to the surface of my skin with sweats and shivers when I allow my thoughts to venture into a scenario where impact evacuation is done poorly, delivering false results, producing inaccurate analyses, inevitably leading to wasteful policy decisions.   

Thankfully my tummy settled as I read through the article,   Ten Reasons Not to Measure Impact—and What to Do Instead, by Gugerty and Karlan.

However, anyone expecting any nifty little shortcuts for rigorously and reliably measuring the impact of an intervention will be disappointed because the ‘What to Do Instead’ parts of the article seem a bit anemic. 

More importantly the authors encourage readers to slow down and make time to:

1. Clearly articulate the type of question you want the evaluation to answer. 
   • Program monitoring questions: we want to learn how well the intervention works.  Gathering data to determine for whom the intervention is working.
   • Impact evaluation questions: take the learning further by asking why the intervention works.   Gathering data to determine why the intervention works for Group A and not so well for Group B. 
2. Gather monitoring data before conducing an impact evaluation.  Meaning, make sure the implementation of the program is sound before attempting to determine if it has made a difference.
3. Determine if conducting an impact evaluation is actually worthwhile.  Think about the ways in which an impact evaluation will (won’t) inform the intervention’s theory of change.

Impact evaluation has the potential to profoundly influence the choices we make to better serve people in our communities. 

Unfortunately, if we ignore the cautions set out by Gugerty and Karlan, it presently runs the risk of becoming a more complicated, expensive, soul-crushing, labor-intensive  way to measure outputs.