That Queasy Feeling ….

by

Finally, an article speaking some truth about impact evaluation! 

Admittedly, the ability to articulate the impact of an intervention has held my rapt attention for the past couple of years.   Yet, contemplating the practicalities of conducting an impact evaluation makes me queasy. To produce reliable data requires resources, time, expertise and a very liberal tolerance for ambiguity.  And some prudent decision-making. 

Impact evaluation is not for everything, everyone, all the time, in every context, for every program or service.

My queasiness lurches towards nausea when I begin to think about how the data from impact evaluation could be hijacked for the benefit of briefing notes and sound bytes itemizing short-term successes and veiled failures. 

My nausea rushes to the surface of my skin with sweats and shivers when I allow my thoughts to venture into a scenario where impact evacuation is done poorly, delivering false results, producing inaccurate analyses, inevitably leading to wasteful policy decisions.   

Thankfully my tummy settled as I read through the article,   Ten Reasons Not to Measure Impact—and What to Do Instead, by Gugerty and Karlan.

However, anyone expecting any nifty little shortcuts for rigorously and reliably measuring the impact of an intervention will be disappointed because the ‘What to Do Instead’ parts of the article seem a bit anemic. 

More importantly the authors encourage readers to slow down and make time to:

  1. Clearly articulate the type of question you want the evaluation to answer. 
    • Program monitoring questions: we want to learn how well the intervention works.  Gathering data to determine for whom the intervention is working.
    • Impact evaluation questions: take the learning further by asking why the intervention works.   Gathering data to determine why the intervention works for Group A and not so well for Group B. 
  2. Gather monitoring data before conducing an impact evaluation.  Meaning, make sure the implementation of the program is sound before attempting to determine if it has made a difference.
  3. Determine if conducting an impact evaluation is actually worthwhile.  Think about the ways in which an impact evaluation will (won’t) inform the intervention’s theory of change.

Impact evaluation has the potential to profoundly influence the choices we make to better serve people in our communities. 

Unfortunately, if we ignore the cautions set out by Gugerty and Karlan, it presently runs the risk of becoming a more complicated, expensive, soul-crushing, labor-intensive  way to measure outputs.

A Story For Systems Thinkers and People Who Like to Laugh

by

The Hillbilly Elegy  is a book I’d shuffled to the bottom of my pile a few times.  The title did little to inspire enthusiasm to read it.  But it seemed to be the one book that always managed to  float to the top, stick out the side or fall off the shelf.

So I read it.  And I’m glad I did.  It’s brimming with examples of how systems impact the lives of people and the choices they make.  

To be clear this is not a book about systems thinking.  It’s compelling ‘beating the odds‘ story about a family trying to escape the poverty-stricken hills of Appalachian Kentucky and ends with the author’s graduation from Yale Law. 

Looking a bit deeper, the story reveals how the ‘odds’ are calculated and the ways in which they systematically work against people like J.D. Vance and his family.

He grew up in a place filled with stories of addiction, abuse, mental illness, imprisonment, trauma, extreme poverty, broken families and death.  Most astonishing are the many ways systems reinforce these stories rather than ameliorate them.

Formative experiences like Vance’s very rarely lead to Yale Law or even a community college diploma.  On the day he was born, the odds favored addiction, mental illness, jail or dying young.

Despite all contrary forces pushing against him, he beat the odds.  How does that happen? How can we make it happen more often?

Vance explicitly points to influences intervening at crucial moments.  His grandmother (Mamaw) providing a safe place from his addict mother.  Teachers, professors, neighbors and other community members who saw his potential and wanted to do what they could.  

I suspect if we delved deeper into his story and explored it from various perspectives, we would uncover less obvious insights.  For example, when Vance graduated high school, he began the process of applying for college but realized he wasn’t ready and joined the Marines instead.   Where does he get the self-awareness at 18 years old to know he wasn’t ready for college and needed the lessons the Marines could teach him?

If you choose to read the book, I encourage you to keep a mental tally of the two things; the number of systems his story encounters and the number of times you find yourself chuckling at the way Mamaw dispenses her wisdom.

My Starting Point for Measuring Impact

by

Over the holidays I was with friends I hadn’t seen in a while and a few did not know about my MS diagnosis and how it lead to volunteer work with EMCN.

“I’ve be helping guide one of their teams through the human-centered design process” I explained “by convening social innovation labs and nudging them to think beyond ‘the way they have always done things‘”

“MS has limited the number of productive hours I have in a day” I continued “so I want them to matter.  This means I’m also keenly interested in evaluating the impact of what we do”

 “Sounds interesting!” one person said “how do you measure the impact of an initiative?” 

A good question to which I realized I did not have a concrete answer.  I alluded to examples I read about and when I still received a glazed look of incomprehension I moved on to technical evaluation jibber-jabber I heard in podcasts. As a last resort I distracted the person with the snack tray.

Which leaves me with an unanswered question. So I started looking more closely at what I actually want to accomplish.

If I were going to dedicate my limited hours in a day to measuring the impact of an initiative, what would that look like?

As of today, this is where my thinking has landed me. (Initiative includes program, project, service, initiative…..)

One.  The initiative swims in the constant churn of complex dynamic challenges.   

Two.  The initiative gains momentum from learning about the ways in which it is not making a difference in people’s lives.

Three.  The initiative is emergent.   

Four.  The initiative is fueled by possibility.

Five.  Any evaluation of the initiative will need to embrace One, Two, Three and Four.

A Guide to Actionable Measurement

by

How organizations make decisions has begun to garner more of my interest as I delve further into the murky undertow of impact measurement.  Recently, I came across  A Guide to Actionable Measurement (17 pages) offering a glimpse into what influences resource and fund allocation at the Bill and Melinda Gates Foundation.  It’s clear. It’s succinct.  It’s different.

To begin, they actually articulate why they are evaluating and how the data will be used.

“Our philosophy and approach emphasize measurement done for a specific purpose or action. We recognize the most elegant evaluation is only meaningful if its findings are used to inform decisions and strengthen our work to improve people’s lives.

Our approach is driven by three basic principles: 1) Measurement should be designed with a purpose in mind — to inform decisions and/or actions; 2) We do not measure everything but strive to measure what matters most; 3) Because the foundation’s work is organized by strategies, the data we gather help us learn and adapt our initiatives and approaches.”  (Actionable Measurement Guide Cover Letter)

Being able to create impactful interventions to complex problems relies on informative evaluation striking an effective balance between learning (improving something) and accountability (proving something).   Both are needed and valuable for understanding how and why an intervention is effective or ineffective.

At present we are super-proficient at accountability evaluation.  How many?  How often?  Numbers in a spreadsheet.

Unfortunately our evaluation efforts often fail to make meaning from the numbers.   In what ways did reaching the target make a difference?   How did the intervention ‘move the needle’ on the problem it is trying to address? 

Putting together an evaluation approach designed to answer these deeper questions can be stymied by the overwhelming feeling of not knowing where to start or the tendency to build something unnecessarily complicated.  Combing through the Guide to Actionable Measurement has revealed a few tips.

Begin by looking at the language being used to describe the evaluation approach.   The Foundation is intentional about including phrases like ‘strategic intent‘, ‘theory of action‘, and ‘actionable measurement‘.  As an example, using strategic intent over strategic plan has an indelible influence on how the strategy is developed, deployed and measured.

Another manageable place to start is The Actionable Measurement Matrix (Exhibit 4, Page 6 of The Guide). It’s an example of how an illustrative visual can connect activities of a single intervention to the broader strategic intent being deployed to address a complex problem.

Finally, the Bill and Melinda Gates Foundation is careful to acknowledge and measure its role in the creation of the problem being addressed.. Externally, they want to know how their activities as an advocate for policy change have impacted the issues they are try to influence.  Internally, they want to know how their interactions with grantees have impacted interventions and ultimately the problem being addressed.

My Latest Insight about Innovation

by

I’ve been reading tons about innovation – particularly social innovation.  I have lists of definitions,criteria, descriptions, and examples of innovation and social innovation. 

But my latest insight came yesterday when I was listening to the a16z Podcast on my way to visit friends.

Amidst the techno jibber-jabber, most of which went over my head, the hosts had a conversation about two types of innovation they see in the tech landscape;  sustaining innovations and disruptive innovations.

A ‘Sustaining’ Innovation occurs within the existing environments and aids to reinforce the existing business model.   Moving online shopping from the desktop computer to the mobile applications on the smartphone. 

A ‘Disruptive’ Innovation will interrupt the existing business model.  Online media and news sources have eviscerated the revenue sources curbside newspapers traditionally used to run their businesses, forcing them to change their business models to survive.

Applying these definitions to the social sector would lead to some valuable, insightful conversations while developing approaches, programs, or interventions aimed at addressed complex problems.

Social Sector Sustaining Innovation – would aid to reinforce the status quo.

Social Sector Disruptive Innovation – would disrupt the status quo.

So a  conversation might begin with the question; 

To what extent is our idea (approach,intervention, service or program) a Sustaining Innovation and to what extent is it a Disruptive Innovation?

To be clear, this kind of conversation is most valuable when the group:

  1. Is open to being honest with themselves about the way(s) in which the idea might be contributing to the problem they are trying to address. 
  2. Sets aside any preconceived notions about either type of innovation being inherently better than the other.

If you are similar to a couple of my friends, and enjoy the techno jibber-jabber, I certainly encourage you to check out this podcast. 

Even if the techno jibber-jabber is not your bailiwick, the podcast is safe way to listen to folks who talk tech without feeling embarrassed about not knowing what’s going on part of the time.

Link to the episode:  a16z Podcast: Technological Trends, Capital, and Internet ‘Disruption’