My Starting Point for Measuring Impact

Over the holidays I was with friends I hadn’t seen in a while and a few did not know about my MS diagnosis and how it lead to volunteer work with EMCN.

“I’ve be helping guide one of their teams through the human-centered design process” I explained “by convening social innovation labs and nudging them to think beyond ‘the way they have always done things‘”

“MS has limited the number of productive hours I have in a day” I continued “so I want them to matter.  This means I’m also keenly interested in evaluating the impact of what we do”

 “Sounds interesting!” one person said “how do you measure the impact of an initiative?” 

A good question to which I realized I did not have a concrete answer.  I alluded to examples I read about and when I still received a glazed look of incomprehension I moved on to technical evaluation jibber-jabber I heard in podcasts. As a last resort I distracted the person with the snack tray.

Which leaves me with an unanswered question. So I started looking more closely at what I actually want to accomplish.

If I were going to dedicate my limited hours in a day to measuring the impact of an initiative, what would that look like?

As of today, this is where my thinking has landed me. (Initiative includes program, project, service, initiative…..)

One.  The initiative swims in the constant churn of complex dynamic challenges.   

Two.  The initiative gains momentum from learning about the ways in which it is not making a difference in people’s lives.

Three.  The initiative is emergent.   

Four.  The initiative is fueled by possibility.

Five.  Any evaluation of the initiative will need to embrace One, Two, Three and Four.

A Guide to Actionable Measurement

How organizations make decisions has begun to garner more of my interest as I delve further into the murky undertow of impact measurement.  Recently, I came across  A Guide to Actionable Measurement (17 pages) offering a glimpse into what influences resource and fund allocation at the Bill and Melinda Gates Foundation.  It’s clear. It’s succinct.  It’s different.

To begin, they actually articulate why they are evaluating and how the data will be used.

Our philosophy and approach emphasize measurement done for a specific purpose or action. We recognize the most elegant evaluation is only meaningful if its findings are used to inform decisions and strengthen our work to improve people’s lives.

Our approach is driven by three basic principles: 1) Measurement should be designed with a purpose in mind — to inform decisions and/or actions; 2) We do not measure everything but strive to measure what matters most; 3) Because the foundation’s work is organized by strategies, the data we gather help us learn and adapt our initiatives and approaches.”  (Actionable Measurement Guide Cover Letter)

Being able to create impactful interventions to complex problems relies on informative evaluation striking an effective balance between learning (improving something) and accountability (proving something).   Both are needed and valuable for understanding how and why an intervention is effective or ineffective.

At present we are super-proficient at accountability evaluation.  How many?  How often?  Numbers in a spreadsheet.

Unfortunately our evaluation efforts often fail to make meaning from the numbers.   In what ways did reaching the target make a difference?   How did the intervention ‘move the needle’ on the problem it is trying to address? 

Putting together an evaluation approach designed to answer these deeper questions can be stymied by the overwhelming feeling of not knowing where to start or the tendency to build something unnecessarily complicated.  Combing through the Guide to Actionable Measurement has revealed a few tips.

Begin by looking at the language being used to describe the evaluation approach.   The Foundation is intentional about including phrases like ‘strategic intent‘, ‘theory of action‘, and ‘actionable measurement‘.  As an example, using strategic intent over strategic plan has an indelible influence on how the strategy is developed, deployed and measured.

Another manageable place to start is The Actionable Measurement Matrix (Exhibit 4, Page 6 of The Guide). It’s an example of how an illustrative visual can connect activities of a single intervention to the broader strategic intent being deployed to address a complex problem.

Finally, the Bill and Melinda Gates Foundation is careful to acknowledge and measure its role in the creation of the problem being addressed.. Externally, they want to know how their activities as an advocate for policy change have impacted the issues they are try to influence.  Internally, they want to know how their interactions with grantees have impacted interventions and ultimately the problem being addressed.

My Latest Insight about Innovation

I’ve been reading tons about innovation – particularly social innovation.  I have lists of definitions,criteria, descriptions, and examples of innovation and social innovation. 

But my latest insight came yesterday when I was listening to the a16z Podcast on my way to visit friends.

Amidst the techno jibber-jabber, most of which went over my head, the hosts had a conversation about two types of innovation they see in the tech landscape;  sustaining innovations and disruptive innovations.

A ‘Sustaining’ Innovation occurs within the existing environments and aids to reinforce the existing business model.   Moving online shopping from the desktop computer to the mobile applications on the smartphone. 

A ‘Disruptive’ Innovation will interrupt the existing business model.  Online media and news sources have eviscerated the revenue sources curbside newspapers traditionally used to run their businesses, forcing them to change their business models to survive.

Applying these definitions to the social sector would lead to some valuable, insightful conversations while developing approaches, programs, or interventions aimed at addressed complex problems.

Social Sector Sustaining Innovation – would aid to reinforce the status quo.

Social Sector Disruptive Innovation – would disrupt the status quo.

So a  conversation might begin with the question; 

To what extent is our idea (approach,intervention, service or program) a Sustaining Innovation and to what extent is it a Disruptive Innovation?

To be clear, this kind of conversation is most valuable when the group:

  1. Is open to being honest with themselves about the way(s) in which the idea might be contributing to the problem they are trying to address. 
  2. Sets aside any preconceived notions about either type of innovation being inherently better than the other.

If you are similar to a couple of my friends, and enjoy the techno jibber-jabber, I certainly encourage you to check out this podcast. 

Even if the techno jibber-jabber is not your bailiwick, the podcast is safe way to listen to folks who talk tech without feeling embarrassed about not knowing what’s going on part of the time.

Link to the episode:  a16z Podcast: Technological Trends, Capital, and Internet ‘Disruption’

How My Experience Becomes My Reality

Imagine you are getting ready to board the plane to the vacation of your dreams.  Just before getting on the plane, a person comes to you and says “I hope you have a great vacation but I just wanted to tell you that when you return, all your pictures will be deleted and you won’t remember anything about the trip.”

What do you do?  Get on the plane anyway?  Take a different vacation?  Go home?  Throw a tantrum?  Cry?

A similar question confronted people in one of Daniel Kahneman’s research studies exploring the cognitive process people go through to make choices.  In his research, he found most people would change their plans because the main purpose of the vacation was to make memories.

Kahneman became curious about discovering the mechanisms or processes our brain uses to make something memorable.  Over the decades of his research, Kahneman made many important discoveries earning him the Nobel Peace Prize and touching off the creation of a new field called Behavioral Economics.

Two of his discoveries, the ‘selves‘ and the Peak-End Rule, are particularly pertinent to the choices we make to ensure experiences with our organization are memorable.

In what ways are we missing opportunities to be memorable to our clients, customers, beneficiaries, funders, donors, stakeholders, partners, volunteers, etc…?

One discovery, the remembering-self and experiencing-self, describes how we make choices and Kahneman uses parenting as an example to illustrate the difference.

Inspired by his research I began asking friends, who recently had children, if they ever regret giving up their freedom to have kids.  Without skipping a beat, they respond by proudly regaling me with stories of first steps and funny first words.

On the whole‘ they would say ‘it is the most rewarding experience of my life

This response is coming from their remembering-self because the answer is informed by the memorable parts of parenting.  They are referencing the highlights over the years to make a judgement about whether having children has been a worthwhile endeavor.

Then the 3 year old has a 5-alarm meltdown making me think he is either on fire or possessed.  With a calm roll of the eyes and a face that momentarily says ‘I might have made a different choice had I known about the volume and frequency of 5-alarm emergencies‘, my friend goes to ensure someone isn’t actually on fire.

Responses like my friend’s facial expression come from the experiencing-self. They rely on information available in the moment and do not have the benefit of past memories.  Have you ever sent text when you were furious?  Sure felt good in the moment……

OK.  Makes sense. Sounds obvious.  We use past experiences to inform future decisions.  This is not new.

For me the new part emerged when he Kahneman began speaking about the Peak-End Rule  

99% of us are not able to remember every moment of an experience.  Instead, our remembering-self identifies positive or negative snippets (highlights) during the experience Kahneman calls Peak moments.  Additionally, we are prone to remembering how an experience ended which he cleverly calls the End moments. (During the podcast and TED Talk, he talks about the colonoscopy study to illustrate this more thoroughly).  The result is a string of memorable moments we use for making decisions in the future.

So naturally the question becomes; how do we ensure people have memorable moments so they continue supporting our organization?

When I think back to all the events I helped organize over the years, I quickly realize the standard we had for success was whether or not everything went perfectly.  Which really meant everything went as we planned.  Everything was perfect.

Kahneman would say trying to achieve perfection leads our decision-making down a blind alley.  Instead we should be asking ourselves how we can create multiple peak moments and be more intentional about the manner in which the event comes to a close.  What ‘material’ can we provide to satisfy multiple remembering-selves from diverse backgrounds, ages and perspectives?

How would intentional consideration of Kahneman’s discoveries influence the ways your organization interacts with its community? 

 

(48 mins)Hidden Brain: Think Fast with Daniel Kahneman

(20 mins)Ted Talks: The riddle of experience vs. memory

 

Etmanski: Advocate with Empathy

When people ask what it is like to work in government I get them to imagine all the work for all parts of government as a giant slow-moving stream.   Since the stream is moving so slowly, it takes deliberate choice to expend the effort to actually get anywhere.   95% of government employees are happy to bob along like driftwood at the whims of the stream.  The other 5% are problem-solvers in boats trying to make a difference by paddling.  Inevitably though, the boats reach a place where the driftwood has clogged the stream and the paddlers must decide whether to work through it or bail out for the shore.

After reading Etmanski’s Advocate for Empathy, I realize my analogy isn’t entirely fair because it leaves out one important group.  There are some pieces of driftwood, tired of bending to the will of the stream, that want to climb into the boats to paddle.

But climbing into the boat means accepting three realities.  First, we can’t change the pace of the steam.  It is so cluttered with competing requests that it can be difficult to tell which direction the stream is flowing.  Second, we can’t change the technology of boats and paddles.  Even though voters expect government to move more rapidly by utilizing motors and GPS, they don’t want to pay for it.  Third, everyone can see you’re paddling effort from the shore and offer their critiques and condemnations.

The result is an intensely risk-averse environment rife with fear, stifling opportunities for innovation, change and impact.

Despite this, people want to paddle, so how do we help them into the boats?  

Etmanski points to Solution-Based Advocacy which focuses on solutions over criticisms and improving the ability of government to make better decisions.  It means acknowledging we are all in the same boat, government included, so we may as well learn paddling techniques from each other.

Five Characteristics of Solution Based Advocacy

  • Searching for a Heart of Gold” – Taking the time to look beyond what our political leaders do, to learn about who they are.
  • Using Strategic Inquiry” – Aligning your agenda with the government agenda by “…discovering the priorities, language and tools of the group you are trying to convince…” (p.116)
  • Cultivating a Network of Champions” – Although we think one champion is great, it just isn’t enough.
  • Solving Problems Together” – Shifting government from the role of parent to partner.
  • Doing it Themselves” – Regardless of how amazing the idea is, there will always be opposing interests being considered decision-makers.  Forging ahead without them can create the space they need to champion your idea and your success in the face of opposition.

Are you a paddler or are you driftwood?

 

Impact: Six Patterns to Spread Your Social Innovation by Al Etmanski is a guide for social innovators to move their idea from localized success to broader systemic impact.

Etmanski: Mobilize Your Economic Power

Etmanski’s first three Patterns to Spread Your Social Innovation have been comfortable to comprehend.  Think and Act Like a Movement, Create a Container for Your Content, Set the Table for Allies, Adversaries are not easy but they make sense.

The fourth pattern, Mobilize Your Economic Power, is not so easy.  It means having conversations about assets, profits, liabilities, customers, marketing strategy and cash flows.  Notice words like funding, grants, beneficiaries and outcomes do not appear.

It makes me a bit squeamish because mobilizing economic power requires the not-for-profit organization to participate in the markets creating many of the injustices it is trying to correct.  I have a natural visceral reaction against the idea.  How can we talk about assets in one meeting and the suffering of people in the next?

It seems wrong.

However, after setting aside my discomfort, I realize Etmanski is encouraging me to uncover new possibilities for systemic impact.  Identifying economic power gives me access to more levers to alleviate or eliminate the suffering of people I serve.  Most importantly, can I really say I am serving people well if I’m not accessing all the avenues open to me just because it makes me squeamish?

What would change about the way you do your work if your organization was presented with a check for $10 million?

Likely your organization would shift mindsets from “scarcity to abundance” and from “victim to agent”.  Etmanski believes we shouldn’t wait for the check.  Find our economic leverage points and use them to become an independent fearless agent for change.

Five Ways to Mobilize Your Economic Power

Admittedly, I became a bit foul as I read through these.  They seem impractical and inaccessible until I read it from the perspective of a person seeking to move an idea from the local context to the mainstream.

  • Influencing the Operational Practice of Business” – really this is about disrupting the way it has always been done.  Etmanski uses an example of the environmental activist shifting her attention from blocking logging trucks to developing a process for recycling paper so the publishing industry would no longer use virgin forest products.  She blocked the logging trucks using the market instead of her body.
  • Commercializing Intellectual Property” – Take stock of your intellectual property and see how it might generate revenue for your beneficiaries/clients.  Charge market value for it!
  • Nurturing a Sharing Economy” – Besides the conventional market, not-for-profits are uniquely positioned to tap into the world of swapping, bartering, and lending.
  • Purchasing Locally” – Buying your office supplies from Walmart might be cheaper, but much of profits associated with that transaction are siphoned out of the community.  Buying your office supplies from a local business increases the likelihood that profits will be recycled into the local community.
  • Starting a Social-Purpose Business” – Becoming an entrepreneur by forming a business that focusses on financial return and social impact.

 

Impact: Six Patterns to Spread Your Social Innovation by Al Etmanski is a guide for social innovators to move their idea from localized success to broader systemic impact.

Etmanski: Set the Table for Allies, Adversaries and Strangers

Have you ever been asked to participate on a committee or in a community meeting and left wondering why were invited?  A situation in which the conversation was swirling around you but you never quite knew how to contribute to it?

Looking back to the occasions in which this was the case for me, I realize the people calling the meeting (conveners) had a broad range of reasons for inviting people (job titles, organizations, to not hurt feelings, to get things done, to access resources, for expertise, to make new connections, because they were told to, and so on).  All of which are valid reasons but it was unclear why I had been invited so I contributed little.

As a convener, we need to be deliberate and thoughtful about the people we invite and the environment in which they gather.  Remember, to have the idea picked up by the mainstream, means inviting supporters, opponents and others beyond the immediate context.  Everyone needs to feel welcome and to be assured that their contributions will be valuable and important.

 

Four Characteristics of Effective Convening

To avoid the gathering in which a few people speak and nothing is ultimately accomplished, take a minute to run through the questions below.

  • Civility” – Has the group crafted (agreed upon) ground rules of conduct by which they can respectfully, openly, and safely contribute?
  • Personal Agency” – As the convener, have you encouraged the best from all participants?  Have you given each participant an opportunity to shine, show their strengths and lead in their own way?
  • Hospitality” – As the convener, are you able to articulate the importance of each participant’s contribution and made them feel like they belong?
  • Curiosity” – As a convener, have you created a group culture in which uncertainty leads to inquiry and eventually new answers?

 

Impact: Six Patterns to Spread Your Social Innovation by Al Etmanski is a guide for social innovators to move their idea from localized success to broader systemic impact.