The Difference between Outputs, Outcomes, and Impacts
Program evaluations are generally done to prove to external stakeholders (funders, partners, authorities) that a program or service is “worth” further or additional support. This usually means providing evidence of value for investment, whether the investment is time, money, energy, or dedication. And yet, traditional program evaluation methods are concerned primarily with “outputs”. These outputs are presumed to represent other positive “outcomes”. For example, if we know how many people completed a program, we assume that most of those have achieved a certain outcome. Then we can imagine that a significant number of those people went on to experience other benefits.
However, “outputs” and “outcomes” are not the same as “impacts”. To identify impacts, we need to use a wider lens. We need to look at what has changed not just for the individual but also for the community, associated agencies, or society as a whole. We don’t often investigate those impacts. That is, we almost never check in with seemingly peripheral stakeholders to see what THEY have witnessed or experienced as a result of an organization’s efforts. We also don’t often ask about the concrete indicators of change or consider the preventative nature of social initiatives. That’s understandable, because it’s hard to investigate negative things that didn’t happen, and it can seem hard to measure unless you have the right methodology.
And yet, social programs and services are intended to bring about a positive change not only for the participants but also for the broader social environment. Or, said another way, they’re designed to reduce, minimize, or end something that’s having a negative impact on someone or something. The avoidance or reduction of the negative is hence value-able. Think of the efforts to get people to stop smoking or exercise more. Some of that is about people’s health, but the effort is also about saving the healthcare system money. “Expensive” can be defined in all sorts of ways, not just in monetary terms, because other things are also valuable to us as a community or society.
That’s what social impact studies look at: what’s the impact of this organization’s activities in the social realm? What is considered valuable to us in this context? If we look beyond the immediate outcomes of a program or service, and look beyond the participants as the sole sources of evaluation data, we can see that social programs have a significant ripple-effect on organizations, communities, and social infrastructure. They have an essential role in society, as much for what they do as for what they help prevent. Social impact studies therefore give us a broader picture and can help us find data that supports the development of better policies and planning because we’re looking not just at the program as an object, but as an agent of change within a larger context.
It makes sense then to identify and talk to different kinds of partners and stakeholders. It makes sense to ask different kinds of questions, and to do academic research to see whether other studies have come to similar conclusions. It becomes possible to put both monetary and non-montary investments on the scale against a more accurate inventory of outcomes and impacts, not just outputs, so that social value can be described more fully. If necessary, the financial cost savings or the “return on investment” can be indicated.
Of course, doing a social impact study will also point out ways of improving the program or service, and will help organizations explore ways of improving the type and amount of impact that they have. Social impact studies even make it possible to work towards greater social innovation through collective impact. As a result, we have a better understanding of individual and collective benefits, and can make improved plans for the future.
- Stakeholder involvement, including broad scoping to identify primary and secondary beneficiaries
- Investigation of “witnessed and experienced changes” using case study methodology, focus groups, unstructured and semi-structured interviews, and participatory research approaches
- Emergent process development to integrate new information as it arises (constructivist grounded theory approach)
- Identification of Crossroads Challenges and alternate story-mapping
- Creation of an Impact Inventory summarizing key individual, group, community and/or systems changes
- Description of Social Impact Snapshots to illustrate witnessed and experienced social impacts
- Assistance with development of data-gathering strategies to support selected indicators of change
- In some cases, Social Return on Investment calculations may be applied
- In-house capacity building through collaborative work, including consciously seeking out opportunities for on-going knowledge exchange
A sample report written by Margerit Roger using Social Return on Investment methodology can be found at: http://www.familydynamics.ca/wp-content/uploads/2015/12/FD-Report-October-2015-Walking-School-Bus-Breakfast-Club1.pdf
A social impact study (without SROI calculations) can be found at: http://www.familydynamics.ca/wp-content/uploads/2013/04/Family-Supports-for-Refugees-Social-Impacts-Study-2016-1.pdf
Recent Related Learning
- McGill University “Social Learning for Social Impact” Group Open Online Course
- Innoweave Impact Accelerator Workshop (hosted by United Way)
- Collective Impact workshop (Tamarack Community)