Systems change has received much attention in recent years as grantmakers have increasingly set out to change underlying conditions that hold systemic challenges in place. The concept is becoming more concrete as foundations, community organizations, and collaboratives have made real change in their communities by using a systems lens. In addition, equity is increasingly recognized as essential for systems change work.
At FSG, we have learned a lot from our partners about what it takes to shift systems. And we’ve often been asked – how do you evaluate systems change? Unfortunately, there is no simple answer to this question. There are many approaches that one could take to effectively evaluate a systems change effort – and finding an effective approach involves:
- Defining the system you are trying to shift;
- Articulating a set of questions that will help you understand how change is happening within the system;
- Determining what approaches have the capacity to surface changes and shifts in the system; and
- Using the evaluation findings as a catalyst for learning and decision making.
A first step is defining the system you are trying to shift. This might initially require your organization to learn more about the concept of systems change. Donella Meadow’s definition of a system is a good place to start: “An interconnected set of elements that is coherently organized in a way that achieves something (function or purpose).” To understand your particular system, mapping is a useful exercise, to get a handle on the organizations (i.e., actors) operating within the system you wish to influence.
Once you have a sense of the system you’re trying to affect, it’s useful to set some parameters for the evaluation. A set of questions will help you focus your inquiry, and might explore:
- Aspects of complexity, including context, relationships, influence, and emerging patterns (read more in Evaluating Complexity: Propositions for Improving Practice)
- The extent to which your effort is furthering equity in your community and how the evaluation itself can further equity (read more from the Equitable Evaluation Initiative)
- Conditions of systems that hold problems in place (read more in The Water of Systems Change) or leverage points in the system (read more from Donella Meadows).
Once you have a set of strategic learning questions, you will need to identify an approach and data collection methods for the evaluation. Given the complex and emergent nature of systems change initiatives, many have benefitted from using a developmental evaluation approach, but depending on how long you’ve been implementing and the nature of your questions, it’s possible a formative or summative approach would be appropriate. We always encourage the use of mixed methods so that both quantitative and qualitative data are collected. As Mark Cabaj writes, “Many results are difficult – even impossible – to capture with quantitative data and methods alone” and thus a principle to adhere to when evaluating systems change efforts is “No numbers without narrative, no narratives without numbers.”
An important distinction between evaluating systems change efforts, versus evaluating programs, is that in systems change work, outcomes rarely fall on a predictable, logically sequenced path. So that means that any evaluation tools that were developed for more predictable or stable programs (e.g., logic models, SMART indicators and goals, benchmarks, RCTs) aren’t the most useful for systems change efforts because they’re assuming a level of control and predictability that just isn’t there. We suggest taking a look at “What We Know So Far About Sets of Principles for Evaluating Systems Change Efforts” from Mark Cabaj and Tamarack Institute that will help you think about the best approach and methods for your evaluation.
As with all evaluations, the evaluation’s findings should be useful to decision-makers and other stakeholders, and should include learning sessions that provide opportunities for reflection and dialogue grounded in the data. It is also important to create space to consider the assumptions underlying the initiative, to understand how the context is shifting, to deliberate on the ways equity is showing up in the work, and what actions (e.g., modifications, expansion, contraction, new relationships) are needed to continue shifting the system(s).
In our work providing strategic evaluation and learning support to foundations and mission-driven organizations, we integrate a systems lens and adaptive approaches into our evaluation designs, as we did in our evaluation of the Grand Rapids Community Foundation’s Challenge Scholars program. And we’ve learned a lot about how evaluation supports systems change.
Systems change work is hard, messy, and unpredictable. Evaluation should be used as a tool to help you gather information, learn, and make decisions. For evaluation to truly support your systems change effort – it needs to be grounded in an understanding of what you’re trying to do, designed to address a set of questions that have real strategic relevance, informed by mixed methods, and supported by structures and a culture that values learning from evaluation findings.
Watch the webcast for Evaluating Systems Change Efforts: Where to Start >
Learn more about FSG’s Strategic Evaluation Services >
