In a recent blog post discussing the importance of good evidence in supporting systems change work, evaluation expert Lisbeth Schorr wrote, “To get better results in this complex world, we must be willing to shake the intuition that certainty should be our highest priority…” Rather, she argues, “it is time for all of us to think more expansively about evidence as we strive to understand the world of today and to improve the world of tomorrow.” [Emphasis added]
At the annual American Evaluation Association Conference (AEA) in November 2015, practitioners, funders, and academics from around the world gave presentations and facilitated discussions around a type of evaluation that is specifically designed to meet this need for a more expanded view of evidence. It’s called developmental evaluation, and, as noted by other commentators, it took this year’s AEA conference by storm.
What is developmental evaluation?
Developmental evaluation (DE) “is grounded in systems thinking and supports innovation by collecting and analyzing real-time data in ways that lead to informed and ongoing decision making as part of the design, development, and implementation process.” As such, DE is particularly well-suited for innovations in which the path to success is not clear. By focusing on understanding what’s happening as a new approach is implemented, DE can help answer questions such as:
- What is emerging as the innovation takes shape?
- What do initial results reveal about expected progress?
- What variations in effects are we seeing?
- How have different values, perspectives, and relationships influenced the innovation and its outcomes?
- How is the larger system or environment responding to the innovation?
DE can provide stakeholders with a deep understanding of context and real-time insights about how a new initiative, program, or innovation should be adapted in response to changing circumstances and what is being learned along the way.
A well-executed DE will effectively balance accountability with learning; rigor with flexibility and timely information; reflection and dialogue with decision-making and action; and the need for a fixed budget with the need for responsiveness and flexibility. DE also strives to balance expectations about who is expected to adapt and change based on the information provided (i.e., funders and/or grantees).
The case for developmental evaluation
Developmental evaluation (DE) has the potential to serve as an indispensable strategic learning tool for the growing number of funders and practitioners that are focusing their efforts on facilitating systems change. But, DE is different from other approaches to evaluation. Articulating what exactly DE looks like in practice, what results it can produce, and how those results can add value to a given initiative, program, or innovation is a critical challenge, even for leaders who embrace DE in concept.
To help meet the need for a clear and compelling description of how DE differs from formative and summative evaluation and what value it can add to an organization or innovation, we hosted a think tank session at AEA in which we invited attendees to share their thoughts on these questions. We identified 4 overarching value propositions of DE, which are supported by quotes from participants:
1) DE focuses on understanding an innovation in context, and explores how both the innovation and its context evolve and interact over time.
- “DE allows evaluators AND program implementers to adapt to changing contexts and respond to real events that can and should impact the direction of the work”.
- “DE provides a systematic way to scan and understand the critical systems and contextual elements that influence an innovation’s road to outcomes.”
- “DE allows for fluidity and flexibility in decision-making as the issue being addressed continues to evolve.”
2) DE is specifically designed to improve innovation. By engaging early and deeply in an exploration of what a new innovation is and how it responds to its context, DE enables stakeholders to document and learn from their experiments.
- “DE is perfect for those times when you have the resources, knowledge, and commitment to dedicate to an innovation, but the unknowns are many and having the significant impact you want will require learning along the way.”
- “DE is a tool that facilitates “failing smart” and adapting to emergent conditions.”
3) DE supports timely decision-making in a way that monitoring and later-stage evaluation cannot. By providing real-time feedback to initiative participants, managers, and funders, DE supports rapid strategic adjustments and quick course corrections that are critical to success under conditions of complexity.
- “DE allows for faster decision-making with ongoing information.”
- “DE provides real time insights that can save an innovation from wasting valuable funds on theories or assumptions that are incorrect.”
- “DE promotes rapid, adaptive learning at a deep level so that an innovation has greatest potential to achieve social impact.”
4) Well-executed DE uses an inclusive, participatory approach that helps build relationships and increase learning capacity while boosting performance.
- “DE encourages frequent stakeholder engagement in accessing data and using it to inform decision-making, therefore maximizing both individual and organizational learning and capacity-building. This leads to better outcomes.”
- “DE increases trust between stakeholders or participants and evaluators by making the evaluator a ‘critical friend’ to the work.”
- “DE can help concretely inform a specific innovation, as well as help to transform an organization’s orientation toward continuous learning.”
Additionally, one participant offered a succinct summary of how DE is different from other types of evaluation: “DE helps you keep your focus on driving meaningful change and figuring out what’s needed to make that happen—not on deploying a predefined strategy or measuring a set of predefined outcomes.”
We hope that these messages and talking points will prove helpful to funders and practitioners seeking to better understand why DE is such an innovative and powerful approach to evaluation.
Have other ideas about DE’s value? Please share them in the comments.
Learn more about developmental evaluation
Developmental evaluation: Applying complexity concepts to enhance innovation and use by Michael Quinn Patton (Guilford Press, 2010).
Developmental evaluation exemplars: Principles in practice by Michael Quinn Patton, Kate McKegg and Nan Wehipeihana. (Guilford Press, 2015).
A Developmental Evaluation Primer. Jamie Gamble. (2008). Montréal: The J.W. McConnell Family Foundation.
Developmental Evaluation Toolkit. Spark Policy Institute. (2015).
DE 201: A Practitioner’s Guide to Developmental Evaluation by Elizabeth Dozois, Marc Langlois and Natasha Blanchet-Cohen. Montréal: The J.W. McConnell Family Foundation.
Evaluating Social Innovation. Hallie Preskill and Tanya Beer. (2012). FSG.
Systems Change: A Guide to What It Is and How to Do It. Ellen Harries, Rachel Wharton, and Rob Abercrombie. New Philanthropy Capital, June 2015.
There are also many DE- related resources available through the AEA e-library and on BetterEvaluation.org.
