The evaluation field, like most established disciplines, operates on a series of assumptions that govern practices, tools, and modes of behavior. However, in recent years, as the social sector has grappled with increasing complexity, improved connectivity (both technological and physical), and rapid change, evaluation has been subtly undergoing a transformation of its own.
Three long held assumptions that have guided evaluation for the past five decades are in the process of being challenged, and three game-changing approaches have emerged that can propel evaluation into the future.
Assumption #1: Formative and summative evaluation approaches are sufficient for telling us what we need to know.
As many of us have tried to apply formative evaluation practices (and assumptions) to complex social innovations, we have increasingly found ourselves frustrated and disappointed. Frustrated in that we collect, analyze, and then sit on lots of information that ends up in a mid-year or year-end report. Disappointed in that our results are barely used and aren’t timely enough to make a difference. We hold information that could be used to inform, reflect on, and make decisions to adapt and change sooner than later, but don’t have a legitimate way to share it.
Game-changer #1: Developmental Evaluation
Developmental Evaluation (DE), which emerged as a newer evaluation approach a few years ago, offers evaluators, nonprofit organizations, and funders a significantly different way of engaging in evaluation practice. Instead of operating against a fixed evaluation plan, along with a linear and predictive logic model, DE offers a much more real-time, learning-oriented, feedback-based, and insight-driven approach to program design and early implementation of a program or initiative.
Assumption #2: Evaluation is generally about the performance of one program or organization at a time
A recent study of 398 nonprofit professional staff found that fewer than 50% were tracking program and outcome data. When asked why they don’t collect more outcome data, more than 75% of nonprofits report that they don’t have the expertise, technology, and time to engage in meaningful monitoring and evaluation activities. So nonprofits are struggling to find their next dollars (amidst increasing competition for scarce resources), trying hard to hang on to quality staff, and passionately trying to provide high levels of services…all without relevant, credible, and useful data about their effects, influence, and impact. What might be possible if they were to pool together their efforts and create a set of common measures and infrastructure for data collection and reporting?
Game-changer #2: Shared Measurement
When groups of organizations come together to co-determine outcomes and indicators that matter, to agree to share their data and learn from each other, they engaged in Shared Measurement. Coupled with a technology platform, these shared measurement systems have the potential of achieving significant progress towards building common understandings, and increasing transparency, accountability and alignment. While shared measurement systems do not replace program level or strategic organization level evaluations, they can provide quantitative and qualitative data on key outcomes and indicators that help track progress, facilitate shared learning, and form the basis and foundation for deeper inquiry.
Assumption #3: Collecting data is the job of professional evaluators and researchers
Over the years, evaluators have worked hard to develop a profession, with standards and ethics, and with codified knowledge and skills. They have fine-tuned their methodology and have made significant improvements in how to design and implement evaluations. However, newer, often digital, data collection methods, and an expanding digital infrastructure are rapidly emerging – allowing diverse groups of people in multiple locations to collect and generate data – often involving no evaluators at all.
Game-changer #3: Big Data
The explosion of social media and the development of an increasingly robust digital infrastructure has created a staggering amount of data. The abundance of data has created the term Big Data, which includes everything from sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, among others. What all of this means for the social sector is still unclear, though it has the potential of being a game-changer in the evaluation field by requiring evaluators to cross boundaries into the use of open source data, and funders to make certain trade-offs with regard to precision and knowing why.
Evaluation continues to offer much to the field. However, there is a need for evaluation to stay current and explore new ideas that are becoming part of its next chapter. While Developmental Evaluation, Shared Measurement, and Big Data are unique approaches in themselves, together they are symptomatic of a profound change in how we conceive of learning and evaluation.
In the coming months, we will be writing, publishing, convening, and curating a series of conversations about these approaches as part of our Next Generation Evaluation project. Stay tuned!
