One of the most used (and abused) terms in the business sector over the past couple of years is “Big Data” referring mainly to the explosion in the volume and variety of data we have begun to generate over the last few years. According to one source, we now generate more data every two days than we did from the dawn of civilization until 2003. However, the massiveness of data is only part of the story. The other, more critical aspect is our ability to process all this data to generate “intelligence.”
Advanced business analytics are now available (e.g. Hadoop) to help pore through quantitative AND qualitative data in seconds, generating insights and trends that would have taken days to gather. At a recent conference in San Francisco organized by the Economist magazine, one heard a few new terms being thrown around to describe this new-found superpower such as:
- machine intelligence, or the ability of computers to learn through analytics and get smarter with time
- reality mining, or the ability to mine digital data to explore trends and understand behavior
- nowcasting, similar to reality mining, a way to understand current trends as they are playing out in real-time
So how is the social sector benefiting from big data? One handicap that is familiar to all of us in the social sector is that data systems and information markets are, to put it mildly, under-developed. Schools struggle to properly keep track of the movement of teachers and students as they enter and leave the school system, leave alone data about their performance. Hospitals have difficulty capturing indicators such as re-admission rates that they are now required to report. Nonprofits may collect data to understand the characteristics and needs of the people that they reach through direct services, but often lack capacity to collect information related to long-term outcomes, such as changes in attitudes or behavior. In this context, the notion of big data and predictive analytics seems like a faraway dream.
But before we close the door on the possibility, let’s take a moment to examine where exactly big data comes from, and how potentially those could be used in the social sector.
- For example, big data is generated as formerly non-digital processes become digitized. Think everything from credit card payments to downloading mp3s to online shopping. As clicks replace bricks, they also generate data that can be aggregated and sorted at the touch of a button. Now think of the possibilities, say in a classroom, as blended learning becomes the norm and every click that that child makes in her journey toward mastery of a content area is recorded and available for analysis. The dream begins to get a little more real.
- Big data is also generated as devices become more integrated into our everyday lives. We are almost immune to the devices such as sensors, cameras and GPS’ that play a role in our everyday lives. These devices are capturing information about us that can reveal key trends. Cut to the social sector and we see some of this technology being incorporated into healthcare. We now see virtual communities abound engaging in conversations about their health, driven by data from devices. Everyone from sleep apnea sufferers sharing data from pressure sensors in their masks, to runners sharing their data from insole sensors in their shoes is making big data a reality in healthcare.
- Finally, we generate lots of big data as we willingly share information with the world through social media. Consider the billions of tweets that are sent out every day. Or check-ins on foursquare. Or updates on facebook. Everywhere we go on social media, we leave “digital breadcrumbs” that point to collective trends. From Hollywood movie studios trying to gauge buzz around a new release to government intelligence agencies picking up chatter on potential flash points, analyzing social media posts is the new parlor game. The social sector isn’t far behind. Ushahidi is an innovative organization that uses aggregated trends from text messages and social media feeds to coordinate immediate response to natural disasters such as the Haiti earthquake and the Japan Tsunami.
There is also evidence that resources are being directed towards big data in the social sector. The Gates Foundation, along with the Hewlett Foundation and Liquidnet, recently launched “Markets for Good,” which is an attempt to discover how the social sector can better share and use information, and upgrade underlying information infrastructure. More and more funders and nonprofits are moving toward shared measurement systems where data is shared across organizations and initiatives.
So what’s possible now? Before we claim big data as the social sector’s new messiah, we should keep in mind that this phenomenon is still brand new and we need to address several issues around privacy, accuracy, and reliability before we can truly move forward. The use of big data in strategic learning and decision making is still quite nascent. Traditional data collection methods such as surveys, interviews and focus groups will continue to have relevance, as they help us go deeper, understand context and test insights. But there is no denying that the big data revolution has indeed begun! How is your organization using big data to change the world?
