Last week, FSG joined over 2500 evaluators from across the country and around the world at the American Evaluation Association 2011 Conference in Anaheim, CA. Ellen Martin, Sr. Consultant, and I facilitated an interactive dialogue in a session entitled Evaluating Community Engagement Using Social Media and the Web (our presentation is part of the AEA eLibrary). We talked with evaluators working in nonprofits, foundations, and as external consultants about how to better integrate social media and web-based analytics into evaluation practice.
Through our work with the John S. and James L. Knight Foundation, we have started using social media and website analysis to evaluate impact. The Knight Foundation’s Community Information Challenge has supported more than eighty projects to create more informed and engaged communities. These projects are relying heavily on the use of websites and social media to reach and engage their target audiences, which may include youth, policymakers, and the general public, among others.
During our presentation, we highlighted the work of GrowWNY, a Community Information Challenge project that aims to engage and connect organizations promoting a healthier environment in Buffalo, NY. GrowWNY is using Facebook, Twitter, Flickr, YouTube, and a new website platform to reach and engage nonprofits, as well as the broader community. Then, in true “think tank” fashion, we asked session attendees to consider a variety of scenarios that included a description of a community engagement project, its goals, its program design and key online and offline activities, and a key evaluation question.
With the help of session attendees, we identified some outstanding questions and key insights about how social media and web analytics could be used to answer questions about who is being reach, how people are engaging, and whether dialogue among residents is growing.
Outstanding Questions:
- Using social media well takes up valuable time and resources. How can evaluators address the matter of which social media activities are getting the greatest return on time, resources being committed?
- IP addresses can be unreliable. What tools are most effective in evaluating whether users are part of the project’s target audience, especially for place-based efforts?
- How do evaluators approach privacy issues when using social media and web-based tools in their practice?
Participants generated several insights about evaluating community engagement using social media and web analytics, as well. Thankfully, some of these insights help to answer a few of the outstanding questions above!
Key Insights:
- Identifying who you are reaching online is a universal challenge. Nonprofits seeking to increase community engagement aim to reach people in a specific geographic location; knowing who their online audience is matters. When asking questions about program reach and engagement, there need to be mechanisms in place to identify online users and participants based on key characteristics (e.g., previous engagement, geographic area, demographics). Registration using a unique system or a Facebook-integrated system is one way of better understanding online reach.
- Social media generates qualitative information through comments and uploaded content that can be used to understand how and why users are engaging. Go beyond page views and counts to understand what people engaging in your program are saying – about their participation, your work, or each other. This could be done by reading and drawing out themes of comments on Facebook or in response to blog posts.
- Tools beyond social media, such as surveys, can help connect offline engagement with online engagement. They might be collected at an in-person event, distributed to an email list, or embedded online. Survey Monkey was mentioned as one tool that’s easy to use and integrates well online.
- There is no one “right” method. A mixed methods approach is needed to evaluate behavior as complex as engagement. While social media and website analytics may help answer some questions, they often are insufficient and should be supplemented by other methods of data collection, as well.
- Consider barriers to online engagement. If your target audience doesn’t have access to the internet or can’t handle one more registration process, then you might have trouble getting people to log-on, comment, or more deeply engage online. These barriers also mean that who you reach online could be very different than who you reach in “real world,” offline activities.
With online tools and resources always changing, we as evaluators need to be nimble and quick to adapt as social media strategies change in response peoples’ online behaviors. You can read more about how we’ve started thinking about approaching questions of reach, engagement, and impact in our report, IMPACT: A Practical Guide to Evaluating Community Information Projects.
