5 Needs and 5 Tools for Measuring Media Impact

Tracy Van Slyke co-authored this article

This spring, National Public Radio launched Go Figure, a new blog authored by members of its Audience Insight and Research Group. In an April 1 post, blogger Vince Lampone wrote, “Nearly all listeners have been moved to take action by NPR at some point in their lives. For instance, two in three have done further research into a topic, most have visited a website, and nearly 25% have become involved with a local or national political issue as a result of listening.”

As Lampone’s post suggests, NPR is just one of many media outlets, researchers, and funders that are currently struggling to answer the pressing question: “How do you know if your media matters?”

Impact Summits

To gather answers, American University’s Center for Social Media (CSM) and The Media Consortium (TMC) organized a series of Impact Summits in seven cities throughout the first quarter of 2010. (In February, I wrote about these Summits for MediaShift as they were just ramping up.)

Hosted with allied organizations in Chicago, New York, Miami, Los Angeles, San Francisco, Washington, D.C., and Boston, they drew together dozens of leading public and independent media makers, funders and researchers, representing “public interest media projects.” These projects range from hyper-local to national, and represent a variety of practices, including investigative journalism, advocacy journalism, documentary film, public and community broadcasting, gaming for change, citizen reporting, and building access and media tools for diverse communities.

These Summits resulted in a new report coauthored by CSM and TMC, Investing in Impact, which outlines the major arguments for assessing impact, synthesizes the five top impact evaluation needs, and proposes five new tools for public interest media assessment. What follows is a summary of that report.

Currently, there is no consensus around what constitutes impact for public interest media. In fact, the very topic can lead to heated exchanges between media producers looking to have clear goals and outcomes in a new media environment, and traditional journalists who suggest that evaluation will threaten their objectivity and limit the focus of their reporting.

The truth is that the field cannot advance without tackling the question of impact head on. To be clear, effectiveness is not synonymous with advocacy. Traditional journalistic values include holding the powerful to account, engaging users in dialogue about issues, and delivering timely, relevant information. These are all trackable outcomes.

Five Needs

What do public interest media need in order to evaluate their impact? We’ve outlined five needs below.

1. Getting on the same page: Developing shared categories of impact assessment
   In order to make meaningful comparisons across projects and inform strategic planning, media makers, funders and project leaders need common rubrics to structure evaluation. Shared benchmarks can also serve as a basis for collaboration or healthy competition among organizations who share common goals.
2. Following the story: Tracking the movement of content and frames across platforms and over time
   For many Summit attendees, influencing the public discourse surrounding a particular issue, perspective or policy is a critical component of their project’s mission. While there may be occasional flashpoints of exposure that are easier to identify, several attendees expressed concern that few tools exist to monitor the “slow burn” of coverage that might not be hot in the moment, but gathers attention and urgency over months or even years.
3. Contextualizing the anecdotal: Refining methods for analyzing shifts in public awareness, deliberation and behavior
   Many Summit attendees offered tantalizing anecdotes about user reaction to their content, but were frustrated in their efforts to connect these accounts with broader trends or concrete outcomes. Few have the funds to conduct surveys or focus groups before and after a project’s distribution, or to dedicate staff hours to tracking down data about users’ contributions, conversations and next steps.
4. Understanding our users: Creating more sophisticated profiles of audience demographics, habits and concerns
   Even many of the largest public interest media projects lack access to high-quality demographic research and user models that would help them to build sophisticated content, distribution and outreach strategies. Attendees at the Los Angeles Summit noted that, in addition to better and deeper information about their current users, they need data about how those users fit into the larger universe of potential audience members.
5. Moving beyond market assumptions: Defining the uses and limitations of commercial metrics schemes for assessing public interest media
   While many Summit attendees are using commercial tools and services to track reach, engagement and relevance, the usefulness of these tools in this arena is limited by their focus on delivering audiences to advertisers. Public interest media makers want to know how users are applying news and information in their personal and civic lives, not just whether they’re purchasing something as a result of exposure to a product.

Five Tools

After we discussed needs, we asked Summit attendees to think creatively about the tools that could really help them understand and communicate their projects’ impact. We then synthesized these ideas and suggestions into five recommended tools:

1. Putting it all in one place: Building a unified social media dashboard
   Many summit attendees expressed frustration with the inconsistency of current social media analysis schemes. “Dashboards” — which combine and analyze a range of data points on one screen — are in wide use across the online media environment. By developing or adapting a unified dashboard that integrates not only site-level metrics, but commonly tracked social media metrics, public interest media makers could develop their own comparative indices. In addition, public interest media makers could work together to develop tailored dashboard categories related to public engagement.
2. Chasing the frame: Building a social issue buzz tracker
   As noted above, many public interest media makers are seeking better ways to track both coverage on specific issues and the movement of their content across various platforms. They hope to pinpoint the “aha” moment, when a story or notion moves from merely interesting to spreadable. A few specialized tools have also been purpose-built for the public interest media sector. For example, Linkfluence tracks the movement of issues across ideologically similar blogospheres to compare how they are trending; the Media Cloud project helps users compare how certain topics are being covered across major news sources. More investment in building a social issue buzz tracker that combines visualizations, traditional public relations tracking, and social media metrics could yield significant new insights and a tighter integration between experiments, evaluation, and strategy.
3. Telling your story of impact: Developing model formats and processes for strategically communicating outcomes
   Ironically, while many public interest media makers are trained and experienced storytellers, they are often at a loss at telling the story of their own impact. Collecting and consolidating creative and effective ways to identify and substantiate impact could help the field to standardize this important process. For example, the American Independent News Network requires its reporters to demonstrate four “impacts” per year — substantive, trackable outcomes such as shifts in policy that follow on the heels of an investigative report, or exposure and trial of corrupt officials. While not all media projects will have such tangible outcomes, public interest media makers could still work together to share templates for reporting impact to funders and users, as well as to capture that information for internal decision making.
4. Asking the right questions: Creating common survey tools for evaluation and audience assessment
   Summit attendees suggested that they would be interested in developing two types of joint survey questions. The first were questions addressed to users of their own projects about impact; the second were questions appended to annual, national polls that could shed better light on media consumption habits and preferences. Within the field, expertise in designing high-quality survey instruments is limited. Partnering with a university or a marketing firm might be one way for media makers and outlets to jointly develop valid surveys to see how their work is engaging users. Contracting with large polling firms is a more expensive and complex prospect; more research is needed to suggest how best to work with them, and what kinds of questions might yield useful data.
5. Identifying networks: Creating a suite of tools that track the growth, health and effectiveness of networks
   While many of the summit attendees are still producing content for legacy platforms, the most cutting-edge projects are eager to develop evaluation tools that measure the growth of networks around their content or platforms. For example, the Public Insight Network is currently seeking evaluators to help them learn more about how their network of citizen sources is changing newsroom habits, providing a more diverse base of sources for stories, and creating community-level impacts tied to coverage. A first step in better understanding this emerging field might be a project to share best practices.

Funding: The Killer App

Based on an analysis of these summits, it seems that the immediate next step is to embark on a set of tool-building projects that will involve multiple funders and outlets. Building tools iteratively and collaboratively will help to surface which shared categories of assessment are most important.

For public interest media impact assessment to advance, however, the field’s foundations and investors are going to need to dedicate more funds to evaluation when funding production. Across the board, Summit attendees identified the need for more support to hire evaluators, purchase access to commercial tracking services, obtain evaluation training, and share best practices.

In addition, several attendees affirmed the need for the field as a whole to work on the sorts of joint frameworks and tools identified in a 2009 report published by FSG Social Impact Advisors, Breakthroughs in Shared Measurement and Social Impact. This would require unprecedented collaboration and resources. As that report warns:

The barriers to developing these systems, however, are formidable. They require a far-reaching vision, millions of dollars in investment, and years of effort by large coalitions of independent organizations.

But such an investment of time and money is worth it, if it helps to productively clarify the relationships between media funding, media production, and social impacts.

Jessica Clark directs the Center for Social Media’s Future of Public Media Project, and is a Knight Media Policy Fellow at the New America Foundation. Tracy Van Slyke is the Project Director of The Media Consortium and was recently named one of “30 Women Making History” by the Women’s Media Center. Together, they are the co-authors of Beyond the Echo Chamber: Reshaping Politics Through Networked Progressive Media, published in February by the New Press. The authors would like to thank the Ford Foundation and The Media Consortium for their support in conducting the summits and associated research.