Is There a Master Metric for Evaluating Public Media?

This article was co-authored by Katie Donnelly

Over the past few months, we’ve been presenting MediaShift readers with a picture of a more dynamic, engaged, public media future. But how are Public Media 2.0 projects measuring their success in informing and engaging publics?

Embracing Digital: A Review of Public Media Efforts Across the United States, a report by Gupta Consulting, gives us an idea of the scope of the challenge:

Very few stations define success with concrete metrics. Most examples are anecdotal. (“I just have a sense.”) What they consider to be “successful” is very subjective. Those that do have an idea of what success means to them include metrics such as page views, unique users, and calls into station when online offerings fail to work.

This lack of clarity is compounded by the fact that Public Media 2.0 projects cross a range of communities of media production practice, from journalism to documentary film to user-generated content to videogames. The various projects that now engage publics around issues are marked by distinct goals and assumptions, sometimes based on the capacities of the platform, and sometimes on the intentions of the makers.

This used to be less convoluted. In the broadcast era, impact was measured most often in terms of audience size, with companies such as Nielsen, Arbitron and the Audit Bureau of Circulations serving as impartial audience talliers for different platforms. Now, in the era of converged media, there are many more moments of media consumption to count, and fewer clear comparisons.

While auditors exist for websites and blogs — for example, Alexa, Quantcast, and Technorati — the tools for tracking audiences online are still a bit blunt. Mark Glaser wrote at length about the problems with web measurement on MediaShift in 2007, noting that “the complexity of web measurement and new technologies makes it nearly impossible for a one-size-fits-all measurement to become the currency of online ratings.” Plaforms like Twitter and increased mobile and app-based media consumption have only further muddied the waters.

What’s more, given social media’s capacities for fostering participation, both public and commercial producers have to track a variety of new indicators, including engagement, online influence, and the inclusion of a range of users in media creation. The ecosystem of communication has shifted significantly. Media makes its way to users not just via selected screenings or broadcasts at fixed times, but through overlapping networks of individuals and institutions, as depicted in the visuals that Jessica developed with her co-author, Tracy Van Slyke, for their recent book, Beyond the Echo Chamber.

For public media makers, commercial measurements designed to track purchases or ad impressions aren’t always enough to assess whether a media project contributes meaningfully to public life. Instead, assessing such projects’ success depends first upon identifying the core stakeholders and intended outcomes, and using both qualitative and quantitative approaches to measure against related goals.

Is it even possible to create a master metric? What are the differences and similarities in evaluating different kinds of projects? Let’s take a look at four communities of practice within public media to see how they are tackling this thorny issue.

Communities of Practice

Traditional public broadcasting
Public broadcasters are still casting about for standards. The Embracing Digital report found that “few station executives can quote quantitative measures of either goals or achievements related to their digital offerings,” not to mention having a sense of how to measure whether they’re meeting a public mission. In fact, some stations were not even able to provide a rationale for creating particular digital offerings. Clearly, accurately measuring impact is difficult — if not impossible — if producers are not able to identify their own motivations.

Within public broadcasting, there is wide variation in the ability of stations to implement evaluation and analyze metrics. The stations in the Embracing Digital report used varying methods, including both quantitative (subscription numbers) and qualitative (user feedback). Stations were excited about the implementation of PBS’s new Comprehensive Online Video Ecosystem (COVE), which will allow them to better track digital downloads. The report concluded that “there is a real need for stations to understand metrics, what data to collect and how to use information to their advantage.”

While some stations may still be catching up, there is help available at the national level for public broadcasting. The CPB-funded National Center for Media Engagement (NCME) has developed a framework for strengthening impact through effective community engagement. This framework includes a spectrum of community engagement that lays out internal and external engagement indicators.

Internal indicators include station culture, station strategy, station commitment and board endorsements; external indicators include community partnerships, interaction, results, and public perception. There is also a diagnostic tool to measure current community engagement levels and a PlanIt! Tool” that helps users identify inputs, activities, outputs, changes in informal systems and networks, and benefits for people. Using these two tools together underscores the importance of incorporating evaluation processes at the beginning of the production process.

Social issue documentaries
Documentary filmmakers are often more explicit about the social and political goals of their work, which opens the door for more creative, outcomes-driven impact assessment. The Fledgling Fund, a foundation that supports social-issue media projects, developed a framework for assessing impact for creative projects with a mission. Noting that “it can be surprisingly difficult to make a firm connection between the power of a film or other media and social change,” the Fund created Assessing Creative Media’s Social Impact.

This toolkit lays out five dimensions of impact, including:

• Quality film/media project
• Increased public awareness
• Increased public engagement
• Stronger social movement
• Social change

The guide includes sample measurements for each of these categories. For example, “increased public awareness” can be measured by audience size, press coverage, and audience diversity. “Increased public engagement” can be measured by website hits, participation in online discussions, and commentaries and response letters. “Stronger social movement” can be measured by the number of advocacy organizations making use of the documentary, screenings with influentials and decision-makers, and the film’s longevity. For more sample measures, see Figure 3, “Sample Measures for Dimensions of Impact,” in the report.

Mission-driven gaming
Games for Change published an evaluation toolkit that includes video commentary from experts in the field about how to assess the social impact of videogames. The gaming community has some advantages when it comes to measuring results. For example, assessment vehicles can be directly embedded into games and data can be gathered with relative ease. Additionally, because gamers often return to games again and again, assessment tools like play diaries can help determine individual impacts, particularly in terms of knowledge acquisition or attitude shifts.

Game producers are currently asking the same qualitative questions as social issue documentarians: Did users engage in dialogue? Did users’ attitudes change? Did they take action? Did a movement develop? Did a policy change?

However, this toolkit also urges game producers to consider expectations, external pressures, and economic realities, noting that producers can face difficulty in getting funders to continue their support without providing them with traditional quantitative metrics, such as how many times the game was downloaded. Interestingly, the accompanying guide for this toolkit encourages game producers to occasionally “Just say no,” and let games stand as art (“art resists metrics”) by avoiding evaluation altogether.

Youth media
Many youth and community media makers are focused more explicitly on empowering users with production tools than creating and disseminating content. This changes their impact assessment routines significantly. In the Youth Media Evaluation Toolkit [PDF file], a guide prepared for the Open Society Institute and the Surdna Foundation by Social Policy Research Associates in 2005, it’s clear that youth media evaluation differs from other communities of practice in that young producers need to be evaluated as much as or even more than their productions. The report states:

Research has found that when youth authentically conceive, develop, and produce news articles, radio commentaries, etc., youth media can foster important individual level outcomes, such as youth voice, critical thinking, research, literacy, writing, media skills and broader youth and career development outcomes.

In fact, research on youth producers has typically been better supported and funded than research on the impact of their productions. However, both can be measured simultaneously by directly involving youth in the evaluation process. This helps assert their position as stakeholders as well as equalize the youth/adult power dynamic.

This toolkit recommends measuring youth media productions by breaking potential outcomes into a series of categories, including:

• Affective/Feel: How has the youth media production affected how people feel about the issues presented?
• Cognitive/Think: How has the youth media production increased awareness and knowledge about the issues presented?
• Behavioral/Act: Has the production led individuals or groups to take action?

Similar to the Games for Change toolkit, this guide reminds practitioners that they must always consider the organization’s mission, the current political and social climates and the available resources (or lack thereof). And like game producers, youth media organizations often do not have the capacity or funding to conduct the types of formal evaluations that funders typically require. In fact, like some traditional public broadcasting stations, many youth media organizations struggle just to document number of viewers.

Shared Questions

While there are some differences among them, these communities of practice have many shared questions when it comes to impact: What are the audience numbers? Have users’ thoughts or actions changed? Has a movement developed? Has policy shifted?

The most pressing question that these communities all share, however, is: What is the best way to measure social impact?

These questions are not unique to public media projects; foundation-funded non-profits and socially responsible investors have been asking them for several years. For example, Social Impact Assessment: A Discussion Among Grantmakers, a 2003 report from The Rockfeller Foundation and the Goldman Sachs Foundation, notes:

The field has yet to establish a common understanding of “social impact” — what it is or how to measure it. Currently, measures of impact vary from funder to funder, and organization to organization. The more sophisticated measurement tools integrate organizational and process metrics with quantifiable outcome data, but in the absence of a common measure (like shareholder value) investors and grantmakers are making it up as they go along.

So for now, it seems there is no master metric for Public Media 2.0. Both qualitative and quantitative tools are needed, and mission needs to play a guiding role in determining what constitutes success for each project. However, public media funders, makers and outlets can still work together to develop more precise and standardized tools and approaches for measuring the elements of impact that they all care about, and that relate back to their social goals.

We’ve identified five elements to explore, which we address in greater depth over at the Center for Social Media’s Public Media 2.0 Showcase. Each of these elements represents a measurable category of activity that helps media projects convene publics around issues:

• Reach: How many people encounter the project across various screens and streams: TV, radio, streaming audio, blogs and websites, Twitter, iTunes, mobile applications, and more?
• Relevance: Is the media project topical within the larger news cycle? Is it designed to stay relevant over several news cycles? Is it particularly relevant to targeted publics concerned with a specific issue, location, or event?
• Inclusion: Does the project address a diverse range of targeted audience, not just in terms of race, but in terms of gender, age, class, geographical location and beliefs? How open is the architecture for participation, collaboration and discussion?
• Engagement: Does the project move users to action: to subscribe to a site, contribute material, to write a letter in response, to pass on a link, donate time and money, sign a petition or contact a leader?
• Influence: Does the project challenge or put the frame on important issues? Does it target “influentials”?Is it it “spreadable” or buzzworthy?

Reach is the most familiar category, but, as we noted above, some public media projects are struggling to even measure this. Many public media projects, across practice communities, are starting to measure engagement through outcomes such as time spent on sites, site subscriptions, and user feedback. The Fledgling Fund recommends social issue documentarians measure relevance through longevity, while social media metrics tools like Addict-o-matic can give some measure of how relevant online media projects are in the real-time web. Gamers, social issue documentarians, and youth media practitioners all aim to track influence — including assessing whether media projects target civic leaders, and help to spawn or support viable social movements.

Over the next few months, we will be holding a series of “impact summits” around the country to explore best practices in assessment for Public Media 2.0 projects. Stay tuned for insights from the funders, makers and media leaders we’ll be talking to.

Jessica Clark is the director of the Center for Social Media’s Future of Public Media project, and the co-author of Beyond the Echo Chamber: Reshaping Politics Through Networked Progressive Media.

Katie Donnelly is the associate research director at American University’s Center for Social Media, where she blogs about the future of public media. When she’s not researching media, Katie spends her time working in the environmental field and blogging about food.