Measuring Non-Profit Newsroom Impact Easier Said Than Done

Non-profit newsrooms, and the organizations that fund them, stand to gain a great deal by knowing the impact of their reporting on local communities.

But impact is not easy to compute, according to a recent report from the Investigative Reporting Workshop (IRW) at American University’s School of Communication in Washington, D.C.

“There is a discussion that has been going on now for a few years … There’s still not a complete consensus,” said Chuck Lewis, co-author on the study. He is the founding executive editor of the non-profit IRW, the largest university-based reporting center in the country.

The report, co-written by IRW intern Hilary Niles, surveys existing studies of newsrooms’ impact on the communities they cover.

Why impact matters

The question has “a sense of urgency,” Lewis said, for any of the 180 foundations that fund journalism in the United States. With the growth in non-profit newsrooms, those 72 organizations need to justify the worth of their original reporting as foundations ensure their investments are well spent.

The need for impact assessment, according to the study, is clear:

“With choice of consumption up to the consumer more than ever before, news organizations must raise the bar of their social value in order to maintain relevance with their audiences and be worth their monetary cost … If journalism doesn’t step up to the job that needs doing today, something else will — and the news industry will whither… what we are talking about is nothing less than the relevance and resonance of original reporting in the 21st century Internet Age.”

Beyond funding, journalists can use impact assessments to improve their ability to track their local communities, Lewis said.

That urge to improve editorially is particularly appealing to investigative journalists, who thrive on reporting that holds governments and corporations accountable, the authors note. Those journalists hope their work changes their community in that regard.

Lewis and Niles write that that’s different from advocacy. Quoting ProPublica’s Richard Tofel, the study notes that “journalism begins with questions and progresses, as facts are determined to answers. Advocacy begins with answers, with the facts already assumed to be established.”

Ways to find impact

The study raises more questions than it answers, but it does provide a number of suggestions on how newsrooms can discover their impact.

• Borrow from the “proven methods” of social science.
• Determine impact measurements based on audience. Different “atomized” communities that consume news content can and will interact with it differently.
• Measure impact on an interpersonal level first, then consider societal, macroscopic impact.
• Examine different metrics based on type of media used to tell the story.
• Define the key indicators of impact and make them “1) specific, 2) observable, 3) understandable, 4) relevant, 5) timebound and 6) reliable.” (“There’s a need for everyone to start speaking the same language,” Lewis said. “And right now, they’re talking at each other.”)
• Collaborate with other newsrooms with similar missions to develop common frameworks, saving time and resources.
• Follow examples of “rigorous” analysis, like those of Wisconsin Center for Investigative Journalism‘s Lauren Hasler, who, according to the study, uses spreadsheets and Google Alerts to track where the Center’s stories have been picked up. She then shares her methods with the Investigative News Network to help other newsrooms develop their own personalized framework.

Barriers to impact assessment

Impact is tough to assess because the measurements of journalism are changing.

“Benchmarks are gone or disappearing,” according to the study. “Horizons morph before our very eyes. New technology is replaced before its potential is even recognized. There is no new normal — yet.”

Reach (how many people consume content) and engagement (how much people interact with content beyond consumption, such as commenting, sharing or annotating), are by far the most common metrics used, though scholars often confuse or conflate the two terms, Lewis and Niles write.

Web analytics are a widespread tool, but they only provide obscure results. (Do you count page views or uniques? Does time on a page mean the reader is engrossed or can’t find what he or she is looking for?

It’s tempting to depend on the siren song of web analytics to determine impact, but the study argues reach and engagement are only two ways to measure impact. Consider that journalism can also have an effect offline (person-to-person conversations, print media, legislatures, etc.), especially because there’s still a large, and self-perpetuating, digital divide. (In 2012, the Center for Public Integrity’s John Dunbar, then at IRW, found that 40 percent of American households don’t have a broadband connection.)

Finally, just because a major societal shift occurs, it doesn’t mean it is entirely, or even at all, the work of that newsroom’s reporting. Lewis also acknowledged “The Long Tail” of impact; it could take months or years for the impact of a story to be realized.

“Look for progress … not credit for it, because a certain amount of this work is done on faith,” Lewis and Niles wrote.

Zach C. Cohen is an editorial intern for PBS MediaShift studying abroad at la Universidad Nacional in Heredia, Costa Rica. Get regular updates from @Zachary_Cohen on Twitter or on his blog.

Full disclosure: The author is a rising senior at American University. He has never had a class with Lewis nor worked at IRW.