How the USA Today Network Is Measuring Impact in Its Newsrooms

A version of this article was originally published by the Media Impact Project.

Measuring the audience response to a journalistic story usually means counting page views and unique visitors, yet that assessment falls short. At my Gannett newspaper in the suburbs north of New York City, we wanted to know more: What happened when we started asking questions (about a story)? After we published a story? When a key decision-maker or a crowd of people saw that story?

Did we spark a change in law, a donation, a citation or the dismissal of an unlawful employee? Did the family who lost their father’s Purple Heart years ago finally find it, with help from the crowd? Did we make a difference?

If you’ve arrived at this post, I probably don’t need to say that journalism matters. For many of us, it’s a calling and a purpose and so on. But while print newsrooms especially have been hesitant to talk about the importance of our work lest we seem boastful, we need to drop the pretense and get comfortable reminding our readers, advertisers and subscribers of the impact we have on our communities — that is, if we wish to hold onto our relevance and our institutional heft.

That’s why we at the Journal News and lohud.com built an impact tracker — to record not the clicks but the real-world change our journalism inspires.

Examples of the impact we track

The tracker primarily serves, but is not limited to, two types of journalism: explanatory and investigative. Whereas the purpose of explanatory journalism is to make something clear (improve understanding, change minds), the purpose of investigative journalism is most often to reveal the secrets held close by those in power.

Documenting the impact of this journalism requires diligence and truthfulness on the part of the reporter; he or she must be able to prove, to a reasonable degree, that the change is the result of his or her work and not coincidence. It also requires time. The impact of a story might not be felt for months or years (for example, if a law is to be changed).

Do it well, though, and you will have information like this at the ready:

• “The Pelham Manor Board of Trustees initially suspended Alfred Mosiello over racist emails he had forwarded. But within days of our revealing some of the emails, the board pressured him to retire.”
• “The story led to Libby Pataki’s resignation as tourism director, the dissolution of one nonprofit, and the reconstituting of another, with a proper board of directors and appropriate oversight.”
• “After the story was published, the Federal Railroad Administration announced that it was considering a rule change that would limit the amount of hours track workers are allowed to put in during a day.”
• “In response to a lawsuit seeking to improve secular education in some Hasidic yeshivas, Assemblyman Kenneth Zebrowski, D-New City, crafted a bill that allows the state to enforce the legal standard of ‘substantial equivalence.’ The lawyer who filed that suit credited our coverage with emboldening plaintiffs to come forward with the complaint.”
• “Our reporting and outreach to U.S. Sen. Charles Schumer’s office led to the Purple Heart being returned to the family of the World War II soldier and POW.”

How we use the information we collect

These stories of impact help our newsroom talk about our work in several important ways:

• In columns, letters and speeches explaining the ways we’ve inspired change.
• For contest entries in which impact is a requirement.
• To add to our understanding of audiences and what moves them to action.

And finally, an area we’d like to explore: How can we use these examples of our impact in promotional and sales materials encouraging support for our journalism?

Why we deemphasize social media and what comes next

Many news organizations and foundations studying impact rightly track social media shares to show reach, influence, and money well spent. While our tracker includes an option to explain the social media impact, we chose not to put our focus there for two reasons:

1. We have other tools that are much better at that than we can be (CrowdTangle, Facebook Insights and Twitter Analytics, to name a few). We use those in concert with the tracker to see a fuller picture of what’s happened with our stories.
2. You get what you measure, and we’re simply more interested in inspiring real-world change.

We started the tracker two years ago as a Google Form because it was simple to spin up and revise as we learned more about impact tracking and how it can help our newsroom. (Much credit for inspiring this first version goes to the wise and thoughtful Lindsay Green-Barber, founder of The Impact Architects and formerly of the Center for Investigative Reporting.)

Today Version 2.0 of the tracker is a Ruby on Rails web application developed by Avram Billig to populate a database. Journalists manually add impacts and tie them to stories, and we can then filter and view by story, author, date, impact, etc. As reporters from multiple newsrooms add their work, it gets categorized and stored in a PostGreSQL database that we query to discover the analytics of impact over time. We’re sharing this version of the software with other USA Today Network newsrooms, and we’re beginning to think about 3.0 features such as connecting impact to traditional analytics and data visualizations. Each iteration of the software is a chance for greater understanding.

Questions about the tracker or how we’re studying impact? Email me at adelgado@gannett.com or message me on Twitter @anjdelgado.

The tracker has been a team effort from the start. Avram Billig turned my visions for the tracker into reality and is the reason reporters and editors think this is all much easier than it really is. Find Avram on Twitter @aabillig. Traci Bauer gave us the support and freedom to develop the tracker, and she helped shape many of its features. She is the vice president/news for The Journal News/lohud.com and The Poughkeepsie Journal. She believes empowering a community with strong journalism is the best way to build loyalty among audiences, and tracking their actions helps measure our own success. Email Tracy at tbauer@gannett.com and on Twitter @tbauer.

Anjanette Delgado is the digital director and head of audience for lohud.com and poughkeepsiejournal.com. You can reach her at adelgado@gannett.com and on Twitter @anjdelgado.