For the past two years, the Center for Investigative Reporting has convened some of the best minds from media, philanthropy and research institutions to discuss how media organizations can effectively track and measure the impact of their work online and in the real world.
When we hosted our “Dissection A” event in Oakland, California, in 2013, the community and practice of impact measurement and evaluation were just starting to coalesce. Now that we’ve made it all the way to “J” in the Dissection series, the community has blossomed into a network of individuals and organizations that are building new tools, employing new methods and pursuing new shared work to help media organizations better understand their influence.
Our 10th and most recent Dissection was our most ambitious yet: a two-day event in Seattle, made possible by support from the Bill & Melinda Gates Foundation. With a packed agenda, participants representing organizations from every continent (except Antarctica) shared tools, strategies and best practices for impact measurement and analysis for nonprofit and commercial media, documentary film and education entertainment.
CIR’s Lindsay Green-Barber was the emcee for Dissection. Photo by Mark Glaser
At the end of the event, attendees reflected on the discussion and imagined a shared future for impact assessment. From this conversation and others in our Dissection series, these three themes have emerged:
1. We have moved beyond “why” and into “how.”
As our conversations about impact have progressed over the past two years, we have noticed an ebb in the philosophical debates about whether and why impact measurement and analysis are important. Now, more newsrooms are working to track and understand real-world changes sparked by their work. Furthermore, they are tying this work together with questions about organizational sustainability. Today, the conversations at Dissection and across the industry focus on specific research questions, methods and tools that can create new possibilities for media organizations of all types to measure their influence.
2. There are many tools and approaches for impact measurement.
A plethora of tools and platforms has been designed specifically for media impact measurement. At Dissection J, particpants got a firsthand look at several of these tools, including:
- Scale Model, a Twitter network analysis tool incubated at Betaworks.
- Media Cloud, a content analysis tool from MIT.
- NewsLynx, an integrated analytics and social network visualization platform.
- CIR’s Impact Tracker, a customizable online tool designed to create qualitative data sets of real-world change.
- Custom tools and analytics designed for organizations such as The Conversation, Medium and Peking University.
Controversy Mapper, one of the tools on Media Cloud, helps reverse-engineer major news stories to visualize how ideas spread, how media frames change over time and whose voices dominate discussions. Screenshot courtesy of Media Cloud.
Now, the challenge for organizations wanting to measure impact is to find and select the most appropriate tool — or tools — based on their own research questions and the types of outcomes they want to measure.
3. Partnerships are key for effective impact measurement.
Many of the most successful case studies in impact measurement and analysis include partnerships between media organizations and research institutions. For example, the Solutions Journalism Network and the University of Texas at Austin’s Engaging News Project partnered to do experiments to understand the effects of solutions journalism content. Through A/B testing of solutions-based stories and traditional stories, they found that, compared with readers of the traditional story, readers of the solutions story felt more inspired or optimistic, were more likely to say they would read more stories from the same newspaper, were more likely to say they would get involved in working toward a solution and were more likely to share the article.
In another example, NewsLynx and the Wisconsin Center for Investigative Journalism shared not only the utility of the NewsLynx, but also how the partnership had contributed to its development. The Wisconsin center explained that it was difficult to understand the relationship among page-level metrics, social media metrics and offline impact. Taking these challenges into consideration, the NewsLynx team built a platform that does just that for any URL. Michael Keller and Brian Abelson detail this process in a white paper from the Tow Center for Digital Journalism.
A case study presented at Dissection from a CIR story
What’s next for media impact measurement?
While the tools and projects highlighted at Dissection J help us understand the tenor of the current impact zeitgeist, the convening was designed to be a space for identifying next steps to continue moving the conversation and community of practice forward. At the end of the event, participants set out three main goals for the next year:
- Continue to expand and strengthen the impact network.
Dissection J benefited immensely from both a continual cohort of individuals steeped in impact measurement and analysis and from a fresh injection of international perspectives. Participants committed to strengthening this network through virtual work groups and in-person events. You can join the new Facebook group for discussing ideas, trends and questions for media impact here.
- Generate more findings that are widely applicable.
The case studies and methods shared at Dissection J generally were about a single project, topic or organization. In order to move toward more generalized findings and understanding of the phenomena associated with media impact, participants committed to sharing intelligence and publishing findings in peer-reviewed journals, articles, white papers and blog posts. That way, media organizations large and small can apply lessons from the experiments and innovation already happening in impact.
- Continue to build upon a shared language and framework.
CIR has proposed a framework for impact analysis, including a glossary of impact. This framework includes input from other media, ranging from single-issue nonprofit organizations to commercial media outlets. Participants at Dissection J committed to further employing this framework in their work, to make communicating about impact easier and more broadly applicable and to move toward more generalized findings.
Up next
CIR is constructing a resource center for media impact which will become a repository for more information about the tools, ideas and approaches mentioned here, as well as others being used in impact measurement and evaluation. If you have a tool or platform that you use, please share it with us so we can include it in our inventory.
This piece originally appeared the Center for Investigative Reporting’s Reveal site.
Green-Barber is the director of strategic research at The Center for Investigative reporting. She works to identify, assess and rigorously test areas of programmatic work where CIR can have catalytic impact through its content distribution and engagement. She leads research and analysis and serves as an expert both internally and for external partnerships. Previously, Green-Barber was an American Council for Learned Societies public fellow and served as media impact analyst at CIR. She earned a Ph.D. in political science from the City University of New York Graduate Center. Her doctoral research, conducted from 2011 through 2013 in Ecuador, focused on indigenous organizations’ use of new information and communications technologies for social mobilization. She also taught political science courses at Hunter College.
