Matt Carlson, Associate Professor at St. Louis University
Matt Carlson, an associate professor of communication at Saint Louis University, was set to announce a collaborative research project that would “connect a lot of dots surrounding news metrics and digital distribution platforms.” He wanted to examine journalism’s embrace of real-time audience data by shining a spotlight on “all the different actors involved, from reporters and editors and news management to engineers and salespersons at data analytic firms to the audience on the other end.”
But first, he needed to find a term that tied everything together. “Measurable journalism” was the solution.
In a special issue of the academic journal Digital Journalism, “Measurable Journalism: Digital Platforms, News Metrics, and the Quantified Audience,” nine researchers explore the implications of these technological and cultural shifts. Carlson, who edited the special issue and wrote an introductory essay, “Confronting Measurable Journalism,” explained his interest in this topic in an e-mail to MediaShift:
“When we think about measurable journalism, we need to keep in mind all the parts that go into it without privileging one over another. Often discussions of news metrics focus on technology, but it is equally about human actions that direct technology to do x and not y.
A concern I have with measurable journalism is when what can be measured takes precedent over what should be measured. These are sophisticated technologies, but they can only ever get to what people do. What we can’t know is what news audiences think or why they do what they do. I am always worried that user data becomes so fetishized that we forget it can only ever be a partial representation. We talk about such complex terms as impact or engagement but then we look for simple measurements.
The idea of measurable journalism can be both promising with the hope of creating journalism that is more accountable to the audience and frightening with the threat of journalists losing control over what is newsworthy. It may bring journalists and their audiences closer together or it might push economic imperatives ahead of journalism’s public service mission. Given these outcomes, what we need is vigilance and a solid understanding of all the forces in play.”
MediaShift discussed these forces in short interviews with the researchers who contributed to the special issue.
Quantified Audiences in News Production: A Synthesis and Research Agenda
Interview with Rodrigo Zamith, assistant professor, University of Massachusetts-Amherst
Why the interest in studying measurable journalism?
There are two main reasons. The first has to do with a professional observation: Journalism is becoming more sensitive to and powered by measurement. We see this in the rise of “data journalism” as well as the proliferation of audience analytics — the latter being the focus of much of my recent work. Those shifts have important implications for how journalism is constructed, performed and rewarded, yet the phenomenon is not yet well understood by scholars (or practitioners). The second reason is personal: I’m a nerd and think in terms of numbers. I find the tensions playing out as quantification and quantitatively oriented actors gain foothold in newsrooms to be fascinating.
What did your research show?
My contribution focused on synthesizing the current literature on audience analytics and metrics, offering new lenses for studying the phenomenon and identifying future research directions for the scholarship. Three arguments stand out in the piece: First, we are witnessing a new wave of audience measurement in journalism (following two waves in the 1930s and 1970s) that is driven by audience analytics (systems that automatically capture information about individuals’ media use). Second, while contemporary journalism is not being driven by quantifications of audiences (i.e., audience metrics), both audiences and quantification are playing far more prominent roles in news production than in the past. Third, scholars and practitioners have become less pessimistic about the impact of audience metrics and now recognize more nuanced impacts on news production as well as opportunities for using them to advance journalistic goals.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
Journalists and educators need to take analytics and metrics seriously. The measurement of audiences will only become more sophisticated and news organizations will face further pressures to make use of those data. Journalists should seek out training on how to use their newsrooms’ analytics suites (e.g., Chartbeat) and/or ask for permission to access the system. Educators need to ensure they incorporate analytics and metrics into their curricula and also provide students the opportunity to engage with those suites (e.g., integrating them into student media offerings, at minimum). In both cases, serious conversations need to be had about how to use those data sensibly — from influencing organizational coverage decisions to developing reward structures for individual journalists. I find the argument that metrics should neither be restricted to the business side nor the primary driver of journalism to be especially persuasive. Analytics are tools that can be put to good use, and that means trying to align journalistic aims like satisfying community information needs with the many data points that analytics can offer. At the same time, those tools can be misused and practitioners should therefore maintain a healthy skepticism and promote robust dialogue.
The Audience-Oriented Editor: Making Sense of the Audience in the Newsroom
Interview with Raul Ferrer-Conill, Ph.D. candidate, Karlstad University, Sweden, and Edson C. Tandoc, Jr., assistant professor, Nanyang Technological University, Singapore
Why the interest in studying measurable journalism?
In the last decades we’ve seen an increasing quantification of journalism, spurred by the use of metrics and analytics. These metrics are effectively the new way by which news organizations make sense of the audience. Combined with the commercial urgency, the need to entice and engage audiences makes the quantification of audience news consumption a key factor to understand the current news production process. Researchers and practitioners should pay more attention to how journalism production is quantified, measured and understood.
What did your research show?
First, the definition of engagement is almost entirely centered on different types of metrics. Second, while audience-oriented editors take part in the editorial process, their role is to help journalists negotiate between the information obtained by their metrics and their journalistic intuition to make editorial decisions. Third, there is a lack of cohesiveness regarding what these newsroom positions are and how they operate. We provide insight on the pervasiveness of metrics and quantification of journalistic processes by offering a more nuanced understanding of a new set of editorial roles.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
The more that metrics are part of news production and the increasing number of audience-oriented editors legitimize and institutionalize metrics. Therefore, understanding the impact of audience measurement on news work requires an analysis of these emerging roles, who act as intermediaries between audiences and the newsroom through their interpretation and valuation of audience data. The reliance on metrics and social media insights questions their capacity to capture the audience. The analytic tools are constrained by what they can measure and rely on likes, shares, number of comments and other audience metrics to define engagement. In this sense, it is user activity and behavior that becomes a proxy for the voice of the audience. This is a limited understanding of the audience, let alone having a dialog with the audience. Editors can assess the performance of their editorial choices as they scrutinize metrics in real time, but they are limited by and reliant on the technological affordances of the tools they use. We argue that this dialog is predominantly informed by metrics and therefore it needs to be understood as such. Metrics are not necessarily a valid way to measure audience engagement and should be used cautiously.
Selecting Metrics, Reflecting Norms: How Journalists in Local Newsrooms Define, Measure, and Discuss Impact
Interview with Elia Powers, assistant professor, Towson University
Why the interest in studying measurable journalism?
My interest is in examining the ways in which journalists think about and ultimately measure their work’s impact. Impact is a buzzword in newsrooms, but there are so many ways to define the term that little can be done to move the conversation forward until there’s more clarity about what journalists mean when they talk about impact. It’s also critical for journalists to feel comfortable publicly discussing their work’s impact, because newsrooms now more than ever need to make the case to audiences and funders that civic-oriented journalism makes a difference in local communities.
What did your research show?
Interviews with journalists from a range of local news organizations in one U.S. city found that they welcome the opportunity to inform audiences and effect change, and they had no issues discussing impact with newsroom colleagues and in promotional materials. However, journalists were generally more hesitant to discuss their work’s impact outside the newsroom — in follow-up news stories, social media posts, interviews, etc. Some journalists were concerned about being perceived as too self-congratulatory or being labeled advocates. There was a perception among some participants that publicizing impact violated journalistic standards of objectivity and detachment. Additionally, journalists had many ways of defining and measuring impact, one of which was audience analytics — although many felt these were more about engagement than enduring signs of impact. Effect-oriented metrics (audience awareness, public discourse, public policy, etc) were widely considered the best to assess impact but among the most difficult to systematically measure.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
First, I propose that measurable journalism should not be limited to quantitative metrics, often the standard by which news coverage is judged. Much of what journalists want to measure has little to do with audience analytics. Second, as I argue in the article, “publicizing the impact of journalism, when facts support such a claim, is central to the journalistic process and necessary for newsrooms to justify their funding…Newsroom policies and professional codes of conduct should clarify that engaging in public discourse on impact is central to the journalistic process, a necessary part of communicating with the public, and a way for newsrooms to justify their funding rather than a sign of self-promotion or
advocacy.”
Dimensional Field Theory: The Adoption of Audience Metrics in the Journalistic Field and Cross-field Influences
Interview with Qun Wang, Ph.D. Candidate, Rutgers University
Why the interest in studying measurable journalism?
Fifteen years ago, I was a TV news anchor and reporter in Beijing, China. I was able to access both quantitative and qualitative audience information due to the nature of the show that I covered: on the one hand, we kept a close eye on the ratings because the show was a signature show in Beijing’s competitive TV news market; on the other hand, we had our ears open to the three hotline phones on which the show relied for audience members to share news tips, feedback and comments like “I don’t like the anchor’s hair!” We sometimes learned a lot from the audience information and sometimes got lost in it. The show ceased years ago, but I have never stopped wondering how the team would deal with today’s audience information in the digital age if the show was still around. This work experience and this particular question have contributed to my interest in measurable journalism.
What did your research show?
In the years that I worked in the newsroom, I already felt that newsroom practices and norms were often a result of the negotiation of different forces. In my study, drawing on French sociologist Pierre Bourdieu’s field theory, I took a historical and relational approach to embed the journalistic field into a wider media ecosystem. Particularly, the study examines how the adoption of audience metrics in news media has been shaped by influences inside and outside the journalistic field in order to understand the origin, driving forces and implications of this trend. I looked at neighboring fields adjacent to the journalistic field, such as the online advertising and online audience research fields, as well as web analytics services working with news media that I identified as the intermediate field to trace the evolution and influence of these fields. I also looked into the journalistic field itself and identified three dimensions — the techno-economic means, the subject of journalism and the object of journalism — that construct the field and serve as sources of internal influence.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
The main takeaway of this study is that the journalistic field is not a static or monolithic arena. Rather, it is a social universe that is subject to and constantly interacts with cross-field influences. Therefore, to understand the root, development and effects of measurable journalism, we may need to inspect our own field and look elsewhere.
Boundary Work, Interloper Media, and Analytics in Newsrooms: An Analysis of Web Analytics Companies’ Role in News Production
Interview with Valerie Belair-Gagnon, assistant professor, University of Minnesota, and Avery Holton, assistant professor, University of Utah
Why the interest in studying measurable journalism?
Journalism has been undergoing a series of fascinating changes for several decades. In particular, social media has challenged the ways in which we measure the success of journalism. Much of that success still rests in financial growth, or at least stability, which itself is increasingly dependent on audience interactions with journalists and the content they produce. We’ve begun moving past a reliance on journalistic or editorial intuition and instead see tangible value in understanding complex web metrics and analytics. So if the latter are beginning to drive journalistic decision making, particularly in news production and professional identity, then they are critical to examine.
What did your research show?
Our most recent research shows that web analytics companies seek to understand and address news production values and norms without assuming responsibility as journalists. We think of these companies, or their employees more specifically, as implicit media interlopers. These are journalistic outsiders, more or less, who are bit more welcome in the journalistic process than previous interlopers (e.g., citizen journalists, bloggers) because of the value they add to news products. These companies also foster profit-oriented norms and values in newsrooms by introducing web analytics as disruptive, connective and routinized in news production. By offering a product that needs to be modified on a continuous basis because of changes in the structure of the web and audience behaviors, web analytics companies foster a milieu of constant experimentation with old and new products. This helps place them squarely in the middle of evolving news organizations that are turning more to disruptors and innovators as they grapple for financial footing.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
Today, like many other technological innovations in newsroom, disruption increasingly comes from innovations from outside companies and individuals. As journalism evolves, scholars and practitioners need to understand more deeply what the values and practices are that these disruptive innovators bring to journalism. We’re not just talking about web analytics here, but rather programmers, app developers, drone hobbyists, and others who are interacting with journalists and news organizations in ways that are giving new meaning to what exactly journalism is and who exactly is doing it.
Engineering Consent: How the Design and Marketing of Newsroom Analytics Tools Rationalize Journalists’ Labor
Interview with Caitlin Petre, assistant professor, Rutgers University
Why the interest in studying measurable journalism?
Way back in 2010, Nick Denton, founder of the now-defunct Gawker Media, said, “probably the biggest thing in internet media isn’t the immediacy of it, or the low costs, but the measurability.” Superlatives are tricky, but Denton was surely right that the unprecedented ability to measure audience behaviors and demographics is a defining characteristic of digital media — one that has major implications for the working conditions in this industry and the kind of journalism that is produced. Media scholars have an urgent role to play in helping to interpret and explain the causes, manifestations and consequences of measurable media.
What did your research show?
In the early stages of my research on the role of analytics in journalism, I kept encountering the same puzzle. Journalists at a wide range of news outlets would profess a profound wariness or even hostility toward analytics tools, often seeing them as a threat to their professional autonomy and integrity. This by itself wasn’t all that surprising: Sociological research has found that workers (especially those, like journalists, who consider themselves to possess some kind of special knowledge or expertise) often resist the implementation of technologies that quantify their performance and rank them against each other.
But even as they regarded analytics tools with suspicion and resentment, journalists didn’t seem to be resisting them very much. On the contrary, many journalists would describe feeling “addicted” to real-time analytics tools, consulting them more frequently than was required or even encouraged by their managers, and scheming about how to boost their stats.
My article aims to figure out why that is. I find that a big part of the answer has to do with something that often gets overlooked in these discussions: the design and marketing of real-time newsroom analytics tools. Newsroom analytics companies engineer their dashboards to provide a user experience that is strongly habit-forming, flattering and emotionally compelling. The resulting products are so “sticky” that explicit managerial coercion to boost traffic (which many journalists would not take kindly to) becomes unnecessary. Once journalists get hooked on looking at real-time analytics tools, they begin to monitor themselves. They also push themselves to work harder and harder in hopes of gaining ever-higher traffic.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
When we have conversations about analytics in journalism, we tend to focus on which metrics are provided: time spent or page views? Scroll depth or uniques? In other words, we assess the merit of each metric and speculate about the kind of journalism it might incentivize. But my findings indicate that anyone seeking to make sense of the role of analytics in contemporary journalism should be just as attentive to the way the data are presented, and the daily experience of using these tools, as we are to the metrics themselves.
The Elusive Engagement Metric
Interview with Jacob Nelson, Ph.D. candidate, Northwestern University
Why the interest in studying measurable journalism?
Over the past few years, a growing number of journalism stakeholders and researchers have argued that newsrooms should make “audience engagement” one of their chief pursuits. This term has many interpretations that stem from one underlying belief: Journalists better serve their audiences when they explicitly focus on how their audiences interact with and respond to the news in the first place. However, those who hope to make audience engagement a larger part of journalistic practice need to first settle an internal debate surrounding how audience engagement should be defined and evaluated. Because the term currently lacks an agreed upon meaning — let alone metric — it has become an object of contestation. The efforts to make audience engagement central to news production therefore present an opportunity to learn how journalism is changing, as well as who within the field have the power to change it.
What did your research show?
My study draws on an ethnographic case study of Hearken, a company that offers audience engagement tools and consulting to about 100 news organizations worldwide. Findings show that news industry confusion surrounding how audience engagement should be defined and measured has left Hearken unable to quantify the benefit of its offerings. The news industry currently privileges measures of audience size, so newsrooms face economic incentives to pursue audience growth (which they can measure) rather than audience engagement (which they can’t). Instead, Hearken’s pitch to newsrooms relies primarily on appeals to intuition. Its employees argue that their interpretation of audience engagement will lead to a better quality of journalism, which will inevitably result in increased audience revenue as well. Though some newsrooms refuse to invest in Hearken’s offerings without proof they will yield some measurable return, others seem eager to take the chance. The success of Hearken’s faith-based approach indicates that many in journalism innately believe the profession should improve its relationship with the audience.
What are the main takeaways for journalists, journalism educators and others who are interested in media metrics?
Hearken’s effort to spread its interpretation of audience engagement is just one piece of an ongoing public contest to determine journalistic practice. There are countless conversations about audience engagement that occur annually at a variety of journalism practice and research conferences. These conversations tend to include editors, reporters, and publishers, but rarely include employees of companies like comScore and Nielsen who are in the business of understanding how audiences behave. What makes this omission confounding is the fact that these firms are having their own conversations about audience engagement. The fact that these conversations are taking place shows that the major players within the news media environment believe that how audiences engage with media is worthy of consideration. On the other hand, the fact that these stakeholders with disparate interpretations of audience engagement have yet to come together reflects just how convoluted the term has become. How the term is ultimately defined and measured will have consequences not just for how journalists produce the news, but also what they expect of public – as well as what the public expects of them.
Elia Powers, Ph.D., is an assistant professor of journalism and new media at Towson University. He writes regularly about news literacy, audience engagement and nonprofit journalism.