Ryan Sholin presented at Poynter’s Measuring Journalism event this month and shared the talk on his blog. He has kindly allowed us to republish it here. Slides are also available.
I love it when the numbers jump off the screen.
In my dozen-plus years in the news business, whenever I’m asked — often in a job interview — what my favorite part of my work is, I tell a little story about the first time I took a serious look at an analytics report for multiple news sites, back at GateHouse Media. Sure, I had been responsible for paying attention to Omniture (or was it still HitBox?) back at the Santa Cruz Sentinel, and we even had a pretty cool heatmap plugin of some sort which let us see how stories on the homepage were doing. (This was years before Chartbeat.) But at GateHouse, I worked with around 125 community newspaper sites.
Months into the rollout of a new content management system — and really, a new way of working, the first time most of these newsrooms were publishing stories in the middle of the day, or breaking any news online at all, and definitely the first time they shot and edited their own video — I took a look at the metrics.
The differences in pageviews and unique visitors jumped off the screen at me. Newsrooms with strong leaders that had some digital experience were succeeding in the new model, as measured by these metrics, at least. It also helped me realize other newsrooms might need more training on the new system!
That experience — that aha moment — is the one I talk about when people ask me what I enjoy most about my work.
I love it when the numbers jump off the screen and answer a question.
Telling the story of metrics
I’m going to talk about three key elements you’ll need to tell a great story about metrics for an internal audience. I’m usually talking about newsrooms, but this should work for most organizations.
- How to identify the metrics that matter for your organization.
Or at least, how to focus on the metrics that make sense for your organization’s goals. - How to make metrics useful — where useful equals actionable.
(I apologize in advance for the business-y paradigm here, and you should feel free to offer improvements on the word “actionable” in the comments, on Twitter, to my face, etc., but I think it’s a good start.)
- How to report smarter, not harder.
Ban the scourge of endless copy/paste from your analytics routines.
Saved to Pinterest by a second grade teacher. Search “data wall” on Pinterest for many, many more like this. They have goals! And KPIs! And graphs! You can, too.
Which metrics matter?
Look, here’s the thing. There are lots of metrics. Not all of them matter. I mean, they do, sure, to somebody, somewhere, yes. But probably not to you.
The metrics that matter to you, dear reader, are the ones you’ve identified as key performance indicators (KPIs) tied to a particular tactic that ladders up to one of several strategies you’ve set for, say, the fiscal year, which in turn ladder up to your vision for the next several years, which of course you’ve aligned with your mission as an organization.
[muffled laughter]
OK, OK, really, I mean, some of you have done this level of strategic planning. But in a newsroom, plans can change pretty quickly. Maybe an election doesn’t go the way most people expect it to, and the focus of your newsroom turns on a dime. Maybe you end up in a running battle with the executive branch, or Congress, or the state legislature over access to public information. Anything could happen.
But, if you’re the one responsible for answering questions like “How are our stories doing?” or “What’s driving subscriptions these days?” or even “How can we get more people to show up to our events?” you’re going to need to set some basic goals and identify the key metrics that indicate how things are going.
I’m not going to do a whole goal-setting workshop here, but suffice it to say there are plenty of methods, lots of ideas about how to do it right, and you can start small.
For example. If increasing the size of your overall audience is a goal, hey, there are some key metrics that will serve you well. Pageviews, New Users, social referrals, there are a bunch that would be good to track.
Let’s look at a big menu of metrics we could be paying attention to:
So that’s too many things. Don’t try to pay attention to all of this at once.
Let’s take a look at the same list, with some highlighted KPIs that might be relevant to your goals as a news organization.
Analytics bingo, anyone?
That’s better, right? If your goal is audience growth, identify the KPIs that will tell you if you’re getting your stories in front of more new users.
If you already have a sizable audience, and you’re trying to get them to come back more often — hey, maybe someday they’ll buy a subscription or make a donation if that’s how you roll — you need to look at a different set of KPIs.
Can you do both at once? Of course, sure, why not. But don’t conflate growth metrics and loyalty metrics if you can help it.
What other goals might your organization have besides audience growth and increasing loyalty? Plenty, sure, including little things like advertising revenue. Use a broader metrics menu like this one as a prop to help talk with the rest of the people in your organization (I’m trying so hard not to say stakeholders) about which metrics matter for which team, right now, based on your current goals.
Jack needed a little more context to gain actionable insights from the available data. We’re pretty big Nightmare Before Christmas fans in my house.
Making metrics useful
Cool, great, so now you’ve identified the metrics that matter, and you’re tracking them closely (oof, more on that shortly), and all this quantitative data is flowing into your dashboards and spreadsheets and you can do the math to show how the numbers change over time. You’ve got graphs! We love graphs! Data visualization is great. But… why did the numbers change over time? What happened? Where did the new users come from? Which stories brought people back for second and third visits within a 30-day period? Is the language in our email newsletter signup form working?
Questions! So many questions.
Your reports should try to answer some of them. And they should include ideas about what someone (editorial, product, business, leadership, engagement, etc.) should do next as a result. Be brave, be bold, make suggestions, especially when you know from the data these actions lead to results.
But at the start, the narrative of your quantitative report might look something like this if you wrote it out in words:
Pageviews increased by 25% in October 2017 compared to September 2017, but they were down 10% from October 2016.
OK, cool, sure, that’s nice. But we have so many questions. So, instead, let’s dig a little deeper for context and provide actionable insights (sorry, not sorry).
Most of our pageview growth this month came from new users reading our Steph Curry story that went viral on Facebook, with triple our usual shares. A KQED page posted it, too, with a ton of engagement in the comments. (BTW, last year this month, local election coverage was through the roof; we were expecting a YoY dip.)
Aha! Now we’ve identified why we had a bunch of pageviews, what kind of pageviews they were, what story drove the traffic, where the readers came from, and — this is important — how much better than normal this was. And we’ve identified a key partner who helped amplify our story. Context! OK, so let’s make this a-c-t-i-o-n-a-b-l-e with some takeaways for the team.
Takeaways:
— Contact the KQED page admins to say thanks, get an email address to send them relevant stories.
— Our reporter should reply to some comments and questions in the thread on the KQED post.
— Follow ups on Steph?
Hard metrics are really important to track in aggregate over time so you can report back to management, the board, your funders, donors, whatever, about how you’ve moved the proverbial needle over time, but they’re not terribly useful on a daily, weekly, or monthly basis.
Unless you unpack them, figure out where your traffic came from (or where it didn’t come from!), find out what worked and what didn’t, and identify follow-up tasks and takeaways to be assigned to team members.
It’s list time.
Three things you can do to make metrics (more) useful inside your organization:
- Include context with quantitative data.
Why did the numbers go up or down, where did the readers come from, what did they read, how is this month/week/day/quarter/hour different from another? Is this unusual? What was different? - Takeaways, takeaways, takeaways
How can we learn from this and what can we do today to take the next steps to either try to make the magic happen again, or to figure out what went wrong and try something different tomorrow. - Know your (internal) audience
Deliver takeaways to the people who can take action today! Find the right cadence for communicating takeaways. Monthly or weekly may not be often enough, depending on your organization and publishing frequency.
There are two more examples of quantitative data / actionable insights comparisons in my slides, if you’re into this sort of exercise.
A live look in at the analyst the night before the metrics report is due.
Report smarter, not harder
We’ve all been there. It’s time to build the weekly/monthly/quarterly report, and screenshots from Google Analytics aren’t going to cut it. (Sidenote: Maybe that would work fine, though?) So it’s copy/paste time. Surely you won’t make any small errors that will undermine the entire premise of your report… Surely…
And beyond the added risk of introducing errors, building reports this way (I call it copypastism) is a waste of your valuable time. Let’s automate the quantitative stuff and save your wildly flailing frog limbs for the aforementioned actionable insights.
Three principles for smarter metrics reporting
- Ban copypastism (copy/paste, endlessly)
I’m tempted to make you all take a pledge to end copypastism in your news organizations within 30 days. - Automate what you can; make time for developing actionable insights.
So we come to automation. Automated, regularly scheduled metrics reports (of the metrics that matter to the internal audiences you’re serving) require a significant amount of up-front effort. Reader, they are worth it. Whether you’re populating a shared spreadsheet, firing off a scheduled email of a PDF report, or routing key metrics into Slack channels, getting the quantitative data in front of people with as little repeated effort as possible will make your life easier. - The right people get the right data at the right time.
Does your revenue team need to know about your top 10 stories of the month? Does your editorial team need to know which email newsletter signup form is working best today? Does your executive team need to know what color subscribe button is winning the A/B test you’re running? (The answer might be “no” to most or all of these questions.) Yes, be transparent enough with metrics so that everyone in the organization can find them if they need them, but don’t present a wall of data to everyone who comes asking. Be selective, and give the right people the right data, as often as it’s — useful!
Ask yourself, is this report actionable, for this audience, at this frequency.
Automate it, for the people.
Here are three tools you can start using tomorrow. All of these are Google Analytics-related, but all have analogues in the world of Adobe Analytics (neé Omniture Site Catalyst).
1. The Google Analytics Spreadsheet Add-on will allow you to schedule clean quantitative reports in Google Sheets, keeping executives and senior editors out of Google Analytics.
We could have a long discussion about whether you should train everyone how to do everything in GA, or whether you should have them bring you their more complex questions, or whether you should find the right hybrid treatment that works for you — every organization is different, and frankly, every journalist is different. But I’ve seen this work for eliminating a lot of copy/paste duties for things like “what were our top 10 stories last week” and much, much more.
2. Google Analytics Custom Reports let you ask more complicated questions and route the answers on a regular basis to stakeholder inboxes.
A real world example: The homepage editor wants to know which stories perform best from the homepage itself. What are people clicking on when they get to the homepage? Create a custom report using page paths and schedule it to be emailed to her on a daily, weekly, or monthly basis.
3. Google Analytics Dashboards can be customized as much as you like and shared with the rest of your organization. Build one for executives, one for editors, and one for the product team.
More automation tools
All of these tools are intended to get you, the analyst, out of copypasteland and into a routine of developing the actionable insights your organization needs to grow and succeed. Along with the above, you may already be using some combination of web, social, and other metrics tools such as Chartbeat, Parsely, Sprout, Hootsuite, Buffer, UTM codes for links, IFTTT or Zapier to connect the dots, or even Slack integrations to deliver data to the right team.
Is all of this elementary to you and you’re looking to take things To The Next Level™? Then it’s time to get serious about some sort of full-on business intelligence tool. Looker and Periscope are two of the leaders in that field right now, but we’re talking about a much different cost and level of effort.
In all cases, putting in the hours to set automated systems up to deliver actionable insights to your team is Worth It.
In conclusion
It’s a good joke.
But it’s also your goal: serious journalists publishing stories that make an impact, paying attention to what the metrics can tell them about how their story landed, who is reading it, and learning how they can improve the next story.
Ryan Sholin, most recently Director of Product and Growth at Chalkbeat, is a product, audience, and analytics professional with more than a dozen years of experience in online news. You can follow him @ryansholin on Twitter.
