Assessing The Impact Of Explanatory Journalism

This article was originally published by the Media Impact Project.

Journalists don’t as a rule have a specific impact in mind when we begin our journalism. We list project goals, but other than bringing awareness to an issue or event we do not identify what we’d like to see happen next.

This is a story about the one time we did.

Rockland County, just northwest of Manhattan, is one of New York’s fastest growing counties. Neighborhoods that used to feel suburban and bucolic now choke with high-density sprawl, multi-family homes rise in backyards next door, traffic is a problem and religious schools pop up on residential streets. This is the lede of our story for the Journal News and lohud.com:

“A generation ago, there were few problems between Ramapo’s small ultra-religious Jewish communities and the gentiles and other Jews who made up the bulk of the town’s population. Things have changed. As the ultra-religious community has grown, Ramapo has become a flashpoint in a continuing conflict over what it means to live in the suburbs. … The conditions fueling that conflict are now threatening to spread beyond Ramapo’s borders. Surrounding communities have taken notice, and they are adopting measures aimed at heading off the strife that has become the norm for their municipal neighbor. No place, it seems, wants to become ‘the next Ramapo.’”

People in Rockland County, one of two counties we cover at The Journal News and lohud.com, argue over this. Things get ugly. Reader comments on the stories we write turn hostile and border on anti-Semitism. No matter what the topic, the conversation inevitably devolves into “us” vs. “them.”

But here’s the real problem, and it’s not religion: Ramapo’s loose zoning, lax enforcement of fire and building codes and largely unchecked development puts everyone at risk.

We wanted to know: Do people understand what’s really happening in Ramapo? If we were to explain the issue — lay it out in long-form story, year by year and decision by decision — could we all (or most of us) finally agree? Could we change minds? If we could achieve consensus, would we stop arguing religion and start attacking the real problems?

So, after years of having the same tired, angry, circular discussion, we set out — in one project — to change the conversation.

We researched the story over several months. We talked to longtime residents, members of the Jewish community, government officials and the county’s Fire and Emergency Services coordinator. We wrote and rewrote, shot video and mapped the region.

“Ramapo nears breaking point” is a long-form piece of explanatory journalism that laid out all of the problems that began decades ago.

How we measured the story’s impact

At the end of our story, we included a short survey to measure the impact of our explanatory journalism — to find out if we successfully proved that the issues in Ramapo are a lack of zoning and safety code enforcement, not religion. (We considered asking for opinion at the beginning of the story to gauge change at the end, but felt that would inhibit readership and we could accomplish what we needed with one survey at the end.)

The survey was four questions, beginning with two easy Likert Scale queries:

1. How strongly do you agree that the issues in Ramapo are a lack of zoning and safety code enforcement, not religion?
2. How strongly do you agree that after having read this story, your opinion of the issues in Ramapo has changed?

Then we expected a steep drop-off in answers to the final two questions because the first takes time and the second raises privacy concerns:

1. How have Ramapo’s issues impacted you?
2. What is your email address?

We built the survey using PollDaddy because it prevents repeat responses and for the data it provides on the back end, including geolocation, a time/date stamp and referrer, and it assigns each respondent a unique PollDaddy ID. This helped us connect their answers from question 1 to question 2, and so on, and get a more complete picture of their response.

We embedded the survey at the bottom of the long-form story. For two days, that was the only place you could find the survey; we didn’t embed it elsewhere or share it on social media. After two days we added it to our editorial, “Ramapo’s shoddy governance is by design.”

Survey results

Surprisingly, nearly 900 people responded. Here’s what they said:

Do people understand what’s really happening in Ramapo? Yes. Most (67 percent) either agreed or strongly agreed with our explanation of the issues, with “strongly agree” ranking the highest (54 percent)

Could we change minds? Yes. Of those 67 percent, 14 percent said our reporting convinced them, while 35 percent said they agreed beforehand.

Finally, we believed we and the community could stop having the same conversation and instead move forward toward solutions.

Emails. Almost a third — 29 percent — gave us their email address for follow-up.

Sources and commentary. Even more surprising than the total number of respondents was the percentage of them — 38 percent — who also took the time to write about how Ramapo’s issues have impacted them. Hearing from homeowners, those who’ve moved away, educators, fire safety officials, people who have tried to move in but claim to have been denied housing gave us a list of future story ideas and sources.

A few wrote to thank us for our reporting, for exposing the issues and doing so in such an unbiased way. A couple said we should have written this years ago. One called it “conversation-framing analysis” and others called for more discussion on the topic. One response shows how we raised awareness and encouraged civic involvement:

“I live in a neighboring town and am concerned about Ramapo’s issues becoming issues here, too. So I will be paying more attention and doing what I can to ensure that we do not fall victim to the same lack of zoning and code enforcement that has plagued Ramapo.”

What we learned

• People will answer a survey, even at the bottom of a long-form story, if they’re invested enough in the issue.
• Think about who you want to answer the survey and who you don’t. Burying our survey at the bottom of the long-form story and not promoting it weeded out casual readers that we didn’t want filling it out anyway.
• Close the feedback loop. Thank your readers and share the results of the survey. We didn’t do this at first, and our audience let us know that wasn’t OK.
• Be upfront about how you’ll use their specific information. Unless you’re clear you will use their comments in a follow-up story, contact people to ask permission before publishing their words. This demonstrates respect and encourages further engagement.
• Use all of the information you gather to learn more about the audience. We cross-checked our survey email list with our subscriber lists and the list of those who attended our project discussion in person to see who we’d reached. We learned that we have room to grow our subscriber base in Rockland when we otherwise would have thought we were talking to existing subscribers.

Anjanette Delgado is the digital director and head of audience for lohud.com and poughkeepsiejournal.com, part of the USA Today Network. Email: adelgado@gannett.com, Twitter: @anjdelgado. Special thanks to Lindsay Green-Barber of the Impact Architects, who helped with survey methodology.