A version of this piece ran at the Duke Reporters’ Lab.
Except for the moment when we almost published an article about comedian Kevin Hart’s plans for his wedding anniversary, the first test of FactStream, our live fact-checking app, went remarkably smoothly.
FactStream is the first in a series of apps we at the Duke Reporters’ Lab are building as part of our Tech & Check Cooperative. We conducted a beta test during Tuesday’s State of the Union address that provided instant analysis from FactCheck.org, PolitiFact and Glenn Kessler, the Washington Post Fact Checker. Overall, the app functioned quite well, with only 2 glitches and mainly positive feedback.
Our users got 32 fact-checks during the speech and the Democratic response. Some were links to previously published checks while others were “quick takes” that briefly explained the relative accuracy of Trump’s claim.
When President Trump said “we enacted the biggest tax cuts and reforms in American history,” users got nearly instant assessments from FactCheck and PolitiFact.
“It is not the biggest tax cut,” said the quick take from FactCheck.org. “It is the 8th largest cut since 1918 as a percentage of gross domestic product and the 4th largest in inflation-adjusted dollars.”
PolitiFact’s post showed a “False” Truth-O-Meter and linked to an October fact-check of a nearly identical claim by Trump. Users of the app could click through to read the October check.
Many of the checks appeared on FactStream seconds after Trump made a statement. That was possible because fact-checkers had an advance copy of the speech and could compose their checks ahead of time.
Working Out the Kinks
We had two technical glitches – and unfortunately both affected Glenn. One was a mismatch of the URLs for published Washington Post fact-checks that were in our database, which made it difficult for him to post links to his previous work. We understand the problem and will fix it.
The other glitch was bizarre. Last year we had a hiccup in our Share the Facts database that affected only a handful of our fact-checks. But during Tuesday’s speech we happened to hit one when Glenn got an inadvertent match with an article from the Hollywood rumor site Gossip Cop, another Share the Facts partner. So when he entered the correct URL for his own article about Trump’s tax cut, a fact-check showed up on his screen that said “Kevin Hart and Eniko Parrish’s anniversary plans were made up to exploit the rumors he cheated.”
Oops!
Fortunately Glenn noticed the problem and didn’t publish. (Needless to say, we’re fixing that bug, too.)
What’s Ahead For FactStream
This version of FactStream is the first of several we’ll be building for mobile devices and televisions. This one relies on the fact-checkers to listen for claims and then write short updates or post links to previous work. We plan to develop future versions that will be automated with voice detection and high-speed matching to previous checks.
We had about 3,100 people open FactStream over the course of the evening. At the high point we had 1,035 concurrently connected users.
Our team had finished our bug testing and submitted a final version to Apple less than 48 hours before the speech, so we were nervous about the possibility of big crashes. But we watched our dashboard, which monitored the app like a patient in the ICU, and saw that it performed well.
Feedback
Our goal for our State of the Union test was simple. We wanted to let fact-checkers compose their own checks and see how users liked the app. We invited users to fill out a short form or email us with their feedback.
The response was quite positive. “I loved it — it was timely in getting ‘facts’ out, easy to use, and informative!” Also: “I loved FactStream! I was impressed by how many fact-checks appeared and that all of them were relevant.”
We also got some helpful complaints and suggestions:
Was the app powered by people or an algorithm? We didn’t tell our users who was choosing the claims and writing the “quick takes,” so some people mistakenly thought it was fully automated. We’ll probably add an “About” page in the next version.
More detail for Quick Takes. Users liked when fact-checkers displayed a rating or conclusion on our main “stream” page, which happened when they had a link to a previous article. But when the fact-checkers chose instead to write a quick take, we showed nothing on the stream page except the quote being checked. Several people said they’d like some indication about whether the statement was true, false or somewhere in between. So we’ll explore putting a short headline or some other signal about what the quick take says.
Better notifications. Several users said they would like the option of getting notifications of new fact-checks when they weren’t using the app or had navigated to a different app or website. We’re going to explore how we might do that, recognizing that some people may not want 32 notifications for a single speech.
An indication the app is still live. There were lulls in the speech when there were no factual claims, so the fact-checkers didn’t have anything new to put on the app. But that left some users wondering if the app was still working. We’ll explore ways we can indicate that the app is functioning properly.
Bill Adair is the Knight Professor for the Practice of Journalism and Public Policy at Duke University and the director of the Duke Reporters’ Lab. He is the creator of Pulitzer Prize-winning website PolitiFact and worked for 24 years as a reporter and editor at the Tampa Bay Times. He is the author of The Mystery of Flight 427: Inside a Crash Investigation.
