Here’s my quandary: If I deliver, as promised, a warts-and-all account of how Agence France Presse’s Twitter-based application the e-diplomacy hub came to be, I risk irritating our developer, getting rapped on the knuckles by my CEO, and provoking the good folks at Twitter who, make no mistake, can pull the plug on our ambitious little project, and others like it, with a digital flick of the wrist.
But the air-brushed version of this story misses all the drama and lessons learned, so let us (gently) throw caution to the wind.
Building the hub
To boost brand and visibility, AFP — a global B-to-B news agency with some 1,600 journalists spread over 190 bureaus — set up a suite of multi-author blogs earlier this year, including one on geopolitics. As the blog’s editor, I saw an opportunity for the news agency to delve into the Brave New World of data journalism. So I squeezed my boss for a small budget and assembled a Paris-based team of freelancers: an investigative journalist specializing in techno-politics just back from a stint as a visiting scholar at UC Berkeley’s J-School; a hacker-cum-developer tapped in 2010 by Julian Assange to design a platform for the WikiLeaks trove of diplomatic cables; and a crack motion designer bored with her day job plugging video games. I had just finished a five-year stint as a science and environment reporter.
My newfound colleagues were so shamelessly young — well under half my age — it made my arthritic knees ache.
With the Arab Spring still unfolding, our mission was to open a real-time window onto the realm of digital diplomacy, showing dialogues between key actors and ranking their influence. Facebook arguably played a greater role in driving the changes in Tunisia and Egypt, but Twitter was the vector through which news flowed to the rest of the world (and back again), and we wanted to harness that.
The application, built with HTML5 rather than the increasingly shunned Flash, seeks to solve Twitter’s needle-in-a-haystack problem: how — in relation to a given topic — to find the relatively few quality drops in the Big Data fire hose. It consists of a series of user interfaces connected via algorithms to a rather large, hand-picked database.
Indeed, stocking the database was something of a moon shot: Our goal was to identify and validate, one-by-one, the 5,000 most influential Twitter accounts in international relations, from presidents and diplomats, to experts and NGOs, to hactivists and even terrorist groups — more than 130 countries and dozens of languages. Crowdsourcing AFP’s global network gave us a good start, but there were a lot of (big) holes to be filled. It took an embarrassing number of person hours.
That Good Ole’ Fashioned Digital Alchemy
With the database in place, Pierre Romera’s algorithmic alchemy came into play. Joan Tilouine, the co-creator of the “Hub,” and I spent caffeine-fueled days imagining new ways to detect patterns and display data, and then Romera and the graphic designer, Elsa Secco, would reality check our ideas and pitch their own. This is not the first time I have led a foray into displaying real-time data, and I’m constantly amazed at how rarely I hear the words “pas possible” in such a setting.
What I do hear a lot is this: “on peut faire mieux” — “we can do better.” And herein lies an obvious but oft-ignored axiom about the development process, especially in data journalism: Get everyone in the same room. An artist, an engineer and a journalist don’t see the world the same way, and even the most sophisticated workflow management software can’t match the yield of face-to-face synergy — better ideas, more efficient.
But not, alas, fool-proof.
Our little team worked day and night ahead of our launch date of June 21, when we were to unveil our creation to a roomful of French journalists evenly split between old-school foreign policy types and new-media mavens — not an easy crowd.
As a warm-up to the main event, I demoed the application to the digital strategy folks at France’s foreign ministry, and the next day to a major media client in France. In the preceding weeks the Hub had been fast and crisp, eliciting “oohs” and “aahs” from everyone who saw it. But now it was sluggish, and bugs began to appear. I managed to tiptoe through these presentations, but knew something was awry.
And then, the night before the press conference, things got worse. Much worse: Ugly blocks of code were popping up in all the wrong places.
Intelligent Toy or Work Tool?
I frantically emailed Romera who — assuming his job was done — was out of town. At first he balked, and rightly so: For weeks, long after the budgetary stream had run dry, I hassled and harassed, coaxed and cajoled him into tweaking and adding features. Axiom No. 2: make sure you’ve got the resources to finish the job.
He finally reported back. “Remember six weeks ago, when you asked me to store all the hashtags running through the app?” he asked by phone. It turned out that the application, like a cargo ship taking on water from an invisible gash below the load line, was sinking under the weight of all this unseen data. Axiom No. 3: be careful what you ask for.
We threw the deadweight overboard and the ship righted itself, just in time. It has been pretty smooth sailing since.
The e-diplomacy hub has been well-received — especially by a cohort that was not its intended audience. We designed the application for what we assumed would be a relatively thin slice of the general public interested in tracking digital diplomacy in action. What we did not expect is that diplomats themselves would perceive it as a working tool. But they do, and now I see why.
Some 500 million tweets run through Twitter every day, double the flow of less than a year ago. Diplomats, along with professionals in all sectors, have come to realize that critical information is buried within this perpetual tsunami of data, but they don’t quite know how to get at it. TweetDeck does not adequately filter out the “noise” that obscures a relative handful of pertinent tweets. Suddenly, for diplomats, here was a platform that did just that, drawing from a Who’s Who database of their peers.
Phase Two
Which brings us to Phase Two of our excellent adventure in data journalism, currently underway: a new round of development, mindful of our dual audience, to shore up the hub and add new features, including analytics on how states and individuals leverage Twitter in international relations. We built the tool, and now we want to use it.
All of this, of course, depends on the good graces of Twitter, a company that has made a lot of developers and startups nervous in recent months by tweaking and tightening the already restrictive rules governing access to its data. This is also one of the reasons I am now as likely to find myself in a meeting with five in-house lawyers (I didn’t know we had that many!) as in the corner café where our project was born. But our evolving relationship with the mothership Twitter is another story, and the plot line — much less the ending — on that one has yet to be written.
Marlowe Hood is the editor of AFP’s thematic blogs, including one on geopolitics, and co-developer of the data-journalism app the e-diplomacy hub. Armed with a graduate degree in Chinese studies, Marlowe began his journalism career in the mid-1980s in China where he reported for the Wall Street Journal and US News and World Report before setting up the South China Morning Post’s first Beijing Bureau. He left China a few months after the June 4, 1989 crackdown and settled into a five-year stint in New York as a freelancer specializing in Asia. He joined AFP in Paris in 1995, working in graphics, web development and, most recently, as a science/environment correspondent covering climate change.