What It Is
ChangeTracker is a project at ProPublica that watches three government websites — Whitehouse.gov, Recovery.gov and Financialstability.gov — for edits, deletions or changes to existing content. Through an RSS feed, Twitter account or daily email digest, ChangeTracker will inform you when a page changes on these sites, and show you what’s been added or removed.
Why It’s Innovative
The Obama administration has already made strides toward greater transparency and better use of technology in government, but has promised even more. It’s important to make sure that President Obama and his people act on those promises. One way to do that, given the increased use of web technology for blogging, press conferences, and sharing information about bills and budgets, is to watch the government’s footsteps online. ChangeTracker makes this possible, and not just for these three government sites — it can be used to track changes on any website.
ChangeTracker is not a piece of software. It’s a combination of cheap and powerful web services. And ProPublica even provides detailed instructions on how you can replicate it to track changes on any website.
Who’s Behind It
Scott Klein is the director of online development at ProPublica and Brian Boyer recently started as an intern.
Boyer actually started work on ChangeTracker a few weeks before his internship began. After attending the grand opening of the RJI Collaboratory, Boyer began building ChangeTracker during a flight layover — and almost missed the plane.
“I’d been thinking a lot about transparent government, citizen journalism, crowdsourcing and Twitter, among other things, and it just occurred to me that this was something that would be neat to make,” said Boyer.
When Klein saw what Boyer had been working on, he saw a solution to a need at ProPublica. Klein explained:
The Obama Administration has set a very high bar both in terms of their own public comments about transparency as well as of their sophisticated use of the web. ProPublica’s mission is to hold the powerful accountable, so using ChangeTracker to keep the administration to those commitments is a pretty obvious step for us to take.
We’ve been wanting a software tool that keeps an eye on website changes since our story by Jennifer LaFleur on the Obama transition team quietly making some significant changes to their website, change.gov, but quite frankly I didn’t think it could be done without a major development effort. When Brian showed us what he’d been working on, we jumped at the chance.
Boyer started his internship at ProPublica in February, and ChangeTracker is his first project. He studied computer science at University of Illinois at Urbana-Champaign, and also worked at various software and consulting firms in Chicago as a developer, software architect, manager, and entrepreneur.
He found out about Northwestern University’s journalism scholarship for programmers through Boing Boing.
“A career change was on my mind, not because I dislike programming — I love to code, but because writing software for hedge funds, litigators, banks and marketing teams is soul-crushing,” he said. “I wanted to do work for the public good. Law and public policy were on my mind, but I figured this was worth a look, so I Googled ‘journalism.’ Lawmakers solve problems from the top-down, journalism feeds democracy from the bottom-up. I was convinced.”
Boyer was one of the first two “programmer-journalist” scholarship winners and finished the program in December. ProPublica is his first job in journalism.
About the Process
Versionista is a web application that scans websites for changes, and generates an easy way to see the differences between versions. Boyer first found Versionista during the presidential campaign when the Wired Threat Level blog posted that McCain’s camp had posted about a change in Obama’s Iraq agenda.
A Yahoo! Pipe scans the list of changes provided by Versionista. It then performs a series of look-ups and transformations on the extracted information and munges it all into an RSS feed. The feed is consumed by FeedBurner, which provides email digests, and gives a little more control over the output. The burned feed is presented to the users, and TwitterFeed is used to syndicate the feed to Twitter.
Challenges
During development, Klein and Boyer wanted to make sure ChangeTracker was useful to anyone, coder or novice. In order to make ChangeTracker replicable, they use very little code. Instead, ChangeTracker relies on the basic concept of the mashup — stringing together existing web services.
Whitehouse.gov is a large and constantly growing website, so changes to the original content can sometimes get lost amongst the addition of new content and changes in navigation and formatting. Boyer spends a lot of time filtering these trivial changes out of ChangeTracker.
3 Questions
Q: What has the response been like?
Klein: So far we’re thrilled. Users are finding it useful and we’ve had healthy growth of Twitter followers, RSS readers, and email subscribers. What we really want to see happen next is for people to copy our work and track other sites. We’ve got our own ideas about other websites that could use this kind of scrutiny. For example, I get a thrill when local news organizations follow-up on ProPublica’s work and shine a light on their own local governments, but there are lots of other sites that need this kind of scrutiny in and out of government. We can’t wait to see what stuff people come up with that we haven’t even thought of.
Q: I’m really interested in ChangeTracker as a tool created by a news organization that can be used in a lot of different ways: internally, by news addicts, and as code that can be taken and adapted for other sites/uses. Did you intend for ChangeTracker to be so multi-faceted?
Klein: Yes and no. It was really pretty simple: We wanted a tool that we could use to track changes to some government websites, and along the way we made sure we weren’t ‘hard coding’ our application into doing only that.
Boyer: Instead of building a big, complicated application, I built a little tool that plays nice with other little tools. In my experience, that approach leads to broader appeal and unexpected uses.
Q: There have been a number of questions about why ChangeTracker wasn’t hand coded and how reliable the services it relies upon are. How would you respond to these concerns?
Klein:ChangeTracker can be replicated, even by novice users, using a straightforward step-by-step recipe. This may be hard to keep going as we introduce more sophisticated features, but we’ll think hard before requiring real programming skills in order to use ChangeTracker.
Versionista was created by Peter Bray. He’s a nerd journalist like us. Although Versionista is still, as far as I know, a one-person startup, it has been solid so far — and it has a profit model. We’re very confident we made the right choice.
Boyer: I think it’s brilliant that such a valuable service as Versionista was created by some dude and costs $9.95/month. Curmudgeons should get over it.
Specs
Changetracker is a mashup using web services Versionista, Yahoo! Pipes, RSS, FeedBurner, email, TwitterFeed, and Twitter.