ScraperWiki Lets You Make Magic Out of Web Data

    by Nicola Hughes
    January 25, 2012

    There’s a wonderful magic wand that every member of a digital newsroom wants to get their hands on. Take control and you can work wonders, untangle the world wide web of information, and even decrease your workload to fit in that extra cup of coffee. “What is this wand?” you ask, and “How can I get my hands on it?”

    It’s the wondrous API (application programming interface). At ScraperWiki, we provide the tools to custom fit your wand to your magical purpose. Learn a couple of incantations in either Ruby, Python or PHP and you can concoct an API of web data — only the relevant data, in the way you want to use it.

    The real power in this way of using data is the ability to fuse APIs from other services and build a whole new tool. For example, this was done by a new startup called Tropo, and here you can see a screencast on how to use Tropo and ScraperWiki to quickly and easily build an airport information system for the Philadelphia International Airport.


    A lot of journalists are using our site to tap into the Twitter API. We think that many more might take up a bit of programming wizardry if they could cast their spells under cloak and dagger. So we have created vaults for making your scrapers invisible to everyone else on the web. Here’s what one looks like. It’s not self-service at the moment, but if you’re interested in gathering and managing data for embargoed stories then get in touch with us here.

    i-974b3f6ba15aaea60fa03ee464fada4d-Screen shot 2012-01-20 at 14.19.51.png


    Also, if you want to work some data magic with developers, journalists, social scientists and the data-curious, keep an eye on our events. The first Journalism Data Camp kicks off at Columbia School of Journalism on February 3.

    Tagged: api developer digital journalists newsrooms scraper scraperwiki web data

    Comments are closed.

  • Who We Are

    MediaShift is the premier destination for insight and analysis at the intersection of media and technology. The MediaShift network includes MediaShift, EducationShift, MetricShift and Idea Lab, as well as workshops and weekend hackathons, email newsletters, a weekly podcast and a series of DigitalEd online trainings.

    About MediaShift »
    Contact us »
    Sponsor MediaShift »
    MediaShift Newsletters »

    Follow us on Social Media