Continuing in their mission to make spending data more accessible and comprehensible, the Spending Stories team and data.gov.uk have just released a reporting tool that will help journalists and analysts pick the freshest and best departmental spending data to work with when exploring the U.K. central government expenditure.
Why spending data gets neglected
Spending data is juicy for journalists, but it gets neglected for many reasons. One key reason is that the shelf-life of a spending dataset is pretty short from journalists’ point of view — if they have to wait six months or even a year for spending data they need for a story, then chances are, the story they were wanting to write will probably have gone stale.
Journalists, campaigners and activists need access to well-structured, machine-readable, and timely data from national as well as sub-national administrations. At OpenSpending, we’re often contacted by journalists with story ideas, or they approach us with a lead. The stumbling stone for them is either lack of information, or worse, data they can’t use because they’re not sure of its completeness. The problem is thus the one of trees falling in a wood: If a transaction is missing from a list, does it mean there was no transaction for that amount on that date, or does it mean the transaction simply wasn’t reported?
These distinctions are important for anyone trying to understand the data — and up to now, they have been pretty tricky to answer. As an attempt to make this a little easier, we’re announcing an automatic reporting tool for spending data (available both on data.gov.uk and on OpenSpending), the result of a collaboration between data.gov.uk and us in order to increase the visibility of the spend data and the ease of browsing the substantial volume of datasets that make up the reporting of government expenditure in data.gov.uk.
The tool lists departments registered as data publishers on data.gov.uk and details how precisely they have followed the HM Treasury reporting guidelines. It will also make the whole of the reported data available for search and analysis both on data.gov.uk and on the OpenSpending site.
The tool is useful both to those using the data and those within government in ensuring departments are reporting on time. It helps check:
- Quality of the data (i.e., adherence to HMT reporting guidelines, well-structured data)
- Status of reporting (i.e., how complete the reports are or if there is a reporting period missing)
Why was this possible?
Having all of these datasets organized under a single catalog at data.gov.uk in simple spreadsheet format, combined with the data.gov.uk team’s work in making the necessary metadata available, enabled the OpenSpending team to create an extraction system to be set up to clean the data on a regular basis. The team then cleaned more than 6,000 column names to add compliance with HMT guidance.
How does it work?
The report generator then highlights in red departments that are registered as a publisher on data.gov.uk but have failed to publish any information on their spending, in yellow those who have published data which cannot be interpreted as spending data (e.g., PDF format or not complying with the template provided by HMT), and green for those departments whose records have been updated as regularly as demanded as per the publication requirements. (Latest data must have been published as recently as a month ago.)
The first stage of this release deals with central departments, which are obliged to report all spending over 25,000 GBP ($40,310). Subsequent stages to follow soon after will monitor local councils and other governmental bodies, which have different reporting requirements. The interface will be useful both inside and out of government, to ensure transparency regulations are met and to better understand where gaps in data may alter the completeness of the picture offered by government data.
- Reporting tool on data.gov.uk
- More extensive list of entities at OpenSpending
- Our briefing with all the details and link to code.
Interested in more regular updates from the Spending Stories team? Join the discussion via the OpenSpending mailing list.
Lucy Chambers is a community coordinator at the Open Knowledge Foundation. She works on the OKF’s OpenSpending project and coordinates the data-driven-journalism activities of the foundation, including running training sessions and helping to streamline the production of a collaboratively written handbook for data journalists.