Sunlight Labs

While this is a little late-- late's better than never for giving thanks. And this year, we've got a lot to be thankful for. Open Data in Open Government is making leaps and strides. The Vice President is talking data quality in government on the Daily Show. ABC News along with Recovery.gov's controversy have brought government data into prime time. It's been a long time since transparency like this has seen this kind of attention.

At this time of Thanksgiving here in the United States I wanted to give thanks for the new and changing government datasets that we have now. Some are truly amazing.

The new Recovery.gov-- which we've written about and even nearly bid on-- has certainly taken the government huge steps forward in terms of disclosing information, but it is not without controversy. The press is questioning the program, pointing to wasteful spending or bad data. The White House fired back with a "reality check"(their words) saying that few of the reports have gone through the "extensive three-week review" and that the data might be particularly misleading at this point.

Earlier this year we started on the Data Commons, a project to merge open government data sets to make them more searchable and usable. Our goal for the initial release is to load state and federal campaign contribution data from The Center for Responsive Politics and The National Institute for Money in State Politics. Along with the raw transactional records, we will be taking the additional step of matching the entities (people, organizations, corporations, etc.) across the data sets. We'll have more posts soon with details about the Data Commons.

To assist us in this effort, we are developing Matchbox, a toolkit for the merging and matching of entities. We have big plans for Matchbox, but want to get feedback from the community as we improve it over the next few months.

Here at the Sunlight Labs, we've focused a lot on the recent bid on version 2.0 of Recovery.gov. This morning on the Labs mailing list, Rusty Talbot of Synteractive, one of the winning contractors, asked for input on the best way for Recovery.gov to publish its data.

Rusty wrote:

The Recovery, Accountability, & Transparency Board wishes to have an open discussion with all interested developers about how data should be made available via Recovery.gov.

As you are all aware, a new version of Recovery.gov will be released soon. From a data standpoint, the initial release of the new site will replicate existing functionality. However, the Board aims to set a new standard of transparency with this site and would therefore like to make the data available in the most convenient and straightforward way (or ways) possible so you can use and analyze official, up-to-date Recovery Act data. We need your input to achieve this goal.

Please let us know how the site could best meet your needs in terms of machine-readable data format(s) and standards, APIs, guidance, training, etc.

This is a great opportunity for all of us who work hard to make government data more open and accessible.

Developers are good at getting bits to line up, importing data and getting great conclusions out of it. Designers are great at making things look great and making those conclusions and bits easily digestible. But in all the apps I've seen, they all ultimately suffer from the same fatal flaw: accuracy.

When we launched TransparencyCorps at the end of June, we ran a few small earmark campaigns, to digitize little batches of earmark request letters that legislators had posted on their websites. These campaigns wrapped up very quickly, and at the same time, the House decided to release earmark request letters en masse, and we didn't have to do our campaigns per-legislator anymore.

Given the demonstrated interest in earmarks, we decided to run a much larger campaign, for all the earmarks released by the House Appropriations Committee, starting with those for the Commerce, Justice, and Science Subcommittee. These were released in a single massive PDF, which I split up into individual 1- or 2-page request letters.

This campaign involved 1,183 letters, and we had the campaign run for 5,537 completed tasks. Total volunteer time, as measured on TransparencyCorps: over 472 hours. That's nearly 20 man-days. Here are the results.

Imagine if Costco announced that they were going to take the Costco experience to Manhattan, and open up convenience stores across the the island. Further, imagine shopping at these new CostCo bodegas, all of 500 square feet, with your giant cart, selecting from what the CostCo bodega has to offer in this limited amount of space! At your local CostCo bodega you have to choose from either 400 rolls of toilet paper, 70 lbs of dehydrated mashed potatoes, or a 6 pack of giant boxes of cereal. That's pretty much all they could store in inventory at the CostCo bodega because they wouldn't have room in 500 square feet for anything else. And good luck carrying all that home!

Sounds absurd, doesn't it?

We scraped all the ideas off of the latest National Dialogue on Recovery.gov to see of we could condense what most of the ideas were about. Here's what we got:



Looks like most of the ideas were about data. People sure do want to set that data free, don't they?