Sunlight Labs

Over a year ago we announced our intention to build scrapers that would collect and sanitize legislative information from all fifty states, an initiative that is now known as the Open State Project.

As of today we're proud to announce a new milestone for the project, version 1 of the Open State Project API. You can start using our API today to get access to information on more than 37,000 bills and 1,600 legislators from the most recent sessions of 10 state legislatures.

At Sunlight we spend a lot of time exploring ways to open up data sets and make them more accessible. The idea is that data enables us to act collectively, making better informed decisions and building a more effective public sector. When we talk about transparency the focus is often on the possibilities that data offers. But this discussion sometimes ignores the fact that translating data into action is hard.

There's a reason for this: data alone doesn't provide answers.

Coming up with solutions to real life problems -- like designing an effective and fair tax code or improving health care -- requires an understanding of how real life works. Unfortunately, more often than not real life is messy and complicated. In order to make sense of this complexity we need models -- approximations of the world that define fundamental mechanics of a given process and reduce it to understandable and meaningful terms.

As Joshua Epstein writes in a clever essay on scientific inquiry, every time we use data to draw a conclusion we also use a model. Sometimes explicitly: when a meteorologist makes a prediction about the weather they use a rigorously designed framework for translating observational data into a forecast. Sometimes not: when I look at the sky and make a prediction I'm using an implicit model based on a mix of past experience and a rather poor understanding of atmospheric processes. Both of us are using models to interpret data and both are based on assumptions about how weather works. I'm just not sure I could explain how mine functions, nor do I have any sense of how well it works.

Having access to good observational data is incredibly important to arriving at useful answers. But well designed and transparent models are equally important. In fact, having a good model is often a prerequisite to determining what to observe and how. If I want to predict the weather should I measure the temperature? Pressure? Wind direction? Where and how frequently? Without a solid theoretical framework it's often impossible to know where to begin and it's even harder to know when I've made a wrong turn.

When we use a model we embed its assumptions into the results. If key assumptions are incorrect, good data turns into supporting evidence for a potentially misguided answer. Or a bad model might drive the collection of useless data.

This post was contributed by one of Sunlight Labs' Google Summer of Code Students, Gabriel Joel Pérez. Gabriel's work is currently being integrated into the core project and the states he has been working on should be available via the Open State Project API later this year. His code is available on github as we work on integration.

Hello! I’m Gabriel, I’m a 4th year student of Computer Engineering from the University of Puerto Rico in Mayagüez. This summer I worked as a GSoC student on developing new scapers for the Open State Project. The states I worked on were Colorado, Hawaii, Washington, Oregon and the territory of Puerto Rico. I really enjoyed the whole experience. The work is very fulfilling as coding in Python is always delightful and fun.

I should say up front that Google's been a great friend to Sunlight: they've helped support our contests, they've sent us phones and Summer of Code students to help our Android development efforts, and when I visited their DC offices a couple of weeks ago they let me eat as much candy as I wanted.

Still, I'd be lying if I said the incredible scope of their success didn't make me a little uneasy. We use Google Apps for our work email, for instance, and YouTube is essential to our video production efforts. We're as dependent as anyone else on Google for search, both as a tool and a source of traffic. I know we're not the only ones to be a bit unnerved at being so reliant on the goodwill of a private enterprise -- and of course over the past few weeks, other voices expressing those concerns have become significantly louder.

So, while we're looking forward to continuing to work with Google, it would be irresponsible for us not to prepare for the unthinkable. I'm happy to say that we've taken the necessary precautions, and today the future seems a bit less uncertain:

Of course, what happens after we run through our 1000 free hours is anyone's guess.

(Many thanks to Pierre Huggins of Rox Chox & Blox Woodworking for lending his awesome fabrication capabilities to this ridiculous project (and to our own sysadmin extraordinaire, Tim, for finding Pierre via HacDC)

Over the past few months we've had the pleasure of working with several developers through Google's Summer of Code program. One of them is Evelina Vrabie, who has contributed her talents to our Android app (and has done so from across an ocean -- Evelina's based in Romania). She was nice enough to write about the experience, and to tease a few of the features she's been working on for the app.

My name is Evelina Vrabie, and for the last four months I've had a great experience collaborating with Eric Mill on the Congress project for Android, as part of the Google Summer of Code 2010 program. Working for the Sunlight Foundation has been an excellent opportunity for me to learn and grow as a capable Android developer...