The Apps for America Judging Process
- Written by
- Clay Johnson
- Date
- 04/21/2009 12:42 p.m.
Since announcing the winners yesterday, a few people have asked for notes about how the Sunlight Foundation selected the winners. The answer is: we didn't. The Apps for America judging process went like this: we got 5 judges to agree to judge the contest-- Adrian Holovaty, Peter Corbett, Xeni Jardin, Aaron Swartz and myself. We built a very lightweight judging app (screenshot) and invited every judge to log in and rate each app according to the attributes we specified on in the contest rules.
4 of those judges showed up to vote, and for the most part every application was voted up by these four judges. The fifth judge, while they agreed to judge this contest, didn't reply to emails or respond to our various inquiries to judge. So we continued the process anyway rather than replacing the missing judge at the last minute.
Each category was rated on a scale of 0-5 with 1 being the lowest score and 5 being the highest score. The categories, as a refresher from the rules were:
- Usefulness to constituents for watching over and communicating with their members of Congress
- Potential impact of ethical standards on Congress
- Originality of the application
- Potential usability of the application
- Code quality of application
Code quality of the application was perhaps the most difficult for our judges to judge. One judge opted to not judge code quality and abstained from judging, which left three judges on that category.
After the judging process was complete, we averaged all the scores, listed them in descending order, and that's the order they appear in on the winner's list.
A few folks have mentioned things along the lines of these apps are "Sunlight Foundation" selected. And while that's partially (25%) the case, the majority of votes cast were from non Sunlight Foundation employees. If there was any bias in this group, I would suspect that it would be a "python friendly" slant between Aaron, Adrian and myself, though I'll point out that our first place winner is a Ruby on Rails application, and our second place one is a CodeIgniter application.
For the next round, I think we could stick with four judges and leave the fifth to the community at large somehow, though we'll have to work that out so that the voting does award merit in the application itself, not just how popular or good of an organizer the entrant is. I think we could also have a more open nominations process for who the judges are.
While no one has complained about the judging process, I think its good to be clear for how applications were judged in this process and to think about how the process can be improved for our next round. We're just learning how to do this, and ideas are welcome!
Discussion
What are Your Thoughts?
Comments have been closed on this post.
First, thanks for your all GREAT efforts in making this happen!
I do have suggestions in the next round to have a mobile-specific category. Our app, Congress Bills is an iPhone app. Since it was still during Apple's review during the judging period, we offered Ad Hoc version of the app providing that judges send their iPhone UUIDs to us. But none of the judges did that. Might be due to the fact that they don't have iPhones. But the truth is that they had to judge the app purely based on the video on vimeo, screenshots, and blog descriptions. Since the mobile device is a very different kind of media than website, there are unique features/advantages exhibited from them. It's sad that the the app did not get a complete look (from our opinion) to be rated.
We are also having an Android version of the app running btw.
I would suggest that in the next contest, state whether you will have sufficient resources in judging mobile applications. Resources such as the availability of the mobile phones, and the models of them. Since the contest rules said any platform, I was actually thinking of building a robot that can etch laser on the wall showing the current activities of the congressmen. Oh well, that went a bit too far :)
Thanks again!
Good for people to know.
Good post.
Some of my friends had built one of the phone-based applications. They had said no one called into their number. They also spoke with another team that did a phone app, and they didn't get any calls either
Based on that and Hong's comment, it sounds like the judges didn't even use the applications they were judging. At least, not all of them.
I participated RailsRumble last year, and it seemed to have similar problem, where judges (the community in that case) were only basing it on cursory glances and clicking. I recall one particular app that was in the top 5 not even functioning aside from the front page.