Sunlight Labs blog

Our next contest: Design for America

Clay Johnson — Wed, 17 Mar 2010 14:39:20 -0400

Our next contest, and first of two contests of 2010 is "Design for America". As we talked about in January, opening government involves more than just developers: we need the art and design community to take data from our government and tell stories about it.

Part contest, part festival, the Design for America contest's intent is to inspire the design community to tell great stories about how our government works, what our government does, and what it could do. It's a contest as much about possibility as transparency, and with categories ranging from infographics to web design, there's plenty for all to compete in. Each category has a $5,000 prize associated with it now, and as we gain sponsors for each category, we'll be increasing the prizes associated as sponsorship allows.

We're really excited to announce some great judges for this contest not only because of their talents but because we're simply big fans. Nicholas Felton famous for his Feltron Annual Reports is judging visualizations around Recovery.gov. Andrew Vande Moere is judging data visualizations from our own community. We're excited to have Nathan Yau from Flowing Data judging USASpending.gov visualizations alongside the New York Times Visual Op Ed columnist Charles Blow. And we'll be announcing a slate of many more judges in the weeks ahead.

On the sponsorship side-- our Platinum Sponsor is Adobe, and it wouldn't be very transparent of me if I didn't mention mention that we've talked about them on our blog once or twice. Adobe and Sunlight share common interest: we want government to be more engaging and we want government data to be available to citizens. We're finding common ground in the design community to do just that and we're appreciate having their support for this contest.

Google is also sponsoring our contest as they've done in the past. We enjoy working with Google and appreciate their support, but more than their support, they make a lot of great tools that make our lives a lot easier-- whether it be Fusion Tables or the public data explorer they're great proponents of opening up government so that the public can see what's going on. Google's sponsoring two of our categories this time around: Best Visualization of How a Bill Becomes a Law, and Best Visualization of Federal Budget/USASpending.gov data.

The contest will end with a showing of the best entries at the Gov2Expo in May, and we'll announce the winners there. We're happy to have the support of O'Reilly, TechWeb and the Gov2Expo team. It will be a great event-- and the showing itself will be free and open to the public (though you'll need to RSVP).

We're really looking forward to this new contest. After Apps for America and Apps for America 2, we're looking forward to seeing how the design community can make a difference with the way people see their government and its data.

Introducing POIA

Clay Johnson — Tue, 16 Mar 2010 13:55:23 -0400

Today Representative Steve Israel introduced the Public Online Information Act calling for government to:

Create an expert committee for all three branches of government to steer government towards making datasets publicly available in meaningful ways.
Direct the executive branch to consider guidelines issued by that committee and for the CIOs of various agencies to do the same thing, and
Place online all publicly available government documents and data held by the executive branch. This includes everything they've got, with a few potential exceptions: classified information, personnel rules, trade secrets, "priviliged inter/intra agency memos", information affecting an individuals privacy, law enforcement records, records of financial institutions, and most importantly, geographical information concerning wells. Those are the same exceptions as in FOIA.

In the event that a document is considered exempt, government must still publish a redacted version, except in very limited circumstances, and even then, they must say that they've withheld a file. (The file's already publicly available, just not online, so you can check up on their claims). All of this is backed up with the ability for you to sue if they don't follow these rules.

So that's the jist the bill does. Why's it important?

First, because of that committee. Hopefully, if it isn't overrun by lobbyists, contractors and vendors, it'll give us-- the citizens-- a seat at the table about how our data is released to the public. That's really important. There's no official means, as it stands now, for the entire government to take our advice and suggestions about data publication. That's a big deal.

Second, unlike the Open Government Directive, which can be revoked by future administration because it is an executive order, this legislation requires agencies to put all information online generated or updated after enactment. It is also broader: the OGD only reaches some executive agencies, this gets all of them. And when this law passes, it'll create a flood of government data.

The technology community needs to band together to get this bill passed. Passing this law means:

Creating small businesses and jobs. Companies form around government data-- whether it be GPS or Customs Imports data, these datasets create jobs.
Empowering non-profits to further their mission. By releasing data, all non profits can further drive their missions-- drive down their costs and increase their effectiveness.
Driving down apathy amongst citizens. Transparency kills apathy-- when people know what's going on, they don't become more apathetic they become more active. They get involved and they try to make change.
A model law has been created that can be adoped by other governments both state and local, and international.

Because the stuff we work on is so obscure and arcane to most of society, this is going to take some work to pass. Like many bills, this could get sent to committee and stay there where it will shrivel up and die. That is, after all, what happens to most bills. We, the tech community, need to make sure people know about the bill, support it, and tell their members of congress to support it. Please-- read the bill, endorse it publicly, and do what you can to help us get it passed into law.

Quantifying Data Quality

Tom Lee — Wed, 10 Mar 2010 09:46:28 -0400

You've already heard me complain about data quality -- how it's a bigger problem than most people realize, and a harder problem than many people hope. But let's not leave it there! Perfect datasets mostly exist in textbooks and computer simulations. We need to figure out what we can do with what we have. In this and other posts, I hope to give the developers in our community some idea of how they can deal with less-than-perfect data.

The first step is to figure out how bad things actually are. To do that, we'll use some simple statistics -- those of you with a strong stat background can skip to the next entry in your RSS reader (or better yet, correct my mistakes in comments).

The GAO provides a good example of how to tackle this kind of problem. They were asked to examine government spending on the nonprofit sector -- a process that ultimately led to this report (PDF). As you might imagine, there are a number of ways that federal dollars make their way to nonprofits, from loan guarantees to tax breaks to medicare payments to nonprofit hospitals. For the most part, each of these is tracked through a distinct system.

Let's confine our work to one of the systems that GAO examined: the Federal Award and Assistance Data System, or FAADS. Along with FPDS, this makes up one half of the data powering USASpending.gov. FAADS tracks grant payments (and some other things that we'll ignore for now). Let's just examine that question: how do we figure out how much federal grant money went to nonprofits?

First we should define the subset of FAADS that deals with the nonprofit sector. If we can do that, and there aren't any other problems, then we should be able to just sum up the records and figure out the totals. There's a fairly obvious way to do this: FAADS records have a "recipient type" multiple-choice field, and one of the possible values is "nonprofit". But of course it's not actually that simple: the full name for the value is "other nonprofit". The field's possible values also include "private higher education". If you take a quick look at some records with that value, you'll see that many of those educational institutions are nonprofits. Worse, a look at the "nonprofit" category shows some suspicious entries. It's worth taking a closer look at the reliability of the values entered into this field.

But we need to get beyond saying the data is "pretty good" or "kind of dodgy" or "really bad". It would be useful to come up with a quantified estimate of how reliable the field is. GAO did this by picking a random sample of records, then checking each one to see whether it was correctly classified. That gave them a precise answer about how good the classification was within the sample -- but what does the quality of the sample tell us about the quality of the larger dataset? To figure this out, GAO calculated the confidence interval of their result.

The idea is pretty simple. Let's say there's a big population out there, and you want to quantify some attribute of it. Unfortunately, it's not practical to examine that value for every member the population. Instead, you'll take a smaller sample of the population, finding the value of the attribute just for its members. How close will that value be to the real value you'd get if you did examine the entire population? It's impossible to say for sure, but thanks to the magic of the Central Limit Theorem, we can look at how erratic the sample's values are and get an estimate of just how good it is at representing the larger population.

You've probably run into confidence intervals before when reading about political polls. "Politician A has a 50% favorability rating, +/- 3 points!" This means that the pollsters are 95% confident that the real favorability rating -- the one they'd get if they asked every relevant person -- is between 47% and 53%. Why are they 95% sure of this, instead of 90% or 99%? Well, 95% is a natural number to use thanks to the properties of the normal distribution, but in truth it's a bit arbitrary -- it's just sort of the stat-industry standard (interestingly, this means that one out of every twenty statistically significant results will be specious, even if the experimenters made no mistakes -- something to keep in mind the next time you read a breathless account of some amazing just-published scientific discovery).

As you might imagine, for a given population there are a few factors that can be tweaked when evaluating the confidence interval: sample size, confidence level, and the width of the confidence interval. As already mentioned, that second term is generally held constant at 95%; but tradeoffs are often made between the other two. You can get a more precise confidence interval by increasing your sample size, for example -- but that usually costs time or money.

But let's get back to the nonprofit problem. The GAO took a random sample of records from the "nonprofit" category and a random sample of all the other records in FAADS. They then turned to whatever supplementary sources they could lay their hands on to figure out if the listed recipient for each record was actually a nonprofit -- they used the IRS Business Master File, the Census of Governments, the Higher Education Directory, and various other subject-specific guides to the nonprofits present within specific economic sectors.

There were three possible results to each examination for a recipient's true nonprofit status: it could be a nonprofit, or not a nonprofit, or the investigation could be inconclusive. After examining the samples, they had a percentage value attached to each of these outcomes, which represented each outcome's share of the sample.

Each of those values can then plugged into this formula:

Where p is the true share of the population that falls within the category (e.g. "is a nonprofit" or "uncertain"), n is the size of the sample, and the p with a hat (called "p-hat", pleasantly enough) is the observed proportion of the sample falling in the category.

Simple! There are a number of assumptions underlying this which hide the statistical machinery: the assumption of equivalence between the standard deviation of the population and that of the sample; the fact that we're after a 95% confidence level. But this is a pretty standard way of going about it, and it seems to be how GAO approached the problem. For the nonprofit category, they reported a confidence interval of (.60-.79), meaning that about 70% of the sample turned out to represent genuine nonprofits, and based upon this they could be 95% sure that the true proportion of nonprofits in the larger population fell between 60% and 79%. They did the same thing with educational institutions, testing how many were nonprofits, and found an interval of (.88-.98).

(Some of you might have noticed that this operation can work backwards: some simple algebra reveals that GAO investigated 89 records in order to get these figures. In fact, the report says that they examined 96. This discrepancy is no doubt partly the result of rounding errors, but it's also probably somewhat attributable to their use of a t distribution, a technique invented at the Guinness Brewery in 1908 to help make reliable predictions even when the sample size is pretty small. It's not worth getting into the specifics here, but the upshot is that we need to adjust that 1.96 value for low values of n.)

So what can we do with this knowledge? It's tempting to say, "Okay, we have a statistically justified range for how many of these records are correctly classified as nonprofits. Let's take the total dollars for that category, multiply it by the high and low end of the confidence interval, and get an estimated range for total grant spending going to nonprofits."

I wouldn't say this is a bad idea, exactly. Certainly this gets you closer to the truth than simply taking FAADS at its word. You ought to add in the weighted sum for the educational institution category too, of course. And you should probably do the same thing for all the other recipient categories, to see how many nonprofits have been miscoded into those buckets. And it would be a good idea to account for that "unknown" category, too -- the records that might or might not be nonprofits.

But even then, we're in dangerous territory if we assume we have an answer. Much of what you read about statistics will be prefaced with "for a random variable" -- when we can't satisfy that assumption, things kind of fall apart. For instance: what if the government is in the habit of giving differently-sized grants to nonprofit recipients versus other recipients? That's something we should test for before we just blindly weight the sum of all grants. Worse: what if we're missing some records from the population? None of the above calculation tells us anything about anything other than the data we have in hand.

In the end, GAO didn't pursue these lines of inquiry. Testing the nonprofit classification left things close enough for government work, as they say. Still, by examining the quality of that classification, GAO peeled back and refined one assumption, and in the process, arrived at a better estimate. Not a perfect one -- there are still a number of assumptions being made here, and they're worth examining, too. But they did get an estimate of how useful one aspect of their data was, and that's a very good thing to know.

Lobbyists and White House Visitors

Clay Johnson — Tue, 09 Mar 2010 16:02:03 -0400

Recently and continuously, the White House has been releasing the "White House Visitor Logs," showing America who is coming in to meet with the President and his staff. At the same time, the Center for Responsive Politics releases cleaned up data on lobbyist filings. We thought it'd be interesting to find the intersect between the names in both sets of data.

Below, you'll find our results along relevant information from both sets of data. Now-- this is important: just because the names match doesn't mean they're the same person. Because the White House doesn't release any other form of identity information besides the name, we're unable to tell whether or not the name in one dataset actually refers to the same person in the other. John Adams in one dataset may be a different John Adams in another.

This is intended to be a starting point for journalists and citizen investigators, but isn't a reliable list of lobbyists who've been to the White House. Instead, it is a list of names of lobbyists who share names with people who have been to the White House and could be the same person. Whether they are or not is up to you to figure out.

Here's the data for your perusal:

White House Visitor Logs & Lobbyists

Thanks to the Center for Responsive Politics for making their data available for us to do this match.

UPDATE 3/11/2010:

We've updated this dataset and removed names of lobbyists who are no longer lobbying. This dataset only includes lobbyists who've filed since 2009. The previous dataset included potential matches for deceased lobbyists and those who haven't been lobbyists in years, and people who couldn't have possibly visited the White House during the current administration as lobbyists. The new data is more refined and more accurate.

Every Non-Profit is an Open Government Non-Profit

Clay Johnson — Fri, 19 Feb 2010 13:39:08 -0400

Often times at Sunlight the non-profit community looks at us strangely. Here in Washington, DC we've probably made more investments in technology than any other non-profit or advocacy organization I've run across. Certainly our mission is focused around the use of technology, so that makes a lot of sense-- we're focused on getting data out of government, doing interesting things with it, and letting you see what happens in Washington better. That means technology investment.

But one question I struggle with is: why doesn't every non-profit do this? It may be a little hyperbolic, but I have a hard time figuring this out. Every single non-profit stands to benefit from OpenGovernmentDataRightNow(tm).

Doesn't Charity Water stand to benefit from some form of data coming out of the United States government? Whether it be data from the EPA on water quality and saftey, or aid spending on water for Africa from the State Department? That data is there. It's waiting for you to ask for it.

Wouldn't the Leukemia and Lymphoma society do well to advocate for the opening of raw data from the department of Health and Human Services? I wonder if the Center for Disease Control tracks rabies data for the Humane Society? Wouldn't open data help Katrina or Haiti relief organizations not only fight waste and corruption on the ground, but also make better decisions on where resources should be provided? That data's there. It's waiting for you to ask for it.

With the exception of increased 403(b) caps or other forms of financial tax incentives, I can't think of a more common issue than open government data that the non-profit community stands to benefit from. Open data can help a non-profit make more informed decisions on how to allocate its very scarce resources, more effectively help those in needs, make better decisions, and drive down a whole bunch of costs.

We spend a lot of time talking about how open data can create billion dollar industries like GPS, but in the case of non-profits the gains are far more immediate and likely both to the organization's bottom line and to increase its impact.

So if you're working at a non-profit I invite you to think about how you stand to benefit from open government data. If you think of some ideas, you should be submitting them to the government. You should join and support our campaign too. You'll be doing your organization a great service and making the world a better place, too.

What if we Google Buzzed Government?

Clay Johnson — Thu, 18 Feb 2010 12:52:06 -0400

Following up on my hypothetical post on what would happen if Government had done the same thing that Google did with Google Buzz, I'd like to imagine something different: what if something like Google Buzz happened to government? What if, out of nowhere, the Executive Branch of government started exposing the most frequent contacts of each Senate Confirmed appointee based on their email inboxes? What would happen if we could, for instance, pull up Rahm Emmanuel's "Buzz" profile and see who he followed and who was following him, based not on his preferences, but based on the frequency of email contacts alone?

The answer is: Rahm would stop using e-mail. He'd use the phone instead. And when we bugged his phone, he'd do face to face meetings. And when we said that all face to face meetings must be on video except when you're in the bathroom, Rahm would put a toilet in the Oval Office and next to every other desk in the White House.

When you turn the lights on in your apartment, the cockroaches don't evaporate, they run under the couch. It's also why ultimately, to keep cockroaches out of your apartment, the answer isn't to keep all your lights on, or even to call an exterminator. It's to clean up after yourself.

Largely what we do in the transparency movement is turn lights on and watch to see where the roaches scatter to. It's important work, because it gives the bad guys less places to hide. Paul's piece on Billy Tauzin for instance, does a great job using the White House Visitor Logs, at shining light on the pharmaceutical industries lobbying effort, demonstrating not only the power dynamics of the lobbying industry but also the importance of the Visitor Logs themselves that the White House released.

Let's not take our eye off the long-game though. Transparency is a value, not an issue. There are a variety of issues related to the value of transparency-- but there's not a single piece of legislation that will cause government to be fully open, accountable and transparent. In the same way an exterminator is useful for solving your apartment's roach problem in the short-term, transparency legislation useful for moving the ball forward but it's only part of the solution.

The heart of what we're trying to change isn't technology or legislation. It's people's minds. You can't legislate how people think. To truly have a more open, honest, accountable government we need to invoke a cultural shift of both the people working for us inside the Government, and within ourselves in how we handle our Government.

To change the values inside of government, the right incentives have to be put in place. Bureaucrats who take risks and make data available should be validated by the public rather than scorned by their bosses. It should be the case that if a bureaucrat errs on the side of being "open" then they'll have a better long-term career success than those who don't. But that's not the case inside of government: too often the opposite is true.

To further that change, we have to connect the people inside who do the right thing and share the values of openness to one another so that they can share the tactics and ideas that have garnered them that success, and to help recruit new people on the inside to affect that change.

Together, we have to play both the short game and the long game to make things happen. Should someone build a Google Buzz like service out of the White House Visitor Logs? Absolutely. It's light shone on our government to make it more accountable. But we also realize that as that's done and as that data source becomes more and more of a source for our storytellers to make stories like Paul's that people inside of the White House will begin to have more meetings outside the White House, and that data will become less and less of an accurate representation of what's really happening.

If I'm to metaphorically call the corrupt forces inside of our government "roaches," then I ought to have a great name for the forces of good inside the government. Let's call the transparency advocates inside of government "kittens". Because everyone loves kittens and because to you and me a kitten might be cute, harmless and playful, but to a roach: kittens eat roaches. And what we need is not only to keep shining lights in the shadows of this metaphorical apartment, but also an army of kittens waiting under the proverbial couch for the roaches to hide.

More About the Door

Tom Lee — Thu, 18 Feb 2010 10:54:20 -0400

The above video -- put together by Noah, Ali & Greg, and featuring star turns by Daniel and Luigi's phone -- shows the current state of the door project I wrote about on Tuesday. It's working pretty well! I think I still need to add a bypass capacitor to improve the circuit's stability, but it's certainly good enough for our uses.

But the electronics are just one part of the system. As I mentioned at the end of that last post, my colleagues did an impressive job of springing into action and building out the systems necessary to turn an SSH-accessible script into a useful interface. Here's how they did it.

Kevin: Once Tom attached the door to our local network I took on the task of making it operable from outside of Sunlight's office via an public-facing API. The API enables any mobile device to become a door "key" by using a native client app or, for the non-smartphone users among us, a Twilio based voice response interface.

We choose to authenticate users based on the device's hardware ID or the Caller ID provided phone number for the voice interface, combined with a PIN number. Once the identity is verified the API triggers the latch code running on the Linksys router, opening the door. I used Django to build the API along with a small administrative interface for our HR staff to create and revoke "keys". For sake of simplicity (and security) our API and admin interface runs on a separate server and connects with the router over our local network. However, for the minimalists out there, it's possible to make this entire system run within the Linksys router.

In addition to offering an endpoint for the native client apps I also provided an interface for Twilio, an impressively easy to use service for integrating telephone access into web applications. Twilio's servers receive the phone call and then request a simple XML "script" from our API describing the response. Our server verifies the Caller ID information provided by Twilio and sends back a greeting message followed by a request for the user's PIN. Twilio then captures the key presses from the phone and submits another request with the PIN data. Assuming the credentials match, our sever unlocks the door and sends Twilio a response welcoming the user. The amazing thing about Twilio is that you can do all that with 30 lines of code. I had a prototype up and running in 15 minutes!

We're open sourcing all our code, including the APIs and administrative interface. The Django application is available here. Keep in mind this code is for demonstration purposes only -- please carefully consider the security implications of offering a web service to your door lock and understand that Sunlight Labs can't assume any responsibility for the use of this code.

Eric: I wanted to make opening the door the simplest experience possible for Sunlight's Android users (and we have many), so I created a native Android app that our G1, myTouch, and Nexus One owners could use. The Android app talks to the same web interface that the Twilio endpoint does, so we can keep our authentication and analytics logic centralized. The permissions are locked to a "Device ID" for the phone, which I get from the CDMA or GSM device inside the phone. In this way we tie access to individual devices, like actual physical keys.

It's a native, Java-based application, that consists of one main screen, and one widget. The first time you want to open the door, you need to open up the main screen from the application tray, select a PIN, try to open the door with it, and fail at it. Then an administrator will see the failed opening attempt, and can enable that device and that PIN for access from then onwards. The PIN gets stored on the phone so the user doesn't have to type it again, though the PIN can be cleared from memory if the user chooses.

For maximum door-opening convenience, a demure widget is included that will open the door in one tap, if the PIN is in memory. This takes advantage of Android's unique desktop and widget interface, one of the best parts of the Android platform.

I've published the source code for the app on Github, if you want to see how it all works. There honestly isn't a whole lot of crazy stuff going on, but finding good Android code examples is surprisingly hard, so if you want to see a basic widget or service in action, there you go.

Josh: It was very simple to create the iPhone application. Using the Titanium Mobile SDK, I was able to quickly throw together an application that would authenticate via the web service. It took about two total hours to write with the majority of that spent playing with the user interface to see what worked best and creating an icon and graphics. The application was passed out to Sunlight employees with iPhones using the ad-hoc distribution model.

Titanium Mobile: http://appcelerator.com
Code for Dorky (Door Key) iPhone Application: http://github.com/jroo/dorky-iphone

ClearMaps: A Mapping Framework for Data Visualization

Kevin Webb — Wed, 17 Feb 2010 11:17:52 -0400

Despite the recent explosion of web based cartography tools, making effective maps for data visualization remains a challenge. While tools like Google Maps are great for helping navigate the world they are often poorly suited for thematic mapping, as many features like roads and cities only get in the way of telling compelling stories with data. In fact, even the distance between places can be a distraction – who cares how far away Alaska is when the goal is to make a simple comparison between US states?

To help overcome some of the limitations with existing mapping tools Sunlight Lab is releasing ClearMaps, an ActionScript framework for interactive cartographic visualization. In addition to giving designers and developers more control over presentation the project aims to address some of the common technical challenges faced when building interactive, data driven maps for the web. ClearMaps is designed as a lightweight, flexible set of tools for building complex data visualizations. It is a framework not a plug-and-play component (though it could be a starting point for those wishing to make reusable tools).

In addition to offering flexibility, ClearMaps attempts to address two of the biggest technical problems encountered when building maps for the web: rendering of vector data in-browser and reducing vector data size for timely loading. Unfortunately there aren't many browser-native approaches to rendering that can scale to the full range of visualization tasks. For example, rendering the boundaries of all 3,100+ US counties isn't performant when using JavaScript and the canvas object. Things only get worse when compromises are made to support non-canvas compliant browsers such as Internet Explore. Data transmission time is equally problematic for complex geometries. At full scale, US Census Shapefiles for county boundaries total 14MB. Even with detail reduced for web display the raw vector data are nearly 2MB in size.

Given these challenges ActionScript becomes an attractive solution. The Flash plug-in's rendering engine is faster and more widely available than in-browser vector rendering. Plus, Flash offers tools for efficiently encoding and compressing binary data streams, significantly speeding up transmission time. Fortunately, with the release of Adobe's open source ActionScript 3 SDK and open source IDEs like FlashDevelop, the proprietary nature of the Flash platform has become less of a concern.

ClearMaps provides an Adobe AIR based encoding tool for translating data from Shapefiles into a compressed binary form and a set of ActionScript classes for decoding and rendering vector data. These tools provide the core functionality required to get from raw cartographic data to a web map with a minimum of glue code (the above demo map requires less than a hundred lines of ActionScript).

However, the library is far from complete. Features like keys and legends are currently missing and much work remains in building an extensible framework for integrating external data sources. If you find these tools useful drop us a line and let us know how you're using the framework. If you have ideas or code for solving common visualization tasks we would love to incorporate them into the library.

Are the American People short on ideas?

Clay Johnson — Tue, 16 Feb 2010 17:07:42 -0400

A couple of developers from the Sunlight Labs community, including one of our Great American Hackathon organizers Jessy Cowan-Sharp, managed to put together something remarkable: OpenGovTracker (source here). The site lets you see where the ideas are coming in across the various agencies from a single dashboard.

What's the synopsis? According to this it's that the American People don't have a lot of ideas. Well-- a lot of agencies are pretty low on ideas. Only 611 ideas have been proposed. Treasury only has a dozen ideas? The best the American people can do is give Social Security 10 new ideas?.

As our the Sunlight Foundation's Policy director stated late last week: now is the time. Request a dataset or submit an idea to government. Here's how.

Our Door Opener (A Science Project)

Tom Lee — Tue, 16 Feb 2010 10:00:23 -0400

Life in the labs has been pretty good since we moved into our current offices. Before, we were spread out over two floors: my team was upstairs in a stuffy law office sublet, and the rest of our colleagues were stuck in a homey but increasingly cramped and run-down space four floors below. Since moving everyone to the third floor we've found ourselves with plenty of room, lots more light and a nicer kitchen. It's just a more pleasant working environment in general.

But there's always room for improvement. For one thing, the new space came with new locks -- ones with really expensive keys. Issuing keys to the entire staff wasn't practical, and coordinating door-opening responsibilities in a way that accommodated team members' occasionally odd schedules was inconvenient. Fortunately, the space also came with the button you see to the right. This button, with a few hacks, now makes it so we can open up our office with our Nexus Ones, iPhones or even SMS.

Located near the reception desk, this button opens an electronic latch on the front door. Pulling the assembly out of the wall revealed the system to be about as simple as possible: the button simply connects two wires. Bridging them with a screwdriver fired the latch (from their small gauge and uninsulated connections, it was obvious we weren't dealing with dangerous voltages, but please don't start pulling cable from your walls unless you know what you're doing).

Connecting two low-voltage wires electronically isn't a particularly hard trick, so I decided it'd be fun to spend some evenings building a system to expose that switch to our network. I've built a few projects in the past that make use of custom router firmwares and Arduino microcontrollers, so I decided to use those tools for the job. I'm a big fan of this approach -- it's a great, inexpensive way to add a scriptable network interface to your microcontroller projects (if you're interested, I talk more about this technique here).

Ultimately, I ran into difficulty getting this particular router's serial connection to work, so I abandoned the Arduino -- it was overkill anyway. Inspired by the excellent new MAKE: Electronics book, I decided to build a circuit to do the job. Here's the finished franken-router, hooked up to the switch:

So how does this all work? Well, let's start with the router. It's a Linksys WRT54GL, a still-Linux-friendly descendant of the WRT54G, the router which jump-started the custom firmware scene. There are a lot of custom router firmwares available -- DD-WRT, Tomato, Gargoyle -- and they can all make your consumer-grade router do some professional-grade things. For this project I used OpenWRT, the system on which many of those other projects are based. It's got a much steeper learning curve than those other distros -- you definitely need to be comfortable with the command line -- but it lets you build a custom firmware with exactly the components you want. With some tweaking I was able to create a firmware that included a stripped-down version of Python, but which still fit into the WRT54GL's meager 2MB flash memory. (Like the Arduino, Python turned out to be overkill, but at the time I expected to need it -- and it's always nice to have it handy.)

With the firmware installed, I was able to SSH into the router and perform some simple manipulations of the system's GPIOs -- General Purpose Input/Outputs. These connect to things like the system's LEDs and switches, and can be controlled in software. I selected a GPIO that didn't seem to be used by OpenWRT -- it illuminates the "DMZ" LED on the front panel -- and wrote a very simple script to control it. I could now flip a tiny light on and off from a network connection. The hard part was over.

Next came the hardware. It was easy to add a pin header to the router's PCB -- it's got holes drilled and ready to go (some of them are connected to router systems -- see here). To this I hooked up ground, power, and a connection to one of the DMZ LED's pins.

A cable then connects those three pins to this breakout board:

There are a few things going on here. That blue cylinder is a relay, and it's the heart of the system. A relay is basically just a switch that can be closed by the electromagnet that's attached to it. They're useful for employing a small voltage to turn a larger voltage on and off. In this case, the switched inputs of the relay are wired up to the connection posts that I added to the outside of the router: when the relay is energized, those posts are electrically connected. When it's not, they aren't. The relay runs on 5 volts, which is why there's a voltage regulator on the board: to reduce the router's 12 volts to 5 (the other components on the board can run on a range of voltages; they're perfectly happy at 5).

The router's GPIO isn't up to the task of controlling the relay directly, so there's some additional circuitry that's needed to turn the GPIO signal into a firing relay. First up: the 556 chip, which contains two 555 timer chips in a single package. When triggered, one of these fires the relay for a set amount of time -- about half a second. The other timer starts up as soon as the circuit gets power, and stays on for almost half a minute. It's responsible for disabling the first timer during the router's boot sequence, when various scripts cycle the LEDs (and would therefore accidentally fire the relay and open the door).

You trigger a 555 by connecting one of its pins to ground, and at first I thought I'd be able to simply hook the GPIO up to the 555's trigger. No such luck: this GPIO, at least, doesn't go to ground when you switch it. Instead it changes between 2 and 3.3 volts (in between is the threshold where the LED happens to light up). The solution to this was to use a voltage comparator chip. This compares its input -- the GPIO -- to a reference value that's supplied on another pin. In this case the reference voltage is 2.5 volts, made with an extremely simple voltage divider (basically just two equal-value resistors in series -- the junction will be at half of the total voltage across them, which in this case is 5. Science!). If the input value is above the reference, it outputs a high value. If it's below the reference, it connects its output to ground. So the comparator's output gets connected to the 555, and voila! We're done.

This is a pretty simple project, but I'm no electrical engineer, so it took me a while to get it working. But eventually I was able to use my SSH client to open the door latch. That's not exactly an ideal interface, though: it's both hard to use and hard to administer. I turned the problem over to my coworkers in the labs, and they built the rest of the system with amazing speed (and, it should be noted, in the middle of a blizzard). They've come up with some pretty impressive solutions -- more on that in the next post!

UPDATE: I've been asked for a schematic of the door-opening circuit; here it is as an EAGLE .sch and as a simple PNG. If you do build this, a word of warning: the initial timer, which is meant to disable the system during boot, behaves somewhat differently in-system than it does on a breadboard. I'm still trying to address this issue -- and have added a bypass capacitor to the schematic in an attempt to do so -- but for us, it's not a particularly pressing issue, as the vulnerability it represents is still substantially harder to take advantage of than it would be to simply smash down the door. Still, if you plan to use this circuit, please be aware of its limitations and understand that you use it at your own risk.