It sounds obvious, but the idea remains revolutionary. For the first time, there would be a single repository for source code that could be shared between the hundreds of agencies, commands, and programs in DOD. Developers would be able to share their work in a familiar, web-based environment. A previous version of forge.mil was pulled for unknown reasons, but the current iteration is based on the TeamForge product from CollabNet. If you’ve used SourceForge, you get the idea. The DOD is the largest consumer, and one of the largest developers of software in the world. Much of this software is redundant, locked up by vendors and integrators, can’t work with other software, and nobody remembers how to maintain it. There’s no doubt forge.mil was long overdue.

It’s dangerous, though,  to assume that forge.mil is a useful template for forge.gov. I think forge.mil could lead forge.gov down the same road as core.gov and other failed attempts to encourage source code reuse in government. To understand why forge.mil can be useful and simultaneously poisonous to forge.gov, you have to first understand how the DOD does software.

COTS vs. GOTS

Before, say, the mid-1990s, much of the DOD’s software was owned by the government. GOTS, or “Government Off-the-Shelf” as it’s now called, was built and maintained by the DOD and its contractors. This was appropriate for some military-specific systems, but the strategy outlived its usefulness when the government could no longer keep up with commercial enterprises. For many pieces of common software, like operating systems, spreadsheets and web browsers, the open market produced more innovative and higher-quality products. So down came the order: use commercial software. COTS (“Commercial Off-the-Shelf”) was ascendant.

Boeing, Lockheed Martin, General Dynamics, and the other integrators no longer write as much software from scratch. Instead, their business model has changed. They now glue commercial software together, and wrote code to fill in the gaps. Their business model, however, remains the same. Instead of seeking rent on the proprietary software they’ve written on behalf of the government, they seek rent on the integration work they’ve done.

This is a lucrative business: the latest UAV is composed of hundreds of software systems, some commercially available, some written by the integrator. If I’m the integrator, I’m the only one who knows how all the pieces go together and I can charge a handsome sum to anyone who’d like to field their technology on my platform. Think of Apple’s locked-up App Store, but flying through the air with a missile strapped to its belly.

There’s nothing nefarious about this, of course. Integrators are doing what the market commands, and controlling access to platforms is an perfectly legitimate business model. It discourages reuse, though, which means that it’s difficult for the DOD to effectively use the software it purchased or developed. You can read more about this strange market and its consequences in the excellent “Losing the Softwar(e)” by my friend and fellow Open Source for America member, John Scott.

Openness as a Desperate Act

So Rob Vietmeyer of DISA decided to borrow from the principles of the open source community and Internet governance. A more open and transparent development process at DISA could remove barriers to reuse, encourage collaboration, and discourage proprietary or closed systems. COTS software is still king, but where the government needs to control its own integration, set its own standards, and exercise stewardship over its own infrastructure, it can still develop its own GOTS solutions — this time, in an open, collaborative manner. Where GOTS was once insular, slow-moving, and highly proprietary, it can now be produced at lower cost and with lower barriers to entry for new innovations. Because this “Open GOTS” is built using familiar open source methods, the projects have a fighting chance of working together.

The Walled Garden and the Moral Hazard

So far, the DOD software problem sounds similar to the government software problem. Don’t misunderstand forge.mil, though. It’s not your typical open source development environment. If you talk with the forge.mil project team, like Guy Martin of CollabNet, he’s quick to correct you if you compare forge.mil to SourceForge. forge.mil is something very different. With all the challenges of this profoundly broken market, forge.mil had to make some serious sacrifices.

The first thing you’ll notice about forge.mil is that you can’t get to it. Access to forge.mil is severely restricted. To get access, you must have an official DOD Common Access Card (CAC) or have a DOD employee sponsor you for an “ECA Certificate”. Dave Wheeler of the Institute for Defense Analysis describes it as “gated development.” On the mil-oss mailing list, this is referred to as the “CAC Wall”.

The CAC Wall has some unintended consequences, and raises some very difficult questions. First, it prevents many well-meaning developers who don’t work for the Defense Industrial Base from helping the projects inside. There are literally tens of thousands of developers behind that wall, and forge.mil keeps that group partitioned from both the public and government employees without CAC or ECA credentials. The bargain is that by keeping these undesirables out, they carve out a sense of safety so skittish program managers are more likely to host their code. The cost of that sense of safety is a much smaller audience than they could muster hosting on more public platforms.

The CAC Wall also creates a dangerous an incentive to split communities. What happens if someone in the DOD wants to hack on a piece of open source code? They’ll host the hack at forge.mil, and the public could never see those hacks again. If I’ve licensed my project under an open source license, it’s because I want others to contribute. If that code disappears behind the CAC Wall, I’m cut off from tens of thousands of DOD developers. This “forge.mil fork” scenario is serious business. It’s a scenario where everyone loses.

Finally, the CAC Wall may create a moral hazard for the developers who live behind it. In the open source community, folks are very careful about what code they commit, and since they’re never really sure who’s a friend and who’s a foe. In a community where everyone’s “trusted,” developers can become complacent, making them more vulnerable to poorly written or hostile code.

Despite these real concerns, the CAC Wall still makes sense for forge.mil. DISA has some very legitimate security concerns about the code that’s being developed. They’ve weighed that risk against the advantages of public scrutiny, cooperation with external projects, and they’ve made the informed decision to keep the code behind the “CAC wall”,  and to their credit they encourage forge.mil developers to contribute patches upstream wherever possible instead of hosting on forge.mil.

forge.mil as Role Model

So we have a kind of Judgement of Solomon in the forge.mil platform. Because of its very unique market dynamics, the DOD needs to take advantage of open source projects, the open source development model, encourage collaboration, and reduce its reliance on proprietary platforms. But for that to happen, DISA had to put all the work behind the CAC wall.

forge.mil has attracted the attention of other agencies. How could it not? The DOD has the worst-case scenario: the gravest problem, the most complex market conditions, and the most dire consequences. So other CIOs take notice and the press on this forge.mil experiment has been relentless.

This brings brings us to the news that the GSA’s Dave McClure is planning forge.gov, a civilian counterpart to forge.mil. This is exciting. This is also terrifying, because they seem to be following the forge.mil model by restricting access to only US citizens. Guy Martin says:

I realize that putting up a barrier to entry in the form of positive identification of US citizenship and a vetting process will irk some who believe that everything should be free and open…

Consider me irked. I want to be perfectly clear about this: the compromises made in forge.mil are dangerous for forge.gov. If forge.gov were to follow the forge.mil “CAC Wall” approach, it will permanently damage the “Open GOTS” movement.

forge.gov cannot be forge.mil

forge.mil may be instructive and inspiring, but it’s a corner-case and fraught with compromises that have diminished its utility. In the case of forge.gov, it would be hosting unclassified code for civilian agencies. There’s no need to create a “trusted” environment. There’s no need to verify the citizenship or security clearance of its participants. The standard open source mechanisms are more than sufficient: only project leaders can commit code to the repository, a semiformal review procedure for patches, and so forth. In any case, I’m struggling to imagine why the repository would be better secured by allowed access to 300 million people. Let’s agree that making US citizenship a prerequisite is counterproductive, unworkable, unnecessary and most important: it’s un-American.

The entire country of Jordan has adopted the VA’s VistA software for their national healthcare system. Countless overseas researchers collaborate with their US counterparts through open source projects. When we share our source code with the world, it improves the quality of the software and is, in fact, a uniquely practical kind of diplomacy.

Since we don’t need to control access to the projects as we do in forge.mil, I have to wonder why we would need a prescribed set of tools for hosting each project. TeamForge is a fine piece of software, but there are literally dozens of viable alternatives. Developers are very picky about their tools, and extremely picky about the version control systems that are at the heart of these code repositories. If we presumptuously select tools on their behalf, we create an unnecessary barrier to entry.

The ideal forge.gov is two forge.govs.

With this in mind, we should break the forge.gov project in two.

First, forge.gov is useful as a catalog of open source projects that are used and created by the civilian government, many of which happily reside on agency websites or public repositories already. Forge.gov could be the way that projects and developers easily find each other. The Freshmeat or Ohloh of government, if you like. Let’s call this the forge.gov Catalog. The forge.gov Catalog would be as inclusive as possible, tracking the progress of every Open GOTS project we can find.

There are still projects that need an infrastructure, of course, and forge.gov could provide that. The forge.gov Repository would be provided by the GSA as a service to agencies, and provide a complete development environment. Here, I’m thinking of github or SourceForge.

I think by splitting these roles, and avoiding the dreaded CAC Wall, we can include the largest possible group of contributors and take best advantage of the excellent open source work that’s already underway.

Am I missing something here? Is there a national security concern that I’m overlooking? Is there an advantage to a homogeneous set of developer tools that I don’t understand? Let me know.

[Thanks to Melanie Chernoff, John Scott and Jim Stogdill for helping to cut this post by more than half.]

I think I was a surprised as anyone when I heard that Larry Lessig was stepping away from Creative Commons. It seemed like a sudden change of direction, because Lessig has been a vocal advocate for freedom and choice for so many years. But as I hear Lessig describe his journey from Creative Commons to Change Congress, I’m reminded of Daniel Okrent’s history of the prohibition movement in the United States, “Last Call”.

In the book, Okrent reminds us that the prohibitionists needed major structural reforms in American politics before they could eliminate alcohol in the United States. The movement was composed and motivated largely by women, who at the time could not vote. The reforms they sought would only arrive once women were given the franchise. Also, the Federal government relied on alcohol taxes for 30%(!) of annual Federal revenue. So suffrage, the income tax, and prohibition are intimately acquainted even if they are, on their face, unrelated.

Susan B. Anthony was a passionate temperance advocate before she became the most famous suffragist. She left the temperance movement, in part, because temperance had many advocates, while suffrage desperately needed leadership. In this light, Lessig’s newest project suddenly makes perfect sense.

I had a chance to talk with Matthew Burton, the former intelligence analyst turned open source cause celebre who just launched a tool that helps frame and understand arguments with imperfect evidence. It’s based on method called Analysis of Competing Hypotheses (ACH), which has been around for quite some time. Matthew and his friend Josh Knowles, though, have a tool that allows the ACH method to be used by multiple participants simultaneously. It’s fascinating stuff, so I’m grateful that he took the time to talk with me.

On a personal note: I’m delighted to see that Matthew is a fellow emdash enthusiast, as you’ll see below.

First, tell me a little about ACH and how you first became interested in the method.

In the fall of 2005, Dick Heuer, the creator of ACH, contacted me after reading an article I’d written for Studies in Intelligence. The article was about how Intelink could benefit by being more like the Web. Dick had been wanting for some time to build a Web-based, multi-user tool for ACH, so he asked me to build it. I spent the following summer at DS&T, interviewing ACH practitioners and trainers.

Intellectually, the most fascinating aspect of this project has been its applicability to groupthink and dissenting viewpoints. When I started, the Intelligence Community was still feeling the effects of Iraq WMD blowback. Dick referred me to a book, “Groupthink: Psychological Studies of Policy Decisions and Fiascoes,” by Irving Janis (ISBN: 0395317045). It’s a fascinating book. Janis evaluates several US policy failures from the 20th century. He not only makes it clear that a groupthink tendency had a hand in misguiding groups of otherwise brilliant men (they were all men); he also pinpoints the moments where much-needed dissent and skepticism were quashed by the desire not to disrupt the camaraderie that comes with consensus.

In my summer at DS&T, I learned that this was a problem with ACH as well. The existing software couldn’t record multiple viewpoints, meaning dissenting opinions evaporated; and analysts had to state their opinions in front of everyone else instead of from the pressure-free environment of their desks. I saw a real opportunity to make a dent, however small, in the problems that lead to intelligence failures.

Are you dogfooding? Has the hyper-rationality of ACH slipped into your real life? Did you apply ACH when you asked your wife to marry you? :)

Ha! Fortunately, I can weasel my way out of that question just by explaining the purpose of ACH. It’s a tool for discovering facts–either has already happened, or what will happen in the future.  When it comes to questions that involve personal preferences, it’s not going to work so well, because its goal is to keep you from thinking subjectively about objective matters, not help you objectify the subjective.

Were you a developer first, or an analyst first?

I actually wouldn’t call myself a developer even now. I maintain the ACH code, but most of it was written by my friend Josh Knowles, a classmate from ITP who’s collaborated with me on several projects. But I’ve been a geek for a long time, having been neck-deep in the Web since I was 15. I took that knowledge–and my expectations for the national security structure’s technological prowess — into my job at DIA. That led to disappointment, so I channeled that negative energy into a desire to change how this place works.

Does your analyst interest complement your developer interests, or are they two separate things?

Most of the development projects I work on are meant to fix a problem that is consuming me, whether it be intelligence analysis, legislative transparency, or political campaigning. I mentioned I’ve been a geek for a long time. But I consciously avoided studying computer science in college because I didn’t want to code for it for its own sake, as an end; I didn’t want to end up building Web sites for online pet stores. Very soon after joining the government, I realized that my interest in technology could be used to further a bigger goal, and I’ve been doing that ever since.

If you developed this tool while a contractor, how did you end up with this code? What kind of hurdles did you have getting a copyright?

Josh and I developed the code ourselves, so that’s why I have the copyright. We did it in collaboration with both Dick and a DC consultancy called Pherson Associates. The Phersons—two retired CIA veterans—have been training analysts to use more structured analysis techniques like ACH for years, so they and Dick were there to make sure the software doesn’t betray the methodology. We built this for an intelligence community platform called A-Space, which was supposed to cater to outside developers, much like Facebook opened its platform to third-party app developers. But it’s been two years since A-Space launched, and neither the development specifications nor the purchasing mechanics have been defined. In that time, our own lives have changed: I’m an entrepreneur handling multiple projects in New York, and the Phersons are neck-deep in their growing training business, so it makes sense for us to take the ACH software in a direction that keeps us from having to maintain dozens of licenses while still allowing us to shepherd it. That means open source.

I know you’ve spent a lot of time thinking about licensing. Can you walk me through your thinking, and how it’s evolved?

It hasn’t evolved so much as led me in infinite loops. My goals for open sourcing the code are in some ways conflicting: I think ACH can help the world make better, more fact-based decisions, so I want it to be as widely adopted as possible. But I’ve been on a personal crusade to change the way the government buys software—namely, they should spend less on it and avoid vendor lock-in. So these two goals pull me simultaneously away from and toward copyleft. Beyond copyleft, the intricacies are so fine that I find myself unable to really comprehend the eventualities that my various options would lead to. When I reach that point in my thinking, I realize that I’m probably overthinking it, so I stop. Then I do it all over again.

It sounds like you feel as though open sourcing the code would lead to less use in the government? Can you unpack that for me a bit?

I feel as though certain FOSS licenses would lead to less government adoption than would other FOSS licenses. Specifically, the government relies on lots of proprietary legacy systems that would be contaminated by GPL code. And any intelligence agency that modifies the code would be loathe to share those modifications back to the world. So according to some people I talk to, the GPL is a non-starter. However, other people say that the copyleft provision would not be invoked simply by providing GPL code to government users, as such code is only being used internally. Different people have different opinions about the GPL’s acceptability in government, which makes me think that the fate of the software depends less on my licensing decision than it does on the worldview of the prospective user.  This realization takes me back to the end of that thought loop, where I feel like I’m overthinking it.

On the commercial side, anyone whose profession requires them to use the facts at hand to figure out a puzzle: pathologists, investigative journalists, detectives, investors. I’m really excited about the non-commercial side. It’s been frustrating to watch political discourse in our country devolve into nonsense that is less about the facts and more about what “feels” right. This attitude is beginning to take hold of the policy process as well, and that’s bad news. I’m under no illusions that ACH matrices will start appearing on the Senate floor, but the idea of using ACH-backed arguments to explain policy proposals to lawmakers is intriguing.  I think this falls in line with the mission of Expert Labs, and I plan to reach out to them and see if we can do something cool together. I also think universities are a good fit for ACH, as college is all about challenging your preconceived notions and teaching you the meaning of rigorous research.

Go to http://competinghypotheses.org. There, you’ll find the repository, the code documentation, and information on how to join the community.

[The site will be live in about two weeks, Matthew and Josh are working furiously on it, I'm sure. For now, you can sign up to get notified when it's up.]

We’re big fans of Stack Overflow, the Q&A community for developers. We think it could be a great platform for answering citizens’ questions about government: navigating bureaucracies and legal codes is very daunting, and finding answers to simple questions—how do I amend my tax return, how do I fight a parking ticket, how do I get a permit—can be extremely frustrating.  Government agencies are experts at making you wait in lines and on the phone. We know that they answer the same questions many times a day, and that private citizens can usually add valuable information (“Don’t go at 1 pm, it’s really busy”) that the government employees can’t or won’t provide.  We’d like to form a community of confused citizens and people who are willing to help them, so in between ongoing projects, we’ve been sketching out how we want to accomplish this — whether it should be done through Stack Exchange, whether we should build our own, etc.

[You can learn more about the project at http://govdecoder.com/.]

Finally, what’s your favorite government open source project?

I’ve never seen it in action, but DHS’s Virtual USA project sounds remarkable. On top of using open source software to build it, the objective of the project is to break another government taboo: sharing information with other agencies and levels of government. Having been an intelligence analyst who relied a lot on mapping tools and was constantly frustrated by the inability to share geographic data even within your own building, it’s apparent that if Virtual USA delivers, it’s going to dramatically change how first responders work.

# # #

I want to thank Matthew for taking the time to speak with me. I think the project is fascinating, and I can’t wait for it to launch.

Conservation Commons is an international effort to facilitate the exchange of environment conservation data, tools, and analysis. They have a number of open source projects for GIS work, and creating tools and protocols for data exchange.

Scientific Linux is a Linux distribution put together by Fermilab, CERN, and many other labs and universities. It provides a set of tools specific to researchers that aren’t easily available commercially. Each lab contributes to the effort, providing a common platform for their tools.

The European Grid Initiative and EU DataGrid are similar to Scientific Linux, but the focus is on providing a common platform for grid computing in the EU. It ties together large-scale computing grids across the EU, providing researchers a stable source of funding, and allowing the collective innovations of each national grid program to be rolled up into a shared platform.

Open Source Drug Discovery is a project sponsored by the Indian government which uses the open source model to encourage international collaboration on the development of useful drugs.

I also stumbled on an EU-funded paper “Collaboration within tool and die making industry through open-source ERP-solution with integrated CRM-functionalities,” the intent of which is pretty clear from the title.

If you wander over to the EU Open Source Observatory and Observatory, they list over 2,000 projects, many in academia, which are using the open source model to facilitate collaboration. This is in part, I think, because the academic environment is already predisposed to sharing work and building on the effort of others. Also, many of these programs are funded by the EU, which has something to do with it as well — the EU wants to make sure that the work it’s funded is available to the public.

It’s sometimes difficult to draw the line between projects that are international, and projects explicitly designed for international cooperation. If I’ve missed your favorite, add it to the comments!

[I want to thank Michael Tiemann and Jan Wildeboer, and Venkatesh Hariharan, who gave me some great pointers. Thanks, guys!]

A small piece, very loosely coupled.

Today, we announced that Red Hat Enterprise Linux is shooting for its 14th Common Criteria certification. My job means I get excited about Common Criteria certifications, which also means I’m unpopular at dinner parties. This certification, though, has me more excited than usual, because it means much more than a rubber stamp from a certification body. With this certification, we’re including the SELinux security system and the KVM virtualization system. In short, it means being able to run many systems on one piece of hardware, and making sure that those systems can’t touch each other.

If you’ve seen me speak in the last few months, you’ve heard me talk about how the modular design of open source projects. Staying modular allows for new features to emerge by combining different, often unrelated components. Also, this “small pieces, loosely coupled” approach brings new features faster because improvements to one component don’t disrupt the whole system. I talk about the Linux community getting their “chocolate in their peanut butter, and their peanut butter in their chocolate,” which is more fun to say than “Linux creates an environment for emergent capabilities.”

Because the architecture permits this kind of innovation, we’re able to solve problems when we weren’t even trying. This combination of SELinux and KVM is a great example: we can now create sandboxes, so that if one computer is infiltrated, the attacker won’t immediately have access to everything on the same machine. This feature, which we call sVirt, is usually interesting only to security nerds. But I think it has the potential to solve some very pressing real-world problems, especially in the military. To illustrate what we’ve made possible, let’s go to Iraq.

The Blue Force Tracker, courtesy PEO-C3T

The tactical elements in Iraq, like tanks, Humvees and unmanned aerial vehicles, run all kinds of systems in a variety of security enclaves. So the Blue Force Tracker has its own housing, the diagnostics have their own berth, and radios have a box of their own. These systems are, for the most part, physically isolated from each other. In part, this is for security. In part, it’s for modularity — I want to be able to take my radio out of the vehicle if it’s disabled and have it still work. It’s also because different companies built each system. For all these reasons, this is about as inefficient as it could be. The most precious resources in a tactical vehicle, like power and space, aren’t being shared.

This scarcity has a very unpleasant side-effect. The scarcity encourages people to make systems that are larger than necessary, so they can capture a “footprint” on the vehicle, and thus seek rent on the space they’ve claimed, which make the problem worse. The shortage of space gets downright tragic when you learn that any individual piece of software is likely using less that 15% of the computing power available, which means that 85% of the power that could be running a computer is instead being burned away as heat.

So power, space, weight, and cooling are inefficient sum-zero competitions on tactical platforms, as long as this scarcity is in place.

The Consequences

For the warfighter, the heat generated by these Balkanized systems, combined with the heat outside the vehicle, means the regular air conditioning is useless. Between running the air conditioners, the computers, and everything else, you can imagine the kind of mileage these vehicles get.

The scarcity also means program managers are not at liberty to adopt the best applications for a particular mission. If a new capability is needed, physical hardware has to be installed. The PM has to account for power, weight, and cooling requirements to add what could be a weightless piece of software. Hardware makes new systems more expensive to procure and slower to deploy.

Integrators who are eager to provide a good idea or better system to the military have to compete with other integrators for space and power on the platform. This friction could prevent the best and most useful software from reaching the field — the integrators should be competing on their capabilities and cost, not real estate.

Inertia.

It’s sad, but this broken market isn’t remarkable, it’s just a grim reality. Having separate boxes works nicely with the acquisition process, and program managers can easily manage each box from each contractor. The physical separation, or “air-gapping”, crudely addresses security concerns by preventing one system from leaking information to another. It certainly encourages modularity. So we live with the platform that we have, and everyone from the contractor to the warfighter has optimized their process around this flawed arrangement.

But look over here…

In the commerical world, though, data centers are embracing virtualization. By hosting many workloads on a single physical machine, we’re able to use 100% of our hardware, instead of just 15%. Virtualization also makes it simple to add new systems to an existing physical infrastructure. Instead of each program bringing their own box, they bring only software, which weighs nothing and consumes no space. As it should.

By using these data center patterns, tactical vehicles can escape the sum-zero trap. Physical space is no longer a limiting factor, removing the perverse incentives for a “land-grab” on the platform. Power use is no longer such a big problem. The heat problem is diminished, because sixteen separate computers can been consolidated onto two redundant computers.

But for all that virtualization can provide the tactical environment, it has not — to date — addressed the security concerns. A secure multi-tenant virtualization environment is still a “dark art” in the security world.

The Open Source community delivers, by accident.

The SELinux Mascot

Which brings us back to the Common Criteria certification and this ingenious and mostly accidental combination of SELinux and KVM.

For the program manager, this means that more and better innovations can be delivered more quickly to a tactical platform — without worrying about space and heat demands.

For the integrators, they can deliver their products on industry-standard Linux and Windows systems using a known quantity. This multi-tenant platform isn’t anything arcane, expensive, or novel: it’s the same Linux they’ve been working with for years.

Warfighters, of course, are relieved of the odious heat problem, but more importantly: it’s easy to re-provision the hardware with the computing workloads they need to accomplish their mission, without requiring the forklift upgrade which makes new workloads so slow and expensive today. It’s even built on a robust, smart open source platform, so we can be sure that this isn’t the last clever new feature or innovative approach. Moving to SELinux and KVM through Linux, we’ll also been in the best possible position to incorporate new ideas.

Now, those of you who work in this market know that there still many obstacles between what we have now and the kind of solution I’ve described. There is the matter of redundancy by design, the need to use system in mounted and dismounted situations, and the security standards for something like this are still murky. Nevertheless, we’re closer than we’ve ever been. And that’s exciting.

And a word on innovation.

A virtualized system like this could solve a bunch of existing problems, but it could also solve problems we haven’t yet anticipated. A Marine with the unimprovable nickname of Major Neutron once said, “Don’t pack it if you can’t hack it.” Jim Stogdill asserts “coding is maneuver.” In other words, the more we can tinker, the more we can adequately respond to a changing environment.

Having a virtual platform in the hands of a warfighter can encourage Major Neutron’s flavor of innovation in the field by providing a safe place to play. You’ll remember the example of SFC Stadtler, who jerry-rigged WiFi from parts he found in the trash. Think what a soldier like SFC Stadtler could do with a safe sandbox within the computers on his Humvee. With properly mediated access to the radios, maps, and other components, he could actually piece together the systems he needs without having to pull wires out of abandoned buildings. Because his tinkering is inside a sandbox, he could play without fear of breaking anything. So sVirt’s sandboxing isn’t just about consolidation or saving power, it’s can also be about enabling innovation at the edge.

So I think you can see why I’m excited. I love the idea that open source community can deliver solutions to practical problems — even when it’s a complete accident.

Your turn.

So what have I missed? What stands between the current platforms and a properly secured virtual platform using this newly certified Red Hat Enterprise Linux? Can you think of other applications for a safe sandbox in virtual environments?

[I want to thank the many folks who gave me comments on the initial draft of this post. It's been greatly improved with your help.]

It splits each page of the PDF into PNG files, and then re-packages the PNG files as a single PDF. Note that I chose 200125dpi resolution — that’s about as low as I could get without it ruining the antialiasing. The resulting file will be, of course, much larger than the original.

[Update: Dave Egts updated the script to clean up after itself and use an even smarter resolution. Thanks, Dave!]

#!/bin/sh

if [ -z "$1" ]; then echo $0 '<input file>'; exit 1; fi;

filename=$(basename "$1")
extension=${filename##*.}
filename=${filename%.*}

echo Creating images... && \
convert -verbose -density 125 -quality 10 -antialias $1 $filename-%03d.gif && \
echo Creating PDF from images... && \
convert -verbose $filename*.gif $filename-flat.pdf && \
echo Removing images... && \
rm $filename-*.gif

For a long time, I did these using the Gimp. It’s kind of a tedious process, especially if you are doing this for multiple images. Even if you automate it with a script, you have to start up the Gimp to get it to work. I wanted a way to do it from the command line, so I could convert a bunch of images at once.

So I wrote this quick script to use ImageMagick instead:

#!/bin/sh

if [ -z "$1" ]; then echo $0 '<filename>'; exit 1; fi;

filename=$(basename "$1")
extension=${filename##*.}
filename=${filename%.*}

convert -channel RGBA -colorspace RGB -background none "$1" \
\( +clone -background none -shadow 60x5+10+10 \) \
+swap -layers merge +repage \
$filename-shadow.png

You’ll note that I always convert to PNG, because it’s important to preserve the transparency. If you convert to a JPG file, it’ll look terrible. So here’s an example. Notice that the shadow isn’t a square box — it actually follows the shape of the visible portion of the image. Shazam!

Before

After

Licenses keep software open.

Open source software depends on its licenses. They are the architecture that keep the communities together. To steal a line from Michael Tiemann: they’re the one thing that stays the same so that everything else can be different. Licenses work very well in the private sector, because everyone is playing by the same rules. They don’t work so well in the public sector.

Our process keeps software closed.

The licenses are fine, but because the simple bargain of the licenses is buried under multiple layers of contracting, bureaucracy and secrecy, they don’t work as well as we’d like. You may have a great project you want to release. You might have an amazing patch to an existing project. You may want to use free software on your program. Whatever the case, you often have a long journey in front of you. It’s nobody’s fault, it’s just how the world works right now. The result: software that should be free disappears into government programs, never to be seen again. So how does this happen?

Tivo-ization.

I know this will come as a shock, but there are programs out there that use free code and don’t abide by the rules. Or maybe the abide by the rules, but have gamed the licenses to their advantage. That doesn’t make them evil, but it does make our lives more difficult. It’s a real problem — the Tivo clauses in GPLv3 specifically address this. It’s common for a program to take free code and lock it up in a black box. Whether it’s on a plane, or in a tank, or in a financial ERP system, we’ll never see that code again.

Secret open source.

Integrators and government programs are, at their core, about risk management. The last thing they want to deal with is something complicated or unusual. Open source is, for them, complicated and unusual. There’s a strong incentive for the lowly engineer who happened to use that Python library or that Linux kernel to hide it. Dan Risacher (godfather of the DOD open source memo) has a great story about the developer who deleted all the copyrights in his source, rather than be subjected to his company’s lawyers.

Tweakers.

This is a behavior that’s not specific to government. Our lowly contractor  improves a piece of software, but considers his contribution too insignificant to contribute upstream. SHAME ON YOU, lowly contractor. For all you know, a hundred other engineers want to fix that same problem, but can’t find the time. You can solve their problem, and you’re choosing to keep it all to yourself. Selfish!

Government is government.

Here’s a structural problem: when the government gives code to the government, like when agencies share code, the distribution clause of the GPL isn’t triggered. This is great for contracting officers, this is terrible for the community. The government can tweak code and it’s not obligated to contribute that code back upstream. This means redundant effort, wasted time, and runs against that architecture, the rules that make our community work.

Government Forges.

We’re going to talk a lot about forge.mil at this conference. I can feel it. Love it or hate it, forge.mil, forge.gov, and all the other government code repositories are here to stay. They’re useful for specific tasks, like encouraging sharing in classified environments. But by their nature, they encourage code to stay in one place. We have to make extraordinary efforts to get code out of the forges and into the public repositories. You need the will and the means, otherwise that code is lost forever.

Security.

Which brings me to security. No matter how well-constructed your license is, it’s trumped by national security. Here, I’m talking about classified use of open source code, but don’t forget ITAR restrictions, which can put your contracting officers in the awkward position of having to fulfill the terms of the GPL and simultaneously not violate the State Department’s export controls.

What can we do?

So we’re beset on all sides. At every stage of the process, from IRADs to distribution, there are barriers to the productive sharing of open source code. The situation has felt hopeless for a long time. I’m here to tell you: it’s not hopeless. We have the power to change how this system works. And we don’t need Congress, we don’t need a memo from the DOD CIO. We can do this all by ourselves.

What do we want from the process?

The point here is not to hack the system. The point is to create a useful community of developers in the DOD. Overcome the regulations, the laws, and the stovepipes. Useful communities, Etienne Wenger tells us, have certain characteristics. Fortunately, these characteristics align nicely with the problems I’ve just described.

Design for Evolution.

We can include language in contracts that ensures the open source software is contributed upstream. It really is that easy. If more PMOs required upstream contribution as part of the contracting process, they could reduce program risk, mitigate vendor lock-in, and make it significantly easier to abide by the spirit of the open source licenses.

Invite different levels of participation.

Dave Wheeler and Dan Risacher are our founding fathers. They’ve been through the law, the FARs, the DFAR, They know this stuff better than anyone. We all need to be as good as they are. We need to make it simple for people to contribute. So, we need to make all these issues easily understood. That’s why I’ve created this flowchart. It’s not the final word, and it hasn’t been anywhere near a lawyer, but this flowchart walks you through all the issues you’ll encounter when you’re trying to release software from inside a government program. Patches welcome.

Develop public and private spaces.

Let’s agree here: forges are for secret squirrel stuff. Everything else should be public. There’s a dangerous trend I’ve noticed — that forges are being treated as de facto “approved product lists.” This is terrible. Think of Red Hat’s Fedora model: forges should be where we experiment, where we play, where we tinker. From the forges comes software that we can use on programs. Keeping the forges away from the APLs frees up the forges, gives them less scrutiny, makes programs less protective of the code that they hold.

Open a dialogue.

Courtesy nachtnebel on flickr, licensed CC-BY-NC-ND-2.0

We all need to be talking about this stuff. The PMs and the COs and the PEOs and the oversight committees won’t come to open source by themselves. They need to be made aware of all the advantages, and all the barriers, before we see structural change.

Focus on value.

Courtesy of adunne on flickr, licensed CC-BY-NC-ND 2.0

As a community, we spend a lot of time talking about what should happen, what structural changes need to be put in place, and so on. I say we stop now. Clay Johnson, when he was kicking off the CrisisCamp for Haiti, said: “Democracy is the enemy of useful work.” What he meant was that consensus through talking and bargaining is a waste of the community’s time. Just get out there and release your code. If we demonstrate the value of our community, and show through useful work that we can change how things work in government, the change will follow behind us. We can’t wait for someone to fix this for us. We have to do it ourselves.

Combine familiarity with excitement.

Courtesy of wannawork, licensed CC-BY-NC-SA-2.0

For a developer, nothing is more important than their tools. There’s a tendency in our industry to solve every problem with a process, and every process needs a tool. Too often, we content ourselves with tools that someone else wrote. They’re outdated or insufficient. Don’t stand for that. Go build your own tools, bring in the best practices from the public open source community and use them. The forges need to serve the developers, not the other way around. Familiar tools will broaden our community, and make us much better developers.

Create a rythym.

Courtesy of drinksmachine on flickr. Licensed CC-BY-NC-ND 2.0

We need to stay vocal. We need to be a thorn in our contracting officer’s side. We need to be the problem children. We also need to hold up our achievements. Too many of us toil in the dark. Start a blog, start a meetup, talk about the great things we’re doing. Help each other out. We need to create a cadence to this movement, and that cadence is what will make us larger and stronger.

Do these three things.

So this was a lot. I think I can distill all of this into three tasks:

• Understand the contribution process. Review the flowchart, talk with your lawyers. Make sure people know they can come to you with questions. We have your back.
• Shout success from the highest rooftop. We’re doing something revolutionary, and we need to be vocal. Congratulate each other publicly. Tell your PM you saved him a million dollars. Tell your CO that releasing the code brought in five new features.
• Find one thing you can set free. If everyone in this room found one piece of government code to set free, to submit upstream, we’d not only have more code to work with, we’d have one hundred open source contribution experts working in the DOD.

Thanks.

But wait! It’s not boring at all, because at the same time as Suricata is released, the Washington Post’s Top Secret Nation series is running. A pall suddenly falls over every aspect of government, especially in security, and especially for Dana Blankenhorn of ZDNet. “Private open source security is not amused,” and neither is Blankenhorn, who is quickly becoming my favorite source of new material:

“The idea seems to be that military contractors will, together, copy the most useful open source tools under their own control, claiming it’s for security, and thus think they are delivering on the Administration’s open source promises while continuing to charge out the wazoo.”

Woah. There’s a lot of red meat in there, so let’s untangle the argument. First, he presumes that someone is getting charged out the wazoo. I don’t know where he gets that impression. Second, he presumes that releasing the code is a cynical act to satisfy an utterly non-existent promise from the Administration. Believe me, if the Administration officially promised anything to do with open source, I’d be shouting it from the rooftops. Third, he presumes that the OISF is not entitled to use open source code, which makes no sense whatever. The whole point of open source software is to share, borrow, and take advantage of the collective intelligence on the community. Fourth, he drags out the increasingly facile cliché of “military contractors” with “ties to government”. As I’ve said before, you can feel how you want about the DOD or the government, but it doesn’t have anything to do with code that’s available to the community.

Here’s another way to tell the story: public and private sector organizations rely on Snort, but are (right or wrong) dissatisfied with it. Maybe it’s the architecture. Maybe it’s the licensing. They decide that they should work together on a different approach. Can we all agree that this is perfectly reasonable? Isn’t this exactly how communities get formed?

Now, this could have gone horribly wrong. All too often, a government contractor will ask the government for money to develop some new technology. Once developed, the government gets rights to use it, but the contractor owns it. The contractor now has a shiny new technology that it can charge both citizens and the government for commercializing. This happens all the time. In some cases, it makes sense. In other cases, it creates monopolies and is an incredible waste of taxpayer dollars. I’ve mentioned John Scott’s take on this before.

But that’s not what happened here. Instead, all the concerned groups got together and decided that they wanted to build something of their own. They wanted to try something new. Instead of keeping that work to themselves, or entering a potentially wasteful procurement process, they decided that they would try their experiment under the open source model. Let me be clear: this is a success story. This is exactly what we want. No inadvertent monopolies, no wasteful contracting, no redundancy, just useful code that everyone can benefit from.

There’s another theme in Blankenhorn’s article, which suggests that the government is trying to starve out private industry. If the government had paid a company to create a proprietary product that it could just as easily have grabbed off-the-shelf from someone else, that’s not just a bad idea, it’s against the rules. Specifically, 41 USC Sec. 253a and FAR 10.001. But that’s not what happened here. First, they wanted to try a new approach that wasn’t available on the open market. Second, the code was released for anyone to use — even SourceFire. No one is getting special treatment. I think it’s very difficult to argue that OISF “shouldn’t” do this without slipping into a protectionism argument that favors SourceFire — which, by the way, doesn’t need any protectionism:

“During the company’s earnings call its officers were quick to note that only 30% of its revenue comes from government and only two-thirds of that comes from the federal government. Even if the OISF took away its federal work, in other words, it would be OK.”

So what’s the problem here? I see a responsible use of government funds, new code in the open source community, and a functioning, competitive market. So thanks to OISF, thanks to DHS for funding the project, and thanks to Snort and SourceFire for your tireless work to improve the IDS field.

I noticed that Brian repeated the misunderstanding that San Francisco had the nation’s first open source policy. I don’t want to diminish his larger argument, but it’s important that we give credit where credit’s due. So for the record:

• May 28, 2003: DOD issues the “Stenbit memo,” which assures readers that open source is commercial software under the law, and can be used in the DOD.
• July 1, 2004: OMB issues OMB-04-16, making clear that open source can be used in the Federal Government
• September 30 2009: Portland, OR is the first city to issue an open source policy.
• October 16, 2009: The US Department of Defense CIO issues a memo reiterating that open source software is commercial software for procurement purposes, and encouraging DOD branches to include open source when they’re picking software.
• January 7, 2010: California‘s open source policy is published.
• February 1, 2010: San Francisco, CA issues their open source policy.

These are just what I could find, of course. If you know of others, let me know! If you’d like to see a comprehensive history of open source battles in national and state governments around the world, CSIS maintains an annual survey intuitively titled “Government Open Source Policies“. Even just skimming it, you’ll be surprised at how little progress the United States has made in open source policymaking.

[Update: David Wheeler was good enough to point out the Stenbit memo at the sister post on opensource.com.]