Originally posted on “Microsoft On The Issues”, 5th November 2009

For governments, the digital revolution presents some important challenges, including how best to ensure that their digital documents will be accessible and readable essentially forever, regardless of how technology evolves. This is a question I encounter often in my role as a Regional Standards Officer, working with various public agencies in Australia and New Zealand that have responsibility for archiving records and other documents.

Fortunately, the challenge of e-archiving has been addressed over the past decade or so through standards development and other collaborative efforts around the world. International standards offer guidance on how to capture documents and how to use metadata to ensure that they can be located and understood by future generations. Published specifications for the Open Document Format (ODF), Open XML and Portable Document Format (PDF) also ensure that documents in these formats will be readable long after the formats are obsolete.

That said, a lot more work is still to be done.

Although ODF, OpenXML and PDF account for the bulk of documents produced today, other formats need to be documented. Microsoft has put documentation for our Office binary formats, for example, into the hands of the British Library for reference by future generations. We encourage other vendors to take similar steps with their storage formats.

New standards are needed to ensure retention of at least some of the massive volume of data produced in social networking activities, such as the micro-blogging increasingly used by elected officials to interact with constituents. Broad collaboration on enhanced standards and processes would help clarify how much of this data needs to be retained, and how to store it.

Microsoft is committed to working with governments and all other interested parties to meet these challenges and preserve the history of this decade, and the next, and the next.

The level of activity around the world in opening up government data is nothing short of astounding. Governments at every level have engaged citizens, businesses and developers in combinations of public discussions and hackfests to look at how the data that they hold can be used in new and exciting ways.

As I talk to friends in government, participate in various events around open data and watch the work that developers are doing in this area I can’t help but think that there is a massive gap between the expectations that government leaders have for open data, the tools that developers are building and the objectives of the general public.

On one side of the conversation you have government ministers and departmental heads. My conversations with this group often come back to ways that data can be used to make government more efficient. Ministers have looked at the success of crowd sourcing in other sectors and are keen to find ways to apply those ideas to the machinery of government.

In many cases ministers are looking for assistance from the community to analyse the cost of building a kilometre of road, maintaining a hospital bed or operating a prison cell, then helping find ways to reduce those costs and increase governmental efficiency.

As an example, in a recent speech to The Institute of Public Administration in New Zealand, Bill English the countries Deputy Prime Minister said;

A second concept for the future is “inside out government”.

Government holds a wealth of information.  Some of it – quite rightly – is sensitive and access should be strictly controlled – tax records for example. 

But in other areas, I see no reason why we can’t turn government inside out, so to speak, and make the same data and information available to those outside of government. 

Government can tap wider resources in the community to analyse and use government data to help solve problems and produce insights. A ministerial committee is exploring this concept.

Inside out government also requires government to be open to good ideas from business.

We want to see ideas generated in the private sector and NGO sector genuinely considered and appraised – not simply ruled out on the basis that these sectors might not understand all aspects of government.

As I have said in previous articles, governments have a business to run and business leaders will look for ways to improve the way that it works using the assets and tools that they have available.

On the other side of the equation we have the tools and applications that are being built with the data that governments are already starting to publish, the theme of many of these applications appears to be somewhat different.

Early applications and much of the conversation that is being driven outside of government tends to focus on government transparency and public control of departmental activity, delivering applications that will help the public understand ministerial expenses, ensure that bills are read in detail before votes are cast and alerting citizens as a piece of legislation that potentially affects them passes through parliament.

The reality is that there is significant benefit underpinning both of these agendas, more efficient government is certainly a good thing, as is a more transparent and accountable government. The work that needs to be done to accommodate both agendas is probably pretty similar, and some of the foundations that we’re seeing from initiatives like NZGOAL will go a long way towards delivering what both sides need.

To those who have worked around government for some time the risks should start to become evident. While the goals of the two agendas may be similar, the language and the end expectations differ dramatically.

In the example above where Minister English talked about inside out government there was a great deal of cheering for the prospect of the New Zealand government publishng more open data but I saw very little mention of how community led projects would help him meet his objectives.

Both sides need to start to listen very carefully to the other, if they don’t then I worry that we’re on a trajectory that will eventually lead to two very unhappy and dissatisfied groups of people.

I’ve used examples from New Zealand in this article, but I don’t see the debate being dramatically different in the many other countries that I’m also following.

The long week last week ended with my joining the Open Government Data barcamp event that several folks have been busy organizing in Wellington.

It is clear from the topics that were discussed that there is lots of great work going on in this area in NZ already.

As I spend more time in New Zealand I’ve noticed that there always appears to be discussion at this type of event around how we don’t want NZ to get left behind in whatever area of new technology is being talked about, in this case I’m pretty confident that it is just the opposite and that the NZ community is much closer to the leading edge than similar groups in other countries.

The output from the day and the details of the conversations will eventually be well documented on the wiki that is being set up, so I thought I would avoid general commentary and just share a thought that was with me as I left the event on Saturday.

Listening to the presentations I was trying to think about three different constituencies and how the topics that were being presented were relevant and would be received by them.

The first group is an obvious one and it is the one hundred and sixty or so participants at the barcamp event itself along with others from around the country who have an interest in this topic, many of whom are already working on solutions of one type or another. For this group the conversation brings obvious benefits, sharing ideas and best practice is always a good thing, as is knowing who else is working on projects that might be able to prevent expending effort on initiatives that may be duplicative.

The second hat I tried to wear was that of the general citizen. Transparent government, open government and participatory government are all great goals, but are not always top of mind for the general citizen (whoever that is). Over the last decade we have learned a great deal about this area while working on various eGovernment solutions, focus groups have frequently told us that what they want is a less intrusive government, and to pay lower taxes – most other noble goals end up being secondary.

Not quite as forward thinking as the citizenry that many of us would wish for, but a frequent reality all the same. For many taxpaying citizens solutions that build upon open government and open data have to slide quietly into processes that they already interact with, or disappear under the covers completely to be dealt with by a third party group or agency.

Finally, as a third group, I tried to think about how government agencies themselves would view the conversation. While we like to think about a government agency as “our government” with a duty to serve us, we also have to consider the fact that an agency is a business much like any other. Money is raised in the form of taxes, grants or loans and gets spent on carefully planned projects.

Additional projects like building applications that use or publish open data sets have to be funded somehow, in many cases that involves cutting existing projects or in extreme cases finding ways of raising more funding.

For this conversation to be highly relevant to government I still think there needs to be some thought around what the business case for an agency would look like, and how ministers would be encouraged to drive policy that makes open data relevant to the way government operates on a daily basis.

It is the intersection of the objectives of at least these three groups that would eventually deliver the environment that would have to exist for open data sets to be an every day reality in New Zealand’s governance structure.

It was a thought provoking day, personally I enjoyed it, hopefully I’ll get invited back to the next one… and next time around I have a few ideas of my own that I would like to throw in to the conversation!

There are a couple of ways of looking at gov20 technologies and their role in national governance, most of the chatter that I’m seeing at the moment is focused on just one side of the opportunity.

Certainly there is a role for gov20 technologies in increasing government transparency, but what about the efficiency of government itself?

As I have said before on this blog, we often think about Government as a monolithic organization that serves faceless services to the tax paying population.

In reality government is much like any other major organization with hundreds of departments, thousands of business processes and tens of thousands of employees. Changes can be complex, they need to be well understood and represented in a business cases that clearly explain the benefits that will be realized through any money that is spent.

To give you an idea of the size of governments, the United Kingdom’s National Office of Statistics presents the UK civil service at around 500,000 full time employees and Statistics Canada shows the Canadian federal government consisting of around 300,000 employees. These numbers are representative of national government employees only, and don’t include state or local employees, it does not need to be said that these are big organizations.

So, I hear you ask, what does this have to do with gov20?

There are several areas where I think more internally focused use of gov20 technology could bring significant efficiency to the way that government works, helping the civil service understand what might be happening in other major agencies, or helping track policy changes coming out of the various executive branches of government.

Implemented correctly these technologies could make government significantly more nimble and better prepared to deliver the external agendas that are currently being discussed in blogs and in the press.

For example, one area where I feel we might be able to see some immediate benefit would be in dealing with the perception that governments are risk averse, instead using technology to ensure that government employees are better informed about the way their business process interacts with others and preparing them to react more rapidly to changing environmental circumstances.

From the outside government can often appear to be slow to react to a changing environment, in my experience the civil service tend to be anything but, in many cases they are dealing with programs of unimaginable size and complexity.

When you are a company delivering a new service you get to make some choices about who your customers will be, what quality of service you might offer and how that service will interact with the rest of your business. Governments rarely get to make those choices, changes to business practice have to be considered in terms of how it will impact processes across the rest of government, and developed in a manner that does not disenfranchise any part of society. As such a change program in Government can be significantly more complex than it would be in the commercial world, and it takes time for organizations to work through that complexity.

I’m surprised that we have not started to see vendors offering social networking and web20 solutions that aim to solve some of these complex issues inside of government, finding ways to use technology to connect a few hundred thousand internal employees of a particular government to provide information and insight on upcoming programs along with a forum to quickly work out any related challenges.

Beginning by tackling some of these internal challenges may help government employees see the value of gov20 technology within their own environments, I’m confident that it would help spark new ideas around how social media and gov20 technologies could be used externally across many thousands of employees, rather than relying on a handful of departmental champions as we appear to be doing in many cases at the moment.

Over time I have a view that solving some of the internal challenges, and then looking at how you might externalize what government learns during that process, will rapidly get us to the full level of interaction that an increasing number of people crave, rather than pushing for potentially premature implementation of gov20 services that may be disconnected from internal processes and valid business cases.

One standard, or multiple standards?

It seems like a pretty cut and dried debate, if there was a single standard for everything in  the software world then life would be simpler. Wouldn’t it?

Back in the early 90s we thought that we had the whole single standard thing pretty much sorted out. For email we would use X.400, for ethernet communications we would use TP4 and so on.

I was working with many of our public sector customers in the UK at the time, and assisted with a number of projects aimed at delivering X.400 enabled mailboxes broadly across the UK civil service.

In the mid 90s along came the internet, bringing with it with the stack of protocols that we are all aware of today.

The agencies that I had been working with mostly had one of two strategies, either a pure X.400 implementation which at this point would be mostly redundant, or a broad strategy of interconnection with Notes, MSMail, Groupwise, X.400 etc., in which cases SMTP was just one protocol in a list of many and as such could be folded into an existing project plan.

Even then it was not the end of the road for X.400, as recently as three years ago we were still seeing the protocol being implemented in military environments where features of the X.400 specification remained crucial for email in the battlefield.

Moving to today, even given the pain that a number of technologists felt during the 90s, the debate around one or multiple standards in a given technology domain still goes on.

In many cases, if we play this debate out in a technology domain that we all know well then it makes very little sense.

Lets use high speed local area networks as an example. There is an obvious choice for a single standard that will work for everybody in this domain, it is TCP/IP. Isn’t it?

Choosing and mandating only TCP/IP has a couple of effects on my ability to communicate with the array of devices that I might find in any corporate datacenter today. Starting with legacy systems, I might have just restricted my ability to talk to those older systems which could be running protocols like IPX, Netbeui or various mainframe communication protocols that have existed through the years. On the other end of the scale, we’re starting to see and increasing number of devices implement the next version of TCP/IP, IPv6.

So for something as simple as networking it seems we have a need for a range of protocols to deal with all of the systems we might be planning to work with. The majority traffic on the wire will obviously be TCP/IP on any network today, but a percentage of it will be something else, and over time I would expect to see the majority migrate to IPv6 or a protocol that has yet to be developed.

This becomes the case for at least three reasons.

First of all, any evolving datacenter will either have existing legacy systems, or will have a need to connect to external third party customers or partners who are using legacy technologies.

The second is that developers (companies and individuals) continue to innovate in every area of the technology itself. Sometimes that innovation is incremental, sometimes it comes from another group and is ostensibly competitive.

The third is that while many standards look similar at the outset (Netbeui and TCP/IP are both just networking protocols for example) it quickly becomes clear as you delve into the details that they exist because at a point in time a development group perceived a scenario that could not be addressed by the many protocols that had already been defined and made available to the market.

My conclusions are by no means empirical, but I do find that this argument tends to fall on the side of preference for a “single standard” for a given technology domain in a handful of scenarios;

There could be a preference for a particular technology. This one is generally pretty obvious, you will hear something like “there should be a single standard in this area, and it must be my standard” thrown into the debate.

Simplicity of implementation may carry the highest possible priority for a small subset of projects, with a relative disregard for factors like legacy integration or future innovation. This is an interesting line of argument, and in my view only works for stand alone solutions that will have a short life span. Solutions that will never have to communicate with a legacy user community or technologies, and do not have to be ready to deal with innovative technology changes that might come along in the future.

Finally, it can sometimes come down to a lack of information being presented to the group making the decision. When you glance over a particular technology domain, choosing a single standard in that area generally looks pretty straight forwards but when you dig deeper it is rarely the case, as with my example of networking protocols.

Many of the software products that have enjoyed success in the market over the last couple of decades have done so in part because of their support for multiple standards and formats, I suspect the same will eventually be true for enterprise and eGovernment technology solutions that are being implemented today.

Early eGovernment history is littered with examples of implementations by departments who were struggling to work out what it meant to provide an online service.

Many of those early sites did little more than provide you with a form that you could download, print, and then drop in the regular mail once it was completed. Others needed you to go and visit a building to pick up a password before you could go online… and so on.

Much of the gov20 activity on the web today feels pretty similar, but as I have mentioned before has the added complexity of an extra layer of participating stakeholders in the form of citizens and businesses.

There is a phenomenal amount of discussion about the high level of participation citizens might enjoy in a gov20 world, yet when look at some of the early projects neither government nor citizens seem to be heading in that direction.

From the side of the citizens I’m witnessing a push for more control of their elected officials and civil servants. As an example, a group called Sunlight Labs are running a program that they call “Apps for America”. It is a terrific program, encouraging developers to build tools that use the increasing amount of data that is being published on data.gov to enrich their civil experience.

There are some fantastically useful apps that are appearing, but the type of functionality that is offered by those tools tells an underlying story about how people are perceiving value in the data that is being published. Many submissions appear to be tools that will ultimately offer the public more control over their government, not necessarily mechanisms to participate in their government’s processes.

Trending officials voting patterns, monitoring their expenses, looking at who they meet with and how they are connected with external organizations… all useful stuff, but not exactly participation.

From the other side of the fence, the government side, there are equally good examples of a lack of two way communication. In many cases departments have not yet managed to muster the resources or design the processes that will be needed to really take advantage of bi-directional participatory gov20 tools.

Instead we see these tools used as huge online megaphones, shouting information out to as many people as possible while spending very little time listening to what those people have to say in return.

Of course, as with many issues that governments have to solve, there is no simple solution. In the example of the Whitehouse twitter feed, just how would you listen to almost a million voices all shouting back to you at once.

Over time, as both sides begin to see the real value in participating together on day to day issues, I’m confident that we will start to see real solutions to these challenges, but for now at least we appear to be some way away from realizing the goals that are front and centre in gov20 discussions and planning…

I’ve mentioned JPEG XR before, along with some links to information about some of the advantages that the digital picture format brings.

It was pleasing to note this morning that the specification has now been ratified by ISO/IEC as IS29199-2.

Bill Crow talks about the process a little on his blog today;

The many innovations delivered in JPEG XR originated from over ten years of research and development at Microsoft.  And through our participation as a member of the JPEG Committee, we’ve been able to deliver additional improvements and create an International standard available to everyone.

So the next question everyone asks is “when will it be available in a camera?”  Since I don’t work for a camera company, I can’t answer that question.  But I’m pleased and proud that because JPEG XR is now a free International standard, there’s very little standing in the way of making that happen!

He also mentions some of the key benefits of the format;

Better compression – JPEG XR offers improved efficiency compared to JPEG, and the type of compression artifacts are often less objectionable than the typical JPEG compression artifacts.  JPEG XR offers a very wide range of compression levels, including perceptively lossless or mathematically lossless compression.  Regardless of your requirements, JPEG XR probably offers a compression option that’s ideal for that scenario.

More image formats – JPEG XR supports 8bpc (bits per channel), 16bpc and 32bpc, as well as several special bit depth formats.  Pixel values can be stored as either integers, scaled fixed point numbers or full floating point values; this provides full support for numerous high dynamic range (HDR) imaging scenarios, as well as support for wide gamut color spaces.  In addition to 3-channel RGB, JPEG XR supports monochrome, CMYK and n-channel formats up to 16 independent channels.  many of these formats also support an alpha channel.   This wide range of image formats allows for dramatically better image quality and allows this single new file format to effectively replace many previous formats that were required for specific scenarios.

Advanced decoding features – JPEG XR provides progressive decoding, allowing lower resolution previews or specific cropped areas to be displayed without the need to decode the entire image.  Additionally, JPEG XR images can be cropped, rotated, flipped and resized (within certain constraints) without ever needing to decode and then re-encode the image.  That means these operations are much, much faster and no additional image quality is lost due to the additional encoding steps.

Marius Oiaga at Softpedia has a short story about the standard, and highlights the fact that it is freely available to potential implementers;

Having been approved as an international standard, JPEG XR is, of course, available for free to all interested parties. In this regard, software publishers and hardware manufacturers can easily integrate support for the new file format into their products. Of course that interoperability is the biggest barrier that stands in the way of the adoption of the new standard. But with Windows already offering support for JPEG XR, the ecosystem of software and hardware solutions built around the operating system could soon follow.

If you want to learn a little more then the press release from the JPEG committee can be found here.

I think this could well be another first for us.

I’ve talked about the importance of the protocol documentation in past posts, but I think this is the first time we have had things written up and available so far in advance of a product launch.

Tom Robertson announced the availability of the Office 2010 protocol documentation this morning on the Microsoft on the Issues blog:

This kind of open access to information is a central tenet in our Interoperability Principles. The aim of these principles is to give all software developers—including commercial ISVs, open source developers, and developers in customer IT departments—technical information that can be useful in building products that work well with our most popular products. All told, we’ve published more than 33,000 pages of technical documentation relating to interoperability with these Office products, and we’ve seen more than 250,000 downloads of the entire range of our interoperability documentation in just the past year.

This model of transparency through access to developer resources has already enabled real-world interoperability in other areas. In December 2008, for example, Microsoft published detailed protocol documentation for Microsoft Exchange ActiveSync (EAS), as part of the expansion of our EAS IP Licensing Program. EAS is the technology that enables mobile devices to synchronize e-mail, calendar and other information with Microsoft Exchange Server. While EAS had been licensed in the past, the posting of this documentation provided consistent, open access for all developers to the Microsoft protocols built into EAS that enable interoperability. As a result, many leading companies—including Microsoft competitors like Apple, Google, Nokia, and Palm—have licensed the EAS protocol patents, making it possible for their customers to synchronize Exchange data on the most widely used mobile phone platforms in the world.

That second paragraph is a great example of how this documentation is being used.

Each year we do something around OSCON that brings us a step closer to delivering on the interop commitments that we have made to the wider IT community.

This year’s Day One announcement is that we’re contributing twenty thousand lines of device driver code to the Linux kernel community for inclusion in the Linux tree, and doing so under GPL v2.

Peter Galli was the first to talk about it over on Port 25:

The drivers will be available to both the Linux community and customers, and will enhance the performance of the Linux operating system when virtualized on Windows Server 2008 Hyper-V or Windows Server 2008 R2 Hyper-V.

In an article posted to Microsoft’s PressPass site, Tom Hanrahan, director of Microsoft’s Open Source Technology Center, notes that this is a significant milestone because it’s the first time the company has released code directly to the Linux community. "Additionally significant is that we are releasing the code under the GPLv2 license, which is the Linux community’s preferred license," he said.

… and Hank Janssen followed with another post on Port 25, where he included the following bullets;

• Yes, our device driver code was released directly to the Linux Kernel
• We released the code under GPL v2
• We are working with Greg Kroah-Hartman so it is ready for the next release of the Linux Kernel, version 2.6.32
• We will continue to update the driver code to enhance interoperability on an ongoing basis, but it’s our hope that other developers in the community will find the code useful and worthy of collaboration.

Microsoft developed the Linux device drivers  to enhance the performance of Linux when virtualized on Windows Server 2008 Hyper-V.  My team and I were responsible for testing and validating the driver components that were contributed for this first release. 

The commentary I have seen so far has been factual and positive.

Todd Bishop has an FAQ with Greg Kroah-Hartman, good answers to questions I know some of you will be asking;

Where the Microsoft code will fit in: "We include all drivers into the main kernel source tree. So you don’t go and download external drivers somewhere, it just all comes included. Now, when you build it, you select all these different options on what you want built and what you don’t want built. Distributions build everything as a loadable module, so that it doesn’t take up any space in your memory, unless you want to use it, unless you have that specific piece of hardware. So in this case, if you happen to want to run Linux on top of (Microsoft) Hyper-V, you just will load these modules and away you go."

John Fontana over at Network World has a fun quote from Jim Zemlin;

"Obviously we are tickled about it," said Jim Zemlin, executive director of the Linux Foundation. "Hell has frozen over, the seas have parted," he said with a chuckle.

And finally Ina Fried has her usual comprehensive view of what this is all about for Microsoft and for Linux;

Although Microsoft has released open-source code in the past, the company has generally favored licenses other than the GPL. That said, the GPL is the way into the Linux kernel and Microsoft wants this code in Linux.

In an article on its press Web site, Microsoft acknowledged the departure. The company has also been going after Linux for years, both on the marketing and legal fronts.

"Today, in a break from the ordinary, Microsoft released 20,000 lines of device driver code to the Linux community," Microsoft said. "The code, which includes three Linux device drivers, has been submitted to the Linux kernel community for inclusion in the Linux tree."

The move comes at a time of mixed signals from Redmond when it comes to Linux. Microsoft has said that the browser-based versions of Office, which are due out next year as part of Office 2010, will support Firefox, bringing at least a portion of Office to Linux for the first time.

Personally, I don’t think our signals over the last few years as being the least bit mixed, a commitment to interoperability is just an issue of the market realties and customer demands of today vs. a decade ago.

Can I say, “I’m still a believer”?

On the flight yesterday I was thinking through the various models that we are seeing emerge for open government data. As with most computer systems you can begin considering a model that should help when it comes to thinking about how we might make raw data available, and then contrast that with the flexibility of what we can do with that data.

For the sake of simplicity I looked at two different examples, one from the United Nations, and a second based upon a proposal that has recently been put forwards by Tim Berners-Lee for the United Kingdom. While both are extremely valuable implementations, it is clear that they present very different options for government.

The United Nations data can be found at http://data.un.org, and at time of writing represents twenty two data sets and over sixty million records. The data can be downloaded as a series of CSV files that can be quickly read into a spreadsheet, database or other tool of the users choosing. Making data available in this form would take less time an effort by the agencies involved, but would obviously still make it possible to build a wide variety of applications using the information available.

The proposal put forward by Dr. Berners-Lee is a little more complex in nature, suggesting that government data should be mapped through RDF and semantically tagged as it is published. In the long run this would represent significant value as individuals and organizations started work to understand and analyze the data but at the same time would represent a much larger project across government to enable the data to be published in the first place.

The overall model might help us evaluate options for implementation plans, end user tools, timelines and milestones and the value that the project would deliver.

In essence it might look a little like the graphic below, of course it would be a lot more useful with a significantly larger set of projects and case studies mapped against it;

As with the early stages of eGovernment I suspect we will see governments take a more pragmatic approach in the short term, implementing systems that fall on the left hand side of this graph, while holding aspirations and a vision that firmly maps to the right hand side.