A quick update on my post from last week on the “Microsoft Shuffle“, where I looked at how Microsoft’s “random” browser ballot was far from random.

First, I’d like to thanks those who commented on that post, or sent me notes, offering additional analysis. I think we nailed this one. Within a few days of my report Microsoft updated their Javascript on the browserchoice.eu website, fixing the error. But more on that in a minute.

Some random observations

Several commenters mentioned that if you search Google for “javascript random array sort” the first link returned will be a Javascript tutorial that has the same offending code as Microsoft’s algorithm. This is not surprising. As I said in my original post, this is a well-known mistake. But it is no less a mistake. If you use Google Code Search for the query “0.5 – Math.random()” lang:javascript you will find 50 or so other instances of the faulty algorithm. So if anyone else is using this same algorithm, they should evaluate whether it is really sufficiently random for their needs. In some case, such as a children’s game, it might be fine. But know that there are better and faster algorithms available that are not much more complicated to code.

Another thing to note is that the Microsoft Shuffle algorithm is bad enough with 5-elements in the array, but the non-randomness gets more pronounced as you increase the length of the array. Regardless of the size of the array, it appears that on Internet Explorer the 1st element will end up in last place 50% of the time. There are other pronounced patterns as well. You can see this yourself this this test file, which allows you to specify the size of the array as well as the number of iterations. Try a 50-element array for 10,000 iterations to get a good sense of how non-random the results can be.

I used that script to run a large test of 1,000,000 iterations of a 1024-element array. The raw results are here. I took that table, and using R’s image() function produced a rendering of that matrix. You can see here the clear over-representation at some positions, including (in the lower left) the flip of the first position to last place. (I’m not quite satisfied with this rendering. Maybe someone can get a better-looking visualization of this same data.)

Evaluating Microsoft’s new shuffle

Sometime last week — I don’t know the exact date — Microsoft updated the code for the browser choice website with a new random shuffle algorithm. You see see the code, in situ, here. The core of it is in this function:

function ArrayShuffle(a)
{
    var d, c, b=a.length;
    while(b)
    {
        c=Math.floor(Math.random()*b);
        d=a[--b];
        a[b]=a[c];
        a[c]=d
    }
}

This looks fine to me. I created a new test driver for this routine, which you can try out here. Aside from being much faster, it is gives much better results. Here is a run with a million iterations:

Raw counts

Fraction of total

And the results of the Chi-square test:

X-squared = 18.9593, df = 16, p-value = 0.2708

Final thoughts

In the end I don’t think it is reasonable to expect every programmer to be memorize the Fisher-Yates algorithm. These things belong in our standard libraries. But what I would expect every programmer to know is:

1. That the problem here is one that requires a “random shuffle”. If you don’t know what it is called, then it will be difficult to lookup the known approaches. So this is partially a vocabulary problem. We, as programmers, have a shared vocabulary which we use to describe data structures and algorithms; binary searches, priority heaps, tries, and dozens of other concepts. I don’t blame anyone for not memorizing algorithms, but I would expect a programmer to know what types of algorithms apply to their work.
2. How to research which algorithm to use in a specific context, including where to find reliable information, how to evaluate the classic trade-offs of time and space, etc.  There is almost always more than one way to solve a problem.
3. That where randomized outputs are needed,  the outputs should be statistically tested. I would not expect the average programmer to know how to do a chi-square test, or even to know what one is. But I would expect a mature programmer to know either find this out or seek help.

Evidently today is National Grammar Day.  I am not a fan.

Like most Americans of my generation I was taught to identify parts of speech, diagram sentences and intone with the rest of the class the mysteries of the three-and-twenty most holy helping verbs: “is, am, are, was, were, be, being, been, have, has, had, do, did, does, may, must, might, can, could, will, would, shall, should”.   Because I was good at it, and felt a call to the service of pedantry, I continued my novitiate in stranger accents, in German, Latin and Greek.

I was well on my path the the priesthood of a grammarian, when in 1992 I abandoned all my vows in a bus in Somerville, Massachusetts, when a drunk showed me what language was really all about.

I’m not one to start a conversation with a stranger –  even a sober one — on public transportation.  But in this case I had little choice in the matter, since this particular gentleman insisted on initiating a debate on the virtues of the Allman Brothers, a subject which I was neither equipped nor inclined to discuss with him.

When I expressed my disinclination to debate, and further, my ignorance of all things Allman, the dear fellow was offended and let out a string of expletives, starting with “Un-freakin’-believable” (albeit with a more emphatic, saltier interposed participial adjective than I can relate to you here) and continuing for several minutes.  Nothing he said was grammatical.  Little was even coherent.  But what I did understand was pure genius.  I wish I had a tape recorder.  As my stop approached, I hesitated a moment, intending to thank the man, offer him my congratulations and laud him as a poet of the first order.  But the smell, as well as my own instinct for self-preservation, held me in abeyance.

Since that day I have been an apostate to grammar.  I think we should all have a range of ways to speak and write,  and should be able to modulate according to circumstances. Language is like a wardrobe.  A man should have jogging shorts as well as a tuxedo.  In the end, language is not about rules.  It is about suiting the words to the occasion, of putting the right words in the right places, and what is “right’ will depend on circumstances.

So down with grammar, down with the rules! Go, split an infinitive, dangle your participles, and like my good friend on the #86 bus, even separate your inseparable prefixes.  To quote Duke Ellington, “If it sounds good, it is good”.  And remember that the goal in the end is expression and understanding.  If you are understood, then you’ve accomplished more than many.

As Gertrude Stein wrote:

Clarity is of no importance because nobody listens and nobody knows what you mean no matter what you mean, nor how clearly you mean what you mean. But if you have vitality enough of knowing enough of what you mean, somebody and sometime and sometimes a great many will have to realize that you know what you mean and so they will agree that you mean what you know, what you know you mean, which is as near as anybody can come to understanding anyone.

March 6th Update:  Microsoft appears to have updated the www.browserchoice.eu website and corrected the error I describe in thi post.  More details on the fix can be found in The New & Improved Microsoft Shuffle.  However, I think you will still find the following analysis interesting.

-Rob

Introduction

The story first hit in last week on the Slovakian tech site DSL.sk.  Since I am not linguistically equipped to follow the Slovakian tech scene, I didn’t hear about the story until it was brought up in English on TechCrunch.  The gist of these reports is this: DSL.sk did a test of the “ballot” screen at www.browserchoice.eu, used in Microsoft Windows 7 to prompt the user to install a browser.  It was a Microsoft concession to the EU, to provide a randomized ballot screen for users to select a browser.  However, the DSL.sk test suggested that the ordering of the browsers was far from random.

But this wasn’t a simple case of Internet Explorer showing up more in the first position.  The non-randomness was pronounced, but more complicated.  For example, Chrome was more likely to show up in one of the first 3 positions.  And Internet Explorer showed up 50% of the time in the last position.  This isn’t just a minor case of it being slightly random.  Try this test yourself: Load www.browserchoice.eu, in Internet Explorer, and press refresh 20 times.  Count how many times the Internet Explorer choice is on the far right.   Can this be right?

The DLS.sk findings have lead to various theories, made on the likely mistaken theory that this is an intentional non-randomness.  Does Microsoft have secret research showing that the 5th position is actually chosen more often?  Is the Internet Explorer random number generator not random?  There were also comments asserting that the tests proved nothing, and the results were just chance, and others saying that the results are expected to be non-random because computers can only make pseudo-random numbers, not genuinely random numbers.

Maybe there was cogent technical analysis of this issue posted someplace, but if there was, I could not find it.  So I’m providing my own analysis here, a little statistics and a little algorithms 101.  I’ll tell you what went wrong, and how Microsoft can fix it.  In the end it is a rookie mistake in the code, but it is an interesting mistake that we can learn from, so I’ll examine it in some depth.

Are the results random?

The ordering of the browser choices is determined by JavaScript code on the BrowserChoice.eu web site.  You can see the core function in the GenerateBrowserOrder function.  I took that function and supporting functions,  put it into my own HTML file, added some test driver code and ran it for 10,000 iterations on Internet Explorer.  The results are as follows:

This confirms the DSL.sk results.  Chrome appears more often in one of the first 3 positions and I.E. is most likely to be in the 5th position.

You can also see this graphically in a 3D bar chart:

But is this a statistically significant result?  I think most  of us have an intuitive feeling that results are more significant if many tests are run, and if the results also vary much from an even distribution of positions.  On the other hand, we also know that a finite run of even a perfectly random algorithm will not give a perfectly uniform distribution.  It would be quite unusual if every cell in the above table was exactly 2,000.

This is not a question one answers with debate.  To go beyond intuition you need to perform a statistical test.  In this case, a good test is Pearson’s Chi-square test, which tests how well observed results match a specified distribution.  In this test we assume the null-hypothesis that the observed data is taken from a uniform distribution.  The test then tells us the probability that the observed results can be explained by chance.  In other words, what is the probability that the difference between observation and a uniform distribution was just the luck of the draw?  If that probability is very small, say less than 1%, then we can say with high confidence, say 99% confidence, that the positions are not uniformly distributed.   However, if the test returns a larger number, then we cannot disprove our null-hypothesis.  That doesn’t mean the null-hypothesis is true.  It just means we can’t disprove it.  In the end we can never prove the null hypothesis.  We can only try to disprove it.

Note also that having a uniform distribution is not the same as having uniformly distributed random positions.  There are ways of getting a uniform distribution that are not random, for example, by treating the order as a circular buffer and rotating through the list on each invocation.  Whether or not randomization is needed is ultimately dictated by the architectural assumptions of your application.  If you determine the order on a central server and then serve out that order on each innovation, then you can use non-random solutions, like the rotating circular buffer.  But if the ordering is determined independently on each client, for each invocation, then you need some source of randomness on each client to achieve a uniform distribution overall.  But regardless of how you attempt to achieve a uniform distribution the way to test it is the same, using the Chi-square test.

Using the open source statistical package R, I ran the  chisq.test() routine on the above data.  The results are:

X-squared = 13340.23, df = 16, p-value < 2.2e-16

The p-value is much, much less than 1%.  So, we can say with high confidence that the results are not random.

Repeating the same test on Firefox is also non-random, but in a different way:

On Firefox, Internet Explorer is more frequently in one of the first 3 positions, while Safari is most often in last position.  Strange.  The same code, but vastly different results.

The results here are also highly significant:

X-squared = 14831.41, df = 16, p-value < 2.2e-16

So given the above, we know two things:  1) The problem is real.  2) The problem is not related to a flaw only in Internet Explorer.

In the next section we look at the algorithm and show what the real problem is, and how to fix it.

Random shuffles

The browser choice screen requires what we call a “random shuffle”.  You start with an array of values and return those same values, but in a randomized order. This computational problem has been known since the earliest days of computing.  There are 4 well-known approaches: 2 good solutions, 1 acceptable (“good enough”) solution that is slower than necessary, and 1 bad approach that doesn’t really work.  Microsoft appears to have picked the bad approach. But I do not believe there is some nefarious intent to this bug.  It is more in the nature of a “naive” algorithm”, like the bubble sort, that inexperienced programmers  inevitably will fall upon when solving a given problem.  I bet if we gave this same problem to 100 freshmen computer science majors, at least one of them would make the same mistake.  But with education and experience, one learns about these things.  And one of the things one learns early on is to reach for Knuth.

The Art of Computer Programming, Vol. 2, section 3.4.2 “Random sampling and shuffling” describes two solutions:

1. If the number of items to sort is small, then simply put all possible orderings in a table and select one ordering at random.  In our case, with 5 browsers, the table would need 5! = 120 rows.
2. “Algorithm P” which Knuth attributes to Moses and Oakford (1963), but is now known to have been anticipated by Fisher and Yates (1938) so it is now called the Fisher-Yates Shuffle.

Another solution, one I use when I need a random shuffle in a database or spreadsheet, is to add a new column, fill that column with random numbers and then sort by that column.  This is very easy to implement in those environments. However, sorting is an O(N log N)  operation where the Fisher-Yates algorithm is O(N), so you need to keep that in mind if performance is critical.

Microsoft used none of these well-known solutions in their random solution.  Instead they fell for the well-known trap.  What they did is sort the array, but with a custom-defined comparison function or “comparator”.  JavaScript, like many other programming languages, allows a custom comparator function to be specified.  In the case of JavaScript, this function takes two indexes into the value array and returns a value which is:

• <0 if the value at the first index should be sorted before the value at the second index
• 0 if the values at the first index and the second index are equal, which is to say you are indifferent as to what order they are sorted
• >0 if the value at the first index should be sorted after the value at the second index

This is a very flexible approach, and allows the programmer to handle all sorts of sorting tasks, from making case-insensitive sorts to defining locale-specific collation orders, etc..

In this case Microsoft gave the following comparison function:

function RandomSort (a,b)
{
    return (0.5 - Math.random());
}

Since Math.random() should return a random number chosen uniformly between 0 and 1, the RandomSort() function will return a random value between -0.5 and 0.5.  If you know anything about sorting, you can see the problem here.  Sorting requires a self-consistent definition of ordering. The following assertions must be true if sorting is to make any sense at all:

1. If a<b then b>a
2. If a>b then b<a
3. If a=b then b=a
4. if a<b and b<c then a<c
5. If a>b and b>c then a>c
6. If a=b and b=c then a=c

All of these statements are violated by the Microsoft comparison function.  Since the comparison function returns random results, a sort routine that depends on any of these logical implications would receive inconsistent information regarding the progress of the sort.  Given that, the fact that the results were non-random is hardly surprising.  Depending on the exact search algorithm used, it may just do a few exchanges operations and then prematurely stop.  Or, it could be worse.  It could lead to an infinite loop.

Fixing the Microsoft Shuffle

The simplest approach is to adopt a well-known and respected algorithm like the Fisher-Yates Shuffle, which has been known since 1938.  I tested with that algorithm, using a JavaScript implementation taken from the Fisher-Yates Wikpedia page, with the following results for 10,000 iterations in Internet Explorer:

Applying Pearson’s Chi-square test we see:

X-squared = 21.814, df = 16, p-value = 0.1493

In other words, these results are not significantly different than a truly random distribution of positions.  This is good.  This is what we want to see.

Here it is, in graphical form, to the same scale as the “Microsoft Shuffle” chart earlier:

Summary

The lesson here is that getting randomness on a computer cannot be left to chance.  You cannot just throw Math.random() at a problem and stir the pot, and expect good results.  Random is not the same as being casual.  Getting random results on a deterministic computer is one of the hardest things you can do with a computer and requires deliberate effort, including avoiding known traps.  But it also requires testing.  Where serious money is on the line, such as with online gambling sites, random number generators and shuffling algorithms are audited, tested and subject to inspection.  I suspect that the stakes involved in the browser market are no less significant.  Although I commend DSL.sk for finding this issue in the first place, I am astonished that the bug got as far as it did.  This should have been caught far earlier, by Microsoft, before this ballot screen was ever made public.  And if the EC is not already demanding a periodic audit of the aggregate browser presentation orderings, I think that would be a prudent thing to do.

If anyone is interested, you can take a look at the file I used for running the tests.  You type in an iteration count and press the execute button.  After a (hopefully) short delay you will get a table of results, using the Microsoft Shuffle as well as the Fisher-Yates Shuffle.  With 10,000 iterations you will get results in around 5 seconds.  Since all execution is in the browser, use larger numbers at your own risk.  At some large value you will presumably run out of memory, time out, hang, or otherwise get an unsatisfactory experience.

Over the years, I’ve seen Mercury, Venus, Mars, Jupiter and Saturn with my naked eyes.  And I’ve seen Uranus and Neptune through a telescope.  But I’ve never seen an asteroid until last night, when I photographed the 2nd largest minor planet, Vesta.

Vesta is currently near opposition, meaning as seen from the Earth, Vesta and the Sun are 180 degrees apart. This makes Vesta well-placed for observation all-night long, and also means that it is around as bright as it ever gets.  So if you are eager to see an asteroid, give it a try some clear night in February or March.

The first step is to find the constellation Leo.   Where exactly Leo is from your location will depend on your latitude.  But generally, if you are in mid-northern latitudes, a good start is to look to the south-east around 10pm.  If you are running Linux, give KStars a try to simulate what the sky will look like.  Once you’ve found where Leo is, take a look at this chart showing the location of Vesta [PDF] over the next few months.

Since Vesta is now around magnitude 6.8, it is too faint to see with the naked eye.   But it should be visible with even the smallest telescope or binoculars.  Compare what you see with the chart.  Don’t expect to see any disk or features.  It will look just like a star.  If you want to be sure, make a sketch of the location of the stars you see in the area, and compare it a night or two later.  If it is Vesta, you should see it noticeably move from night to night.

Instead of using binoculars or a telescope, I decided to photograph Vesta.  With some trial and error I found a technique that worked rather well.

1. Obviously, you need a camera that can record the faint stars without so much noise that it overwhelms them.  I don’t think a little compact digital camera will work well here.  It is all about light gathering ability of the lens.  The little lenses in a compact camera will not work well here.  But a DSLR can work well, especially if you have a fast lens.  Use the fastest (lowest f-ratio) that you have.  If the lens is too “soft” at its wide open setting, you can also try stopping it down a stop.
2. The focal length of the lens is not that important.  A big telephoto lens is not going to help here.  You are not going to be able to make Vesta look any bigger than just a pin point of light.  At the other extreme, if you use a wide-angle lens, your image scale might not allow you to resolve individual stars well.  You want something in the “normal” range, not telephoto, not wide angle.
3. You will need to take a longer exposure, several seconds long, so a tripod is a must, to hold the camera steady.  You also want to use a cable release or remote trigger to avoid camera shake.  If you know how to set the mirror-up lock on your camera, that can help as well.
4. You want to take as long of an exposure as you can while avoiding the star images streaking (from the earth’s rotation).  This will depend on the local length of your lens and the resolution of your camera.  Try exposures in the range of 2-30 seconds.
5. You will want to set the ISO speed as high as you can without the noise level getting too high.  What is too high?  Use your judgment.  At some point the noise overwhelms the star images.
6. If you are in a suburban location, you may find that the light pollution causes the image to “fog up”.  If so, reduce the ISO speed or decrease the exposure.

In my case, I used a Pentax K20d at ISO 800 with a 50mm f/1.4 lens and a 4-second exposure.  The results are as above, which I’ve annotated, as well as a the blown up details   When I compared to the chart the extra “star” was obvious.  It doesn’t look like much, but that is Vesta, nearly 600km in diameter, orbiting somewhere between the orbits of Mars and Jupiter.

Good luck!

Related articles by Zemanta

• Get Set for a Possible Glimpse of an Asteroid (spacefellowship.com)
• Vesta interest (blogs.discovermagazine.com)
• Viewing Vesta (space.com)

• Danish Open Source Vendors declares victory in open standards war

  “The mood at this year’s general meeting was joyous. In late January 2010, OSL could declare victory in maybe the most important and hard fought battle that OSL has been part of since its formation.

  On 29 January 2010 the Danish Parliament (Folketinget) decided unanimously to place the ODF standard as the only compliant open standard for editable documents on the list of open standards to be used by the Danish public sector.”

  tags: ODF

• Brent R Brian: Questions for Microsoft

  “Now, what about all the customers that chose to use Microsoft’s Office product just to get the OOXML feature? If they suffer loss because of this Microsoft compensate them? What about the documents they created with OOXML, will they have to be converted? What about the people that refuse to let Microsoft remove the patent violating code to protect their investments and documents? Could i4i sue them?”

  tags: OOXML

• Use the tools that make the job easier | Apps Meet Ops – CNET News

  “Because we have ever less time to build apps and services, and because the human resources needed to get the job done are now the gating factor (not system resources), there aren’t too many good counter-arguments to the idea of up-leveling our data and tasks, then using the most powerful tools we can find. “

  tags: ODF

• odtPHP – OpenOffice documents generation with PHP

  “OdtPHP is an oriented object libary for PHP 5+. It allows you to generate automatically OpenOffice text documents from templates. You can use it directly within your PHP scripts (without OpenOffice).”

  tags: ODF

• File formats, alphabets and public money: did you know that… | Stop!

  “File formats are the rules that define the meaning of all the sequences of bits that you can find inside a computer file. If the format of a file isn’t really open, that is completely known and useable by everybody without paying fees or asking some permit, that file will surely readable without errors only with one or very few software programs and only until those programs exist. Therefore, their authors will be able to ask whatever price they want for every new version of their program(s), especially if citizens are trained in public schools to know (only) those same programs. “

  tags: ODF

• Gnumeric 1.10 Release Brings Better Tools

  “It’s been just over two years since the last stable release, but the Gnumeric team is still going strong. The project has a new stable series 1.10.x. This release removes the 65,536 row restriction on spreadsheets and includes many new functions, better OpenDocument Format (ODF) support, new statistical analysis tools, and a new utility for searching spreadsheet files.”

  tags: ODF

• maximkulkin’s spreadsheet at master – GitHub

  ODF with Ruby?

  tags: ODF

Posted from Diigo. The rest of my favorite links are here.

Summary

In this post I take a look at Microsoft’s claims for robust data recovery with their Office Open XML (OOXML) file format.  I show the results of an experiment, where I introduce random errors into documents and observe whether word processors can recover from these errors.  Based on these result, I estimate data recovery rates for Word 2003 binary, OOXML and ODF documents, as loaded in Word 2007, Word 2003 and in OpenOffice.org Writer 3.2.

My tests suggest that the OOXML format is less robust than the Word binary or ODF formats, with no observed basis for the contrary Microsoft claims.  I then discuss the reasons why this might be expected.

The OOXML “data recovery” claims

I’m sure you’ve heard the claim stated, in one form or another, over the past few years.  The claim is that OOXML files are more robust and recoverable than Office 2003 binary files.  For example, the Ecma Office Open XML File Formats overview says:

Smaller file sizes and improved recovery of corrupted documents enable Microsoft Office users to operate efficiently and confidently and reduces the risk of lost information.

Jean Paoli says essentially the same thing:

By taking advantage of XML, people and organizations will benefit from enhanced data recovery capabilities, greater security, and smaller file size because of the use of ZIP compression.

And we see similar claims in Micrsoft case studies:

The Office Open XML file format can help improve file and data management, data recovery, and interoperability with line-of-business systems by storing important metadata within the document.

A Microsoft press release quotes Senior Vice President Steven Sinofsky:

The new formats improve file and data management, data recovery, and interoperability with line-of-business systems beyond what’s possible with Office 2003 binary files.

Those are just four examples of a claim that has been repeated dozens of time.

There are many kinds of document errors.  Some errors are introduced by logic defects in the authoring application.  Some are introduced by other, non-editor applications that might modify the document after it was authored.  And some are caused failures in data transmission and storage.  The Sinofsky press release gives some further detail into exactly what kinds of errors are more easily recoverable in the OOXML format:

With more and more documents traveling through e-mail attachments or removable storage, the chance of a network or storage failure increases the possibility of a document becoming corrupt. So it’s important that the new file formats also will improve data recovery–and since data is the lifeblood of most businesses, better data recovery has the potential to save companies tremendous amounts of money.

So clearly we’re talking here about network and storage failures, and not application logic errors.  Good, this is a testable proposition then.  We first need to model the effect of these errors on documents.

Modeling document errors

Let’s model “network and storage failures” so we can then test how OOXML files behave when subjected to these types of errors.

With modern error-checking file transfer protocols, the days of transmission data errors are a memory.  Maybe  25 years ago, with XMODEM and other transfer mechanisms, you would see randomly-introduced transmission errors in the body of a document.  But today the more likely problem would be that of truncation, of missing the last few bytes of a file transfer.  This could happen for a variety of reasons, including logic errors in application-hosted file transfer support , to user-induced errors from removing a USB memory stick with uncommitted data still in the file buffer.  (I remember debugging a program once that had a bug where it would lose the last byte of a file whenever the file was an exactt multiple of 1024 bytes.)  These types of error can be particularly pernicious with some file formats.  For example, the old Lotus WordPro file format stored the table of contents for the document container at the end of the file.  This was great for incremental updating, but particularly bad for truncation errors.

For this experiment I modeled truncation errors by generating a series of copies of a reference document, each copy truncating an additional byte from the end of the document.

The other class of errors — “storage errors” as Sinofsky calls them — can come from a variety of hardware-level failures, including degeneration of the physical storage medum or mechanical errors in the storage device.  The unit of physical storage — and thus of physical damage — is the sector.  For most storage media the size of a sector is 512 bytes.  I modeled storage errors by creating a series of copies of a reference document, and for each one selecting a random location within that document and then introducing a 512-byte run of random bytes.

The reference document I used for these tests was Microsoft’s whitepaper, The Microsoft Office Open XML Formats.  This is a 16-page document, with title page with logo, a table of contents, a running text footer, and a text box.

Test Execution

I tested Microsoft Word 2003, Word 2007 and OpenOffice.org 3.2.   I attempted to load each test document into each editor.  Since corrupt documents have the potential to introduce application instability, I exited the editor between each test.

Each test outcome was recorded as one of:

• Silent Recovery:  The application gave no error or warning message.  The document loaded, with partial localized corruption, but most of the data was recoverable.
• Prompted Recovery: The application gave an error or warning message offering to recover the data.  The document loaded, with partial localized corruption, but most of the data was recoverable.
• Recovery Failed: The application gave an error or warning message offering to recover the data, but no data was able to be recovered.
• Failure to load: The application gave an error message and refused to load the document, or crashed or hanged attempting to load it.

The first two outcomes were scored as successes, and the last two were scored as failures.

Results: Simulated File Truncation

In this series of tests I took each reference document (in DOC, DOCX and ODT formats) and created 32 truncated files corresponding to 1-32 bytes truncation.  The results were the same regardless of the number of bytes truncated, as in the following table:

Results: Simulated Sector Damage:

In these tests I created 30 copies of each reference document and introduced a random 512-byte run of random bytes,  with the following summary results:

Discussion

First, what do the results say about Microsoft’s claim that the OOXML format “improves…data recovery…beyond what’s possible with Office 2003 binary files”?  A look at the above two tables brings this claim into question.  With truncation errors, all three word processors scored 100% recovery using the legacy binary DOC format.  With OOXML the same result was achieved only with Office 2007.  But both Office 2003 and OpenOffice 3.2 failed to open any of the truncated documents.  With the simulated sector-level errors, all three tested applications did far better recovering data from legacy DOC binary files than from OOXML files.  For example, Microsoft Word 2007 recovered 83% of the DOC files but only 47% of the OOXML files.  OpenOffice 3.2 recovered 90% of the DOC files, but only 37% of the OOXML files.

In no case, of almost 200 tested documents, did we see the data recover of OOXML files exceed that of the legacy binary formats.  This makes sense, if you consider this from an information theoretic perspective.  The ZIP compression in OOXML, while it compresses the document at the same time makes the byte stream denser in terms of the information encoding.  The number of physical bits per information bits is smaller in the ZIP than in the uncompressed DOC file. (In the limit of perfect compression, this ratio would be 1-to-1.)  Because of this, a physical error of 1-bit introduces more than 1-bit of error in the information content of the document.  In other words, a compressed document, all else being equal, will be less robust, not more robust to “network and storage failures”.  Because of this it is extraordinary that Microsoft so frequently claims that OOXML is both smaller and more robust than the binary formats, without providing details of how they managed to optimize these two opposing and complementary qualities.

Although no similar claims have been made regarding ODF documents, I tested them as well.  Since ODF documents are compressed by ZIP, we would expect them to also be less robust to physical errors than DOC, for the same reasons discussed above.  This was confirmed in the tests.  However, ODF documents exhibited a higher recovery rate than OOXML.  Both OpenOffice 3.2 (60% versus 37%) as well as Word 2007 (60% versus 47%) had higher recovery rates for ODF documents.  If all else had been equal, we would have expected ODF documents to have lower recover rates than OOXML.  Why?  Because the ODF documents were on average 18% smaller than the corresponding OOXML documents, so the fixed 512-byte sector errors were proportionately larger impact in ODF documents.

The above is explainable if we consider the general problem of random errors in markup.  There are two opposing tendencies here.  On the one hand, the greater the ratio of character data to markup, the more likely it will be that any introduced error will be benign to the integrity of the document, since it will most likely occur within a block of text.  At the extreme, a plain text file, with no markup whatsoever, can handle any degree of error introduction with only proportionate data corruption.  However, one can also argue in the other direction, that the more encoded structure there is in the document, the easier it is to surgically remove only the damaged parts of the file.  However, we must acknowledge that physical errors, the “network and storage failures” that we looked at in these tests, do not respect document structure.  Certainly the results of these tests call into question the wisdom of claiming that the complexity of the document model leads it to be more robust.  When things go wrong, simplicity often wins.

Finally, I should observe that application difference, as well as file format differences, play a role in determining success in recovering damaged files.  With DOC files, OpenOffice.org 3.2 was able to read more files than either version of Microsoft Word.  This confirms some of the anecdotes I’ve heard that OpenOffice will read files that Word will not.  With OOXML files, however, Word 2007 did best, though OpenOffice fared better than Word 2003.  With ODF files, both Word and OpenOffice scored the same.

Further work

Obviously the field of document file robustness is a complex question.  These tests strongly motivate the thought that there are real differences in how robust document formats are with respect to corruption, and these observed differences appear to contradict claims made in Microsoft’s OOXML promotional materials.  It would require more tests to demonstrate the significance and magnitude of those differences.

With more test cases, one could also determine exactly which portions of a file are the most vulnerable.  For example, one could make a heat map visualization to illustrate this.  Are there any particular areas of a document where even a 1-byte error can cause total failures?  It appears that a single-byte truncation error on OOXML documents will cause a total failure in Office 2003, but not in Office 2007.  Are there any 1-byte errors that cause failure in both editors?

We also need to remember that neither OOXML nor ODF are pure XML formats.  Both formats involve a ZIP container file with multiple XML files and associated resources inside.  So document corruption may consist of damage to the directory or compression structures of the ZIP container as well as errors introduced into the contained XML and other resources.    The directory of the ZIP’s contents is stored at the end of the file.  So the truncation errors are damaging the directory.  However, this information is redundant, since each undamaged ZIP entry can be recovered in a sequential processing of the archive.  So I would expect a near perfect recovery rate for the modest truncations exercised in these tests.  But with OOXML files in Office 2003 and OpenOffice 3.2, even a truncation of a single byte prevented the document from loaded.  This should be relatively easy to fix.

Also, the large number of tests with the “Silently Recover” outcome is a concern.  Although the problem in general is solved with digital signatures, there should be some lightweight way, perhaps checking CRC’s at the ZIP entry level, to detect and warn users when a file has been damaged.  If this is not done, the user could inadvertently work and resave the damaged work or otherwise propagate the errors, when an early warning of the error would potentially give the user the opportunity, for example, to download the file again, or seek another, hopefully, undamaged copy of the document.  But by silently recovering and loading the file, the user is not made aware of their risky situation.

Files and detailed results

If you are interested in repeating or extending these tests, here are the test files (including reference files) in DOC, DOCX and ODT formats.  You can also download a ZIP of the Java source code I used to introduce the document errors.  And you can also download the ODF spreadsheet containing the detailed results.

WARNING: The above ZIP files contain corrupted documents.  Loading them could potentially cause system instability and crash your word processor or operating system (if you are running Windows).  You probably don’t want to be playing with them at the same time you are editing other critical documents.

Updates

2010-02-15: I did an additional 100 tests of DOC and DOCX in Office 2007.  Combined with the previous 30, this gives the DOC files a recovery rate of 92% compared to only 45% for DOCX.  With that we have significant results at 99% confidence level.

Given that, can anyone see a basis for Microsoft’s claims?  Or is this more subtle?  Maybe they really meant to say that it is easier to recover from errors in an OOXML file, while ignoring the more significant fact that it is also far easier to corrupt an OOXML file.  If so, the greater susceptibility to corruption seems to have outpaced any purported enhanced ability of Office 2007 to recover from these errors.

It is like a car with bad brakes claiming that is has better airbags.  No thanks.  I’ll pass.

• Q&A: IBM’s Alistair Rennie on the big picture for Lotus | Between the Lines | ZDNet.com

  “Ultimately, new types of documents are possible. I don’t see my kids creating content 5 to 10 years from now going into a dumb text editor and doing a presentation. I see them using rich media and aggregating small bits of data from the Web. The office suite box is constraining. People should interchange content over time, use collaborative Web-based editing and components that can be shared. Symphony is fundamental to changing and enabling open standards.”

  tags: ODF, Symphony

• ODF Import | drupal.org

  “ODF Import allows a user to import ODF files into drupal nodes. Currently the module can import content from ODT files only. No style information is imported in current release.
  Future releases will support other ODF formats as well as importing of styles from an ODF document.”

  tags: ODF

• WP-United User Manual: Help Needed!

  “You can download the manual from SVN, here. It is in ODF, and it would be great if it could stay that way — if you are using software by The Man, you can download an ODF add-in to open and save the format. (For downloads, I will convert it to PDF, and probably many other formats. The key is keeping the source in ODF).”

  tags: ODF

• Digital Investigation : Data concealment and detection in Microsoft Office 2007 files

  “As more offenders attempt to conceal incriminating data or stolen information, it is important for forensic examiners and computer security professionals to know where to look for concealed information. This paper demonstrates how data concealment in Microsoft Office 2007 files is possible. The Office Open XML (OOXML) format forms the basis of Microsoft Office 2007, and an individual can use OOXML to define customized parts, relationships, or both within a Microsoft Office 2007 file to store and conceal information”

  tags: OOXML

• OpenOffice.org 3.2: 10 Years in the Making

  “File format support gets a boost in 3.2, including better compliance with the Open Document Format (ODF). One of the sad facts about ODF support, at least at the moment, is that multiple suites support ODF but do it differently — thus making the interoperable format rather less than interoperable. OpenOffice.org supports a “superset” of ODF 1.2, and the latest release will now warn users if they are using Extended features that may not be supported by other suites. “

  tags: ODF, OpenOffice

• Jochen Friedrich’s Open Blog: Relevant link of today: OOXML not suitable for Norwegian government

  “A study published by the Norwegian “Direktoratet for forvaltning og IKT” (Agency for public adminstration and ICT) comes to the result that OOXML is not suitable for being used by the Norwegian government. The study is available online in Norwegian. Amongst the reasons that are given are that the standard with its more than 6000 pages is not appropriate; it is not suitable for collaboration; and there is only one software that can implement it and produce the respective file format. Norway recommends pdf for electronic document exchange of documents that don’t need to be edited and ODF (open document format) for all other documents.”

  tags: ODF

• http://odp-view.googlecode.com/svn/trunk/src/test1.html

  “Web based Open Office Presentation Viewer -v0.0000 – A effort to make a pure Javascript ODP Viewer”

  tags: ODF

• monkeyiq: KOffice & RDF: Who, What, When, Where?

  The first of a series of posts on what KOffice is doing to support the new metadata framework in ODF 1.2. Some good work shown here in the videos.

  tags: ODF, RDF

• Advogato: Proprietary File Formats conflict with Equal Opportunities

  tags:  standards

• Bob Sutor: What would ODF support for WordPress look like?

  tags: ODF

• Lotus Solutions Development Lab: Lab 04: ODFDOM: Generating ODF Documents from a Notes Agent

  tags: ODF, Notes

• Gwennel – A WYSIWYG and WYSIWYM editor

  “Gwennel is a free WYSIWYG and WYSIWYM editor for Windows supporting natively the Open Document Format.”

  tags: ODF

• OSOR.EU: Danish state administrations to use ODF

  “The open standard ODF is recognised by many European member states. Next to Denmark it is also a national standard for public administrations in Belgium, Germany, France, Lithuania, Sweden and the Netherlands. ODF is recommended by Norway and it is one of the document standards at NATO.”

  tags: ODF

• Copenhagen Post: State dumps Microsoft

  “After years of deliberation, parliament has voted to stop using Microsoft’s Open Suite file format and switch to ‘Open Document Format’”

  tags: ODF

• ZDNet: IBM launches Lotus Symphony 3 beta; Office alternatives pile up

  “Jeannette Barlow, Symphony’s product manager, called the latest release of IBM’s free office suite ‘the most significant investment and enhancement from that original delivery.’”

  tags: Symphony

• Denmark adopts ODF and PDF/A – ZDNet.co.uk

  tags: ODF

• Integrating OpenDocument Format with Microsoft Office Document Formats

  “In this guide, you’ll learn how to implement and integrate the OpenDocument Format into your environment where other file formats are also in use. Topics include advantages of ODF, best practices for common scenarios, developing an action plan for ODF adoption, document management systems, and collaborating with Microsoft Office users.”

  tags: ODF

• Doc or Docx?

  “For those using older versions of Microsoft Word, or other non-Microsoft word processing software, the new .docx format can be a real pain. It has caused dissension in some workplaces. “

  tags: OOXML

• lpOD: Git repository access available!

  lpOD is a powerful new python API for reading, writing or manipulating ODF documents.

  tags: ODF

Posted from Diigo. The rest of my favorite links are here.

Salt and Fresh-ground Pepper (photo by author)

A major milestone was reached for the OASIS ODF TC today.  The latest Committee Draft of ODF 1.2 Part 1 was sent out for a 60-day public review.

“What does this mean, and why should I care?” you might be asking.  That’s a fair question.

First, a quick review of the OASIS standards approval process.  The stages look like this:

1. TC creates one or more Working Drafts
2. A Working Draft may then additionally be approved as a Committee Draft
3. A Committee Draft may then be additionally approved as a Public Review Draft
4. After addressing received public comments, a Committee Draft may be approved as a Committee Specification
5. Finally, a Committee Specification may be voted on by OASIS as an OASIS Standard

There is the possibility of iteration at most of these stages.  So we’re not done with ODF 1.2.   There is still work to be done, but we are certainly in the endgame now.

Also, it is important to remember that ODF 1.2 has been factored into three “parts”:

• Part 1 specifies the core schema
• Part 2 is OpenFormula (spreadsheet formulas)
• Part 3 defines the packaging model of ODF, and went out for public review back in November

Part 1 is by far the largest of the 3 parts, at 838 pages.  Here is a high-level view of what is covered:

1. Introduction
2. Scope
3. Document Structure
4. Metadata
5. Text Content
6. Paragraph Elements Content
7. Text Fields
8. Text Indexes
9. Tables
10. Graphic Content
11. Chart Content
12. Database Front-end Document Content
13. Form Content
14. Common Content
15. SMIL Animations
16. Styles
17. Formatting Elements
18. Datatypes
19. General Attributes
20. Formatting Attributes
21. Document Processing
22. Conformance
23. Appendix A.  OpenDocument Relax NG Schema
24. Appendix B.  OpenDocument Metadata Manifest Ontology
25. Appendix C.  MIME Types and File Name Extensions (Non Normative)
26. Appendix E.  Recommend Usage of SMIL
27. Appendix G.  Acknowledgments (Non Normative)

If any of this interests you I’d encourage you to take a look at the draft and submit comments per the process defined in the public review announcement.  I expect few will review the entire specification, but even if you can review only a chapter of particular interest to you, or even do a random page review, that will help.  We’re looking for any reasonable feedback, from typographical errors, to ambiguities to new feature proposals.  It is all good.

You can follow the incoming comments via the TC’s comment list, or unofficially via the ODFJIRA Twitter feed.

Now, I know that a vigorous public review, with many reviewers and many comments, is seen in some quarters as inconvenient and troublesome.  It is thought better (in those circles) for standards to sail by, unread, unchallenged and unimplemented.  I do not subscribe to that view.  I ask you to not be gentle on the ODF 1.2 public review draft.  Send us a lot of comments, so we know where we need to improve.  Send us a lot of defect reports, so we know what to fix.  Send us a lot of feature proposals, so we know what to do next.  Short of joining the ODF TC directly, this is the best opportunity to give us feedback.

Sunset at First Encounter Beach, Eastham, Massachusetts (photo by Rob Weir)

As my readers have no doubt heard by now, today the EC cleared Oracle’s proposed acquisition of Sun Microsystems.  This will undoubtedly have a significant impact on all Sun employees,  many of whom I have worked with toward common purposes, on standards or open source projects, and whom I am proud to call my colleagues.  I wish them all best of luck.

I know a little of what they will be going through in the weeks and months ahead, having worked for Lotus Development Corporation when IBM acquired it in 1995.  So here is my unsolicited advice.  Some day they will come.  It may be in weeks, maybe months from now, maybe early in the morning, maybe after hours or on the weekend.  But that day will come.  They will come and strip the Sun logo from the wall in the lobby and replace it with the Oracle logo. Watch for that day, that narrow window of opportunity.  Save the logo.  It is your trophy, your icon, your totem.  You will always be Sun.  Don’t let them take it away.

Captain Galaxy and Commander Glarcon are locked in mortal combat.   Each mans a battle tank armed with N photonic missiles which move at the speed of light.   They move toward each other at constant velocity=v on a 1-dimensional track, unable to stop or reverse direction.  Assume v << c.  The probability of scoring a kill with a missile is described by a function f(d) which monotonically increases from 0 to 1 as the distance between the tanks decreases from infinity to 0.  If the distance closes to 0 and no missiles are fired, both tanks are destroyed in the collision.    Assume each combatant attempts to maximize their own probability of survival.

Note that this is not strictly a zero-sum game, since it is possible for neither player to survive.  But it is impossible for both to survive.

The state of the game is thus described by three variables:

• d=distance between the players
• N1= number of own missiles remaining
• N2= number of opponent’s missiles remaining

A strategy S(d,N1,N2) would describe a combatant’ actions (shoot or don’t shoot) for all possible states.

1. If each player has exactly one missile what is the optimal strategy?  Clearly, if the first player shoots and misses, the 2nd will win by waiting for d to approach 0 and then make a last minute shot.
2. What if each player has exactly two missiles?
3. What if each player has N missiles?

It may simplify the problem to assume f(d) is proportionate to 1/d or 1/d^2 and then solve the general case.

The ground is blanketed in snow, and will remain so for most of the next 3 months. The temperature ranges from cold to frigid. It is hard to think of spring, but now is the time when the garden planning begins, when I start making lists, drawing plot diagrams, calculating planting times, and anticipating, in my mind, the eventual blooms of summer and the harvest of next fall.

This is also the time when my mailbox fills up with catalogs from plantsmen and nurseries, offering new cultivars, rediscovered heirloom varieties and all manner of new garden gadgets.  When possible I prefer to support local nurseries.  Their plants are chosen to be well-adapted for my regional conditions.  Of course, the downside is that if you buy locally, then you’ll be buying the same plants as your neighbors, and who wants to have their landscaping look the same as everyone else on the street?

Of the dozens of catalogs I receive every winter, there are a few that I return to year after year, the ones where I find the immense variety I crave,  as well as high quality service.  The following is my “top 10 list” of seed and plant catalogs suitable for New England:

1. Fedco Seeds (Maine)   Their new 137 page catalog covers seeds, tubers and organic growing supplies.  The emphasis is on hardy and short-season varieties suitable for northern gardens.
2. Nourse Farms (Massachusetts) Strawberries, raspberries, blackberries, currants, gooseberries live plants, including new introductions.  They are my first choice for berries.
3. Territorial Seed Company (Oregon)  Very informative catalog, 167 pages, should be a model for others.  Clearly indicates which seeds are open pollinated, which are F1 hybrids, which are organic, disease resistance, days to maturity, etc.   Most seeds are available in different packet sizes, ranging from small samplers to larger packets for small commercial growers.
4. Artistic Gardens / Le Jardin du Gourmet (Vermont)   Their 50 Sample Herb Packet Special is a regular annual purchase for me.  At any other seed company you pay $3 or so for a packet containing enough basil seeds for a pesto factory.  Total overkill for a mall garden, and expensive if you want to plant a wide variety of herbs.   But for a very reasonable price Le Jardin du Gourmet offers a bundle of  50 herb seed packets, each very small, but providing “just enough” seeds for my small garden.  You have no control over the variety, so if you absolutely must have Genovese Basil, this is not for you, but in turn for giving up control over the variety you get large selection of culinary and traditional medicinal herbs for a very reasonable price.  I find it also gets me to grow herbs I might not otherwise try, like lovage or horehound.
5. Seed Savers Exchange (Iowa) This is the open source project of the seed world.  Individual amateur gardeners and small farms preserving 12,613 varieties of heirloom, ethnic and unique vegetable varieties, growing from seed, and harvesting and preserving the seed to pass on to the next year.  The resulting huge diversity of seeds are shared with members for the cost of postage. The variety is staggering.  If they don’t have it,  it probably doesn’t exist (or is protected by a plant patent).  A subset of the most popular heirloom varieties is for sale on their website, but to get the full phonebook-thick catalog of member-offered seeds, you need to join.
6. White Flower Farm (Connecticut)  The catalog, fully photographed in luscious detail, is a feast for the eyes.  Strong on perennials (herbaceous and woody) and bulbs.  I use it as much to generate ideas as I do for ordering.
7. Johnny’s Selected Seeds (Maine)  I go to Johnny’s especially for their new vegetable variety introductions.  For example, they introduced “Bright Lights” Swiss chard, one of my favorites.   Their catalog always has something new to look forward to.
8. Raintree Nursery (Washington) If you want something unusual in the fruit department, this is where to go.  From medlars to hardy kiwi to saskatoons, they have it.  Their catalog has also the cultural information you need to be successful, including recommendations on compatible pollinators.
9. Bluestone Perennials (Ohio) Along with the White Flower Farm catalog, this is my main source for perennials.  The PlantFinder search engine on their website is great for finding plants for a special niche, like “Zone 6 hardy, deer resistant groundcovers with blue flowers”.
10. Totally Tomato (Wisconsin)  The name says it all.  If you want a specific tomato variety and no substitute will do, this is the place to go.  For example, their catalog lists 41 different cherry tomato varieties.  This is the “long tail” of the lycopersicum world.

Related articles by Zemanta

• Maria Rodale: It’s Time to Order Seeds (huffingtonpost.com)
• Almost Spring, in my dreams (sutor.com)

The 2009 wall calendar is now tossed in recycling bin, and I look to 2010 with renewed energy and dedication.  But I did want to take once last parting look at 2009, from the perspective of this blog’s server logs.

Top Blog Posts

1. Update on ODF Spreadsheet Interoperability (May 2009)
2. ODF Lies and Whispers (June 2009)
3. A Game of Zendo (July 2006)
4. A follow-up on Excel 2007 SP2’s ODF support (May 2009)
5. The Final OOXML Update: Part I (October 2009)
6. The Formats of Excel 2007 (January 2007)
7. The Final OOXML Update: Part III (October 2009)
8. Taking Control of Your Documents March 2009)
9. The Battle for ODF Interoperability (May 2009)
10. The Chernobyl Design Pattern (October 2006)

Top Browsers

• Firefox: 57.3%
• Internet Explorer: 22.9%
• Safari: 5.2%
• Mozilla 4.7%
• Chrome: 3.8%
• Opera: 3.4%
• Mobile (various browsers): 1.4%
• Konqueror :1.3%

Top Operating Systems

• Windows: 62.1%
• Linux 26.8%
• Mac 9.7%
• Mobile 1.4%

What We Commemorate

Commemorative stamps memorialize recently-deceased presidents, mark important anniversaries, acknowledge national institutions, boast of engineering, scientific and artistic achievements and celebrate victory in war and in peace.  Historically, U.S. stamps have portrayed the country as we like to imagine it is, or was.

If history is written by the victors, then that portion of history represented on our stamps is written by the Citizens Stamp Advisory Committee (CSAC).   As we look at the commemorative issues of 1959, we might ask three questions:

1. What is the ostensible subject of the stamp, the real historical event?
2. What was notable about the subject matter in 1959?  What recommended this for commemoration?  How did the ethos of 1959 color the portrayal of the subject?
3. How is the subject viewed today?  Is it of enduring interest?  For example, was it widely celebrated in 2009?

The Lincoln Sesquicentennial

The sesquicentennial of the birth of Abraham Lincoln in 1959 was celebrated by a redesign of the Lincoln Cent (retiring the “wheat back” and introducing the “Lincoln Memorial” reverse) and with a series of four commemorative stamps.    The first stamp (actually issued in 1958) marked the 100th anniversary of the Lincoln-Douglas debates.  I covered that stamp last year with the other 1958 stamp issues.

The Final Report of the Lincoln Sesquicentennial Commission gives many details of these stamps and the ceremony around their release:

The second stamp in the Lincoln Commemorative series was of 1-cent denomination and the first-day issuance was held at Hodgenville, Ky., on February 12, 1959.  Hodgenville was chosen as the site since it is the nearest post office to Lincoln’s birthplace. A commemorative event had already taken place here when on December 27, 1958, Commission Chairman Cooper, Senator Thruston B. Morton, and Representative Frank Chelf, all of Kentucky, formally presented to Carl Howell, president of the Hodgenville Chamber of Commerce, the Post Office Department’s cancellation die hub, especially prepared for commemorative use at the Hodgenville Post Office during Lincoln Year. The die was inscribed : “Lincoln’s Birthplace, Sesquicentennial, 1809- 1959″

Mr. Howell turned the die over to Postmaster Russell Parker, who began using it on January 1, 1959, to cancel all mail issuing from the Hodgenville Post Office.  At this ceremony, also attended by Mr. George M. Moore, executive assistant to the Postmaster General, announcement was made of the first-day issue of the new 1-cent Lincoln stamp. This second stamp in the commemorative series was arranged vertically, printed in green and measured 0.84 by 1.44 inches in size. It bore the head of Lincoln from the famous portrait by George Peter Alexander Healy. After Lincoln’s election to the Presidency in 1860, he sat for a portrait by this famous American artist. This portrait, known as the “Beardless Lincoln,”  is owned by the Corcoran Gallery of Art in Washington, D.C, and was used for the design of the stamp. Initial quantity issued: 90 million.

If you look at the original Healy portrait you will notice that Ervine Metzl’s stamp design departs just slightly from a literal copy.  The features are the same, but the pose has been adjusted with a slight twist to the neck and tilt to the head.  Compare the angle of the shoulders in the original compared to Metzl’s pose.  Posing, like fashion, changes over time.

The third stamp in the commemorative series was of 3-cent denomination, arranged vertically, the same 0.84 by 1.44 inches in size, and maroon in color. It featured the sculptured head of Lincoln by Gutzon Borglum which was completed in 1906 in marble and is now in the rotunda of the Capitol in Washington, D.C. The first-day sale ceremony was held at the Cooper Union in New York on February 27, 1959, as a prelude to Cooper Union’s own centennial year and marked the 99th anniversary of Lincoln’s address there. It was on February 27, 1860, that Lincoln, speaking in the Great Hall of the Cooper Union, delivered what has come to be known as his famous “right makes might” speech. As he concluded his address, the campaigner said :

‘Let us have faith that right makes might, and in that faith, let us, to the end, dare to do our duty as we understand it.’

As a memento of this first-day-issue ceremony, the printed program carried a picture of Lincoln delivering his Cooper Union Address on the front cover and the first-day cancellation on the inside. The Honorable Robert F. Wagner, mayor of New York City, gave greetings; Postmaster General Summerfield delivered the address; and Dr. Edwin S. Burdell, president of the Cooper Union, responded. Initial quantity of stamps issued was 90 million. There were 1,576,866 stamps sold in New York City on this first day of issue and 437,737 covers canceled.

Compare the stamp design with a photo of the Capitol’s Borglum Lincoln bust.   The name Gutzon Borglum may be more familiar to you as the sculptor of Mount Rushmore.

The fourth and final stamp in the Sesquicentennial series was placed on first-day sale at a special ceremony on the steps of the Lincoln Memorial in Washington, D.C., May 30, 1959. This date also marked the anniversary of the completion of the Lincoln Memorial, the most popular shrine in the United States. The stamp, printed in blue, arranged horizontally and measuring 0.84 by 1.44 inches, featured a drawing by Fritz Busse of the head of Lincoln in the Memorial sculpted by Daniel Chester French.

Cosponsored by the Lincoln Sesquicentennial Commission and the Post Office Department, the stamp dedication ceremony drew an audience of several hundred persons, including diplomats. Cabinet officers, Members of Congress, Lincoln enthusiasts, and scores of visitors to the Nation’s Capital. Following the Presentation of Colors by the Joint Services Color Guard and an invocation by Maj. Gen. Frank A. Tobey, Chief of Chaplains, U.S. Army, the audience heard a stirring eulogy to Mr. Lincoln by John B. Fisher, a member of the Commission’s Executive Committee. Secretary of the Interior Fred A. Seaton- delivered an address of welcome, Deputy Postmaster General Edson O. Sessions gave the dedication address, and Mrs. Katharine McCook Knox, an honorary member of the Commission, presented to the Post Office Department a reproduction of the Healy portrait of Lincoln on which the i-cent stamp in the series is based. L. Rohe Walter, special assistant to the Postmaster General, presided.

Oregon Statehood Centennial

The Oregon Statehood issue, Scott #1124, was designed by illustrator Robert Hallock.  The wagon is unhorsed, signifying that the travelers have arrived at their destination, perhaps having traveled along the famed Oregon Trail.  The star represents statehood.  You can see Mt. Hood on the right.

Ten Years of NATO

Only 10 years?   This seems like an unsubstantial achievement.   Or was this beating expectations?  Keep in mind the world of 1959: Castro in charge in Cuba, the first American dies in Vietnam, the Space Race heating up, and de Gaulle withdrawing France from NATO in 1959, preferring to develop their own force de frappe deterrent.  So I’m not sure what exactly we were celebrating.

The design here is by Stevan Dohanos, a social realist illustrator known for his Saturday Evening Post covers.

Arctic Explorations

This design by George Samerjan marks two events:  the 50th anniversary of Admiral Peary reaching the North Pole, and the 1959 sub-polar transit (Operation Sunshine) of the nuclear submarine “Nautilus”.

(Btw, it is well worth a trip to Groton, Connecticut to visit the Submarine Force Museum.  If you are not claustrophobic you can take a self-guided tour of the Nautilus.)

World Peace through World Trade

There is an IBM connection on this one.  Former IBM president — and a giant of industry and commerce — Thomas J. Watson, Sr., coined this phrase in an address to the International Chamber of Commerce in 1937.  It then became an IBM advertising slogan which was featured for many years on a 30-foot sign on the side of 590 Madison Ave in NYC.

It is interesting that the slogan was pre-war, though the stamp is post-war. Certainly the optimism of world peace in 1937 was not very prescient.  But post-war there was a string of successes including Bretton Woods (1944), GATT (1947) and the Marshall Plan (1948).  For a generation that saw two world wars and world-wide depression, there was a lot to be optimistic about in 1959.

The design by Robert Baker, features a globe and laurel.

Centennial of the Comstock Lode

Silver was discovered in Utah Territory (now Nevada) in 1859 on the slopes of Mt. Davison.   The commemorative stamp, designed by Robert L. Miller and W.K. Schrage is based on an old print which I have not been able to identify.

Of course, this is a romanticized view of the original event, which was soaked in blood and whiskey. No mention is made of the most significant social result of the silver rush, namely the instability introduced into the US currency system, then based on gold and silver, by the rapid increase in the silver supply.  Left uncorrected this would have lead to inflation, which was favored by those in debt.  This became the largest political issue of the day, culminating in the  Coinage Act of 1873 (which de-monetized silver) and the 1896 presidential campaign of the populist William Jennings Bryan and his renowned “Cross of Gold” speech:

If they dare to come out in the open field and defend the gold standard as a good thing, we shall fight them to the uttermost, having behind us the producing masses of the nation and the world. Having behind us the commercial interests and the laboring interests and all the toiling masses, we shall answer their demands for a gold standard by saying to them, you shall not press down upon the brow of labor this crown of thorns. You shall not crucify mankind upon a cross of gold.

Opening of the St. Lawrence Seaway

The St. Lawrence Seaway was opened in 1959, a joint U.S.-Canadian venture that connected the Great Lakes to the sea, an event considered momentous enough to warrant a contemporary stamp issue.  The design of the stamp was also a collaborative venture, by Arnold Copeland, Ervine Metzl, William H. Buckley and Gerald Trottier.  There was a corresponding Canadian stamp featuring the same design which was issued simultaneously.

The 49-Star Flag

Alaska joined the Union as the 49th state on January 3rd, 1959.  The design of the Union (that portion of the flag which has the stars) was updated to include the 49th star, but this design was in use only for a short period of time, since Hawaii became the 50th state later that year.  So the 49-star flag was only officially used from July 4, 1959 to July 3, 1960.  It must have been good business for flag makers that year.

So what happens if a new state is added in the future, say Washington, DC, or Puerto Rico?  USC Title 4, Chapter 1, § 2 covers this:

On the admission of a new State into the Union one star shall be added to the union of the flag; and such addition shall take effect on the fourth day of July then next succeeding such admission.

What about the design?  Where exactly do you put a 51st or 52nd star?  The US Army’s Department of Heraldry has that covered, with contigency designs for 51-56 stars.  Just in case.

Soil Conservation

Now here’s a topic you don’t hear a lot about these days.  Errosion.  Soil conservation.  Contour plowing.  We don’t hear much about it these days, mainly because the movement, lead by the U.S. Department of Agriculture’s  Soil  Conservation Service after the 1930’s “Dust Bowl”, was successful.   By plowing fields parallel to the natural contours of the land, water run-off was reduced, preventing topsoil errosion, improving the efficiency of irrigation and increasing crop yields.

Centennial of Drake’s Oil Well

When you think of oil wells, you typically don’t think of Pennsylvania.  But that is where a lot of the early work happened, including the first US oil refinery (Pittsburgh) and Colonel Edwin L. Drake’s early oil well (some claim it to be the first in the U.S.) in Titusville.  Of course, there were no automobiles in 1859.  Coal was king.  So what kind of “petroleum” industry was there?  Kerosene, mainly for lamps, as a replacement for whale oil.

American Dental Association Centennial

The American Dental Association resulted from a merger of several smaller associations at a convention in Niagra Falls in 1859.  The Association’s 1959 centennial celebrations in New York City made the headlines due to an unplanned intersection with cold war politics.  The Association had booked the Waldorf Astoria for their celebrations, but the day before their event, they received a call from the City asking if they would relinquish that space so the City could hold a luncheon in honor of the visiting Soviet Premier Khrushchev.   The Association refused to give up their space, a stance warmly supported by Vice-President Nixon, who was a featured speaker at their event.  Probably a wise choice.   Remember, this was a time when many thought that fluoridation of water was a communist conspiracy.  If the ADA had yielded publicly before Khrushchev it would have given the lunatic fringe more fodder.

Sometime over the next two weeks I’ll be migrating An Antic Disposition over to WordPress, introducing a new visual theme, and relocating to a new hosting company.  This will allow some additional capabilities that I look forward to enabling down the road.

My plan is to preserve all of the comments during the migration, not to break any incoming links, and to minimize any downtime.  That is the plan.  But minimal downtime is not the same as zero downtime, so don’t be surprised if you see me not here, at least occasionally.

One last thing to check, dear reader, especially if you follow me via my feed.  Around a year ago I wrapped my feed via FeedBurner.  If you have subscribed since then, you should be fine, since that FeedBurner URL will continue to work.  However, if you still subscribe to my old original Blogger feed (http://www.blogger.com/feeds/11236681/posts/full) then you will need to resubscribe with the new URL:

The main feed is: http://feeds.feedburner.com/robweir/antic-atom

I’ve completed the migration from Blogger to WordPress.  It went easier than anticipated.  The posts and comments came over without problems.  I think I was able to preserve almost all of the post permalinks.  I’ll monitor the logs for 404 errors and add 303 redirects to fix any remaining URL mismatches.  However, I have not made any attempt to preserve the URLs for the archive or tag pages.  For the various legacy feeds, I’ve redirected all of them to the FeedBurner feed.

By the time you read this (actually probably by the time I finish writing this post) a ballot approving the Public Review Draft of ODF 1.2, Part 1 will have passed.  Part 1 is the largest of the three parts of ODF 1.2, and reaching a Public Review Draft status is a major accomplishment.  Expect to see an official notice of the start of the public review period over the next few days.

But as we look forward to ODF 1.2, and then beyond to “ODF-Next”, it is worth giving some consideration to what we do with ODF 1.1 and ODF 1.0.

Today, if you surveyed ODF implementations, you would find that the preponderance of them write ODF 1.1 documents by default. Twelve months ago many of them wrote out ODF 1.0 format, and in another 12 months I predict most will be writing out ODF 1.2 format by default.

So what does this mean for ODF the standard?

Every 5 years each ISO standard undergoes what is called “Periodic Review”.  The outcome of this review is to classify the standard as one of: confirmed, revised, stabilized or withdrawn.   If it is confirmed, it means the standard is of continued relevancy and is still undergoing maintenance.  Revised means it is currently undergoing revision and periodic review is not necessary.  Stabilized means it “has ongoing validity and effectiveness but is mature and insofar as can be determined will not require further maintenance of any sort”.  And a standard is withdrawn (the most extreme option) if it has been declared unsafe, has a non-RAND patent asserted against it, or is “no longer in use”.

Some of the nattering nabobs in SC34 (e.g., Alex Brown) are floating the idea that ODF 1.0 should be withdrawn from ISO, claiming it is not implemented and not relevant.  At the recent SC34 meeting in Paris this view was echoed by a Microsoft participant (one of many) in the meeting, who additionally urged that a motion to withdraw ODF 1.0 be brought forward at the Stockholm SC34 Plenary in March.

I think this shows an extraordinarily poor understanding of how documents and document format standards work.  ODF is not a standard for a transient phenomenon, like a network or telephone protocol standard, that is no longer relevant when the last producer of the network protocol is gone from the market and the last signal fades from the wire.  ODF specifies a document format, and documents persist and remain relevant so long as the documents and their owners remain.

Additionally, and especially in public sector use, there are regulatory or statutory requirements for how long documents (records) must be preserved.  Some for 3 years, some for 7, some for 30 years, and some records must be preserved forever.  Just because ODF 1.2 comes along does not make ODF 1.0 retention and public access requirements go away.

Although most major ODF editors now write out new documents in ODF 1.1 format by default, they all are able to read and process ODF 1.0 documents as well.  So they are all “consumers” of ODF 1.0 and conform to the ODF 1.0 standard.  This occurs at the same time they are also conforming ODF 1.1 “producers”.  So it is absolutely false to say that there are no ODF 1.0 implementations today.  There are many, including OpenOffice, Symphony, Google Docs, KOffice, even Microsoft Office.  The are all ODF 1.0 consumers.

We should also consider the needs of new word processors that implement ODF, since there are still a few that do not support ODF yet, like Apple’s iWork.  When they eventually implement ODF they will want to implement write support (“producer” conformance) for the current version of ODF, as well as read support (“consumer” conformance) for earlier versions of ODF.  So to enable competition in this space, and allow for new players, we must preserve access to the relevant legacy standards.  Otherwise we would be perpetuating the type of information exclusion we typically associate with Microsoft, in the decades when they restricted access to their legacy formats.

In any case, it is still puzzling to me why some are pushing for the very unusual and extreme action of withdrawing ODF 1.0 from ISO.  This doesn’t pass the sniff test.  Something is rotten here.  This is an anti-competition, anti-user, anti-adopter and overtly political move, lead by Microsoft employees and Microsoft consultants in the Microsoft-dominated JTC1/SC34.   (I wish I had a pump big enough to drain that swamp.)  Ironically, by questioning the relevancy of ODF 1.0, they will cause many more to question the relevancy of SC34.

At some point, I agree that stabilization may be something to consider in the future.  But for now, ODF does not fit in that category because it is actively undergoing maintenance. SC34 members, including Alex Brown, have submitted defect reports against ODF 1.0, and the OASIS ODF TC is responding to them.  It is quite reasonable to expect that ODF 1.1 and ODF 1.2 will be broadly implemented at the same time as ODF 1.0 continues to undergo corrective maintenance.  That is the nature of document format standards like ODF.  Their relevancy, as perceived by users and adopters of the standard, is determined by the mass of legacy documents in the format, not on whether their current word processor saves in that format by default.

[12/15/09 Doug Mahugh today wrote to the OASIS ODF TC list, apprently concerned that this blog post might be misread as an official statement of the OASIS ODF TC.  I've attempted to dispel such notions in my response on that list.  As I've made abundantly clear on my Who is Rob Weir page, "The postings on this site are my own and don't necessarily represent the positions, strategies or opinions of any of my employers or the organizations I'm associated with".

My practice is simple:   I am not speaking as OASIS ODF TC Co-Chair, unless I am posting ODF TC agendas, minutes or similar official ODF TC notices to the ODF TC's mailing list, or when I explicitly sign my name with the title, "Co-Chair, OASIS ODF TC".]

Image via Wikipedia

There is more OOXML controversy in the news, this time in Denmark. I don’t claim to understand all the nuances of the accusations, since I don’t read Danish, and Google Translates makes it sound at times like a discussion about loaves of rye bread or something, but the gist of it, as I can surmise from this account, is whether Office 2010 will “support the complete ISO-approved version of OOXML”. Microsoft’s spokesperson says it will. Mogens Kühn Pedersen, chair of the Danish Standards Committee, says it will not.

This is the kind of dispute where you can go around in circles with for days and not reach agreement. The problem is they are arguing over words, not facts, and they do not agree perfectly on the meaning of the words. Words like “support” and “complete” and “conform” are used in different ways, with different meanings and intents.

Let’s try to escape the equivocation and instead try to establish the underling facts. I can’t promise that this will clarify the situation any. In fact I suspect we’ll end up even more confused about what exactly Office 2010 actually supports. But replacing a false certainty with an honest uncertainty is progress of a kind. It gives us something we can build on.

First, we need to acknowledge that OOXML entered ISO as one standard, and was transformed, via the BRM and ISO ballot, formally into 4 standards, ISO/IEC 29500 Parts 1, 2, 3 and 4. Within these parts are are several different conformance targets and conformance classes. In particular, these 4 standards encompass two different and incompatible schemas for many of its features: “Strict” and “Transitional”. What was submitted in the Fast Track is essentially the “Transitional” schema. What was created by the BRM was the “Strict” schema. This is where Microsoft made most of its “concessions” in order to turn “No” votes into “Yes” votes. So things like support for spreadsheet dates before the year 1900, the elimination of VML graphics, etc., these are all in the “Strict” schema. All the legacy “DoItLikeWord95″ garbage was in “Transitional” only. Several NBs voted to approve OOXML because the assertion that “Transitional” would not be written in documents produced by future versions of MS Office. The promise was that it was…well…transitional, for moving legacy binary documents into XML. Few people want to support two different document standards (both ODF and OOXML) in the first place. But to require support for two different and incompatible versions of OOXML — that is simply intolerable.

In any case, because of these two conformance classes, anyone who claims that their product supports “OOXML” in an unqualified sense, without stating which conformance target or conformance classes they are supporting, is not stating anything of substance. It is like trying to buy an electrical plug adapter by just saying “I need electricity”. Merely saying “conformance to OOXML” means nothing. You need to state the conformance targets and classes that you support. Remember, the conformance language of OOXML is so loose that even a shell statement of “cat foo.docx > /dev/null” would qualify as a conformant application. I assume that Office 2010 supports at least that.

Of course, the alleged assertion that Office 2010 supports OOXML “completely” is a bit more problematic. What exactly does this mean? Does this mean that Office 2010 supports all conformance classes and targets of all 4 parts of OOXML? Including being a Strict consumer? A Strict producer? That would be a good thing, IMHO, if it were true. But that is not what WG4 was recently told in Seattle, where they were told that Office would not write out Strict documents until Office 16. That would put it out to the middle of the next decade, assuming the typical 3-year Office release schedule.

So I’ll lay out my assertions (with the caveat that Office 2010 is not complete and shipped yet) as:

• Office 2010 will conform to the Transitional consumer and producer classes defined in the OOXML standards. Any bugs that are found in the shipped version of Office 2010 will be “fixed” by retroactively changing the standards to match what Office actually does, as is currently being done by Microsoft-packed SC34/WG4 committee with similar bugs found in Office 2007’s OOXML support.
• Office 2010 will not have conforming support for OOXML Strict producer or consumer classes.
• Office 2010 will write dozens of non-interoperable, proprietary extensions into their OOXML documents, extensions which are not defined by the OOXML standards and which have not been reviewed or standardized by any standards committee and which will not be fully interoperable with other OOXML editors, or even with previous versions of MS Office.

So instead of arguing over the meaning of “support” and “complete” I suggest some alternate questions for Microsoft, to give them the opportunity to clarify exactly what kind of support for OOXML will be coming in Office 2010:

1. Exactly what ISO/IEC 29500:2008 conformance classes and targets will Office 2010 conform to?
2. Is this contingent on first changing the conformance requirements of the published ISO/IEC 29500:2008 standards to match what Office 2010 actually supports? Or is there a commitment to support the published standards as they was approved by JTC1 national bodies? In other words, is Microsoft committed to conform to the standards, or are we back to changing the standards to “conform” to Microsoft?
3. Will Microsoft Office 2010 write out only markup that is fully described in the OOXML standards? Or will it write out proprietary markup extensions that are not fully defined in the standards? In other words, will Office 2010 be “strictly conformant” with the ISO/IEC 29500:2008 standards?

The problem you run into here is that there are really two different OOXML standards: the new and improved OOXML Strict conformance class, the one that was “sold” to ISO NBs, the one that garnered the approval votes, and then the old ugly one, the “haunted” specification, the Transitional conformance class, supported only by Microsoft Office. Anyone considering adopting OOXML should have perfect clarity as to which one they are adopting, especially since these are two very different standards, both formally and logically. Just as it is problematic to speak about OOXML support in a product without stating which conformance classes and targets are supported, it is equally a defect of any adoption policy to be loose in what version of OOXML is being proposed for adoption.

IMHO, if you must state a requirement for OOXML (along with ODF), at least specify it clearly, and state a requirement for “strict conformance” (meaning no extensions) of the Strict conformance classes of ISO/IEC 29500:2008. To do otherwise is to essentially specify a requirement for the use of Microsoft Office and Microsoft Office alone.

A major milestone for ODF 1.2 was reached on Friday. Part 3 of ODF 1.2, which specifies document packaging (how a document’s XML, images and metadata are combined into a single file and are optionally encrypted or signed), went out for a 60-day public review period. This public review period will run through January 12th, 2010. A public review is a necessary OASIS procedure before a Committee Draft can be approved as a Committee Specification and then as an OASIS Standard.

The official announcement of the review has more information, including links to download the public review draft and information on how to submit comments on the draft.

Compared to the packaging specification used in ODF 1.0 and ODF 1.1, the main differences are:

1. We’ve split this material into its own specification, since these packaging conventions are more widely applicable, and in fact have been more widely used than just in ODF. For example, the International Digital Publishing Forum (IDPF), who standardize the increasingly important ePub digital book format, use ODF’s packaging as the base of their Open eBook Publication Structure Container Format (OCF) 1.0 specification.
2. We’ve added digital signature support (chapter 4) based on the W3C’s XML Digital Signature Core, including the ability to use standardized extensions such as XAdES.
3. We now have an RDF-based metadata framework with OWL ontology for the manifest file (chapter 5).
4. We include a more detailed conformance definition has been added, including conformance targets for packages, producers and consumers, as well as a separate conformance class for extended packages.
5. Generally, a redraft of the specification to ISO style guidelines.

This specification is only 34 pages long, so if you’re at all interested please give it a look  between now and January 12th, and send along any comments via the office-comment list. Anything that improves the specification is welcome, from reports of typographical errors, to technical omissions or errors, to suggestions for future features. It is all good.

And if you want to follow along, you can track the incoming comments in several ways:

• Subscribe to the office-comment list mentioned above.
• View the archives of the off-comment list.
• View the public review comments we’re tracking in JIRA. I have a python script that scrapes the office-comment list and enters them into JIRA. This will be more complete than the office-comment list because it will includes additional comments from ODF TC.
• I have another python script that takes each newly entered issue from JIRA and sends it out via Twitter. So you can follow all new ODF issues by subscribing to @ODFJIRA. Depending on your Twitter reader, you might be able to mark some issues as “favorites” and return to them later to see how they have been resolved.  (While you’re at it, you might also follow me, @RCWeir)

Also, keep your eye open for the announcement of a public review for ODF 1.2, Part 1 (ODF Schema) and Part 2 (OpenFormula), which will be ready for review soon.

This is Part III of an 5-part series on the state of OOXML today. Previous to starting this series, I had not posted about OOXML in over a year. Part I showed how Microsoft, despite their promises that control of OOXML would be handed over to an independent, international committee, have instead stuffed the committee that maintains OOXML (JTC1/SC34/WG4) with Microsoft employees. And in Part II I looked at how the final published text of OOXML failed to account for all BRM decisions, and described the steps that ISO was taking to remedy this obvious procedural flaw.

In this Part I’ll look at how Microsoft is using their dominance in SC34 to push through hundreds of changes and additions to OOXML, in a misuse of a procedure intended for correcting drafting errors, to make OOXML “conform” to Microsoft’s monopoly product.

Let’s start by taking a look at the OOXML defect log [PDF] that SC34/WG4 uses to track their large list of errors and omissions discovered in the published standard. This defect report currently amounts to over 800 pages, longer than the entire ODF 1.0 standard. But it is well worth downloading and browsing through.

Some of these changes will be made in Technical Corrigenda while others are proposed for Amendments. What is the difference? SC34/WG4 itself made the distinction clear, in a presentation (N 1187 for those with access) it made to the SC34 Plenary in Prague, where it outlined its practice for deciding which changes would be made in corrigenda versus amendments:

All of the following criteria should be met for the defect to be resolved by Corrigendum:

1) WG 4 agrees that the defect is an unintentional drafting error.
2) WG 4 agrees that the defect can be resolved without the theoretical possibility of breaking existing conformant implementations of the standard.
3) WG 4 agrees that the defect can be resolved without introducing any significant new feature.

Unless all the above criteria are met, the defect should be resolved by Amendment.

These are reasonable criteria and no objections were made when these guidelines were presented to SC34.

A key procedural point is that in ISO/IEC it is the JTC1 NBs who are the consensus body that has the authority to create international standards. All ballots which create or substantial modify standards must be approved by JTC1. This includes DIS ballots, FDIS ballots, FDAM ballots and DTR ballots. So standards, technical reports and amendments are ultimately approved or disapproved by JTC1 NBs. Although subcommittees in JTC1, such as SC34, provide the technical expertise and author and review work, they are not the standardizing authority. The exception that proves the rule is with corrigenda, which are authored and approved entirely at the SC level. However, this small area of autonomy in defect correction comes with carefully delineated bounds. A SC can author, approve and publish corrigenda by itself, but only to make corrections.

So if we look at JTC1 Directives 15.4.2.2, we read (with my emphasis) “A technical corrigendum is issued to correct a technical defect…. Technical corrigenda are not issued for technical additions which shall follow the amendment procedure…”. And in 15.4.1 “technical addition” is defined as: “Alteration or addition to previously agreed technical provisions in an existing IS.”

So amendments, which require approval by JTC1, are used for altering or extending the provisions of a standard, while corrigenda are used to correct errors introduced in drafting or publication. This dichotomy is common in other standards organizations. For example, in OASIS, a technical committee is able to approved and publish “Approved Errata” but these are restricted to changes that do not break conformance of existing implementations. Anything beyond that is considered a substantive change to the standard and requires review approval by the OASIS membership.

Clear enough? In fact, in many cases WG4 appears to follow this important distinction. Some of the proposed changes are simple and benign. For example, some BRM issues were fixed, but in being fixed caused informative example markup in the standard to be incorrect. A quick fix of these items via corrigenda is most welcome.

However, in other cases (in fact most of the cases), the Microsoft-dominated WG4 appears to have overstepped the permissible bounds for corrigenda, and indeed gone far, far beyond what it stated it would be doing in corrigenda. Let’s look at a few examples.

(Sadly, the general public is not given access to the text of the draft corrigenda (the DCOR) but those on the inside can follow along by reading N 1252 in the SC34 document repository.)

Let’s start by looking at items 16, 17, 36, 52, 53 and 133 in DCOR for ISO/IEC 29500 Part 4. These make changes and additions to the WordProcessingML schema. Deletions are noted in red strikethroughs, and additions in blue:

This is not correcting a drafting error. This is not correcting a publishing error. This is a substantial addition to the schema as you can see above.

It is argued, in the defect log, that this change is needed because, without it, ISO/IEC 29500 cannot represent change tracking in mathematical equations. However, this is exactly the type of change that WG4’s guidelines and JTC1 Directives exclude from corrigenda and place into amendments. The schema of OOXML is certainly an “agreed technical provision of an existing IS”. So how can adding math change tracking support to the schema be anything other than an “addition to previously agreed technical provisions”? And how can anyone in WG4 believe that adding dozens of lines to the schema can be done “without the theoretical possibility of breaking existing conformant implementations of the standard”? What about, for example, applications that were programmed to use the published OOXML schema, such as any application that uses a validating parser, or an schema-directed editor, or a program that generates code stubs from the schema, or does XML-to-relational DB mapping? Not only is there a theoretical possibility of breaking such applications, there is a theoretical certainty.

(Ironically, it should be noted that Microsoft was very keen to beat up on ODF for not having change tracking for mathematical equations, all while hiding the fact that OOXML lacked complete support for this feature as well.)

Another example, #122 in the DCOR.

It changes a type in chart, from a byte to an int and in doing so extends its allowed range considerably. How did anyone think that this was a change that was “without the theoretical possibility of breaking existing conformant implementations of the standard”? Isn’t there enough theoretical and practical expertise in WG4 to know that changes like this break compatibility?

For this change the rationale in the defect log explains the logic of it:

The standard states that the ST_Period simple type uses the XML Schema ST_Period data type and supports a range 2–255.

These observations are incompatible with existing documents and should be updated to reflect such prior art.

And so on and so on. If you search through the defect log, you will see the phrase “existing documents” used dozens of times. That appears to be how many discussions in WG4 end. It shuts down debate like an appeal to “national security” or “executive privilege”, arguments that trumps all others. It doesn’t matter what WG4 previously told SC34, or what JTC1 Directives say, if ISO/IEC 29500 does not match what Microsoft Office actually writes out, then this is by definition a drafting error, and the standard will be “corrected” to conform with MS Office. Let that sink in for a little, until you realize how backwards this is.

I invite you to go back to the defect log [PDF] and search for “BRM”. You will find several oddities. For example, among these proposed changes are some that actually reverse BRM decisions. Yes, you heard me correctly. SC34/WG4, the Microsoft-dominated committee that maintains OOXML, is undoing various BRM decisions that enabled OOXML to be approved in the first place. Why? Well, of course, to make the standard conform more to Microsoft Office.

For example, take DR 09-0159 “General: Unintended incompatibilities between Transitional schema and Ecma-376″ or DR 09-0275 “BRM: serial date representation” with this comment:

Although this text is in accord with the detailed amendments resolved at the BRM, it is against the spirit of the desired changes for many countries. We believe that due to time limitations at the BRM, this change was made without sufficient examination of the consequences, and was made in error by the BRM (in which error the UK played a part).

(Norbert Bollow, a member of the Swiss NB, has some good analysis of the return of the leap year bug in spreadsheets. And Jomar Silva with the Brazilian NB tracks some additional breaking changes on a wiki.)

Ah, So WG4 is now interpreting the “spirit of the BRM” through their shamanic communion with the ISO Weltgeist, and each time their oracle come back with the same response: “Change OOXML so it ‘conforms’ to Microsoft Office 2007″. How convenient for Microsoft.

For most standards, multiple vendors work together to improve interoperability and to increase their conformance with the standard. But with OOXML a single vendor stuffs the committee and works to make the standard better conform to Microsoft’s monopoly product.

So although Microsoft Office does not conform to ISO/IEC 29500 today, I have no doubt that within a few months it will fully conform. But not a single line of code will have changed in the Office product. Office 2007 will be retroactively made to conform to ISO/IEC 29500. What will happen is the standard will be modified to match that single vendor’s products, by misapplication of an ISO procedure intended for fixing minor drafting errors.

So why go through all this trouble? I believe this is all about getting the OOXML standard “corrected” so Microsoft can push for it to get it officially adopted around the world. The only reason they’ve held back so far is because MS Office does not actually implement ISO/IEC 29500 today. So it would have been counter productive for them to push for official adoption. However, once this oversight is remedied, by changing the standard to match their product, then watch out.

The side effect, perhaps unintended, is that the OOXML standard is thus clearly marked to be unstable and unsuitable for adoption or implementation. With 800 pages of defects and more being found, and a Microsoft-dominated committee that changes the standard with no objective technical justification, the exact contents of the OOXML standard is tentative, uncertain and temporary. Four corrigenda documents and two amendment documents are currently being balloted, including many breaking changes. More corrigenda and amendments are on the way. There is no provision for a version attribute or any other indicator to declare which of the multiple incompatible versions of the standard a document conforms to. What competitor would risk implementing the standard, knowing that Microsoft dominates WG4, which has shown it is willing to change the standard at Microsoft’s whim? The risk is simply too large. A competitor would simply be putting their head in the lion’s mouth.

And at the same time WG4 rushes to make OOXML conform to Office 2007, Microsoft is moving on with Office 2010, now in technical preview. Office 2010 will be extending OOXML in hundreds of places. Where is SC34 in this? Where is the new work proposal for OOXML 1.1? Where the are discussions? The drafts? None of this exists. If Microsoft wanted to, they could have submitted these changes to SC34 at the recent meeting in Seattle, but they preferred to reserve discussion of the Office 2010 changes for a private meeting in Redmond the day after the SC34 Plenary ended, a snub to SC34 and their fictional control of OOXML.

So Microsoft is now off extending OOXML, and this whole ISO escapade with OOXML seems for naught. (I hear also that Microsoft is also backing off the submission of their Extensible Page Specification (XPS) to ISO as well, saying that “an Ecma Standard is good enough”.) It appears that Microsoft got what they wanted from ISO and is moving on. Who said it would last more than a night? As my grandmother used to say, “Why buy the cow when you can get the milk for free?”

In any case, the future looks like something like this:

• ISO/IEC 29500:2008’s future is uncertain. If the whole i4i patent thing goes against Microsoft, the standard will probably need to be withdrawn.
• ISO/IEC 29500 with Corrigenda and Amendments will eventually line up with Office 2007 SP2 sometime in 2010/11.
• But before that happens, Office 2010 will ship with hundreds of extensions that are not described in ISO/IEC 29500 but are documented in proprietary “implementation notes” on Microsoft’s web site.
• “Office 15″ will ship sometime around 2013. It will have further proprietary extensions to ISO/IEC 29500, also not standardized. Office 15 will still be supporting “transitional” OOXML, just like Office 2007 and Office 2010 did. “Transitional” OOXML is the variation that has all the deprecated crud, like VML and “DoItLikeWord95″ in it.
• “Office 16″ will ship sometime around 2016. It will finally support the “strict” schema of ISO/IEC 29500, but with 3 generations of proprietary extensions layered on top of it. And that assumes ISO/IEC 29500 actually still exists. In 2015 — 5 years after its last amendment — it will be up for “periodic review” in ISO and may be withdrawn if it appears to have been abandoned by Microsoft.

The pattern is clear: OOXML will be extended by Microsoft much faster than it will be standardized and corrected by ISO. This will make the ISO version of OOXML, currently not supported by Microsoft, even more irrelevant in the future.

In Part I of this OOXML update, my first post on the topic in over a year, I showed you how Microsoft maintains strong control over the OOXML standard. Despite their earlier promises that control of OOXML would be handed over to an independent, international committee, a look at attendance records reveals that the committee that maintains OOXML (JTC1/SC34/WG4) consists mainly of Microsoft employees, who outnumber any other company or organization on the committee 10-to-1.

In this, Part II of my OOXML update, I’ll tie up another loose end from the immediate aftermath of the DIS 29500 BRM.

Let’s start by casting our memories back to April, 2008. The BRM was over. NBs had reviewed thousands of pages and submitted thousands of defects. And Microsoft/Ecma made thousands of responses. And the BRM approved thousands of changes. Then, as a final step, NBs were asked if they wanted to change their vote from their original September 2007 vote, based on the changes made to the standard by the BRM.

Curiously, NBs were asked to make their final decision without actually seeing the text of the standard they were being asked to approve. ISO leadership denied requests from several NBs, a formal SC34 resolution requesting this text, as well as NB appeals, all which asked to have access to the “final DIS” text that would eventually be published. The ISO chief, in his response to the NB appeals, called the final text of OOXML “irrelevant” (prophetic words, indeed!) and would only permit NBs to have access to a list of over 1,000 resolutions from the BRM, many of which gave great editing discretion to the Microsoft consultant who would eventually produce the final text of the specification.

I discussed why the lack of a final DIS text was a problem back in May 2008:

We are currently approaching a two month period where NB’s can lodge an appeal against OOXML. Ordinarily, one of the grounds for appeal would be if the Project Editor did not faithfully carry out the editing instructions approved at the BRM. For example, if he failed to make approved changes, made changes that were not authorized, or introduced new errors when applying the approved changes. But with no final DIS text, the NB’s are unable to make any appeals on those grounds. By delaying the release of the final DIS text, JTC1 is preventing NB’s from exercising their rights.

Would you make thousands of changes to code and then not allow anyone to test it, and then release it internationally? Of course not. Doing so would amount to professional malpractice. But that is essentially what ISO did with OOXML.

Well, guess what happened? Indeed, the published text of OOXML failed to carry out all of the editing instructions made by the BRM. Several of the BRM resolutions were ignored altogether. Several others were applied inconsistently or erroneously. Although I am aware of no systematic review of all 1,000+ BRM decisions, some NBs have gone back and reviewed the published text against BRM decisions that should have addressed their own NB’s reported comments. They have found many “discrepancies” and these have now been reported as defect reports [PDF].

Whether the flaws in the published text are intentional or accidental, grave or minor, does not really matter here. Errare humanum est. The problem is that it was 100% predictable that human error would cause problems like this when dealing with text changes of this volume. The issue is not whether there will be errors introduced. The presence of many errors was guaranteed. The question was whether NBs are entitled to base their vote on all relevant information, including the final text of the standard, or whether relevant information, indeed the most relevant information — the text of the specification they are voting on — may be withheld from inspection. For ISO leadership to deny NBs the ability to review the very text they were voting was irresponsible.

The good news is that ISO leadership has changed since this debacle, and JTC1 is currently revising Fast Track procedures, in part to deal with the abuses of DIS 29500. One of the changes they are making is to the Fast Track procedure will require that the final DIS text be distributed to NBs before the final vote. This is progress and it is good to see these changes made, though it is unfortunate that it required a failure before such obvious and prudent precautions were instituted. Leadership entails foreseeing and preventing problems, not simply reacting to them. In any case, the NBs that appealed to ISO on the basis of not being allowed to see the final text should feel vindicated now. The NBs of India, Brazil and South Africa were right.

In Part III of this Update, I’ll bring the story up to the present day, and in Part IV I’ll update the story through the year 2016.