File Formats Blog

Moving day

2009-08-09T08:20:00.002-04:00

This blog is now located at Wordpress. Old posts will remain here for the foreseeable future, but all new posts will be at the new location.

The reason for this move is the increasingly capricious policies of Blogspot (i.e., Google) in declaring blogs to have "objectionable content." I haven't been affected by that, not counting the time they mysteriously declared this a "spam blog" several years ago along with dozens or hundreds of other blogs, but I'm hoping Wordpress will be a more reasonable host.

If you're using http://www.mcgath.com/fileformatsblog to bookmark this blog, then you won't be affected, except that you won't even see this post and the look will be a bit different.

You'll have to register for a Wordpress account if you want to comment there. There may also be an Open ID option.

JHOVE 1.4

2009-08-03T15:17:00.000-04:00

JHOVE 1.4 is now available on SourceForge. The main change is that PDF/A compliance is more accurately identified than before, and is based on the final standard rather than a draft.

JPEG XR is ISO standard

2009-07-31T08:58:00.002-04:00

JPEG XR, formerly known as Microsoft HD Photo, is now an international standard, as reported in a JPEG press release.

Thanks to Bill Crow's blog for calling this to my attention.

JHOVE2 workshop

2009-07-22T13:50:00.002-04:00

A workshop on JHOVE2 will be held after the conclusion of iPres 2009 in San Francisco, on October 7, 2009. This will include, for the first time, a presentation of the prototype code.

Jumping the gun on HTML 5

2009-07-01T08:25:00.000-04:00

With the release of Firefox 3.5, references to the "HTML 5 standard" are on the rise in tech news. CNET refers to "new standards such as HTML 5." The Christian Science Monitor says "this latest version [of Firefox] adds support for the HTML 5 web standard." BNET Technology has a whole article on HTML 5 without once suggesting that it isn't in final form.

But as the World Wide Web Consortium notes, HTML 5 is still a long way from being settled. The latest working draft states: "Implementors should be aware that this specification is not stable. Implementors who are not taking part in the discussions are likely to find the specification changing out from under them in incompatible ways." (Emphasis in the original.)

This most likely means that there will be several different "HTML 5"s from different vendors, and that it will be a long time before they all come close to agreement. HTML 5 was intended to close the gap among different implementations by specifying more aspects of the language and basing it on an object model rather than a syntactic one, but we're probably in for more of the same techno-Babel.

DROID 4.0

2009-06-11T10:19:00.001-04:00

The National Archives (UK) has released DROID 4.0, the latest version of its file format identification tool.

The focus of the latest major release is the inclusion of the DCS (Digital Continuity Service) and Planets, Collection Profiler. DROID now runs in two modes. The original file identification mode, and the new “profile” mode that allows users to obtain file format information gathered from large distributed sources of digital files, providing users with aggregated statistical data and reports to help them take appropriate management decisions regarding risk associated with such large collections of files.

JHOVE 1.3 is out

2009-06-06T08:16:00.002-04:00

JHOVE 1.3 is now available. This version includes fixes to some serious bugs in the PDF module. It now has a much lower rate of spurious rejections. Read the release notes for full details.

I've already made a few post-release changes to source code, removing a stack dump for debugging purposes which I'd inadvertently left in, and updating version information where I'd forgotten to. Sigh ... I need to put all the configuration information in one place, instead of having it scattered through a dozen source files, before the next release.

WARC is ISO standard

2009-06-05T09:49:00.002-04:00

The WARC (Web Archive) format is now an ISO standard, available for a mere 118 Swiss francs. Or you can grab a near-final draft (PDF) for free.

Found by way of digitizationblog.

W3C rescinds four drafts

2009-05-21T15:21:00.002-04:00

This is unusual; W3C has rescinded four XHTML-related drafts. This means that they roll back to the previous versions. The rescinded drafts are:

Survey results on JPEG2000

2009-05-19T14:53:00.003-04:00

The results of a University of Connecticut study on how libraries are using JPEG2000 format are available online. Thanks to Digitization 101 for the link.

Spoofing characters

2009-05-08T08:00:00.000-04:00

You wouldn't think there's a security issue in the UTF-8 character set, but there is, in an indirect way. In a note on a Sun Java blog, it's explained that in the old (20th century) definition of UTF-8, some characters could be represented by more than one byte sequence. For example, any ASCII character (0X01 through 0X7F) could be preceded by 0XC0 without changing its interpretation. This could create problems if security filters are looking for certain characters in order to stop cross-site scripting or SQL injection; the spoofed characters could get past such filters if they don't take the alternate byte representations into account. For this reason, the current UTF-8 requirements specify that the shortest byte representation of a character is the only legitimate one.

This change was made in 2000, but not all implementations of UTF-8 have caught up. Sun has only recently fixed this in Java, with JDK7, Open JDK 6, JDK 6 update 11 and later, JDK5.0u17, and Java 1.4.2_19. (If you're using Java 1.3 or earlier, you're probably stuck, but why would you do that?)

PDF with DRM

2009-05-05T16:20:00.001-04:00

PDFZone discusses ProtectedPDF, a way of incorporating digital rights management in PDF. According to the article, ProtectedPDF files can be read in Acrobat Reader without any plugins, which presumably means that the files are fully PDF-compliant. Internet access to Vitrium's server is required to open documents. While the article doesn't say so, the only way I can think of doing this is through JavaScript.

The value or abusiveness of DRM depends on how it is used. For short-term use of a document, it can be a sensible way of limiting access. For limiting the distribution of content while supposedly allowing recipients permanent access, it doesn't work so well. If Vitrium ever goes out of business or gets tired of supporting ProtectedPDF, then all the rights-managed documents become inaccessible. "Buying" DRM-protected content, as opposed to renting it, is a dubious proposition. The effectiveness of DRM is limited; if you can get content on your computer, there's always a way to copy it.

Assuming JavaScript is required, users must enable it to view ProtectedPDF documents. (A third-party FAQ confirms this.) This exposes users to JavaScript-based vulnerabilities in Acrobat Reader, as well as possible loss of privacy. The article notes that ProtectedPDF files can report back on who's reading them. People are aware of these risks when using a web browser, less so when using a PDF reader.

As a technical concept, it's very interesting that DRM can be implemented within the PDF specification. How it works out in practice remains to be seen.

HTML 5 update

2009-04-24T15:02:00.002-04:00

W3C has published new working drafts of:

The last four are newly broken out from the HTML 5 specification. Or should we call it Ginormica?

CLIR absorbs DLF

2009-04-17T18:47:00.002-04:00

The Digital Library Federation is being merged into the Council on Library and Information Resources (CLIR), effective July 1, 2009. The DLF Spring Forum isn't affected by this.

GDFR + Pronom = UDFR

2009-04-08T15:35:00.002-04:00

For some time, people working in digital preservation have been wondering what was happening with the Global Digital Format Registry (GDFR). The answer was: Not much. But a lot of discussions were happening on how to combine the openness of GDFR with the maturity of Pronom. Now it can be told: Planning for the Unified Digital Formats Registry (UDFR) has been announced.

The proposal and roadmap (PDF) states:

There are two major efforts underway to create a format registry with complimentary strengths and weaknesses. PRONOM, created by The National Archives (TNA) in the UK, has a strong technological base, and has been building a database of original information about various digital formats. PRONOM at this point however is owned and maintained by a single organization, making it vulnerable to changes in that institution. The Global Digital Formats Registry (GDFR) effort, hosted by Harvard University, has developed a model for a registry based on shared governance, cooperative data contribution, and distributed data hosting. However, GDFR is technically less far along in development, and has not yet begun database building. Given the paucity of resources in the digital preservation community it would be highly unfortunate if these efforts were to compete for resources. Therefore a group of involved and interested institutions have agreed to join together to create a single shared formats registry drawing on the individual strengths of the two existing efforts. The initiative would:

* be technically based on the existing PRONOM system and database; * create a community governance model for the registry involving all institutions willing to contribute to its development; * develop a mechanism for the distribution of the registry data in such a way as to support local extensions and additions to the database; * develop both technical and organizational support for distributed input to the registry, including some form of quality vetting of contributed data.

The start of the working group came at iPres in London last September, but I couldn't talk about it publicly then. The working group includes members from British Library, the California Digital Library, the Harvard University Libraries, the National Archives, the National Library of Australia, the National Library of New Zealand, Portico, and Tessella.

HUL has registered the domain udfr.org, but there is no content up yet.

Keeping track of JHOVE2

2009-03-31T15:15:00.003-04:00

Sheila Morrissey and Stephen Abrams mentioned to me that some people have been looking at the old JHOVE2 proposal on the HUL website, even though it's far out of date. I've just removed a couple of links to it on the JHOVE website. Here's a link to the current and funded proposal (DOC format).

PDF/A conference

2009-03-24T14:57:00.004-04:00

The third international PDF/A conference will be held in Berlin on June 17-19, 2009. Here's the announcement (PDF, naturally).

Reconstructability and digital preservation

2009-03-23T06:45:00.001-04:00

In my post on Preservation vs. format tolerance, I briefly addressed the issue of how strictly format compliance should be required for preservation purposes. Here I'd like to expand on that. A frequently-mentioned idea in digital preservation is that people in the future might need to examine old files armed only with the specification for the format. Following this view, files worthy of preservation should strictly follow the specification, so that our descendants can write a file reader that implements the specification and open the ancient documents without difficulty.

There are several problems with this view. First, many specifications aren't unambiguously written. It's often unclear what is a requirement and what is desirable. (The TIFF specification has this problem.) Second, future programmers won't just give up if their software won't open a file; they'll look to see what's going wrong and will be able to work around some problems. Third, not all deviations make the file unreadable; some cause the loss of only limited information. For instance, a malformed date in TIFF or PDF may cause problems reading the date, but won't otherwise impact the ability to read the file.

JHOVE 1.x takes a "one strike, you're out" approach to compliance, with certain exceptions. If a file violates the specification in any way, it's ill-formed or invalid and you're told only that it's a "bytestream." (We have compromised this approach on specific features, but it's still the basic approach. One character with the high bit set in an ASCII file, for instance, says it's not well-formed ASCII.)

JHOVE2 improves on this by introducing the concept of assessment. According to the JHOVE2 functional requirements (version1.3), "Assessment is the process of determining the level of acceptability of a digital object for a specific purpose on the b[asis] of locally-defined policy rules. Since these rules are configurable, assessment is considered (vis-a-vis validation) a subjective determination." From what I've seen, this concept hasn't been very well nailed down yet, though I'm sure Stephen Abrams has thought a lot more about it than what I've read.

What I'm proposing here is that a metric of assessment should be reconstructability, the degree to which a file can be reconstructed in spite of non-standard aspects in its content or structure.

To a certain extent, estimates of reconstructability require guesswork about what information people will have in the future. In the case of TIFF, will they have just the 6.0 specification? Will they have the Photoshop supplement? The contents of technical mailing lists? But we can recognize that some deviations are easier to reconstruct by guesswork and common sense than others. If strictly following byte-alignment requirements makes files look broken, it won't be hard to try relaxing the requirement. If being unable to decipher a field means losing just one piece of metadata, that's not usually a disaster.

Conversely, a file can be completely well-formed and valid, yet have reconstructability issues because of external dependencies. Interestingly, all schema-based XML files have this problem. Schemas are necessarily external to the data file, and the schema's location is a URI which might easily become worthless in the future. Sometimes schemas change, even though they shouldn't; for instance, the schema for MIX metadata used to be the one for MIX 0.2, but was replaced with the schema for MIX 2.0, breaking the validation MIX 0.2 documents. So unless an XML file is packaged together with its schema, it should lose points for reconstructability. On the positive side, anyone with knowledge of the ASCII character set and a few representative files to study should be able to reconstruct the basic rules of XML, even without documentation of the format, and if the tags reflect their intended meaning, a file can be largely self-documenting.

HTML provides a worrisome case. The large majority of the HTML on the Web is defective in one more more ways, and browsers use ingenious tricks to approximate the intent of the author. Read a file with the wrong browser, and it may be a complete mess. Then there's the matter of external links, which may include JavaScript and CSS files that are essential to the presentation. HTML accounts for a large portion of the digital information available today, but it presents severe reconstructability problems.

This post is just a first shot at describing the idea, and it's possible I'm repeating what someone else has said (if so, someone please tell me). But I hope it will be a springboard for discussion.

Preservation vs. format tolerance

2009-03-18T00:15:00.001-04:00

In doing a search for blog posts about JHOVE, I came across a very interesting discussion of the conflict between exact format checking and error tolerance. "Postel's law" implies that readers should overlook errors in files whenever feasible. But what should digital repositories do? Should they be completely strict, because we don't know how future readers will work? Or should they overlook some harmless errors rather than reject otherwise perfectly good documents? JHOVE uses a "one strike, you're out" approach, which is often less then ideal. But accepting any document which doesn't break the current Adobe Reader would be reckless. The balance is tricky. Some errors are deadly, some harmless, and validation software should really be able to distinguish degrees of harm.

Services which ban MP3 format

2009-03-16T06:09:00.002-04:00

Over at Taking License, I've posted a discussion of Internet services which, for reasons beyond my comprehension, have a specific prohibition on the MP3 file format.

Planets Testbed

2009-03-05T12:35:00.004-05:00

The Planets project has announced a web application, the Planets Testbed, designed to allow evaluation of digital preservation strategies. Testers can use various digital preservation tools (including JHOVE) and migration paths. People with an interest in digital preservation from libraries, archives, museums, and so on are invited to take part. Institutions will be able to register and carry out experiments.

I should mention that the login page triggers some urgent warning messages in Firefox, since it uses a self-signed security certificate. Since you won't be giving them any credit card numbers or other sensitive information, you shouldn't have to worry about that.

By the way, congratulations to Adrian Brown on his appointment to the Parliamentary Archives.

New working drafts from XML working security group

2009-02-26T16:07:00.002-05:00

The XML security working group of the W3C has released eight new working drafts. These deal with crypotographic algorithms, use cases, key derivation, and best practices.

ODF-Next

2009-02-23T12:15:00.000-05:00

OASIS has started to solicit input for "ODF-Next," the version which will follow ODF 1.2. This input will go on concurrently with the finalization of 1.2. Proposals must be received by March 31 to be considered in the ODF-Next requirements report, which is expected to come out around May 1. Proposals previously submitted to the ODF TC's public comment list don't need to be resubmitted.

See discussion by Rob Weir.

JHOVE 1.2

2009-02-11T12:30:00.002-05:00

JHOVE 1.2 is now up on SourceForge. This release has some bug fixes, and includes scripts which were accidentally left out of the 1.1 release.

JHOVE2 functional requirements

2009-02-04T07:00:00.000-05:00

The latest (version 1.3) functional requirements for JHOVE2 have been posted.