dive into mark

Why do we have an IMG element?

2009-11-03T21:38:59Z

On February 25, 1993, Marc Andreessen wrote:

I’d like to propose a new, optional HTML tag:

IMG

Required argument is SRC="url".

This names a bitmap or pixmap file for the browser to attempt to pull over the network and interpret as an image, to be embedded in the text at the point of the tag’s occurrence.

An example is:
<IMG SRC="file://foobar.com/foo/bar/blargh.xbm">

(There is no closing tag; this is just a standalone tag.)

This tag can be embedded in an anchor like anything else; when that happens, it becomes an icon that’s sensitive to activation just like a regular text anchor.

Browsers should be afforded flexibility as to which image formats they support. Xbm and Xpm are good ones to support, for example. If a browser cannot interpret a given format, it can do whatever it wants instead (X Mosaic will pop up a default bitmap as a placeholder).

This is required functionality for X Mosaic; we have this working, and we’ll at least be using it internally. I’m certainly open to suggestions as to how this should be handled within HTML; if you have a better idea than what I’m presenting now, please let me know. I know this is hazy wrt image format, but I don’t see an alternative than to just say “let the browser do what it can” and wait for the perfect solution to come along (MIME, someday, maybe).

Xbm and Xpm were popular graphics formats on Unix systems.

“Mosaic” was one of the earliest web browsers. (”X Mosaic” was the version that ran on Unix systems.) When he wrote this message in early 1993, Marc Andreessen had not yet founded the company that made him famous, Mosaic Communications Corporation, nor had he started work on that company’s flagship product, “Mosaic Netscape.” (You may know them better by their later names, “Netscape Corporation” and “Netscape Navigator.”)

“MIME, someday, maybe” is a reference to content negotiation, a feature of HTTP where a client (like a web browser) tells the server (like a web server) what types of resources it supports (like image/jpeg) so the server can return something in the client’s preferred format. The Original HTTP as defined in 1991 (the only version that was implemented in February 1993) did not have a way for clients to tell servers what kind of images they supported, thus the design dilemma that Marc faced.

A few hours later, Tony Johnson replied:

I have something very similar in Midas 2.0 (in use here at SLAC, and due for public release any week now), except that all the names are different, and it has an extra argument NAME="name". It has almost exactly the same functionality as your proposed IMG tag. e.g.

<ICON name="NoEntry" href="http://note/foo/bar/NoEntry.xbm">

The idea of the name parameter was to allow the browser to have a set of “built in” images. If the name matches a “built in” image it would use that instead of having to go out and fetch the image. The name could also act as a hint for “line mode” browsers as to what kind of a symbol to put in place of the image.

I don’t much care about the parameter or tag names, but it would be sensible if we used the same things. I don’t much care for abbreviations, ie why not IMAGE= and SOURCE=. I somewhat prefer ICON since it imlies that the IMAGE should be smallish, but maybe ICON is an overloaded word?

Midas was another early web browser, a contemporary of X Mosaic. It was cross-platform; it ran on both Unix and VMS. “SLAC” refers to the Stanford Linear Accelerator Center (now the SLAC National Accelerator Laboratory). SLAC hosted the first web server in the United States (in fact the first web server outside Europe). When Tony wrote this message, SLAC was an old-timer on the WWW, having hosted five pages on their web server for a whopping 441 days.

Tony continued:

While we are on the subject of new tags, I have another, somewhat similar tag, which I would like to support in Midas 2.0. In principle it is:

<INCLUDE HREF="...">

The intention here would be that the second document is to be included into the first document at the place where the tag occured. In principle the referenced document could be anything, but the main purpose was to allow images (in this case arbitrary sized) to be embedded into documents. Again the intention would be that when HTTP2 comes along the format of the included document would be up for separate negotiation.

“HTTP2” is a reference to Basic HTTP as defined in 1992. At this point in early 1993, it was still largely unimplemented. The draft known as “HTTP2” evolved and was eventually standardized as “HTTP 1.0” (albeit not for another three years). HTTP 1.0 did include request headers for content negotiation, a.k.a. “MIME, someday, maybe.”

Tony continued:

An alternative I was considering was:

<A HREF="..." INCLUDE>See photo</A>

I don’t much like adding more functionality to the <A> tag, but the idea here is to maintain compatibility with browsers that can not honour the INCLUDE parameter. The intention is that browsers which do understand INCLUDE, replace the anchor text (in this case “See photo”) with the included document (picture), while older or dumber browsers ignore the INCLUDE tag completely.

This proposal was never implemented, although the idea of text-if-an-image-is-missing is an important accessibility technique which was missing from Marc’s initial <IMG> proposal. Many years later, this feature was bolted on as the <img alt> attribute, which Netscape promptly broke by erroneously treating it as a tooltip.

A few hours after that, Tim Berners-Lee responded:

I had imagined that figues would be reprented as

<a name=fig1 href="fghjkdfghj" REL="EMBED, PRESENT">Figure </a>

where the relation ship values mean
EMBED	 Embed this here when presenting it
PRESENT	 Present this whenever the source document is presented
Note that you can have various combinations of these, and if the browser doesn’t support either one, it doesn’t break.

[I] see that using this as a method for selectable icons means nesting anchors. Hmmm. But I hadn’t wanted a special tag.

This proposal was never implemented, but the rel attribute is still around.

Jim Davis added:

It would be nice if there was a way to specify the content type, e.g.

<IMG HREF="http://nsa.gov/pub/sounds/gorby.au" CONTENT-TYPE=audio/basic>

But I am completely willing to live with the requirement that I specify the content type by file extension.

This proposal was never implemented, but Netscape did later add arbitrary embedding of media objects with the <embed> element.

Jay C. Weber asked:

While images are at the top of my list of desired medium types in a WWW browser, I don’t think we should add idiosyncratic hooks for media one at a time. Whatever happened to the enthusiasm for using the MIME typing mechanism?

Marc Andreessen replied:

This isn’t a substitute for the upcoming use of MIME as a standard document mechanism; this provides a necessary and simple implementation of functionality that’s needed independently from MIME.

Jay C. Weber responded:

Let’s temporarily forget about MIME, if it clouds the issue. My objection was to the discussion of “how are we going to support embedded images” rather than “how are we going to support embedded objections in various media”.

Otherwise, next week someone is going to suggest ‘lets put in a new tag <AUD SRC="file://foobar.com/foo/bar/blargh.snd">‘ for audio.

There shouldn’t be much cost in going with something that generalizes.

With the benefit of hindsight, it appears that Jay’s concerns were well-founded. It took a little more than a week, but HTML5 did finally add new <video> and <audio> elements.

Responding to Jay’s original message, Dave Raggett said:

True indeed! I want to consider a whole range of possible image/line art types, along with the possibility of format negotiation. Tim’s note on supporting clickable areas within images is also important.

Later in 1993, Dave Raggett proposed HTML+ as an evolution of the HTML standard. The proposal was never implemented, and it was superceded by HTML 2.0. HTML 2.0 was a “retro-spec,” which means it formalized features already in common use. “This specification brings together, clarifies, and formalizes a set of features that roughly corresponds to the capabilities of HTML in common use prior to June 1994.”

Dave later wrote HTML 3.0, based on his earlier HTML+ draft. HTML 3.0 was also never implemented (outside of the W3C’s own reference implementation, Arena), and it was superceded by HTML 3.2. HTML 3.2 was also a “retro-spec” — “HTML 3.2 adds widely deployed features such as tables, applets and text flow around images, while providing full backwards compatibility with the existing standard HTML 2.0.”

Dave later co-authored HTML 4.0 and developed HTML Tidy, and went on to help with XHTML, XForms, MathML, and other modern W3C specifications.

Getting back to 1993, Marc replied to Dave:

Actually, maybe we should think about a general-purpose procedural graphics language within which we can embed arbitrary hyperlinks attached to icons, images, or text, or anything. Has anyone else seen Intermedia’s capabilities wrt this?

Intermedia was a hypertext project from Brown University. It was developed from 1985 to 1991 and ran on A/UX, a Unix-like operating system for early Macintosh computers.

The idea of a “general-purpose procedural graphics language” did eventually catch on. Modern browsers support both SVG (declarative markup with embedded scripting) and <canvas> (procedural direct-mode graphics API), although the latter started as a proprietary extension before being “retro-specced” by the WHATWG.

Bill Janssen replied:

Other systems to look at which have this (fairly valuable) notion are Andrew and Slate. Andrew is built with _insets_, each of which has some interesting type, such as text, bitmap, drawing, animation, message, spreadsheet, etc. The notion of arbitrary recursive embedding is present, so that an inset of any kind can be embedded in any other kind which supports embedding. For example, an inset can be embedded at any point in the text of the text widget, or in any rectangular area in the drawing widget, or in any cell of the spreadsheet.

“Andrew” is a reference to the Andrew User Interface System (although at that time it was simply known as the Andrew Project).

Meanwhile, Thomas Fine had a different idea:

Here’s my opinion. The best way to do images in WWW is by using MIME. I’m sure postscript is already a supported subtype in MIME, and it deals very nicely with mixing text and graphics.

But it isn’t clickable, you say? Yes your right. I suspect there is already an answer to this in display postscript. Even if there isn’t the addition to standard postscript is trivial. Define an anchor command which specifies the URL and uses the current path as a closed region for the button. Since postscript deals so well with paths, this makes arbitrary button shapes trivial.

Display Postscript was an on-screen rendering technology co-developed by Adobe and NeXT.

This proposal was never implemented, but the idea that the best way to fix HTML is to replace it with something else altogether still pops up from time to time.

Tim Berners-Lee, March 2, 1993:

HTTP2 allows a document to contain any type which the user has said he can handle, not just registered MIME types. So one can experiment. Yes I think there is a case for postscript with hypertext. I don’t know whether display postcript has enough. I know Adobe are trying to establish their own postscript-based “PDF” which will have links, and be readable by their proprietory brand of viewers.

I thought that a generic overlaying language for anchors (Hytime based?) would allow the hypertext and the graphics/video standards to evolve separately, which would help both.

Let the IMG tag be INCLUDE and let it refer to an arbitrary document type. Or EMBED if INCLUDE sounds like a cpp include which people will expect to provide SGML source code to be parsed inline — not what was intended.

HyTime was an early, SGML-based hypertext document system. It loomed large in many early discussions of HTML, and later XML.

Tim’s proposal for an <INCLUDE> tag was never implemented, although you can see echoes of it in <object>, <embed>, and the <iframe> element.

Finally, on March 12, 1993, Marc Andreessen revisited the thread:

Back to the inlined image thread again — I’m getting close to releasing Mosaic v0.10, which will support inlined GIF and XBM images/bitmaps, as mentioned previously. …

We’re not prepared to support INCLUDE/EMBED at this point. … So we’re probably going to go with <IMG SRC="url"> (not ICON, since not all inlined images can be meaningfully called icons). For the time being, inlined images won’t be explicitly content-type’d; down the road, we plan to support that (along with the general adaptation of MIME). Actually, the image reading routines we’re currently using figure out the image format on the fly, so the filename extension won’t even be significant.

I don’t really know why I wrote this. It wasn’t what I set out to write. That happens. But I am extraordinarily fascinated with all aspects of this almost-17-year-old conversation. Consider:

HTTP still exists. HTTP successfully evolved from 0.9 into 1.0 and later 1.1. And still it evolves.
HTML still exists. That rudimentary data format — it didn’t even support inline images! — successfully evolved into 2.0, 3.2, 4.0. And still it, too, evolves. HTML is an unbroken line. A twisted, knotted, snarled line, to be sure. There were plenty of “dead branches” in the evolutionary tree, places where standards-minded people got ahead of themselves (and ahead of authors and implementors). But still. Here we are, in 2009, and web pages from 1990 still render in modern browsers. I just loaded one up on my Android phone, and I didn’t even get prompted to “please wait while importing legacy format…”
HTML has always been a conversation between browser makers, authors, standards wonks, and other people who just showed up and liked to talk about angle brackets. Most of the successful versions of HTML have been “retro-specs,” catching up to the world while simultaneously trying to nudge it in the right direction. Anyone who tells you that HTML should be kept “pure” (presumably by ignoring browser makers, or ignoring authors, or both) is simply misinformed. HTML has never been pure, and all attempts to purify it have been spectacular failures, matched only by the attempts to replace it.
None of the browsers from 1993 still exist in any recognizable form. Netscape Navigator was abandoned in 1998 and rewritten from scratch to create the Mozilla Suite, which was then forked to create Firefox. Internet Explorer had its humble “beginnings” in “Microsoft Plus! for Windows 95,” where it was bundled with some desktop themes and a pinball game. (But of course that browser can be traced back further too.)
Some of the operating systems from 1993 still exist, but none of them are relevant to the modern web. Most people today who “experience” the web do so on a PC running Windows 2000 or later, a Mac running Mac OS X, a PC running some flavor of Linux, or a handheld device like an iPhone. In 1993, Windows was at version 3.1 (and competing with OS/2), Macs were running System 7, and Linux was distributed via Usenet. (Want to have some fun? Find a graybeard and whisper “Trumpet Winsock” or “MacPPP.”)
Some of the same people are still around and still involved in what we now simply call “web standards.” That’s after almost 20 years. And some were involved in predecessors of HTML, going back into the 1980s and before.
Speaking of predecessors… With the eventual popularity of HTML and the web, it is easy to forget the contemporary formats and systems that informed its design. Andrew? Intermedia? HyTime? And HyTime was not some rinky-dink academic research project; it was an ISO standard. It was approved for military use. It was Big Business. And you can read about it yourself… on this HTML page, in your web browser.

But none of this answers the original question: why do we have an <img> element? Why not an <icon> element? Or an <include> element? Why not a hyperlink with an include attribute, or some combination of rel values? Why an <img> element? Quite simply, because Marc Andreessen shipped one, and shipping code wins.

That’s not to say that all shipping code wins; after all, Andrew and Intermedia and HyTime shipped code too. Code is necessary but not sufficient for success. And I certainly don’t mean to say that shipping code before a standard will produce the best solution. Marc’s <img> element didn’t mandate a common graphics format; it didn’t define how text flowed around it; it didn’t support text alternatives or fallback content for older browsers. And 16, almost 17 years later, we’re still struggling with content sniffing, and it’s still a source of crazy security vulnerabilities. And you can trace that all the way back, 17 years, through the Great Browser Wars, all the way back to February 25, 1993, when Marc Andreessen offhandedly remarked, “MIME, someday, maybe,” and then shipped his code anyway.

The ones that win are the ones that ship.

Thank you for giving me the opportunity to explain this to you

2009-10-20T19:57:09Z

Recently, someone did the unthinkable: they published their own version of Dive Into Python and got it listed on Amazon.com. This apparently caused a small firestorm within Apress, the exact details of which I am not privy to, but which (I am told) became a somewhat larger firestorm after the Apress executives realized they had no legal recourse, and asked my opinion on the matter. You see, the book is published under the GNU Free Documentation License, which explicitly gives anyone and everyone the right to publish it themselves. (I was about to write “gives third parties the right,” until I realized that there are no third parties because there are no second parties. That’s kind of the point.)

This didn’t use to matter, because publishing on paper used to require a serious up-front investment in, well, paper. “Freedom of the press” was reserved for those with an actual press, and distribution costs were decidedly non-trivial. Publishing a book commercially just wasn’t practical for anyone but, well, a book publisher. That’s no longer the case. Copies can be purchased online, printed on demand, and drop-shipped to the customer — up-front investment be damned. And that’s for printed books; e-books are even easier.

Software had this problem first, by virtue of its non-corporeality. How many people are selling Free Software on eBay? We deride these sellers as “scammers,” but in truth the only time they run afoul of the law is when they attempt to rebrand your software without acknowledgement, or when they fail to abide by some other intentionally inside-out clause of the license that you chose in the first place (e.g. selling GPL’d binaries without offering source code).

Still, there’s a qualitative difference between letting people download your own work from your own site, and watching other people try to profit from it. But it is precisely this difference that strikes at the heart of the Free Software/Free Culture ethos. Part of choosing a Free license for your own work is accepting that people may use it in ways you disapprove of. There are no “field of use” restrictions, and there are no “commercial use” restrictions either. In fact, those are two of the fundamental tenets of the “Free” in Free Software. If “others profiting from my work” is something you seek to avoid, then Free Software is not for you. Opt for a Creative Commons “Non-Commercial” license, or a “personal use only” freeware license, or a traditional End User License Agreement. Free Software doesn’t have “end users.” That’s kind of the point.

The aforementioned Apress executive told me that he did not understand why I would be willing to work with a publisher but then be happy about their competition. This is what I told him:

I enjoy working with publishers because it makes me a better writer. But I don’t write for money; I write for love (or passion, or whatever you want to call it). I choose open content licenses because this is the way I want the world to work, and the only way to change the world is to change yourself first.

I don’t know where that leaves you as a business. But you’ve made a good amount of money on the original “Dive Into Python,” despite the fact that it’s been available for free online for 8 years. A German translation of Dive Into Python 3 is being published this quarter by Springer/Germany [a division of Apress' parent company] almost simultaneously with the English edition — much sooner-to-market than it would have been under a closed development process. (And an Italian translation was just released yesterday. You should snap that one up too before someone else does!) So maybe the problems you perceive are really opportunities in disguise.

So I am grateful for this anonymous soul who woke up one day and said to herself, “You know what I should do today? I should try to sell copies of that Free book that Pilgrim wrote.” Grateful, because it afforded me the opportunity to remind myself why I chose a Free license in the first place. My Zen teacher once told me that, when people try to do you harm, you should thank them for giving you the opportunity to forgive them. In this case it’s even simpler, because there’s nothing to forgive, just explain. She’s redistributing the work that I explicitly made redistributable. She’s kind of the point.

Translation From MS-Speak to English of Selected Portions of Tony Ross’ “Distributed Extensibility Submission”

2009-10-05T19:56:15Z

[Source Distributed Extensibility Submission from Microsoft - 30 September 2009]

It is a common practice for authors, tool vendors, and library authors to want to extend languages to represent additional information that can’t be adequately described by the standard grammar. … Here are a few examples that apply to HTML:

A JavaScript library processes custom tags in a browser and turns them into custom controls dynamically on the page.

We would very much like our proprietary Custom Tags to validate.

A browser wants to allow custom behaviors to be defined in one module and attached automatically to custom elements.

We would very much like our proprietary HTML Components to validate.

An author includes processing instructions in the document that will be processed by a server before delivering the document to a user agent.

We would very much like SharePoint’s proprietary InfoPath processing instructions to validate.

A HTML document editor adds information about tool settings so that a subsequent editing session can continue with the same settings.

We would very much like Word’s “export as HTML” output — which is so proprietary that it has spawned an entire cottage industry dedicated to “cleaning” it — to validate.

In many cases, the language customizations are used for small niche applications and don’t require the burden of centralized standardization. Instead these extensions are defined in a distributed fashion among groups of interested developers or authors.

We like making up our own shit.

A Distributed Extensibility model for standard HTML is desirable because it means that user agents from different vendors that adhere to the standard can be assured of correctly processing mark-up that contains extensions without destroying the integrity of the document.

Other people keep fucking up our shit.

The proposal as stated closely matches behavior that Internet Explorer has had for a number of releases, reducing compatibility concerns.

I am high as a kite.

Supporting distributed extensibility means providing a standard repeatable mechanism for creating these extensions without the need for centralized agreement.

We’re going to keep making up our own shit, whether it validates or not.

(with apologies to John Gruber,
who did it first and did it best)

Postscript: I am not “against” extensibility, or distributed extensibility, or decentralized extensibility, or whatever we’re calling namespaces this week. I just think it’s funny that some people read this proposal and immediately jumped to The Glorious And Infinitely Extensible Semantic Web With Unicorns And Ponies, when it’s really just an attempt to get the HTML Working Group to bless a collection of Microsoft-proprietary technologies that they’ve jammed into Internet Explorer over the years.

Recently

2009-10-01T02:49:51Z

Most of my recent writing has happened elsewhere.

Dive Into Python 3
Dive Into HTML5 (in progress)
Various articles about HTML5 on the WHATWG blog, including

That last article came about during the creation of mimesniff, my open source Python 3 library that implements the HTML5 Content-Type detection and character encoding detection algorithms.

If none of that is your cup of tea, here is a picture of my dog Beauregard, enjoying the beautiful North Carolina summer weather.

Open for your input

2009-08-04T14:57:47Z

On June 3, 2009, Janina Sajka, Chair of the Protocols and Formats Working Group of the W3C Web Accessibility Initiative (WAI), wrote:

The following consensus was reached by Protocols and Formats Working Group during its teleconference of Wednesday, 3 June 2009 …

We note that summary is often used as a technique for accessibility support where governmental regulations require governmental web sites to be accessible. … [link to Guide to the Section 508 Standards for Electronic and Information Technology, Subpart B - Technical Standards, Web-based Intranet and Internet Information and Applications (1194.22), Section (g) "Data Table"]

If summary is removed [from HTML 5], U.S. Government web sites, might find it more difficult to conform to HTML 5. We further note that Section 508 regulations apply to U.S. state and local governments, and that similar accessibility requirements are emerging in Canada, the U.K., the E.U., Australia, and elsewhere.

Guide to the Section 508 Standards for Electronic and Information Technology, Subpart B – Technical Standards, Web-based Intranet and Internet Information and Applications (1194.22), Section (g) “Data Table”:

Web developers who are interested in summarizing their tables should consider placing their descriptions either adjacent to their tables or in the body of the table, using such tags as the CAPTION tag.

On June 4, 2009, Ian Hickson, editor of the HTML 5 specification, replied:

As far as I can tell this concern is unfounded; the <caption> attribute is in fact encouraged by the very same government (as quoted above) to be used exactly as HTML5 recommends in a manner consistent with the goals of the summary=”" attribute.

HTML 5, Editor’s Draft as of this writing:

For tables that consist of more than just a grid of cells with headers in the first row and headers in the first column, and for any table in general where the reader might have difficulty understanding the content, authors should include explanatory information introducing the table. This information is useful for all users, but is especially useful for users who cannot see the table, e.g. users of screen readers.

Such explanatory information should introduce the purpose of the table, outline its basic cell structure, highlight any trends or patterns, and generally teach the user how to use the table.

There are a variety of ways to include this information, such as:

In prose, surrounding the table

In the table’s caption

In the table’s caption, in a details element

Next to the table, in the same figure

Next to the table, in a figure’s legend

Authors may also use other techniques, or combinations of the above techniques, as appropriate.

If a table element has a summary attribute, the user agent may report the contents of that attribute to the user.

On July 7, 2009, Janina Sajka, Chair of the Protocols and Formats Working Group of the W3C Web Accessibility Initiative (WAI), wrote:

PF responded on these questions formally. We would appreciate the basic human courtessy of acknowledgment. If you don’t like what we said, please speak to that. But kindly don’t simply ignore us. [link to the June 3, 2009 message announcing the consensus of the Protocols and Formats Working Group]

On July 7, 2009, Ian Hickson, editor of the HTML 5 specification, replied:

That e-mail received a reply some weeks ago: [link to Ian Hickson's message of June 4, 2009]

Is there a formal reply to that e-mail?

On July 7, 2009, Janina Sajka, Chair of the Protocols and Formats Working Group of the W3C Web Accessibility Initiative (WAI), replied:

No, we don’t make formal replies to individuals.

On August 2, 2009, John Foliot wrote:

I maintain that it is not the role of the HTML WG, and the editor in particular, to be offering this guidance, especially when it contradicts the consensus position of the W3C Group chartered to speak on web accessibility issues. Simply put, you are messing in somebody else’s yard, and it is against W3C process to be doing so. If HTML WG feel that they have compelling evidence and data that suggests that the WCAG guidance needs to be reviewed and revised, there is a process for that.

On August 3, 2009, Ian Hickson responded to John Foliot:

I didn’t want to be the one to have to explain this to you, but nobody else is doing so, so here goes: The W3C process doesn’t actually require that working groups agree, or not contradict each other. The WAI’s mission is not binding on other working groups.

On August 3, 2009, at approximately 7:51 pm, Roy Fielding wrote:

I have no opinion on the value of @summary other than noting the likelihood that its support will be required for some FIPS or government statute for accessibility, and therefore deprecating it within HTML5 will just make HTML5 look stupid.

Web developers who are interested in summarizing their tables should consider placing their descriptions either adjacent to their tables or in the body of the table, using such tags as the CAPTION tag.

On August 3, 2009, at approximately 7:59 pm, Roy Fielding wrote:

[A]uthors are clearly not served by a specification that tells them caption and summary are the same and all such information must be relegated to caption. As an implementor of content management systems used by government agencies in several different countries, I will not conform to any HTML specification that deprecates or fails to define @summary.

Web developers who are interested in summarizing their tables should consider placing their descriptions either adjacent to their tables or in the body of the table, using such tags as the CAPTION tag.

On May 23, 2006, Joe Clark wrote:

The Web Content Accessibility Guidelines Working Group [part of the W3C Web Accessibility Initiative (WAI)] is the worst committee, group, company, or organization I’ve ever worked with. Several of my friends and I were variously ignored; threatened with ejection from the group or actually ejected; and actively harassed.

In response to Joe Clark’s article, John Foliot wrote:

I can attest to knowing a regular participant to the [Web Content Accessibility Guidelines] WG discussion list who has been shut down and ignored on more than one occasion, and I personally have been dismissed by other working groups within the W3C. … So the behavior and treatment described by Joe is not unknown any time you strongly voice an opinion counter to the internal W3C herd.

On August 2, 2009, John Foliot wrote:

I have submitted an alternative [HTML 5] Draft document for consideration; one which I believe rightly returns the role of author guidance for creating accessible content to the W3C WAI – the group officially chartered by the W3C to speak to these matters. It is a question of respect.

On August 4, 2009, Ian Hickson asked:

Are you saying that for you, it is more important that HTML5 not contradict other W3C specifications, than it be that HTML5 address accessibility problems with the HTML language?

On August 4, 2009, John Foliot responded to Ian Hickson:

You needs to stop contradicting WAI, even if you have proof that WAI might need to update their guidance. …

Contradictory information *harms* the overall outreach aspect of teaching people how to create accessible web content, and I speak from the position of one who actually does that for a living, and have been doing so for close to a decade. THE MESSAGE WE SEND TO THE WORLD’S WEB DEVELOPERS MUST BE CONSISTENT!

Web developers who are interested in summarizing their tables should consider placing their descriptions either adjacent to their tables or in the body of the table, using such tags as the CAPTION tag.

On August 3, 2009, Roy Fielding wrote:

John [Foliot]’s point is that the W3C has a group specifically tasked to make accessibility recommendations.

On August 3, 2009, David Baron responded to Roy Fielding:

Has that group weighed in in this debate, in response to the evidence presented? Or is it just that an out-of-date (i.e., not updated in response to newer evidence) recommendation of that group is being cited?

On August 3, 2009, Roy Fielding responded to David Baron:

I don’t know — it isn’t a relevant question. The group exists [link to W3C Web Accessibility Initiative (WAI)] and seems to be open for your input.

This is the house

2009-07-06T19:25:53Z

This is the house
This is the house
The house of confusion
That Jeffrey built.

These are the tags
That live in the house
The house of confusion
That Jeffrey built.

These are the slashes
That we all ignore
That end the tags
That live in the house
The house of confusion
That Jeffrey built.

This is the namespace
That doesn’t do squat
When we parse the slashes
That end the tags
That live in the house
That Jeffrey built.

This is the header
That nobody sees
’Cause they see the namespace
That doesn’t do squat
When we parse the slashes
That end the tags
That live in the house
That Jeffrey built.

This is the spec
That nobody uses
’Cause we use the header
That nobody sees
’Cause they see the namespace
That doesn’t do squat
When we parse the slashes
That end the tags
That live in the house
That Jeffrey built.

Say goodbye to the spec
That no longer exists
And stick with the header
That nobody sees
And drop the namespace
That doesn’t do squat
And skip the slashes
That end the tags
That live in the house
That live in the house
The house of confusion
That Jeffrey built.

Fuck the foundries

2009-04-21T17:08:47Z

The more I read about embedded web fonts, the more I crystalize my thinking. Take, for example, this latest “A List Apart” article where Jeffrey Zeldman interviews David Berlow:

Zeldman: Let me put it another way. I want to use your ITC Franklin in a site I’m designing, but I’m not willing to violate my end user licensing agreement. How do we resolve this impasse, from your perspective?

Berlow: The next step is for those who control the font format(s) to define and document a permissions table to be added with all due haste to the OpenType, CoolType, TrueType, and FreeType formats. …
Zeldman: How can type designers and web designers work together to persuade the engineers who control the formats to modify the code to include a permissions table?

Berlow: [W]eb designers flat-out refused to part with real type, which has filled the web with type as graphic files, scaring the bejesus out of a lot of engineering people. … How important dynamically rendered type is to design and use on the web must now be clear. In addition, the only other option — that the type industry cede its intellectual property to the public without permission — is not going to happen. With no upgrade penalty to any applications, or change in usage by the public, the permissions table is the only invisible (type-like) solution.

I like how he focuses on the publisher’s end of the problem — “gee, all we have to do is define this permissions table, that sounds easy.” What he fails to mention is that every font-consuming application on every platform on every computer on Earth will need to be “upgraded” to “respect” this permissions table. Because otherwise they’re not really permissions, are they? They’re just useless bits taking valuable chunks out of my metered bandwidth plan. Like the bozo bit without the bozo.

This, then, is my current thinking about embedded web fonts:

FUCK THE FOUNDRIES

Seriously. Fuck them. They still think they’re in the business of shuffling little bits of metal around. You want to use a super-cool ultra-awesome totally-not-one-of-the-11-web-safe-fonts? Pick an open source font and get on with your life.

I know what you’re going to say. I can hear it in my head already. It sounds like the voice of the comic book guy from The Simpsons. You’re going to say, “Typography is by professionals, for professionals. Free fonts are worth less than you pay for them. They don’t have good hinting. They don’t come in different weights. They don’t have anything near complete Unicode coverage. They don’t, they don’t, they don’t…”

And you’re right. You’re absolutely, completely, totally, 100% right. “Your Fonts” are professionally designed, traditionally licensed, aggressively marketed, and bought by professional designers who know a professional typeface when they see it. “Our Fonts” are nothing more than toys, and I’m the guy showing up at the Philadelphia Orchestra auditions with a tin drum and a kazoo. “Ha ha, look at the freetard with his little toy fonts, that he wants to put on his little toy web page, where they can be seen by 2 billion people ha h… wait, what?”

Let me put it another way. Your Fonts are superior to Our Fonts in every conceivable way, except one:

WE CAN’T FUCKING USE THEM

Soon — and I mean really fucking soon, like “this year” soon — there will be enough different browsers in the hands of enough different people that can use any possible font on any possible web page. And then a whole lotta people will start noticing fonts again — not just Your People, just also Our People. People who couldn’t tell a serif from a hole in their head, but they’re gonna be looking for new fonts. People who are just savvy enough to be tired of Comic Sans will be looking for a new font to “spruce up” their elementary school newsletter, which, in an effort to Love Our Mother (Earth), they now publish exclusively online.

And maybe, just maybe, they’ll stumble across Jeffrey Zeldman’s excellent interview with highly talented David Berlow and think, “Wow, this guy has over 300 fonts! That’s awesome! Where can I download them?” And boy, won’t they be surprised to learn that those 300 fonts can only be used offline. Epic fail.

Dynamic web fonts are coming. Actually they’re already here, but most of Our People haven’t noticed yet. But they will, and that’s going to be a huge boon to somebody. I see you’ve decided that it won’t be you. Well, have fun shuffling your little bits of metal around. The rest of us will be over here, using the only fonts we’re allowed to use: Everything But Yours.

hhgregg: DOA

2009-04-07T15:32:55Z

SR-2020 Mystery Receiver © mysteryreceiver / CC

Jason Scott: Brick and Morte.

But honestly, I don’t know how someone says “the best way for me to sell this stuff is to go into a building, deal with the endless hassles in today’s over-licensed and over-regulated world, get everything working, go into debt, open the door and hope like hell I make more than $3000 in the next 30 days.” Like, now, in the present day.

2009 may not be the best year to complain about over-regulation, but that’s not what I want to talk about.

Two things happened recently at the local strip mall (Beaver Creek Commons): Circuit City closed and hhgregg built a brand new building and opened in it. My father and I like to lightly chatter about business stuff, and Circuit City has been going downhill for a while now. Every single time the topic of Circuit City’s imminent demise comes up in conversation, one of us will interject “NO ONE WILL MISS YOU!” before letting the other finish their sentence.

Still, having never been to an hhgregg, my wife and the boys and I sauntered in after a rare occasion of eating out. As it happens, we need a new very-low-end stereo receiver. Not a whole integrated system, mind you; just something that can sit between our SqueezeBox and the two speakers in our dining room so we can listen to music while eating dinner. This is not a difficult concept.

hhgregg does not sell receivers. Whole integrated systems, yes. Not parts.

But they do sell $10 cables… “on sale” for $100. (But they’re premium!) And $200 DVD players that play… DVDs (not even AVIs). And washing machines — I swear to God, over half the space in this brand spanking new store-in-a-box was devoted to refrigerators and washing machines — ranging from $1200 to $1700. Payment plans available. I asked a salesperson about the difference between a $1200 washing machine and a $1700 washing machine. He glanced furtively at the information card and mumbled, “More bells and whistles.”

Recession-busting pro tip: buy your $10 cables for $10 at Monoprice, buy your electronics at NewEgg, and don’t buy a $1700 washing machine from anyone.

Dive into history, 2009 edition

2009-03-27T21:56:07Z

Again, this is more for my benefit than for yours. If I don’t write this down, I’ll forget it.

Dive Into Python 3 was commissioned in January 2009 by Apress, who published the original Dive Into Python in 2004. Upon agreeing to contract terms, I registered a ten-year lease on diveintopython3.org and immediately published a draft table of contents.

The original DiP was written in DocBook XML. As I’ve mentioned before, I chose DocBook XML because I wanted to learn XML and XSL, and DocBook seemed to be Just The Thing for technical documentation. There was also a bit of self-grandeur involved. I was writing a book For The Ages, so it was important that it be in a Format Of Forever. And in the short term, I could transform The Format Of Forever into useful (but lowly) Output Formats, so I could do unimportant things like publish it online.

For The Ages turned out to be about 10 years. The Format Of Forever is still going strong, but Python itself changed so quickly that it didn’t matter.

Oh, and there was one other little thing that happened between 2000 and 2009: search stopped sucking and took over the web. Kids today may not remember, but it used to be hard to find stuff on the web. Once you found it, you wanted to download it so you could read it offline.

Remember being “offline”?

Anyway, I now realize that there were some hidden assumptions behind my design decisions in 2000. Some of those assumptions turned out to be wrong, or at least not-completely-right. Sure, a lot of people downloaded DiP, but it still pales in comparison to the number of visitors I got from search traffic. In 2000, I fretted about my “home page” and my “navigation aids.” Nobody cares about any of that anymore, and I have nine years of access logs to prove it.

So, I am writing DiP3 in pure HTML and, modulo some lossless minimizations, publishing exactly what I write. This makes the proofreading feedback cycle faster — instead of “building” the HTML output, I just hit Ctrl-R. I expected it to make some things more complicated, but they turn out not to matter very much.

Some examples:

DocBook autonumbers callouts in code blocks. In HTML, if I insert a callout in the middle of a code block, I have to renumber the other callouts manually. This has happened a few times, but it’s not something I do all the time.
When I used DocBook, I used custom XML entities for each code fragment in a code block. (i.e. I show a complete function, then I have separate code blocks to show and talk about individual lines of code.) In HTML, I copy-and-paste. This hasn’t bit me much yet because I haven’t had to refactor much code. Then again, the XML entities technique was also difficult after refactoring, and it required escaping to XML’s weird quoting rules.
DocBook let me “chunk” the HTML output differently — by section, by chapter, or everything in one big file. I published by section online and let people download as one big file. Nobody cared, and I have logs to prove it. Furthermore, the chapter was always the atomic unit — I would take one concept or one piece of code and build lessons around it for an entire chapter. Splitting up a chapter into multiple pages was just annoying for people who came in from search engines and landed in the middle of a chapter wouldn’t immediately understand the context. And everyone comes in from search engines.
Related to the previous point, HTML is not an output format. HTML is The Format. Not The Format Of Forever, but damn if it isn’t The Format Of The Now. Looking back on it now, the HTML output from DocBook was atrociously inefficient. Tables-for-layout for callouts and tips, empty anchors everywhere, and lots of other markup cruft everywhere. Writing my own HTML, I can put in only the markup I need — and it turns out I need very little. Callouts are now ordered lists, tips are now blockquotes, etc. Putting an entire chapter on one page sounds bloated, but consider this: my longest chapter so far would be 75 printed pages, and it loads in under 5 seconds. On dialup.

Furthermore, I am no longer under the illusion that this book will be useful forever. Python will either continue to evolve or it will die; either way, static documentation has a shelf life. Today’s cutting edge code is tomorrow’s mainstream code is next year’s legacy code. DiP’s shelf life was about 10 years. I am supremely confident that the HTML I’m writing today will still be readable 10 years from now, and after that it won’t matter because I’ll have to rewrite the whole damn book anyway.

See you in 2020 for Dive Into Python 4!

Accessibility is a harsh mistress

2009-03-22T01:05:37Z

Sam Ruby:

This blog entry has an [inline SVG] image with a text alternative. Who does it benefit?

Short answer: no one, but you have to do it anyway.

Long answer: As far as I know, none of the commercially available screenreaders support SVG in any way, much less reading the title of an SVG image included inline in an XHTML page (as opposed to, say, linked from the src attribute of an <img> element, or embedded in an <object> element). Nonetheless, you have provided a text alternative for the image, and theoretically, that could be presented to a user in place of (or in addition to) the image. You have therefore fulfilled your moral duty, even though no one actually benefits from it. Welcome to the wacky world of access enablement.

The concept of access enablement is not complicated. In the physical world, it works like this: I build the ramp, you bring the wheelchair. I don’t have to provide you with a wheelchair; it’s up to you to procure one. Nor do I have to teach you how to use your wheelchair to get up my ramp. Nor do I have to push you up the ramp when you arrive. If your wheelchair happens to break at the bottom of my ramp, you can’t sue me for being inaccessible. I did my part: I built the ramp; everything else is Somebody Else’s Problem.

For better or for worse, this concept got translated directly into the virtual world of software. Just as there are standards that define the minimum width and maximum slope of wheelchair-accessible ramps, so too there are standards for building accessible software and authoring accessible content. In the desktop software world, priority #1 is to keep track of the focus. In the web authoring world, it’s to provide text alternatives for any non-text content. The exact techniques vary by medium. For the HTML <img> element, the guidelines say you must provide an alt attribute and (potentially) a longdesc attribute. For SVG, they mandate a <title> child element and (potentially) a <desc> element.

The interesting part is not what the guidelines say, but what they do not say.

No thought of complexity for authors. What are the chances that the author is even qualified to write a long description for a complex graph? Or captions for a video? And if they’re not, how much would it cost to pay someone else to do it? How long would it take for them (or someone they hire) to implement it? Would that delay cause other problems or present other opportunity costs?
No thought of implementation cost for authoring tool vendors. Is it reasonable to expect authors to provide text alternatives to a photo they take with their cellphone and upload to a photo sharing site? Is it reasonable to expect tools to enforce this?
No mention of implementation cost for client software. Can screenreader vendors justify the cost of implementing and maintaining support for rarely-used features, e.g. reading the title of an inline SVG image when only one site in the world actually does that? What about the cost of implementing workarounds for bogus data for popular-but-misused features?
No mention of how end users would learn about the feature.

So here’s the crux of the problem: nowhere in the process of defining an accessibility feature is there any consideration for how often it would be used, how often it would be used correctly, what would happen if it were used incorrectly, how much it would cost to implement it, or how users would learn about the feature. In short, there is no cost-benefit analysis.

Now, some features are simple and easy and popular, so these questions never come up. If enough authors use them and tool vendors implement them and end users learn about them, then everything works. But not every feature is simple or easy or popular; a lot of them are waaaaay down the “long tail” of usage + implementation + education. So far down that, in any other field, you would start talking about the law of diminishing returns. But in accessibility, there is no such limit.

Some concrete examples: most browsers don’t expose information about the access keys available on a page, and most authors don’t define access keys in their pages, and those that do often conflict with other browser, AT, or OS-level shortcuts. Most images aren’t complex enough warrant a long description, and most authors who try to offer a long description get it wrong. But it is just assumed that users who would benefit from them will somehow learn of their existence and be motivated to find software that supports them (assuming they can ever find a page that uses them).

The accessibility orthodoxy does not permit people to question the value of features that are rarely useful and rarely used.

When this orthodoxy collides with reality, the results are both humorous and sad. When I was an accessibility architect at IBM, I assisted in the final stages of ensuring that Eclipse’s Graphical Editing Framework was fully accessible to blind people. This involved ensuring that all possible objects were focusable, all possible actions were keyboard-accessible (including drag-and-drop), and all possible information about nodes and connectors was exposed to third-party assistive technologies via MSAA. It was mind-numbing work, full of strange edge cases and bizarre hypothetical situations, not unlike the one Sam is struggling to understand. During one particularly difficult teleconference, an Eclipse developer muttered something like, “You realize no one is ever actually going to do any of this, right?” There was an awkward silence as the people who had spent their lives in the trenches of access enablement contemplated the very real possibility that no one would ever benefit from their work.

Back to Sam’s question. Few authors publish in true XHTML mode, fewer still include inline SVG images in their XHTML, and fewer still include titles or descriptions in those images. But in theory, you can imagine a situation where a web author publishes in true XHTML mode, and the author includes an inline SVG image within an XHTML page, and an end user is using a browser that supports true XHTML, and that user is using a hypothetical screenreader-of-the-future that implements support for the <title> and <desc> elements within inline SVG images within XHTML pages, and that user stumbles across that page. It’s theoretically possible, therefore you have to do it. Period. End of discussion.

Now go retrofit text alternatives into every SVG image you’ve ever published, or an accessibility advocacy group who has never visited your site will sue you on behalf of all the users you’ve been disenfranchising. All zero of them.

If it fails for some, it should fail for all

2009-03-18T22:47:43Z

On the topic of <canvas> accessibility, John Foliot writes:

Finally, I propose that any instance of <canvas> that lacks at a minimum the 2 proposed mandatory values be non-conformant and not render on screen.

When pressed for an explanation, John continues:

Actually, yes, I have proposed this form of draconian response before.

It’s about consequences: until such time as there are real consequences for slack developers/tools that allows content to exist that is incomplete, then there will be content that is incomplete – it’s a simple as that. Why would <img src="path..." /> be any more complete than <img alt="Photo of a leprechaun" />? I mean, clearly, anyone processing that info in their user-agent will ‘get’ the intent of the author, right? Yet today, the first example will render in the browser, the second delivers a ‘fail’. Ergo (to me) there is a problem of inequity here that must be addressed – if it fails for some, it should fail for all.

If it fails for some, it should fail for all.

John also believes that Flickr is (or at least should be) illegal because it allows people to publish inaccessible content. I’ll pause for a moment and let that sink in.

.
.
.

It’s important to understand just how extreme these views are, even within the accessibility community. I was a professional accessibility architect for several years; before that, I took an intense interest in web accessibility; long before that, I was a relay operator for AT&T. A few years ago, I had the pleasure of attending the annual CSUN accessibility conference, where I helped staff the Mozilla booth and talked about Firefox’s accessibility features to anyone who would listen. So I have had the opportunity to speak to a great many people who cared about accessibility, authored accessible content, wrote accessible software, designed accessible hardware, and provided accessibility services.

In all that time, in all those conversations, with all those people, I have never heard anyone say, “Seriously, you know what we should do to make the world more accessible? Fuck over all the sighted people.”

I know that most of the people who care about accessibility do not have the time or resources to follow the daily machinations of the HTML 5 working groups. That’s fine; standards take a long time and require a lot of attention, and most people have day jobs somewhere else. But I have observed a kind of “conventional wisdom” taking hold in this wider community — that the HTML working group doesn’t care about accessibility, that any and all proposals are rejected, that the views of “experts” are simply dismissed out of hand.

I think it would be wise for people who truly care about accessibility to take a closer look at the so-called “experts” who are participating on their behalf, and to understand exactly what these people are proposing. It’s true that some of their proposals have not been adopted, but it’s not because some cartoonishly monocled villain enjoys being mean to them. It’s because the proposals are insane.

Immersed in the Googleplex ball pit

2009-01-11T21:29:04Z

A gentle introduction to video encoding, part 5: constraints

2009-01-09T02:23:31Z

[Part of an ongoing series.]

I had lunch with my father the other day, and I explained this series as well as I could to someone who didn’t start programming when he was 11. His immediate reaction was, “Why are there so many different formats? Why can’t everybody just agree on a single format? It is political, or technical, or both?” The short answer is, it’s both. The history of video in any medium — and especially since the explosion of amateur digital video — has been marred by a string of companies who wanted to use container formats and video codecs as tools to lock content producers and content consumers into their little fiefdoms. Own the format, own the future. And when I say “history” — well, it’s still going on. Tried to play a Windows Media Video on Mac OS X lately? The codec and container support is out there, but it’s not baked in. Want to watch movie trailers on Apple.com? Please install QuickTime. And so forth and so on. The only thing that was pre-installed on both platforms was Flash, so when a few startups dipped their toes into the Internet video waters, the ones that used Flash Video won despite it being an objectively inferior codec. (Some revision of Flash 9 added support for H.264 video, AAC audio, and the MP4 container, which is what YouTube HD uses.)

So that’s the politics. But there are also technical barriers. As with all engineering, video encoding is primarily about constraints. I can think of 10 just off the top of my head:

CPU capacity for decoding and playing in real time. This is one of the most important constraints, since video is meant to be watched in real time. That sounds simple, but it’s incredibly complex. Every video you’ve ever watched in your entire life had to be decoded and played in real time. Otherwise it stutters and the viewing experience sucks. And we’re talking about video here; if the viewing experience sucks, there’s nothing left. Some codecs are just more complex than others, and that translates into higher system requirements to decode videos in real time. As I’ve mentioned before, some codecs are now decoded by specialized hardware. iPhones have a little chip inside them that understands H.264 Baseline Profile; without that, the iPhone would need a Core 2 Duo processor to play movies, and it would have a battery life of 10 minutes.
Codec compatibility. Normal people won’t download codecs or plug-ins just to watch a dog on a skateboard, or even to watch a trailer for a $100 million blockbuster. (Sadly, they will download plug-ins for porn, but those are invariably trojan horses. Or so I’ve read. Moving on…) The phone in your pocket can probably play AMR ringtones, maybe MP3 ringtones, but probably not Vorbis ringtones (unless you have an Android phone) — and you probably couldn’t download new codecs even if you wanted to (which, I must reiterate, nobody wants to). Apple and Real Networks tried for years to corner the web video market, but 99% of schmucks with a browser have Flash, so Flash video won on the web. Meanwhile, Firefox 3.1 will ship with support for the <video> element but will only support Theora and Vorbis in an Ogg container — even if your underlying operating system ships with other codecs.
CPU capacity for encoding. Encoding takes a long time. Taking my home movie from iMovie to a DVD used to take 8 hours on a Powerbook G4 laptop. These days you can rip a DVD movie with Xvid in 30 minutes, or you can rip it with a more complex codec with all optional features turned on, and maybe it’ll still take 8 hours. It’ll look better, but will it look 16 times better? If you’re only doing it once, maybe you don’t care. If you’re running YouTube and people are uploading 13 hours of video every minute, maybe you do. CPU cycles aren’t free; at that scale, they’re not even cheap. (That’s a real statistic, by the way; I got it from the page on the Google intranet entitled “What can we tell non-Googlers?” and it’s accurate as of September 2008.)
Acceptable delay between recording and delivery. In my own experience, videos I’ve uploaded on YouTube are available within minutes, which is just mind-boggling when you consider the volume. If you’re re-encoding a live stream, even a few minutes delay is probably unacceptable. That means you’ll need a faster encoder, a less complex codec, or lower quality settings.
Audience size. It’s not a big secret that lots of video on the Internet looks like crap. Partly that’s because the video uploader uploaded crappy video, but it’s also because most Internet videos are only watched by a few people, and it’s just not a worthwhile tradeoff to spend 8 hours re-encoding it. On the other hand, if you’re mastering a DVD that’ll get sold to 10 million people, you’ll probably use higher quality settings.
Screen dimensions. DVDs can’t store high-def 1920 x 1080 video because the standard doesn’t allow for it, which makes perfect sense because it was designed around the screen resolution of standard-def TVs. Blu-Ray ups the limit, but there’s still a limit. Screen sizes vary more for PC video, but there will always be practical upper limits depending on your audience.
My bandwidth. If you’re streaming or downloading video, some percentage of your audience is probably living in a third-world country like the United States, with limited broadband access, slow speeds, and monthly bandwidth caps. Larger file size = longer wait to play = fewer videos watched overall.
Your bandwidth. Obviously every bit I download is a bit that you upload, and bandwidth ain’t free either. “When I get a little money I buy bandwidth; and if any is left I buy food and clothes.” Or something like that.
Hard limits on storage size. As I mentioned before, physical media has upper limits on total size. Commercial DVDs can hold upwards of 9 GB, which seems like a lot but really isn’t. Blu-Ray maxes out at 50 GB, which seems like a lot but really isn’t.
Patents / licensing costs. Did I mention that most popular video codecs are patent-encumbered? This is why Wikimedia uses Theora exclusively, and why Firefox can ship a native Theora decoder and but won’t ever ship H.264.

…and that’s the short list.

All of which leads me to the Zen of video encoding, which is this:

There is no right or wrong. There is only what works and what doesn’t.

If you can find even one combination of tools, delivery devices, and target platforms that satisfies your constraints and still accomplishes your goals, congratulations. You’re ahead of 99% of the people who’ve tried.

Tomorrow: specifics on common devices/platforms and their limitations.

A gentle introduction to video encoding: the slides

2009-01-09T20:46:49Z

My tech talk went extremely well. Lots of people, lots of great questions. I can’t share the video because we talked about some internal Youtube stuff, but here are the slides:

Download as OpenDocument presentation

A gentle introduction to video encoding, part 4: captioning

2009-01-07T22:18:50Z

[Part of an ongoing series.]

The first thing you need to know about captions and subtitles is that captions and subtitles are different. The second thing you need to know about captions and subtitles is that you can safely ignore the differences unless you’re creating your own from scratch. I’m going to use the terms interchangeably throughout this article, which will probably drive you crazy if you happen to know and care about the difference.

Historically, captioning has been driven by the needs of deaf and hearing impaired consumers, and captioning technology has been designed around the technical quirks of broadcast television. In the United States, so-called “closed captions” are embedded into a part of the NTSC video source (”Line 21″) that is normally outside the viewing area on televisions. In Europe, they use a completely different system that is embeddable in the PAL video source. Over time, each new medium (VHS, DVD, and now online digital video) has dealt a blow to the accessibility gains of the previous medium. For example:

PAL VHS tapes did not have enough bandwidth to store closed captions at all.
DVDs have the technical capability, but producers often manage to screw it up anyway; e.g. DVDs of low-budget television shows are often released without the closed captions that accompanied the original broadcast.
HDMI cables drop “Line 21″ closed captions altogether. If you play an NTSC DVD on an HDTV over HDMI, you’ll never see the closed captions, even if the DVD has them.

And accessible online video is just fucking hopeless. (And no, it won’t change unless new regulation forces it to change. When it comes to captioning, Joe Clark has been right longer than many of you have been alive.)

So even in broadcast television, captioning technology was fractured by different broadcast technologies in different countries. Digital video had the capability of unifying the technologies and learning from their mistakes. Of course, exactly the opposite happened. Early caption formats split along company lines; each major video software platform (RealPlayer, QuickTime, Windows Media, Adobe Flash) implemented captioning in their own way, with levels of adoption ranging from nil to zilch. At the same time, an entire subculture developed around “fan-subbing,” i.e. using captioning technology to provide translations of foreign language videos. For example, non-Japanese-speaking consumers wanted to watch Japanese anime films, so amateur translators stepped up to publish their own English captions that could be overlaid onto the original film. In the 1980s, fansubbers would actually take VHS tapes and overlay the English captions onto a new tape, which they would then (illegally) distribute. Nowadays, translators can simply publish their work on the Internet as a standalone file. English-speaking consumers can have their DVDs shipped directly from Japan, and they use software players that can overlay standalone English caption files while playing their Japanese-only DVDs. The legality of distributing these unofficial translations (even separately, in the form of standalone caption files) has been disputed in recent years, but the fansubbing community persists.

Technically, there is a lot of variation in captioning formats. At their core, captions are a combination of text to display, start and end times to display it, information about where to position the text on a screen, fonts, styling, alignment, and so on. Some captions roll up from the bottom of the screen, others simply appear and disappear at the appropriate time. Some caption formats mandate where each caption should be placed and how it should be styled; others merely suggest position and styling; others leave all display attributes entirely up to the player. Almost every conceivable combination of these variables has been tried. Some forms of media try multiple combinations at once. DVDs, for example, can have two entirely distinct forms of captioning — closed captioning (as used in NTSC broadcast television) embedded in the video stream, and one or more subtitle tracks. DVD subtitle tracks are used for many different things, including subtitles (just the words being spoken, in the same language as the audio), captions for the hearing impaired (which include extra notations of background noises and such), translations into other languages, and director’s commentary. Oh, and they’re stored on the DVD as images, not text, so the end user has no control over fonts or font size.

Beyond DVDs, most caption formats store the captions as text, which inevitably raises the issue of character encoding. Some caption formats explicitly specify the character encoding, others only allow UTF-8, others don’t specify any encoding at all. On the player side, most players respect the character encoding if present (but may only support specific encodings); in its absence, some players assume UTF-8, some guess the encoding, and some allow the user to override the encoding. Obviously standalone caption files can be in any format, but if you want to embed your captions as a track within a video container, your choices are limited to the caption formats that the video container supports.

And remember when I said that there were a metric fuck-ton of audio codecs? Forget that. There are an imperial fuck-ton of caption formats (i.e. multiply by 9/5 and add 32). Here is a partial list of caption formats, taken from the list of formats supported by Subtitle Workshop, which I used to caption my short-lived video podcast series:

Adobe Encore DVD, Advanced SubStation Alpha, AQTitle, Captions 32, Captions DAT, Captions DAT Text, Captions Inc., Cheetah, CPC-600, DKS Subtitle Format, DVD Junior, DVD Studio Pro, DVD Subtitle System, DVDSubtitle, FAB Subtitler, IAuthor Script, Inscriber CG, JACOSub 2.7+, Karaoke Lyrics LRC, Karaoke Lyrics VKT, KoalaPlayer, MacSUB, MicroDVD, MPlayer, MPlayer2, MPSub, OVR Script, Panimator, Philips SVCD Designer, Phoenix Japanimation Society, Pinnacle Impression, PowerDivX, PowerPixel, QuickTime Text, RealTime, SAMI Captioning, Sasami Script, SBT, Sofni, Softitler RTF, SonicDVD Creator, Sonic Scenarist, Spruce DVDMaestro, Spruce Subtitle File, Stream SubText Player, Stream SubText Script, SubCreator 1.x, SubRip, SubSonic, SubStation Alpha, SubViewer 1.0, SubViewer 2.0, TMPlayer, Turbo Titler, Ulead DVD Workshop 2.0, ViPlay Subtitle File, ZeroG.

Which of these formats are important? The answer will depend on whom you ask, and more specifically, how you’re planning to distribute your video. This series is primarily focused on videos delivered as files to be played on PCs or other computing devices, so my choices here will reflect that. These are some of the most well-supported caption formats:

SubRip
SubStation Alpha
MPEG-4 Timed Text
SAMI
SMIL

SubRip

SubRip is the AVI of caption formats, in the sense that its basic functionality is supported everywhere but various people have tried to extend it in mostly incompatible ways and the result is a huge mess. As a standalone file, SubRip captions are most commonly seen with a .srt extension. SubRip is a text-based format which can include font, size, and position information, as well as a limited set of HTML formatting tags, although most of these features are poorly supported. Its “official” specification is a doom9 forum post from 2004. Most players assume that .srt files are encoded in Windows-1252 (what Windows programs frequently call “ANSI”), although some can detect and switch to UTF-8 encoding automatically.

Because .srt files are so often published separately from the video files they describe, the most common use case is to put your .srt file in the same directory as your video file and give them the same name (up to the file extensions). But it is also possible to embed SubRip captions directly into AVI files with AVI-Mux GUI, into MKV files with mkvmerge, and into MP4 files with MP4Box.

You can play SubRip captions in Windows Media Player or other DirectShow-based video players after installing VSFilter; in QuickTime after installing Perian; on Linux, both mplayer and VLC support it natively.

SubStation Alpha

SubStation Alpha and its successor, Advanced SubStation Alpha, are the preferred caption formats of the fansubbing community. As standalone files, they are commonly seen with .ssa or .ass extensions. They have a spec longer than three paragraphs. They are actually miniature scripting languages. A .ass file contains a series of commands to control position, scrolling, animation, font, size, scaling, letter spacing, borders, text outline, text shadow, alignment, and so on; and a series of time-coded events for displaying text given the current styling parameters. It has support for multiple character encodings.

The playing requirements for SubStation Alpha captions are almost identical to SubRip. The same plugins are required for Windows and Mac OS X. On Linux, mplayer prides itself on having the most complete SSA/ASS implementation.

MPEG-4 Timed Text

a.k.a. “MPEG-4 Part 17,” a.k.a. ISO 14496-17, MPEG-4 Timed Text (hereafter “MP4TT”) is the one and only caption format for the MP4 container. It is not a file format; it is only defined in terms of a track within an MP4 container. As such, it can not be embedded in any other video container, and it can not exist as a separate file. (Note: the last sentence was a lie; the MPEG-4 Timed Text format is really the 3GPP Timed Text format, and it can very much be embedded in a 3GPP container. What I meant to say is that the format can not be embedded in any of the other popular video container formats like AVI, MKV, or OGG. I could go on about the subtle differences between MPEG-4 Timed Text in an MP4 container and 3GPP Timed Text in a 3GPP container, but it would just make you cry, and besides, technical accuracy is for pussies.)

MP4TT defines detailed information on text positioning, fonts, styles, scrolling, and text justification. These details are encoded into the track at authoring time, and can not be changed by the end user’s video player. The most readable description of its features is actually the documentation for GPAC, an open source implementation of much of the MPEG-4 specification (including MP4TT). Since MP4TT doesn’t define a text-based serialization, GPAC invented one for their own use; since their format is designed to capture all the possible information in an MP4TT track, it turns out to be an easy way to read about all of MP4TT’s features.

MP4Box, part of the GPAC project, can take an .srt file and convert it into a MPEG-4 Timed Text track and embed it in an existing MP4 file. It can also reverse the process — extract a Timed Text track from an MP4 file and output a .srt file.

On Mac OS X, QuickTime supports MP4TT tracks within an MP4 container, but only if you rename the file from .mp4 to .3gp or .m4v. I shit you not. (On the plus side, changing the file extension will allow you to sync compatible video to an iPod or iPhone, which will actually display the captions. Still not kidding.) On Windows, any DirectShow-based video player (such as Windows Media Player or Media Player Classic) supports MP4TT tracks once you install Haali Media Splitter. On Linux, VLC has supported MP4TT tracks for several years.

SAMI

SAMI was Microsoft’s first attempt to create a captioning format for PC video files (as opposed to broadcast television or DVDs). As such, it is natively supported by Microsoft video players, including Windows Media Player, without the need for third-party plugins. It has a specification on MSDN. It is a text-based format that supports a large subset of HTML formatting tags. SAMI captions are almost always embedded in an ASF container, along with Windows Media video and Windows Media audio.

Don’t use SAMI for new projects; it has been superceded by SMIL. For historical purposes, you may enjoy reading about creating SAMI captions and embedding them in an ASF container, as long as you promise to never, ever try it at home.

SMIL

SMIL (Synchronized Multimedia Integration Language) is not actually a captioning format. It is “an XML-based language that allows authors to write interactive multimedia presentations.” It also happens to have a timing and synchronization module that can, in theory, be used to display text on a series of moving pictures. That is to say, if you think of SMIL as a way to provide captions for a video, you’re doing it wrong. You need to invert your thinking — your video and your captions are each merely components of a SMIL presentation. SMIL captions are not embedded into a video container; the video and its captions are referenced from a SMIL document.

SMIL is a W3C standard; the most recent revision, SMIL 3.0, was just published in December 2008. If you printed out the SMIL 3.0 specification on US-Letter-sized paper, it would weigh in at 395 pages. So don’t do that.

QuickTime supports a subset of SMIL 1.0. WebAIM provides a nice tutorial on using SMIL to add captions to a QuickTime movie.

dive into mark

Why do we have an IMG element?

Thank you for giving me the opportunity to explain this to you

Translation From MS-Speak to English of Selected Portions of Tony Ross’ “Distributed Extensibility Submission”

Recently

Open for your input

This is the house

Fuck the foundries

hhgregg: DOA

Dive into history, 2009 edition

Accessibility is a harsh mistress

If it fails for some, it should fail for all

Immersed in the Googleplex ball pit

A gentle introduction to video encoding, part 5: constraints

A gentle introduction to video encoding: the slides

A gentle introduction to video encoding, part 4: captioning

SubRip

SubStation Alpha

MPEG-4 Timed Text

SAMI

SMIL

Further reading