^* All tests were run under a load of 10 concurrent threads.

Resources

• Getting FastCGI setup on Dreamhost - Jay Allen
• Running Movable Type on OS X with FastCGI - Neil Turner

The Open Media Profile for Open Search specification was written for people and companies wishing to integrate their content into Vox and outlines the components and protocols necessary to achieve a successful integration.

This is a specification for Vox’s Open Media Profile for Open Search.

Originally authored by Tatsuhiko Miyagawa, edited by Byrne Reese.

What your API should implement

What information do we need to integrate your service to Vox? Here’s the brief list:

• The URL endpoint of API (e.g. http://yourhost.example.com/api/search)
• The format of API request and responses (e.g. URL parameters as request and Atom/1.0 response)
• Which media data you’d like us to integrate on Vox (e.g. Photo, Video, Book, Audio)
• What query parameter is needed to search for specific terms
• Metadata of each item found: Title, URL, Thumbnail URL, width, height, file size etc.

We have chosen to utilize existing open protocols and standards, OpenSearch 1.1 with Atom 1.0, as we feel they provide the most flexibility. They also have plenty of existing tools available to developer to ease the process of integration.

OpenSearch Description Document

The first step is to author an XML document that specifies the URL that will act as the endpoint for the OpenSearch API protocol. You might optionally specify names of the parameters and response format. Here’s an example:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
  <ShortName>Video Search</ShortName>
  <Description>Use Example.com to search the Video.</Description>
  <Tags>video foo</Tags>
  <Contact>admin@example.com</Contact>
  <Url type="application/atom+xml"
       template="http://example.com/?q={searchTerms}&amp;pw={startPage?}&amp;format=atom"/>
  <OutputEncoding>UTF-8</OutputEncoding>
  <InputEncoding>UTF-8</InputEncoding>
</OpenSearchDescription>

Save this XML and publish it so that it is accessible via a publicly accessible URL, like ”http://example.com/opensearch.xml”, and make sure it declares a MIME type of ”application/opensearchdescription+xml”. Technically speaking, Vox will ignore the MIME type, but we consider it important to adhere to the requirements of the OpenSearch API and encourage you to setup the MIME type anyway.

Search API

Once this Open Search document has been published we’ll be able to introspect your interface in order to query your endpoint. The query is accomplished by sending an HTTP GET request to the URI specified in the OpenSearch description document above. Using the example above, the URL we would query would be ”http://example.com/?q=Vox&pw=1&format=atom”, if we use ”Vox” as a search query. Your application searches the assets (e.g. Video) matching with the query (”Vox”) and returns the response in Atom 1.0 format.

Example Search Response

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
    xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
    xmlns:media="http://search.yahoo.com/mrss/">
  <title>Example.com Search: Vox</title>
  <link href="http://example.com/search/Vox" />
  <updated>2003-12-13T18:30:02Z</updated>
  <author>
    <name>Example.com, Inc.</name>
  </author>
  <id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>
  <opensearch:totalResults>4230000</opensearch:totalResults>
  <opensearch:startIndex>1</opensearch:startIndex>
  <opensearch:itemsPerPage>10</opensearch:itemsPerPage>
  <opensearch:Query role="request" searchTerms="Vox" startPage="1" />
  <link rel="alternate" href="http://example.com/search/Vox" type="text/html"/>
  <link rel="self" href="http://example.com/?q=Vox&amp;pw=1&amp;format=atom" type="application/atom+xml"/>
  <link rel="search" type="application/opensearchdescription+xml" href="http://example.com/opensearch.xml"/>
  <entry>
    <title>Foo bar baz</title>
    <link href="http://example.com/show/video/123" />
    <link rel="alternate" type="application/atom+xml" href="http://example.com/atom/123" />
    <id>tag:example.com,2006:video-123</id>
    <updated>2003-12-13T18:30:02Z</updated>
    <content type="text">
      Vox rocks blah blah blah ...
    </content>
    <category term="Vox" scheme="http://example.com/tags/Vox/" label="Vox" />
    <category term="cat" scheme="http://example.com/tags/Cat/" label="cat" />
    <link rel="license" type="text/html" href="http://creativecommons.org/licenses/by/2.5/" />
    <media:content url="http://example.com/data/123.flv" fileSize="123456" type="video/x-flv" />
    <media:player url="http://example.com/data/123.swf" height="200" width="400" />
    <media:thumbnail url="http://example.com/thumb/1223.jpg" width="75" height="50" />
  </entry>
  <entry>
  ...
  </entry>
</feed>

That is a lot of XML to digest. So let’s step through the document looking at it a little bit at a time.

<opensearch:totalResults>4230000</opensearch:totalResults>
<opensearch:startIndex>1</opensearch:startIndex>
<opensearch:itemsPerPage>10</opensearch:itemsPerPage>
<opensearch:Query role="request" searchTerms="Vox" startPage="1" />

This says the search for ”Vox” might have 423,000 results in total, and the current response is returning the first page of the search results with 10 items per page. It also describes what the current request is. We’d need these information to paginate to the next page (with ”pw=2” parameter set) if users want to get more data.

<link rel="alternate" href="http://example.com/search/Vox" type="text/html"/>

You can optionally have a link to the HTML version of the search result. Currently it’s ignored on Vox.

<link rel="self" href="http://example.com/?q=Vox&amp;pw=1&amp;format=atom" type="application/atom+xml"/>
<link rel="search" type="application/opensearchdescription+xml" href="http://example.com/opensearch.xml"/>

These link tags declare the current URL of the response itself, and the URL of the original OpenSearch description document. We’re currently ignoring these data but would like to use them for a sanity check in the future.

<entry>
...
</entry>

The <entry> element describes each item found in your database. You can have multiple entries in one feed so be sure that you set the opensearch:itemsPerPage element as described above.

<title>Foo bar baz</title>
<link href="http://example.com/show/video/123" />

The title and link (HTML version) of the item are essential. We use these fields as the name of the asset as it appears within their Vox Library (under the Organize tab). If it has a link element we’ll add a link to the original site on the asset details page.

<link rel="alternate" type="application/atom+xml" href="http://example.com/atom/123" />

This link element represents the ”alternate” form of the entry, as Atom format. It is critical to have a representation of every item within your search results, as it allows Vox to get more specific information about any specific item in the search results. In the example URL above (”http://example.com/atom/123”) Vox will be able to retrieve a document that looks similar to the following (as defined by the Atom Publishing Protocol):

<?xml version="1.0" encoding="UTF-8"?>
<entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:media="http://search.yahoo.com/mrss/">
  <title>Foo bar baz</title>
  <link href="http://example.com/show/video/123" />
  <link rel="alternate" type="application/atom+xml" href="http://example.com/atom/123" />
  <id>tag:example.com,2006:video-123</id>
  <updated>2003-12-13T18:30:02Z</updated>
  ...
</entry>

The URL form of the alternative Atom representation would be arbitrary. It could be something like ”http://example.com/atom?id=123” or whatever, as far as we can programatically access via link rel=”alternate” element.

<id>tag:example.com,2006:video-123</id>

atom:id is another very important element under the entry. This needs to be something unique in your system. This examle uses Tag URI (RFC 4151) to indicate that this video has the ID 123 in your site. You can also use the permanent link of the HTML version, like

<id>http://example.com/video/123</id>

That is of course if you can guarantee that the URL is unique and will always refer to the same object. Vox parses this id element to associate a unique and permanent identifier for the asset within your system. Vox will then be able to detect duplicates in the user’s library, and to retrieve rich metadata about the asset at a later point in time.

<updated>2003-12-13T18:30:02Z</updated>

This shows when this asset was last modified in a significant way. You can also refer to the <published> element to determine when the asset was created.

<content type="text">
  Vox rocks blah blah blah ...
</content>

The text content of the asset.

<category term="Vox" scheme="http://example.com/tags/Vox/" label="Vox" />
<category term="cat" scheme="http://example.com/tags/Cat/" label="cat" />

The <category> elements are used to list the ‘tags’ associated with the asset.

<link rel="license" type="text/html" href="http://creativecommons.org/licenses/by/2.5/" />

You can optionally declare the license of the item using the corresponding Creative Commons URI.

<media:content url="http://example.com/data/123.flv" fileSize="123456" type="video/x-flv" />

Elements prefixed with the “media” namespace/prefix are part of the Media RSS components of the feed. The <media:content> element is used to indicate the URL at which the original content associated with the asset can be found. In this example the URL is a Flash Video file. We’re not currently downloading Video data like this on Vox, but we are doing it for image files if you’re providing Photo assets. This is necessary for us to create a better thumbnail on our end, without linking photos directly to your site.

<media:player url="http://example.com/data/123.swf" height="200" width="400" />

The <media:player> element specifies the embeddable Flash player URL, if you’re providing Videos or Audio assets.

<media:thumbnail url="http://example.com/thumb/1223.jpg" width="75" height="50" />

The <media:thumbnail> element specifies the thumbnail URL we show to the user when a search is conducted. The width and height attributes express the width and height for those thumbnail images.

References

• OpenSearch 1.1
• RFC 4287 The Atom Syndication Format
• RFC 4151 Tag URI
• Google GData API Protocol
• Media RSS
• Atom License Extension

Of course I use the term "hack" loosely here because there is nothing hackish about it - the technique utilizes completely kosher plugin API calls introduced in Movable Type 3.3.

And insert this at the bottom of the same file:

But then I saw the code samples at Yahoo's User Interface Library. Not only did their code samples have syntax highlighting, but they also included:

• line numbers
• alternating background colors for each line of code
• a facility to print the code sample
• the ability to quickly view the raw text of the code

Curious as to how they may have achieved these effects I discovered that the code sample "widget" (for lack of a better term) was the product of an open source Javascript Syntax Highllighter utility. Not only that, but using the tool was dead simple.

All one has to do is wrap a block of code in a textarea HTML element according to the tools instructions, and then run a javascript function will convert the text within the textarea into a nicely formated, fully functional, highlighted block of code. Let's see it in action.

Here is how I used to publish code samples online: I would take my text, escape special characters, and then wrap the code in a <pre> tag and a <code> tag. Then with a little CSS I would style the block of text like so:

#!/usr/bin/perl
# This is a test script written in Perl
my $who = "World";
if (@ARGV) {
  $who = $ARGV[0];
}
print "Hello $who!\n";

Here is the same code sample wrapped in a textarea element and processed through the Syntax Highlighter:

The HTML used to produce this output is as follows:

It didn't take long for me to apply this style across all the code samples on the developer site. And not just code samples, I also used to make it easier to read HTML and XML fragments as well because the utility is capable of highlighting a number of languages and markup.

It is an excellent tool and one I plan to use more of in more of my projects. It just makes writing about code easier and it makes the code I write about prettier. What's not to like?

So how do we keep all of our customers and partners up to date in a way that is cost effective, scalable, easy to implement (for everyone) and easy to support?

So we did what FedEx did, we centralized our content distribution system by creating the “Six Apart Update Stream.” Six Apart Updates allows anyone to connect to steady and endless stream of content generated by our users. The content of the stream is broadcast using the Atom Syndication Format so that subscribers can easily parse out and listen for the content that is of interest to them.

But be warned, we have often referred internally to this “stream” as a “fire hose” because there is still a tremendous amount of data that is broadcast through this service. But despite the torrent you can at least rest assured that the stream is extraordinarily unlikely to overwhelm your system. Why? Because as soon as your client falls behind in reading the stream’s contents, the service will stop transmitting to your client until it can catch up; and as soon as it does the service will indicate to your client how many posts they missed.

What follows is a simple Perl script, that illustrates how easy it is connect and consume the stream. It utilizes Miyagawa’s XML::Atom::Stream perl module to connect to the stream and display all the titles of all the posts created by Six Apart users and customers.

Note: We would love to build a library of clients capable of consuming this stream, so if you have written a client in .NET, PHP, Ruby, Java, or any other language, please share it with us. We would love to link to it and share it with others.

Resources

• Six Apart Updates Homepage

Clients

• Perl
• Python

I emailed with Nick about this new module briefly and asked him a few questions. Nick is in email, as he is person: to the point.

Because people may not be familiar with you, what tags would you use to describe yourself?

married, software engineer, perl hacker, blogger, cocoa developer

I developed the interface that allows us to communicate with FeedBurner in a way which enables our users to effortlessly use both FeedBurner and TypePad together. Like most of the people that I work with, I’ve always been a big supporter of giving back to the community. Creating a library that other developers could use just felt like a good idea.

Write as little code as possible and keep the footprint small. When modules like this are used in larger applications I try to keep things simple for the sake of the developer using it. I also have a tendancy to write test code first. It allows me to plan the behavior and logic beforehand.

Immediately on the TODO is to add support for the FeedBurner Awareness API. I think being able to get statistical information on your feeds can go a long way. Everybody likes to see the numbers, or at least know they are they if they did want to see them.

I don’t know of anyone outside of Six Apart using this module, but I hope I’ve made it easy to integrate.

The test suite ‘Net-FeedBurner/t/*’ covers most all of the common tasks performed with the API. Use it as a quick reference.

Resources

• Net::FeedBurner on CPAN
• Nick Gerakines’ Homepage
• FeedBurner’s Homepage
• FeedBurner’s Feed Management API Reference