metajack.im

Strophe Tutorial: Pubsub Weather Alert Demo

2009-08-27T10:08:00Z

Strophe has been getting more popular lately, and I keep finding it in new places. Boris Okner has just written an interesting demo application showing off the power of the real-time Web and Strophe. He didn't just stop with the demo, either; he has provided a detailed walk-through of how it all works as well.

The demo is an imaginary system for weather alerting that allows users to subscribe to various weather stations and receive updates in real-time. It uses Strophe and JavaScript for the front end and ejabberd and some custom Erlang modules for the back end. The XMPP API exposed by the server is similar to standard XMPP pubsub, which can be browsed via service discovery.

I also really like Boris's description of XMPP's advantages to tackling this problem:

Another important thing is ability to reuse XMPP server features. For instance, there is no need to have separate database of registered users and their subscriptions, as this is already maintained by XMPP server. All changes in user's presence (i.e. logged in/out, disconnected) and component presence (restart, disconnection etc.) are also taken care of. It is possible to easily implement IM features, such as live support or chat between users.

I'd add that if he had used standard pubsub, there is very little protocol design work that needs to be done for this type of application. Pubsub and service discovery already have all the standard usage models covered as well as many error cases. It's a lot easier to implement a well thought out protocol than to design your own and re-discover all the non-obvious issues.

Give Boris's demo application a spin if you're interested in what is possible with the real-time Web and how it works.

JavaScript Function Tricks for Making Callbacks Better

2009-08-06T21:02:00Z

Have you ever needed to pass an extra parameter to a library callback? Perhaps you were confused the first time your method callback was invoked but didn't have the right this pointer. Someone recently e-mailed me asking how to do this in Strophe, and I figured it was time to finally write up the techniques I use to solve these problems.

In these days of ever-larger JavaScript applications, callbacks have become extremely common and important. Unfortunately, JavaScript doesn't have bound methods like Python and not every JavaScript library allows passing arbitrary state to callbacks, so getting the right state in callbacks can be irksome. It doesn't have to be this way though; JavaScript has plenty of primitives available to build tools that make callbacks easier.

Bound Methods

In JavaScript, if you pass a method to a callback, that object that the method belongs to is lost. The this pointer will be the function object itself, not the actual object that it would be if you called obj.foo(). You can make your own bound methods, though, and some JavaScript implementations, like Firefox, even have this built in as Function.prototype.bind().

The bind() method takes the function object it is called on, and returns a new function object. The new function will invoke the original function with this set to the parameter passed to bind().

For example, to create a Python style bound method, you'd write:

var bound_method = some_obj.some_func.bind(some_obj);

Now when bound_method() is invoked, some_func() will have access to some_obj through the this pointer, exactly as if it had been invoked as some_obj.some_func().

This is extremely useful because your code can store state in the object as normal, and not have to worry about passing it along through the callback chain. When your method is invoked, it will know exactly where to find the state it needs.

Prepending Arguments

A lot of frameworks pass a well known set of parameters to callback functions, but sometimes, you need a little extra piece of information. Wouldn't it be nice if you could create a callback that prepended some arguments to the libraries normal parameters? You can!

Here's an example using Strophe:



// non-standard stanza handler which takes an extra argument at the beginning
function on_message(response, message) {
    var m = $msg({to: message.getAttribute('from'), type: 'chat'})
                .c('body').text(response);
    connection.send(m);
}

// register the handler for different cases
connection.addHandler(on_message.prependArg('Hi!'), null, "message");
connection.addHandler(on_message.prependArg('Hola'), null, "message");

Two <message> stanza handlers are registered for the XMPP connection; the first responds with "Hi!" and the second with "Hola!". prependArg() is used to add an extra argument at the beginning of on_message()'s argument list.

The code for prependArg() appears below.



Function.prototype.prependArg = function (arg) {
    var func = this;

    return function () {
        var newargs = [arg];
        for (var i = 0; i < arguments.length; i++)
            newargs.push(arguments[i]);
        return func.apply(this, newargs);
    };
};

Strophe uses both bind() and prependArg() to make sure that XMLHttpRequest events are sent along with the proper state since hte XHR implemenations don't allow passing along arbitrary data with callbacks. I found found bind() to be extremely useful when trying to build large JavaScript projects, as bound methods can make a lot of tasks much easier.

Please leave a comment if you have a favorite function manipulation trick in JavaScript!

Updated ejabberd-snapshot Packages for Jaunty and Intrepid

2009-05-06T16:57:00Z

My Launchpad PPA now has updated packages for ejabberd fresh from the Subversion repository. These are now available for both Intrepid Ibex and Jaunty Jackalope.

For those of you running Intrepid, I also recommend you upgrade Erlang to R12B5 which is also available in my PPA.

Next up on my list of packaging tasks is to make Erlang R13B packages for Jaunty and to update the Wokkel packages to a recent version.

Enjoy!

Tape Your Stuff to the Web

2009-04-07T10:40:00Z

I work on a lot of JavaScript projects that make heavy use of AJAX. Most of these need no server infrastructure except reverse proxies (due to JavaScript's same origin policy). I got tired of setting up nginx configuration files, so I wrote a simple Web server that publishes the current directory to the Web along with reverse proxies. I call it Tape because I use it to tape my stuff to the Web.

Tape is extremely easy to use. Just type tape, and the current directory is accessible via http://localhost:8273 (8273 for t-a-p-e). Want to proxy your local BOSH connection manager? Type tape -P /xmpp-httpbind=http://localhost:5280/xmpp-httpbind and go! You can also drop a .taperc file with proxy information into the current directory and Tape will use that so you don't have to type much.

Tape is inspired by SproutCore's sc-server command. While reading the change logs for SproutCore, I noticed that this command recently got reverse proxy support. That seemed like it would be incredibly useful for me, and Tape was born.

Tape is written in Python and makes use of Twisted's Web server features. It clocks in at a little over 150 lines of code.

This is going to my make development work flow so much simpler. I hope it will help you as well.

jQuery and Strophe: Made for Each Other

2009-03-13T15:50:00Z

Strophe is a beautiful library for building XMPP applications in JavaScript. It has a lot of nice features to help you build stanzas and respond to various event stanzas. It is built on top of the DOM, and unfortunately, the DOM is not always fun to work with. Luckily, we have an excellent DOM slicer and dicer that we can use alongside Strophe, the jQuery library.

The Hard Way

Let's take a look at what the code looks like for the traditional DOM method of manipulating incoming XMPP stanzas.

Our first example stanza handler is for receiving service discovery information, or in XMPP parlance, disco#info stanzas. Our code will pull out the feature information from the stanza, and pass that along to someone else.



function onDiscoInfoResult(stanza)
{
    var result = []
    var features = stanza.getElementsByTagName("feature");
    for (var i = 0; i < features.length; i++) {
       result.push(features[i].getAttribute("var"));
    }

    doSomethingWithFeatures(result);
}

The disco code isn't too bad, but the DOM gets a lot worse the more complicated the manipulations.

Next, let's look at Strophe's echobot example.



function onMessage(msg) {
    var to = msg.getAttribute('to');
    var from = msg.getAttribute('from');
    var type = msg.getAttribute('type');
    var elems = msg.getElementsByTagName('body');

    if (type == "chat" && elems.length > 0) {
        var body = elems[0];

        log('ECHOBOT: I got a message from ' + from + ': ' + 
            Strophe.getText(body));

        var reply = $msg({to: from, from: to, type: 'chat'}).cnode(body);
        connection.send(reply.tree());

        log('ECHOBOT: I sent ' + from + ': ' + Strophe.getText(body));
    }

    // we must return true to keep the handler alive.  
    // returning false would remove it after it finishes.
    return true;
}

This still isn't terrible, but we can see already that there are a lot of calls to long functions and dealing with the results of getElementsByTagName. Looping has to be done manually. If we need a specific element of the hierarchy, then digging it out of the DOM is pretty painful.

The Easy Way

jQuery makes DOM manipulations really, really simple. You can dig out very precise pieces of the DOM, manipulate them in aggregate, and it even takes care of much of the pain of looping for you.

Here are the same two examples using jQuery.



function onDiscoInfoResult(stanza)
{
    var result = $(stanza).find('feature').map(function () {
        return $(this).attr("var");
    }).get();

    doSomethingWithFeatures(result);
}

In the disco info example, jQuery's map() function has made it really easy to grab a bunch of things at once, do something to them (in this case, pulling out a specific attribute value), and then throw the results into an array.

Notice that jQuery is happy to take our XML stanzas as input, just by passing them into the $() function. It then works just as normal, except with our single stanza as its document instead of the HTML page.

Here's the echobot example:



function onMessage(msg) {
    $(msg).find("message[type='chat'][from]:has(body)")
        .each(function () {
            var body = $(this).find("body:first").text();
            var from = $(this).attr("from");

            log('ECHOBOT: I got a message from ' + from + ': ' + body);

            var reply = $msg({to: from, type: "chat"})
                .c("body")
                .t(body);
            connection.send(reply.tree());

            log('ECHOBOT: I sent ' + from + ': ' + body);
        });
}

The jQuery code is shorter and clearer because it doesn't have to do any of the busy work of dealing with DOM. Using jQuery's powerful selectors, we have:

picked the correct element out of the stanza
checked that the stanza made sense
done nothing when the stanza doesn't match

This code is actually more robust than the other version because we never bothered to make sure there was a from attribute originally.

jQuery is not Just the Inspiration

jQuery's design has certainly influenced Strophe, most notably with the builder syntax. It is not just an inspiration, however, as its powerful selectors and looping make it perfect for using to slice and dice incoming XMPP stanzas.

I'm sure it was never intended for such a purpose, but it is a testament to jQuery's good design that it just works, even in domains that are unexpected.

I have to say that working with XMPP stanzas in JavaScript is by far my preferred environment due to jQuery's powerful selectors and Strophe's really convenient builders. I hope that this will eventually find its way into Python, Ruby, Erlang, and other languages that do heavy XMPP lifting.

Strophe in Action: Drop.io's Awesome XMPP Powered App

2009-03-12T16:19:00Z

Drop.io just announced that their collaboration service is now a real time Web application. I got word from them that this amazing new stuff is powered by XMPP via Strophe. They have certainly added a lot of elbow grease, passion, and magic on top to make this possible, but this is a terrific example of what I envision with XMPP powered Web applications.

To get the full effect, I highly recommend watching their real time demo screencast.

They have this to say about the technical details:

Using the Jabber (XMPP) protocol and through a chain of events mediated by JavaScript, BOSH and XMPP drops are now updated for all users viewing that drop in real-time. We are using ejabberd, which is known for its high level of compliance with XMPP. On the front end we use the Strophe javascript library, which uses a technique called Bidirectional-streams Over Synchronous HTTP (BOSH) to connect to a Jabber server.

Each drop is assigned its very own chat room on our Jabber setup, and whenever a user views the drop, Strophe automatically logs the user into the drop’s chat room. Whenever an event such as a file creation occurs, drop.io's application servers send an XMPP message to the drop’s chat room describing the event.

This is not far off from the structure of Chesspark, and shows just how powerful a few XMPP technologies can be when combined well. One neat side effect of this is that they will have a public API for free that has robust support in many languages already. Maybe some day they will enable Drop.io features in arbitrary chat rooms as well!

Congratulations to the whole Drop.io team on a successful launch of a very cool product!

Upcoming XMPP Talks and Events

2009-03-11T13:48:00Z

There are a lot of XMPP events in the near future, and we'd love to see you at a few of them.

The first XMPP Monthly Meeting (March 12) will be in the jdev@conference.jabber.org MUC room. Peter has posted more details here.
The SXSW XMPP Meetup (March 17) will be in Austin, Texas. Details can be found on the meetup page. There are already 15 people going, and about the same amount marked as interested. It's looking like it will be a great event. Besides myself, there will also be Julien from Notifixio.us and Marcus from Minggl. I'm sure a lot of interesting discussions will take place regarding XMPP and Web applications.
IETF 74 (March 22-27) will be in San Francisco, and two sessions will have XMPP focused content, both on March 23rd. The first is the App area open meeting in which we'll be talking about BOSH, WebSockets, and Bayeux. The second is the XMPP BOF session where various XMPP related topics will be covered. Peter and I will both be there, and I believe some of the Isode guys as well, one of whom is the first XSF member to become an IETF Area Director.
The all JavaScript JSConf (April 24-25) will be in Washington DC. I'll be giving a presentation there on building XMPP powered Web applications.
The Glue conference (May 12-13) will be in Denver. XMPP makes for great glue, and accordingly, Peter and I will be on a panel along with others including Seth Fitzsimmons of Yahoo's Fire Eagle team. This conference should be really great, and if you use the code "spkr09" to get a $100 discount on the already cheap conference.
OSCON 2009 (July 20-24) will be in San Jose, CA this year instead of the usual Portland, OR location. The XSF is planning to have the 7th XMPP Summit during the conference, and there are some XMPP related talks shaping up. It should be even more fun than last year!

I'm sure this is not even a complete list; the number of events where XMPP is popping up is growing every day. Let me know if you've heard of any that I've missed!

XMPP Complexity Versus Sophistication

2009-03-09T20:59:00Z

Outside of the XMPP community a perception seems to exist that XMPP is too complex or too difficult, usually in reference to HTTP. I don't believe this is an accurate perception, and I've been wanting to address it here for some time. Of course, simpler systems can be built, but in most cases, these simpler systems are missing critical features such as security, extensibility, or maturity. I think that once these features are addressed, what you end up with will be markedly similar to XMPP.

Security

The original XMPP protocol was not nearly as secure, and because of this, much simpler. The IETF required that we address this issue when we went through the standardization process. TLS, SASL, and other methods replaced ad-hoc SSL, plain text authentication, and server dialback. Any alternative protocol will probably need to be similarly hardened.

Jayson Vantuyl recently talked about this on the CloudForum list:

XMPP comes with a built-in, non-optional security mechanism. This is really going to have to be a part of any Cloud. Passing around authentication tokens is lovely and all, but having it at the transport level just makes a lot more sense. After all, it's what HTTP does. XMPP just makes it a bit easier to work with because it's not optional, not an afterthought, and uniformly accessible without regard to which server you're using (e.g. extending HTTP auth isn't realistic given all the servers out there).

Extensibility

I've seen several people mention recently that JSON might be a better encoding for something like XMPP. XML however, does offer several present day advantages.

XML tools and libraries are well deployed, well tested, and pretty much ubiquitous. JSON parsers are getting there, but still have a ways to go. Also, a JSON parser would need to accept streaming data, and my guess is that few of them currently do, just as few XML parsers used to deal with such data.
XML namespacing provides the core of XMPP's extensibility. This does make the encoding more complex, but for good reason. JSON would have to develop something similar, but I think the XML solution would look elegant in comparison. It is easy to put new elements and attributes in existing ones. How does one do this in a JSON array without breaking previous implementations?
XML is human readable, making debugging easy. This was a huge win for HTTP. Many times on the XMPP lists, developers can post short snippets of wire traffic, and the problems are easily identified. Here JSON is on the same page, but binary encodings are not.

Maturity

It may seem strange at first to talk of XMPP's maturity, but the fact is that many clever minds have been thinking and working on it for a decade. Each implementation comes with bugs, but these are rapidly disappearing as well, just as they did for HTTP.

A hypothetical new and simpler protocol would need to recreate a lot of this work, especially in the area of publish-subscribe. One thing I've been very happy about with XMPP is that the hard work (designing the protocol) has already been done. I'm well equipped to create plugins, fresh implementations, or bug fixes; protocols, on the other hand, are quite hard to get right.

Design versus Implementation

I think people desire something simpler so that it is easier to implement, but they don't always realize that the hard part is not the implementation, but the protocol design. It is an incorrect optimization in many cases to design for ease of implementation by subtracting robustness or sophistication.

XMPP is Beautiful and Getting Better

Once you get a connection established and authenticated, XMPP is simple and beautiful. With HTTP, getting to this point is fairly trivial in 99% of cases, as authentication is usually ignored or that sophistication is moved directly into the applications.

I think what people are frustrated by is deploying XMPP and dealing with the strict security requirements.

Deploying XMPP is getting better, just as HTTP did. A decade ago, getting Apache running was not huge fun. Nowadays, almost everything comes with a simple built in Web server. It's not that HTTP is easy, it's that the implementations and documentation for those implementations improved dramatically. XMPP is making huge strides on these issues.

Peter, in the same email thread as Jayson above, offered this:

If your complaint against XMPP technologies is that the existing software implementations can be difficult to deploy, then the solution is not to throw out all of the work that has been done over the last ten years to harden and secure XMPP's core layers and also develop numerous extensions to that core. Instead, the solution is to build a slimmed-down implementation in one of your preferred code languages.

People are starting to do just this. We have Wokkel (which will get folded into Twisted Python), Prosody (written in Lua), and synapse (written in Ruby) already.

XMPP Meetup at SXSW 2009 in Austin

2009-03-06T10:10:00Z

If you're interested in XMPP and will be in Austin on March 17th, 2009, please join us at the XMPP meetup. Everyone is welcome, XMPP users, developers, and anyone who is interested in learning more.

I've been trying to organize an XMPP meetup in Austin during SXSW, and thanks to the efforts of Marcus Irven of Minggl and the Conjunctured co-working space, the event is officially happening.

I'll be there with a few coworkers from Stanziq and Chesspark along with Marcus from Minggl, and I believe there will be a few current and former Googlers there as well.

Please let me know if you'd like to do a small presentation on something cool you are working on. I hope to meet some new people and have a great time talking about XMPP.

An Octavo of Stanzas

2009-02-17T23:00:00Z

While at FOSDEM, Peter suggested that XMPP needs a collective noun for stanzas. We have a pack of dogs, a herd of cattle, and (my favorite) a murder of crows.

Two good ideas were suggested:

an octavo of stanzas

a folio of stanzas

See Wikipedia for the meanings of octavo and folio. Personally, I like the former. What about you?

Some Thoughts on Serialization

2009-02-16T20:00:00Z

I've written before that XML serialization tends to be the main bottleneck in XMPP apps. I haven't benchmarked other software stacks that involve serialization, but I'm guessing it's expensive everywhere. Here are some thoughts on serialization that I've had recently.

Eliminating serialization is better than making it faster. This is an obvious statement, but getting rid of serialization is not always easy.
The XMPP component protocol adds two serialization steps in most cases. One serialization to send data to the component over the socket and one for the components response back to the server. If you want to avoid this extra work, the component logic should be implemented in a server native fashion such as an ejabberd module.
Using XMPP over some other protocol stack will probably add serialization costs. For example, you might implement an XMPP pubsub service on top of an AMQP based backend. This means that you have an extra serialization cost of converting to the other protocol.
As everyone optimizes their code, we'll see more and more native serialization plugins, application code that speaks server native data structures, and application code that passes around already serialized chunks (the original data off the wire). Punjab with cserialize is an example of an application using a native serializer and passing around already serialized chunks.

Make Your Twisted XMPP Apps Scream

2009-01-26T23:50:00Z

XML serialization tends to be the most expensive part of many XMPP applications, and Twisted Python apps are no exception. Now you're Twisted apps can scream with a new C based XML serializer for Twisted.

Note: To speed up ejabberd, please see Speed Up Ejabberd, which talks about a similar solution for Erlang.

The standard serialization code in Twisted can process approximately 2,100 elements per second on my Macbook Pro. It's not great, but it's a reasonable start.

A few years ago my team and I spent a few days improving this code. The result was a pair of patches that traded off memory for speed by caching serialized fragments. This code can process about 3,150 elements per second on the same machine. This is an improvement factor of 1.5.

Last week, the Twisted team improved this work in preparation for merging it into the code base. They managed to achieve about 3,700 elements per second - an improvement factor of 1.75x over the original code.

While they were working on this, I realized that it would be fairly easy to write a C module that could serialize the Twisted XML objects directly. It took a little longer than expected, but you can find such a serializer at the cserialize GitHub page. This code achieves over 7,500 elements per second, which is a factor of 3.5x faster than the original code.

I've yet to really optimize this code, so I think more gains can be made. I'm hopeful that I can make it about twice as fast as it is now. My original version, which was missing namespace support, was about 15 times faster than the original Python code. I'm not quite sure where the slow down came from, but as soon as I figure out how to profile Python C modules, I'll track it down.

This code is fresh, but it passes almost all of the Twisted tests. The few that don't pass appear to be legitimate bugs in the test cases, not the serialization code.

I've created bug #3633 to try and get this integrated upstream. In the meantime, we'll be enjoying our faster games at Chesspark and speedier searches at Stanziq.

We Don't Need No Stinkin' Web Frameworks

2009-01-25T23:06:00Z

Web frameworks can be nice, but they can also get in your way. One of the pleasant side effects of implementing more and more functionality over XMPP is that we've eliminated the need for Web frameworks at Stanziq. This is extremely liberating.

How do we survive without them? Here are some of the ways:

Web frameworks may offer a lot of functionality, but XMPP has a excellent, diverse set of extensions that offer a higher layer to application developers. Web frameworks are still essentially stuck on basic protocol plus ad-hoc commands. XMPP provides pubsub, presence, discovery, and much more.
Communication is done via XMPP through Strophe instead of polling a database or using work queues. This is a step up in efficiency.
Configuration is stored in pubsub nodes instead of relational databases. One awesome consequence of this is that all subscribers get instant notification of configuration changes, sort of like a broadcast SIGHUP.
Front end apps, administrative code, and internal utilities are all just JavaScript. This makes them trivial to develop and test locally, and we don't need any special deployment code.
The whole system is more decoupled because there is no middle interfacing layer. The backend speaks XMPP, the frontend speaks XMPP, and they both use standard XMPP layer protocols to do work.

Of course, this isn't the right fit for every problem; Web frameworks solve a lot of problem domains quite well. For those areas where it fails, it sure is nice to avoid the hassle and use an XMPP only stack.

Some Thoughts on Logging

2009-01-24T23:30:00Z

I'm not a huge fan of syslog. It works well enough for logging on the local host, but it has been extremely fragile for us when logging from several hosts. One thing we have been thinking about at Stanziq is trying to find a better solution.

There are tools like Spread or protocols like AMQP. The basic idea behind these tools is publish-subscribe, so it should be no surprise that XMPP fits this problem domain well. This idea came from Ken during a conversation we had about logging infrastructure; I was still looking at the list of possible solutions when he made the obvious connection.

XMPP offers:

publish subscribe semantics
reliable delivery
baked in presence
multiple server implementations and client libraries in every language
a whole suite of tools for managing commands, configuration, and extensibility

Some people might argue that solutions like AMQP are better, and for this, I think Peter has summed it up well in his post on AMQP and XMPP.

I look forward to playing with a new XMPP based logging system, and I'm sure I'll post more about it as it starts to materialize.

Blogging with Git, Emacs, and Jekyll

2009-01-23T22:22:00Z

A few months back I switched this blog to Jekyll. Jekyll is a static blog generator; it transforms a directory of input files into another directory of files suitable for a blog. The management of the blog is handled by standard, familiar tools like creating and renaming files, the text editor of your choice, and version control. Jekyll combined with Git and Emacs has made blogging feel very natural to me.

What is WordPress, Really?

WordPress and other blogging systems are simplified content management systems. Many of the features they provide relate to a blog's minimal need for dynamic content. The front page needs to change as new things are added, and each post must change as comments, trackbacks, and edits are made.

Along they way they often made it harder to just write. While they keep providing more and more sophisticated text editing capabilities, most of us who write or code are quite comfortable in our own editors. No matter how good the text editing widgets become, they are not Emacs or vim or TextMate.

Blogs have simple APIs for using external products, but even these products are often not great text editors. When the real text editors have added the ability to speak these APIs, the functionality feels bolted on and clunky. The reason is that we are used to interacting with documents in a file system, and the current blogging APIs are much less elegant than the familiar, simple API of a file system.

Managing documents is something nearly everyone using a computer has gotten quite good at, and the tool support for this is immense. We have automatic indexing systems like Spotlight, powerful search tools like grep, and history tracking and analysis tools like Git.

WordPress and others would do well to concentrate on the areas where blogging software is really needed and expose the document management part via WebDAV or something similar. In the end, WordPress is a transformation layer to take some set of documents and turn it into a blog as well as a system to provide commenting, trackbacks, and search.

The Post-WordPress World

Let's look at the key areas where blogging software is necessary. We have transformation of content, interaction with content (commenting and such features), and search. In all these areas superior, domain specific tools have emerged.

Search is dominated by Google, but there are other services that can also be used. I have been really underwhelmed by the search capabilities found in self-hosted blogs, but Google's site search is really good.

Interaction services started appearing as spam became a problem for blog comments. Now we have full featured services like IntenseDebate and Disqus which provide excellent interaction features like email integration, comment aggregation, moderation and filtering, and portable identity.

Now we come to transformation of content, which turns out to be the only missing piece that on really needs.

Jekyll Generates Blogs

Jekyll fills this last piece of the blogging feature puzzle. It takes a tree of documents and transforms that into a blog.

At a basic level, Jekyll goes through the directory tree and translates files from a variety of formats like Markdown and Textile into HTML. Each file can contain a small YAML header with metadata. These metadata blocks tell Jekyll which template to use during rendering and define the title and other properties of the document.

A few directories in Jekyll are special. _posts contains all the blog posts, with each post being named in a common format such as 2009-01-14-blogging-with-git-emacs-and-jekyll.markdown. _drafts contains documents that are ignored by Jekyll. _layouts contains rendering templates that are used during the transformation process.

My own templates use Disqus for comments and Google for search, but it is easy to mix and match the services you want in your own templates.

The resulting generated site has a standard URL structure of /2009/01/14/blogging-with-git-emacs-and-jekyll/ and the front page and RSS or Atom feeds are just simple templates that loop over the latest group of posts.

Managing and Publishing with Jekyll

You can use Jekyll by hand, but it is meant to be used alongside a version control system. Whenever you commit or push to the repository, it runs the jekyll command to generate the blog from the latest data.

Git is a popular choice here, but Subversion or Bazaar would also do a fine job. One thing I like about Git is that I can set up a remote on my main machine to my Eee laptop, and a corresponding remote in the other direction. Both machines have the origin remote set to my blog's server. This means I can easily push and pull the blog around between machines and I always have access to the entire blog no matter what machine I'm on.

I also wrote some Emacs glue to help automate my own blogging workflow. I start new drafts with C-c b n and then move drafts to the _posts folder with C-c b P. I can also easily access the list of drafts or posts when I'm searching for something I've previously written. This kind of glue is easy to write in any language because it leverages the familiar platform of the file system.

Why I Love This System

Now that I've been using this system for several weeks, I'm really loving it.

I'm using the same tools I always use for the rest of my work to write and manage my writing.
I always have access to my entire blog, which is awesome when I am crafting posts on plane rides and need to reference some previous content.
If I find some manual thing in my workflow, I can use any number of common tools to automate or customize it. Everything is just text and files.
Comments are now just email threads to me, which means I don't have to use some clunky interface. Once again, I can use tools which are already part of my daily work habits.

If you've ever thought about blogging, or blogging more, but didn't want to mess around with blogging clients, Web interfaces, or installing or maintaining blogging software, you should give Jekyll a try.