Dev @ AboutUs.org

Cassandra Truncate Means Slow Test Suites

2011-08-22T00:00:00-07:00

We’ve been using Cassandra as the primary data store on several applications for a while. Of course this means that it’s integrated into our test, and ci environments. A couple days ago Brad and I noticed that one of our test suites, which had been running in about 1 minute was now taking 10 minutes. It was painful.

We tracked the slowness down to this line in our Rspec spec_helper.rb file:

config.before(:each) { $cassandra.clear_keyspace! }

The intention here was to clear the database before each test ran, to ensure that tests were isolated from each other. Great, except as we added more Column Families this got slower and slower, until the call was taking ~1 second per test case. About 90% of the time in the test suite was being spent deleting data from cassandra. Oh the pain…

Luckily we got a tip from thobbs in the #cassandra irc channel. He said:

doing a get_range() while deleting everything is faster that’s what I do for most of the pycassa test cases

He pointed me to an example in pycassa.

Translated into ruby it looks like this:

config.before(:each) do
  $cassandra.schema.cf_defs.each do |cf|
    $cassandra.get_range(cf.name, :key_count => 10000).each do |(row_key, _)|
      $cassandra.remove(cf.name, row_key)
    end
  end
end

With this change the test suite was back down to ~1 minute run time. And the people rejoice.

Managing Cassandra's Schema from Rails

2011-07-18T00:00:00-07:00

The other day I was tracking down a bug on one of our rails projects that uses Cassandra as its primary data store. I needed to find a way to manage changes to the Cassandra schema across all of our development, and production boxes.

Rails normally uses ActiveRecord Migrations to manage an application’s schema. Obviously these don’t work with Cassandra since it is not a relational database. Fortunately we were able to quickly create a system for managing Cassandra’s schema from the rails app. It’s brand new so YMMV, but I thought the approach might be useful to others trying to use Cassandra from rails.

Declarative Schema

Column families and their attributes are defined in a YAML file. Our project has a schema.yml file that looks like this:

---
ReportHandles:
  comparator_type: 'UTF8Type'
Reports:
  column_type: Super
  comparator_type: 'UTF8Type'
Runs:
  column_type: Super
  default_validation_class: 'BytesType'

This file defines all of our column families and the properties such as comparator_type which we care about. This differs from Rails’ built-in approach to migrations. Instead of defining a series of migrations that are applied in order, we declare the schema that we want, and the system brings Cassandra in line with it. This felt like a simpler approach for our use case, and made sense since Cassandra (not being a relational db) has a simpler, more flexible schema system, and less dependency between column families than, say, MySQL tables. You might look at active_column if you want Cassandra migrations that are closer in style to ActiveRecord.

To accomplish the actual schema changes we have a Schema module that looks like this:

module Schema
  def self.migrate
    config.each do |(name, properties)|
      migrate_column_family(name, properties)
    end
    wait_for_schema_agreement
    schema
  end

  def self.schema_agreement?
    cassandra_client.schema_agreement?
  end

  def self.config
    @config ||=  config!
  end

  def self.config!
    YAML.load_file(File.join(Config.root, 'db', 'schema.yml'))
  end

  def self.wait_for_schema_agreement
    return if schema_agreement?
    secs = 90
    Timeout.timeout(secs) do
      puts "waiting up to #{secs} seconds for schema agreement"
      until schema_agreement?
        print '.'
      end
      puts
      puts "done"
    end
  end

  # Migrate a column family to a desired state.
  # NB. Only properties that are explicitly declared are set.
  # Removing a value
  # from properties will not reset it back to default, it will
  # leave it in its
  # current state.
  def self.migrate_column_family(column_family_name, properties = {})
    wait_for_schema_agreement
    cf_def = find_or_initialize_column_family(column_family_name)
    properties.each do |property, value|
      cf_def.send "#{property}=", value
    end

    if column_family_exists? cf_def.name
      cassandra_client.update_column_family cf_def
    else
      cassandra_client.add_column_family cf_def
    end
  end

  def self.find_or_initialize_column_family(name)
    cf_def = column_family(name.to_s) || (
      cf_def = CassandraThrift::CfDef.new
      cf_def.keyspace = cassandra_client.keyspace
      cf_def.name = name.to_s
      cf_def
    )
  end

  # SCHEMA INTROSPECTION
  def self.column_family_exists?(name)
    column_families.map(&:name).include? name.to_s
  end

  def self.schema
    cassandra_client.schema
  end

  def self.column_families
    schema.cf_defs
  end

  def self.column_family(name)
    column_families.detect{|cf| cf.name == name.to_s}
  end

end

It assumes that you have a cassandra_client method defined, and that the Keyspace you’re managing exists. At some point I may clean this up and release it as a gem but, like I said, for now YMMV. I’ve omitted the specs for brevity, but you can see them on gist.

Cluster schema does not yet agree…

When I first ran this against a clustered Cassandra setup I started getting cluster schema does not yet agree messages, followed by a failure. It turns out Cassandra has the concept of schema agreement. This makes perfect sense when you consider that Cassandra is distributed, and fault tolerant. Schema changes have to be propagated through the cluster, until eventually all nodes agree. It seems that until there is agreement, you can’t make further schema changes. To deal with this the Schema manager waits for schema agreement before making changes to the column family:

def self.migrate_column_family(column_family_name, properties = {})
  wait_for_schema_agreement
  cf_def = find_or_initialize_column_family(column_family_name)
...

Waiting for the schema to propagate usually just takes a second and is easy to do:

def self.wait_for_schema_agreement
  return if schema_agreement?
  secs = 90
  Timeout.timeout(secs) do
    puts "waiting up to #{secs} seconds for schema agreement"
    until schema_agreement?
      print '.'
    end
    puts
    puts "done"
  end
end

Integrating with Rake and Capistrano

I wanted to keep the workflow around schema management as close to the rails conventions as possible. The schema can be brought up to date locally by running rake cassandra:migrate which is defined as:

namespace :cassandra do
  desc "Migrate to the current cassandra schema"
  task :migrate do
    require File.expand_path(File.join(Config.root, 'lib', 'schema'))
    Schema.migrate
    puts "Migrated to this Schema:"
    puts *Schema.column_families.map(&:inspect)
  end
end

Hooking it up to our Capistrano deploy was also easy. By overriding the deploy:migrate task this gets hooked into our deploy process in place of ActiveRecord Migrations.

namespace :deploy do
  task :migrate do
    run "cd #{release_path} && RAILS_ENV=#{stage} rake cassandra:migrate"
  end
end

Now running the standard cap deploy:migrations task deploys the codebase and brings the Cassandra schema up to date.

Getting Started Exploratory Parsing

2011-07-03T00:00:00-07:00

A parser reads text to discover structure and meaning. For example, a C language parser can read a C program and understand in a real sense everything that the program has to say. Contrast this to a pattern matcher, such as regular-expression matching, which can find fragments of a program useful in editing but can’t keep track of enough context to make sense of a whole program.

We often use the unix grep utility to look through large files. By applying a regular-expression match to each line, grep is able to report just the lines of interest. When we allow ourselves to grep repeatedly, driven by our curiosity, responding to each answer grep provides with another question, when we do this we are exploring.

The internet is full of text that defies understanding in any sense with simple pattern matching. In response AboutUs built an environment for exploring the internet interactively, using parsers constructed on a whim, returning matches in within the context described by the explorer.

Exploring the World Fact Book

The AboutUs exploratory parsing environment has been released as open source on GitHub. Included with the release are scripts to download The World Factbook and English Wikipedia as sample texts for exploring. Let’s take a look at the Factbook.

When exploring, we start by looking for something that we know is there. We’ll start by looking for short strings of characters without any regard for where they are.

char = << ......... >>

Our parser uses a dot (.) to match any character. We say we’re looking for a few of them. And when we find them we want to see them so we add “eye-balls” around the dots to tell the parser to remember some of the matches.

This says that our parser found over a quarter million matches. When we ask to see some, it shows us the text on the right. This is just a sample match. When the sample was taken the text shown in green had been matched, and that highlighted in yellow is the specific match sampled. This data looks like keys and values separated by colons.

Let’s look for keys and values by describing what we think we know about the file. We’ll offer the parser an alternative for text that isn’t key-value pairs as we understand them.

fact = key value | other_char
key = whitespace << word+ >> ':'
value = << ( !key . )+ >>
word = [A-Za-z]+ ' '*
whitespace = '\n' ' '*
other-char = << . >>

Now we’re getting pretty specific as to what we mean by key and value. We say 1) a key starts with whitespace, 2) the key has one or more words, 3) the words end with a colon, and 4) we only care to have eyeballs on the words of the key. Is happy to read the whole file.

The parser found 19,498 keys and an equal number of values. Makes sense. When we look at the sample keys we find a few surprises. Most are capitalized but not all. The subcategories of Imports are lower case. Interesting. Lets see how wide spread this convention is.

key = whitespace << ( upper | lower ) >> ':'
upper = << [A-Z] word+ >>
lower = << word+ >>

When we look at some lowers we see the expected “commodities” and “partners” and one more outlier, the login instructions to get the Factbook from Project Gutenberg. Let’s separate out the familiar to see what other lower-case keywords might exist.

lower = << ( familiar | other-key ) >>
familiar = << ( 'commodities' | 'partners' ) >>
other-key = << word+ >>

Now we’re down to 77 out of 20 thousand keys. Sampling these we see more that make sense and should be added to the familiar category. We also see a few places where our parsing rules are clearly not working as intended. We could tighten up the rules by saying just how much whitespace we expect before a key, or how many words we expect, or just discover the words that should be familiar and ignore the rest. We have options.

We also have other branches in our parse to explore. We haven’t even begun to parse the values. We could, for example, select out “Climate” and see how many ways climate is described in the Factbook. Maybe we do the same for “Terrain”. Maybe we correlate phrases we find within the two and get some insight into how the two are related. We don’t have to just sample parser matches. We can take the text of interesting matches and feed that into other programs.

Get the Exploratory Parser

We’ve been using a tool made out of two parts, both of them available to other programmers under AboutUs on GitHub. One is our fork of Ian Piumarta’s peg/leg parser generator. The other is our parsing experiment management system.

The parser generator is written in C and could be rough going for programmers who haven’t studied compilers at some point in their lives. We’ve only modified peg/leg as we found our unusual approach to parsing was not anticipated by Ian. Ian provides documentation on his website.

The experiment manager is a web application written in Ruby to run under Mac or Unix. We run it on our laptops and in Amazon’s EC2 cloud. We’ve described how we install it in our GitHub ReadMe. Your Mileage May Vary.

Thinking Different

We think we’ve opened up a whole new way to use technology. This can happen when one takes some assumed requirement and reverse it. Wiki, for example, reversed the assumption that only the owner should edit the pages of a web site. Parser generators have traditionally been used to describe exactly what should be written and anything else is a “syntax error”. Wiki allows writers to write what they think makes sense. With exploratory parsing we now have a way for the parser writer to discover what has been written after the fact. This inversion of control mirrors the original thinking behind wiki. Let those who know write as they see fit. Trust people to be regular enough to create lasting value. Use the power of our modern computers and networks to organize that value.

Agile vs. Open Source: They're Actually a Little Different

2011-05-17T00:00:00-07:00

Last night I was hanging out at the North Portland Coder’s Night (nopoconi) and Ward and I got to talking about the difference between Agile Software Development and Open Source Software Development. I’d always thought of the two together, but Ward had some good insight into the differences in these two cultures and methodologies that I realized fit well with my own experience.

While there’s a lot of similarities between Agile and OSS philosophies, there’s a key fundamental difference. Agile methodologies like Extreme Programming are about getting the right people in the room. Face to face communication is much more efficient, says XP, and the best way to get results is to have a small, empowered team of smart people in clear and constant communication.

Open Source starts with a different idea; code should be available and editable by anyone. Letting the people in the room make all the decisions isn’t fair to the people that aren’t in the room. What’s most important is that the processes are in place that allow people all over the world to contribute to the codebase and improve the project. This means being set up to review patches, and have discussions (in IRC, forums, wikis) that anyone can participate in. While agile strives to put a tight-knit team in close quarters, OSS strives to create a community that anyone can participate in, despite their location or circumstance.

While we were talking about this, I began to realize I’d seen these two patterns in lots of software companies. For example, at AboutUs, we all work in one big room together (Ward once called it a Wiki space in video). We have a small team. When we have a question we can yell across the room. That’s agile. A good friend of mine works on the Firefox team at Mozilla. His team members are all around the world and do most of their communication over IRC, skype, and through issue trackers.

A lot of times we talk about building software as a choice between Big Design Up Front and Agile methodologies, but in fact there’s a lot more options out there that we can draw upon to build great software.

Rodents Of Unusual Size

2011-05-05T00:00:00-07:00

“You keep using that word. I do not think it means what you think it means.”

At AboutUs there is no formal code review process. We collectively own the code and we pair a ton, so most code gets seen by at least two people before it hits production. We also have a Big Red Light to keep us in line. To keep even better coverage, we have a daily habit of reviewing and commenting on the commits made by others in Github. This is incredibly useful. Most of the chatter is just run of the mill knowledge sharing, but sometimes some good conversations pop up. One of these conversations started the other day when one of our devs used that Princess Bride quote above in response to a commit.

One of the benefits of working at AboutUs is that we have Ward Cunningham as an advisor and contributor. Ward was CTO when I came on board and although he’s moved on to other projects he’s kept a close relationship with AboutUs. Ward often chimes in on commits, and this conversation was no different. Along with his usual pearls of wisdom he mentioned this:

You really can learn something new every day. Chatting in Github commits is a great way to do that.

One and a Half Minds are Better: Learning by Pairing

2011-05-03T00:00:00-07:00

My name is Brad Heller and I’ve just started at AboutUs.org as an agile developer! AboutUs.org has a pretty interesting way to ramping their new guys: Pairing! To be honest, it’s not not really all that different from how we normally do our work, but it is a great way to learn about a company’s technology if you’re new.

In reality, a lot of agile shops introduce people to their stack via pair programming, either formally or informally. Here are a few notes I’ve compiled from my experience thus far.

Use a workstation that has multiple input devices.

This is good general advice for pairing. At AboutUs, we have pairing stations with two keyboards, two mice, and one gigantic monitor. This allows you to jump in quickly when you feel inspired, without having to shuffle hardware.

Use a tool set that you are both familiar with.

You want to learn the technology, not the tools (unless, of course, the tools are critical to the technology). Spending a lot of time explaining how the tools work interrupts the “knowledge stream” you build in doing the work together.

Figure out a system that allows you to take notes without getting lost.

This is actually a lot more difficult than it sounds. When you’re pairing, the person in the know often moves pretty quickly, so taking notes can be hard. I don’t know how many times I’ve asked the same question for the Nth time because I didn’t take good notes!

Be aware of your partners working style and pair with someone who’s compatible.

Avoid problems that stem from the different working styles by pairing with a mentor who works similarly to you. For example, I tend to want to move very quickly through problems by trying many solutions and rolling back anything that doesn’t work. Then, when I do find a working solution, I reflect on it to figure out if it’s optimal or not. I’ve found that Thomas and I work well together (as far as I’ve been able to tell, anyway) because his working style is similar to mine.

Don’t be afraid to drive.

Few people can learn by watching alone. You have to get your hands dirty if you want to learn! Besides, this is the best time dive in, as you’ve got a saftey net in case you mess up (erm, that is when you mess up).

Agile Metrics: Tracking Numbers that Matter

2011-05-02T00:00:00-07:00

Many agile teams keep track of velocity as a measure of their performance.¹ This is a useful measure, but it shouldn’t be the only one. This past week we realized that another performance metric of an agile team should be how much work it avoids doing.

During a discussion with stakeholders last week, we realized that much of the work in our queue was a temporary fix for a problem we would ultimately solve at a later date. Thanks to the stakeholders keeping their own planning queue in a visible location, the development team was able to see that the proper fix for the problem was scheduled for just a few weeks later. We engaged the primary stakeholder in a discussion about the requirements of the proper fix, and deemed that it would be no more work than the temporary fix. Naturally, we started work on the proper fix, abandoning the temporary one — and eliminated nearly a week’s worth of work!

Velocity is a great metric to track, but perhaps agile teams should also get in the habit of tracking work avoided on account of successful communication. After all, the best code is no code at all.

This, of course, is not the recommended use for velocity, but remains a method of measuring performance.
↩

Meet The Kanban Board

2011-04-29T00:00:00-07:00

A lot of people talk about agile. We try to be agile. Working with one of the signers of the Agile Manifesto doesn’t hurt.

Our day to day process revolves around a plexiglass kanban board. I’d like to show it to you.

Meet the kanban board.

The kanban board does a lot for us. It radiates information. It’s been called an agile innovation.

It helps us remember what we’re working on.

We know who’s working on what.

We can see what’s coming up too.

We try to keep Martin on low hanging features.

It tells us when too much is going on.

And sometimes it breaks and we have to change it.

It gives us some time to decide if things are really done.

And it asks us to show them off when they are.

After spending so much time working with this board it’s hard for me to imagine us doing our work without it. We’ve had to change and tweak it many times. As the way we work evolves, and our team changes we’ve had to adapt it to better reflect our work. The goal is to be able to look at it, at any moment, and know exactly what the status of things are. It isn’t always perfect, but it cuts down a lot on the effort we must put towards coordinating our efforts. We keep trying to get more aspects of our work reflected on the board, and that’s almost never a mistake. Everyone in the company can use it to see and communicate their work, and move cards across.

Playing With Sedgewick's Data

2011-04-22T00:00:00-07:00

Pittsburg University CS Professor Robert Sedgewick suggests students will learn better and stay engaged when writing programs that make sense of real data. I suggested to the newly forming Portland Data Science meetup that we might likewise benefit from exploring data together and gave Sedgewick’s state adjacencies as an example. Here is where I found the data and the first few lines of the file:

http://introcs.cs.princeton.edu/data/contiguous-usa.dat

AL FL
AL GA
AL MS
AL TN
AR LA
AR MO
AR MS

I saw this when I was reading something else Sedgewick wrote and thought, what would graphviz do with this?

It did pretty well, I’d say. In fact it did too well. How did graphviz know that WA was in the northwest?

You can see that it did get the northeast upside down. That’s comforting. I know more about geography than graphviz.

This raises the question, what little extra bit of information would allow a much better map? Some ideas:

the length or orientation of the border
the size of the state in square miles
the state’s voting record

Here is the perl program I used to convert the dataset to dot format:

@lines = `cat contiguous-usa.txt`;
open D, ">contiguous-usa.dot";
print D "graph US {\nnode [style=filled,color=yellow]";
for (@lines) { print D "$1 -- $2;\n" if /(\w\w) (\w\w)/; }
print D "}\n";

I consider this kind of programming a warm-up for serious work. But warming up is very important in a field where the opportunities are so diverse.

Our Continuous Integration Is A Big Red Flashing Light

2011-04-19T00:00:00-07:00

Continuous Integration is a pretty important part of any agile workflow. Having your ci server run your test suite whenever anyone pushes code means never having to argue about who broke the build.

A couple months ago we took it a step further. Instead of just having the CI server send an email when the build breaks, we have a big red light in our office which starts flashing.

The ingredients:

A big red flashing light I found a Goodwill
A Teensy 2.0 microcontroller donated by Ward Cunningham
A vacation light timer, hacked by Matt Youell
A mac mini, running bash and curl

How we made it work:

On my way back into Portland one weekend I stopped at the Goodwill and spotted a red light on a shelf with wires hanging out of it. I ponied up the $6.99 to buy it, having no idea if it actually functioned. I took it home and proceeded to test it.

Realizing this would be the perfect addition to our CI setup, I enlisted to help of active DorkBotPDXer Ward Cunningham. He gave me a Teensy 2.0 that he’d gotten from Teensy creator Paul Stoffregen. He also talked me into going to the next DorkBot meetup, to get some help using it.

I went to the meetup and sat down with Paul to get some help with the Teensy. I flashed his USB Serial shell onto the microcontroller. With this code installed on the Teensy I could plug it into a USB, and toggle the built in LED on or off with shell commands like this:

echo "d6=1" > /dev/cu.usbmodem12341
echo "d6=0" > /dev/cu.usbmodem12341

Progress.

Now that I could control the Teensy’s LED programmatically, the next challenge was hooking it up to the big red light. Matt Youell had done electrical work in the past. Apparently powering a tiny LED is dramatically different from running a 120V light, which opens the risk of frying your equipment, fires, and fire marshals. Despite this Matt offered to help. He knew we’d need some type of relay. He took the setup home, and came back a few days later with it wired to vacation light timer he had lying around.

For a while after he wired it we were nervous we would come in Monday morning to a burned down office but it’s been working great for months now.

The last step was getting the whole rig hooked up to our CI server. We use CruiseControl.rb to run our builds. I wrote a simple bash script that hits the server and toggles the pin 6 (the light) on the Teensy based on the response. It looks something like this:

#!/bin/sh
# Monitor cruisecontrol and trigger red light when there's a broken build.
# Also turn the light on when we don't get a 200 response from the server.

bad_requests=0
while [ true ]; do
 ci_url=http://ci.aboutus.com/XmlStatusReport.aspx
 response=`curl -i --max-time 5 -s -u user:pw $ci_url`

 # count how many times we've gotten a non-200 response from ci
 if [ `echo $response | grep 'HTTP/1.1 200 OK' | wc -l` -ne 1 ] ; then
   bad_requests=`expr $bad_requests + 1`
 else
   bad_requests=0
 fi

 # turn the light on when there's a build failure or we've had 3 consecutive
 # non-200 responses from the ci server.
 if [ `echo $response | grep 'lastBuildStatus="Failure"' | wc -l` -gt 0 ] \
      || [ $bad_requests -gt 2 ]; then
   (sleep 1; echo "d6=1") > /dev/cu.usbmodem12341
 else
   (sleep 1; echo "d6=0") > /dev/cu.usbmodem12341
 fi
done

I set this up as a startup item on our reception computer and bam!, we were done.

The complete setup looks a little like this:

Why It’s Awesome

Having this set up has been a big win for our team. It makes it even more obvious when someone has broken the build, and decrease the amount of focus we need to devote to monitoring the CI server. It’s an in-your-face “don’t deploy now” indicator which is great for a team that typically pushes code to production several times per day.

It’s also had the interesting effect of making the non-developers we work with aware of how continuous integration works and why it’s important. Now they know when the build is broken as soon as we do. They know we’re running tests, and that the test failures control the light; a big flashing red light never means good. Tests are important. They protect us and they help us developers, and the whole company be more agile.

Sort == Srot

2011-04-18T00:00:00-07:00

This may be something well-known to a lot of people, but it’s one that I just recently found out about. We were working with large volumes of data for our redirectory feature on (http://www.aboutus.org). For part of this we had to sort about 100 GB of tab-separtated data. Because we wanted to do the simplest thing that would work, GNU sort came to the rescue.

For those of you who aren’t familiar with sort, it is a standard UNIX/Linux tool that sorts the input it gets on standard in, and returns the new list on stdout. It has all kinds of options to control the sort order, algorithms used, etc., but some of them aren’t too obvious. For the basic list, check out the man-page.

Before I go on, a brief exercise. Lexically sort the following lines:

 foo
 far
 f.o

If you guessed “f,o, far, foo”, congratulations! You’re a sane, normal person. If, on the other hand, you guess “far, f.o, foo”, you are GNU sort.

This was a serious problem for us because it meant that some of our data was out of order, making our processing scripts angry. We would have go-daddy.com stuffed in the middle of a long list of godaddy.com’s. Not good.

The solution lies in a little-known environment variable that controls your terminal language and character set, “LC_ALL” (actually, LC_LOCALE but LC_ALL works too). If you set that bit of magic to “C”, things work as you would expect.

By default on most modern linux installs, the system locale defaults to US-english using UTF-8 character set. Beyond setting language and such, it also sets things like how characters are ordered as far as a computer is concerned. For whatever reason, punctuation is handled as “higher” than letters, even though they are “lower” in the standard character set charts, and because sort uses strcmp, character ordering is very important.

So there you go. If you are trying to sort a ton of data, and it’s not coming out sorted, give export LC_ALL=C a try. It worked for me!

Initial Commit

2011-04-08T00:00:00-07:00

This is the first post for AboutUs’ programming blog. The blog’s powered by git, jekyll and nginx. To create a post we create a text file in the git repository in our favorite editor (i.e. vim). The posts go in a _posts directory and look like this:

devblog [master*] $ ls _posts

_posts:
2011-04-08-initial-commit.markdown

Jekyll is used to convert the content textile, markdown, and html templates into a static html site.

devblog [master*] $ jekyll
Configuration from /www/aboutus/devblog/_config.yml
Building site: . -> ./_site
Successfully generated site: . -> ./_site

Jekyll’s --auto option is pretty nice for development. It regenerates the site every time you save a file.

devblog [master*] $ jekyll --auto
Configuration from /www/aboutus/devblog/_config.yml
Auto-regenerating enabled: . -> ./_site
[2011-04-11 20:49:52] regeneration: 8 files changed
[2011-04-11 20:50:22] regeneration: 1 files changed
[2011-04-11 20:50:27] regeneration: 1 files changed

On the server jekyll generates static html files for nginx to serve.

It barely has any dependencies.

devblog # gem install jekyll --no-rdoc --no-ri
Building native extensions.  This could take a while...
Successfully installed liquid-2.2.2
Successfully installed fast-stemmer-1.0.0
Successfully installed classifier-1.3.3
Successfully installed directory_watcher-1.4.0
Successfully installed syntax-1.0.0
Successfully installed maruku-0.6.0
Successfully installed jekyll-0.10.0
7 gems installed

There’s a git post-receive hook that regenerates the site whenever someone pushes a change.

devblog # vi .git/hooks/post-receive

#!/bin/sh
#
GIT_REPO=/www/aboutus/devblog.git
TMP_GIT_CLONE=/tmp/devblog
PUBLIC_WWW=/www/aboutus/devblog

git clone $GIT_REPO $TMP_GIT_CLONE
jekyll --no-auto $TMP_GIT_CLONE $PUBLIC_WWW
rm -Rf $TMP_GIT_CLONE
exit

Nginx serves the static files in /www/aboutus/devblog.

Deploying is just a git push.

devblog [master] $ git push
Counting objects: 23, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (12/12), 925 bytes, done.
Total 12 (delta 6), reused 0 (delta 0)
remote: Initialized empty Git repository in /tmp/devblog/.git/
remote: Configuration from /tmp/devblog/_config.yml
remote: Building site: /tmp/devblog -> /www/aboutus/devblog
remote: Successfully generated site: /tmp/devblog -> /www/aboutus/devblog
Killed by signal 1.
To blog@devblog:/www/aboutus/devblog.git
   12aa781..ab54f06  master -> master

Adding Syntax Highlighting

To add syntax highlighting I just had to install Pygments, which is a cool python project.

I did this with easy_install.

$ easy_install Pygments