Repeatable Systems

Writing CoffeeScript Modules for Node.js

relistan — Mon, 10 Dec 2012 20:42:30 +0000

If you write a module in CoffeeScript, you probably intend for it to be available to the wider Node.js community and not limit it to working in CoffeeScript projects. This is not hard to do and there are several successful strategies. The chief among these are:

Compile to Javascript before pushing to the npm repository, so you are just uploading native Javascript
Compile to Javascript on installation, using the coffee-script npm package

There are potential advantages to both but for Troll-opt, I chose to compile my code at installation time rather than beforehand. There is not much overhead to doing this and the advantage is that it makes it rather easy to facilitate continuous integration (i.e. with something like Travis CI) because you can rely on the configuration in package.json to tell the CI system how to install and run the tests.

Here is all it takes to distribute things this way:

Get the right configuration in package.json
Add a .npmignore file and a .gitignore with the right settings

The first of those really only involves making

CoffeeScript Testing on Travis CI

relistan — Sat, 08 Dec 2012 20:18:25 +0000

I have recently been working on Troll-opt, a powerful but simple command line parser for Node.js that was inspired by William Morgan’s Trollop gem for Ruby. I wanted to get CI running and Travis CI is a free for open source service that makes continuous integration fairly painless. It wasn’t obvious how to get it running with CoffeeScript, though, so here is what I learned.

I had already set up a Cakefile that builds and runs the tests on my project. I’m testing with Pivotal’s Jasmine, which is a lot like RSpec and thus very familiar to me, and a darn nice tool. In combination with jasmine-node, it works great from the command line, too.

The Cakefile looks like this:

{spawn} = require 'child_process'
{print} = require 'util'
fs      = require 'fs'
 
spawnAndRun = (command, args, callback) ->
  subproc = spawn(command, args)
  subproc.stderr.on 'data', (data) ->
  process.stderr.write data.toString()
  subproc.stdout.on 'data', (data) ->
  print data.toString()
  subproc.on 'exit', (code) ->
  callback?() if code is 0
 
test = (callback) ->
  spawnAndRun 'jasmine-node', ['--coffee', 'spec'], callback
 
build = (callback) ->
  fs.mkdir 'lib', 0o0755
  print "compiling..."
  spawnAndRun 'coffee', ['--compile', '--output', 'lib', 'src'], callback
  print "\n"
 
task 'test', 'Run all tests', ->
  test()
 
task 'build', 'Build the Javascript output', ->
  build()

There’s no rocket science there, then. It just creates a build and a test task that make it easy to invoke from Travis CI. You could alternatively do this with a custom script, or with commands directly inserted in the Travis CI configuration (see below). So, on to the actual integration.

First, you need your package.json file to contain some things it might not otherwise. You need all the things you should have normally, including devDependencies to make sure this module can be built and tested. But then, I added a scripts section which tells npm and also Travis CI, what to to do test and install this:

"devDependencies": { 
    "jasmine-node": ">=1.0.26", 
    "coffee-script": "latest" },
"scripts": {
  "test": "./node_modules/.bin/cake test",
  "install": "./node_modules/.bin/cake build"
},

The devDependencies guarantee that CoffeeScript and Jasmine are installed. The install script ensures that our code has been compiled to Javascript and put into the lib directory.

Note the paths. These assume that you are installing your npm packages locally, hence the ./node_modules/.bin path which is where npm puts binaries from installed packages. The “test” command will be invoked by npm when you run npm test. This ensures that Travis knows how to test your application.

Travis CI uses a YAML file to configure it, called .travis.yml. We need to tell Travis that we are using Node.js so that it doesn’t try to build a Ruby project. Then, we need to tell it which commands to run. In this case, we are going to install the coffee-script package locally to make sure that cake is in the right path. The before_script definition will call a script before any tests are run. before_install, will likewise run before the normal npm install is called on the dependencies in the package.json.

language: node_js
node_js:
  - 0.8
  - 0.6
before_script:
  ./node_modules/.bin/cake build
before_install:
  - npm install coffee-script

Be sure to login to Travis and set up your GitHub account and authorize the commit hook from GitHub to trigger it. This is a few simple steps. Now, when you push to GitHub, your CoffeeScript should get compiled, and your tests should get run. Status will be reported on Travis, or optionally via an image you link into your README.md on GitHub.

A Works-Anywhere Config For SSH Tunneling

relistan — Sun, 22 Apr 2012 14:44:02 +0000

This is not a new topic but I think this is a novel way of handling it. I’ve had a number of solutions for SSH’ing through a jump host over the years. Some have worked better than others. I recently built a setup that seems to work very well and that I am happy with so I thought it was worth sharing.

Generally you have a jump box if you have a remote network of isolated hosts and you want to have a security pinch point on inbound SSH sessions. You generally fortify the jump host as a bastion host and only allow SSH sessions on the remote internal network when they are inbound from the jump box. This is a fairly common architecture and I’ve seen it at a score of companies. What becomes a pain is initiating two ssh sessions every time you want to get to a host inside the remote network, or if you want to scp a file, you end up copying it twice. So people often set up scripts for getting around this and connecting directly to the remote internal host over some kind of SSH tunnel. Now you either need a config entry for each host, or a remote domain name that you can use to wildcard a Host entry in your SSH config. But then you are not just typing the hostname each time you connect, you are also typing the domain name. It’s annoying.

What I’ve set up is, I think, much better. It connects a backgrounded tunnel to the jump host, running a SOCKS proxy. All future connections are then tested to see if they would work directly, and if so they are connected. Otherwise they are proxied over the SOCKS tunnel to the jump host. It’s simple, and having used it in production now for awhile, it seems to work pretty well. I don’t have to use the domain name for the host on the other side of the tunnel, and I don’t have to manage ssh config records. Here’s the only entry you need in your SSH config:

You’ll want to replace ‘username’ and ‘jumphost’ with your actual remote username and jump host name. You may also want an SSH config entry for that host to make sure it uses your RSA key for authentication. And here is the script that runs it all, to be installed in ~bin/ssh-proxy.sh:

Enjoy!

Discovering Pending Rails Migrations With Git

relistan — Sat, 31 Mar 2012 13:31:56 +0000

At MyDrive, Gavin and I came up with what I think is a pretty novel solution to discovering new Rails migrations at deployment time. We deploy often but not on every commit. We usually know about new migrations from reading the commits—but no one is perfect and sometimes you miss a commit or three. The standard checks for pending migrations involve deploying the new code and verifying the schema version against the database before reloading the app. This works but it’s slow and you don’t find out about the migrations until the code has been pushed to at least one system. Automating this is still a good idea. But it’s often nice to find out about pending migrations right up front. Sometimes you just don’t want to deploy right now if there are pending migrations.

But, do we need to talk to the DB to find this out? Doesn’t git tell us what has changed in our code? Why yes, it does.

We tag each deployed ref with a git tag representing the deployment environment (Capistrano stage) and the time and date. This is tells us what code was running in any environment at any time. We now also have a moving git tag that identifies the last deployment to any environment. After each deployment we move the tag to the current HEAD ref. Finding new migrations is now as simple as git diff on the db/migrate tree against the previous deployment tag for this environment. Capistrano does this as part of the deployment and lets us choose to abort if we don’t want to deploy with migrations. It’s not perfect: it will generate false positives on any change to any file in the migrations tree, but old migrations are rarely changed. Here’s what it looks like:

Notably this needs to be done from a clean tree because you will be changing git tags on the local installation. So it should either be done from a deployment host or from a clean checkout of your code. It’s best to detect this in your deployment scripts as well so that you don’t get into trouble. If there is interest, we could probably release this as a Capistrano plugin. If you decide to implement it yourself, here are some notes on helpful git commands for doing so.

Update: I have released two gems to do this for you. The first is called capistrano-deploytags and handles all of the tagging for you at deployment time. The second, capistrano-detect-migrations does what I described above. It requires the functionality from capistrano-deploytags.

HTTP Header Hell Starring X-Real-IP and X-Forwarded-For

relistan — Wed, 15 Feb 2012 20:58:03 +0000

It’s always fun to spend a whole day debugging something that should be simple. Actually I think it’s always the things that should be simple that end up in a day of debugging. Sharing tales of woe can sometimes help people. Or at least people can laugh at your misery. Here’s one such tale.

Happy Beginnings

It all starts out like your average fairy tale. For one of our production apps, we have a setup with a load balancer and some app servers behind it. In this case the load balancer is HAproxy and the app servers are running Rails with a Sinatra application mounted, all on top of Phusion Passenger on Nginx. This is a great setup for production systems.

The Unhappy Middle Bit

No story that is worth reading has a happy middle, so here’s where it went wrong. The load balancer system needs to handle SSL termination which HAproxy does not support. HAproxy is, however, a great load balancer through which I have in other jobs run absolutely massive traffic without issue. It has the benefit of actively monitoring your servers so that it knows they are not responding before some request gets hung up checking for you. It has great capability for routing traffic based on all kinds of HTTP header information. Finally, it has a great stats page that gives you a lot of live information about the services it is handling. We wanted to use HAproxy.

There are a number of solutions for running HAproxy where SSL termination is needed. The best of these is this right at hand. Nginx supports SSL termination, is really lightweight, and is event-based. It scales to massive proportions without much trouble. At an unnamed previous employer we were doing 35,000 rpm in production through a single Nginx install. I know Nginx works fine as a load balancer, but it’s nowhere near as nice to run as HAproxy in production.

But… one final requirement, self-imposed for purposes of debugging, was that the app server logs actually contain the original source address of the client. This now means that the original IP address needs to be relayed from Nginx to HAproxy, to Nginx, to Rails and Sinatra. The easiest way to do that is to set HTTP headers like X-Forwarded-For or X-Real-IP on the load balancer. X-Forwarded-For is more common and lots of things muck with it. I thought, to avoid trouble, let’s just use X-Real-IP in the SSL terminator’s nginx.conf. HAproxy will leave it alone and pass it along to Nginx and Rails/Sinatra on the app servers. I can have Nginx log it on the app servers and it will be available to put in the production.log as well.

WRONG.

This all seemed to work fine in Rails. Just as expected. Alas, any attempt to connect to the Sinatra apps mounted on the Rails installation resulted in 403 and an entire page body consisting of the word “Forbidden”. This was from our Sinatra app as well as from Resque-web.

First clue: connecting directly to HAproxy without going through the SSL-terminating Nginx works as expected.

Second clue: a tcpdump of the traffic sent to HAproxy from Nginx shows that both X-Real-IP and X-Forwarded-For are set but only X-Forwarded-For is set by HAproxy.

Poking around with curl and netcat reveals that I only have the problem when both headers are present. Then after poking at this for awhile I discover that it only doesn’t work when they are both set and NOT the same. What’s going on here? Well Nginx is diligently setting X-Real-IP as expected, but HAproxy is configured to add X-Forwarded-For (leftover from a previous config). So X-Real-IP is always the real client and X-Forwarded-For is always the IP address of the load balancer.

The Happy Ending

So what? There is a gem you perhaps don’t know about that is getting invoked in your application stack. The culprit: rack-protection. This gem does a lot of sanity checking and validation on requests headed into a Rack stack. It is included in Rails but something is Rails overrides this particular behavior. Sinatra triggers it, even mounted on top of Rails. Grepping around revealed this test case in rack-protection:

  it 'denies requests where IP and X-Forward-For are spoofed but not X-Real-IP' do
    get('/', {},
      'HTTP_CLIENT_IP'       => '1.2.3.5',
      'HTTP_X_FORWARDED_FOR' => '1.2.3.5',
      'HTTP_X_REAL_IP'       => '1.2.3.4')
    last_response.should_not be_ok
  end

So this behavior is a, ermm, “feature” of rack-protection. Knowing is half the battle–or well, all of it. A quick one-line deletion from the HAproxy config, a new Chef run on the load balancer and we have a working app stack, SSL and all.

You may now either sigh with relief because I’ve helped you solve the same problem or laugh at my pain. For the therapeutic release of your laughter I do charge a small fee. Simply post your bank account numbers in the comments section and it will be withdrawn automatically.

Kiwi Wikitext Parser at Wikimedia Data Summit

relistan — Wed, 02 Feb 2011 03:09:35 +0000

I will be presenting the Kiwi wikitext parser at the Wikimedia Data Summit at O’Reilly’s headquarters in Sebastopol, CA on Friday. Kiwi is a formal parser written using a PEG and relying on the Peg/Leg tool from Ian Piumarta. It’s in C and supports most of MediaWiki’s syntax in a more or less tolerant manner. This was a side project at AboutUs that Thomas Luce started one Saturday. I joined up with him shortly afterward and we have now got a mostly complete parser. There is a lot of room for improvement, but it is fast, works in semi-production at AboutUs.org, and we hope it is a promising example of what can be done with MediaWiki’s syntax. We’re hoping we can build some community support behind by presenting at the Data Summit. If you want to check out the parser first hand you should visit Sam Goldstein‘s Ruby/Sinatra-based wiki site at DrasticCode.com.

Kiwi has been released under the permissive BSD 3-clause license and can be found on GitHub.

Convert Chef Cookbooks to Puppet Modules

relistan — Wed, 19 Jan 2011 04:03:33 +0000

What if you, for any number of reasons, needed to convert Chef cookbooks to Puppet modules and it were convenient to do it in an automated way? How hard the task would be to convert the cookbook depends largely on how off the ranch the recipes are with respect to the Chef DSL. Since Chef recipes are just Ruby code, you can write whatever you want into the recipe. But if the recipes were written in a way that mostly sticks with the Chef DSL you can get fairly clean Puppet output in a really simple way: take advantage of the fact that Chef’s DSL is Ruby, and write a system for evaluating the Chef DSL and generating Puppet syntax output. I found this to be a surprisingly legitimate case for heavy meta-programming as much of the Chef DSL translates pretty well directly to Puppet.

Hopefully no one is ready to start a flame war at this point. I am not suggesting that anyone use one system or the other. I recently had the need to convert a lot of site-specific Chef code to Puppet so I wrote a tool to save me a lot of time. The general approach is to convert as much of the Chef code to a module formatted for use in Puppet as possible. You will need to edit the module when it is done being converted. Chef makes an assumption about the relationship between resources based on order. Puppet does no such thing. So you will likely need to managed the dependencies yourself. However, if you specified them explicitly in Chef, they will work in Puppet out of the box in most cases. There are currently only three options you need to specify to run the tool:

Usage: convert.rb [options]
-c, --cookbook COOKBOOK          Chef Cookbook directory (e.g. contains /recipes, /attributes...)
-o, --output-dir OUTPUT_DIR      Output directory (where modules are written)
-s, --server-name SERVER_NAME    The name of the Puppet server to be used in puppet:// URLs

Here is what the converter handles:

Converts cookbooks to a Puppet module directory structure
Converts recipes to mostly-right Puppet manifests in the proper location
Copies templates to the correct location in the Puppet module
Downloads remote_file resources into the Puppet module’s files directory
Tries to replace Chef’s node[:some][:var] variables with $some_var format
Edits templates to replace variable substitutions with the same format
Makes an attempt to replace not_if and only_if blocks with Puppet equivalents

I am still working with the tool and it will probably evolve over time. But it has a pretty good success rate now. Here’s some sample output:

$ ./convert.rb -c /tmp/chef-cookbooks/cookbooks/xen/ -o /tmp/asdf -s localhost.localdomain
Cookbook Name:   xen
Recipes Path:    /tmp/chef-cookbooks/cookbooks/xen/recipes
Templates Path:  /tmp/chef-cookbooks/cookbooks/xen/templates/default
Files Path:      /tmp/chef-cookbooks/cookbooks/xen/files/default
Output Path:     /tmp/asdf/xen
Working on recipe... /tmp/chef-cookbooks/cookbooks/xen/recipes/default.rb
Working on recipe... /tmp/chef-cookbooks/cookbooks/xen/recipes/dom0.rb
Fetching /xen/xen-Config.mk from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/xen-Config.mk...
Fetching /xen/linux-2.6.31.12.tar.bz2 from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/linux-2.6.31.12.tar.bz2...
Fetching /xen/patch-kernel.sh from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/patch-kernel.sh...
Fetching /xen/xen-3.4-testing.tar.bz2 from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/xen-3.4-testing.tar.bz2...
Fetching /xen/xen-patches-2.6.31-12.tar.bz2 from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/xen-patches-2.6.31-12.tar.bz2...
Fetching /xen/make-initrd.sh from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/make-initrd.sh...
Fetching /xen/initramfs_conf.tar.bz2 from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/initramfs_conf.tar.bz2...
Fetching /xen/07_xen from some-bucket.s3.amazonaws.com to /tmp/asdf/xen/files/07_xen...
Modifying template /tmp/chef-cookbooks/cookbooks/xen/templates/default/xen-kernel-config.erb...

Find it on GitHub: http://github.com/relistan/chef2puppet