Gerhard Lazu web feed

Deploy a Ruby application with Capistrano, rvm & bundler

2011-09-12T00:00:00+01:00

The wastelands of “sysadminia”

If you want to get your code out there, you - as a developer - need to match your most fierce adversary: the “sysadmin”. It can’t be done from the comfort of your shiny Mac development environment, you need to go out in the wastelands of “sysadminia” where the fans howls 24/7 and sysadmins roam with their mighty htop, tmux and pkill. There will be no clever plugins, no IDEs, no snippets, just plain text. You will need an ally.

There are 2 weapons in particular which make all the conventional sysadmin tools look cheap in comparison: Chef and Puppet. You should pick one and make it your horse: it will make a life and death difference in the wastelands. Both of them are fine choices, go with whatever makes you most comfortable, but know this: Chef comes with a knife. I don’t use it myself, true men fight with their bare hands, but just in case you’re a stabby proc -> person, it will suit you.

So let’s assume that you’ve chosen Chef, your chances are already looking good. Preparation is everything, so let’s pick up some gear first.

Githubia

The best place for finding the most awesome gear is Githubia. It’s a world of magic with unicorns and eternal queues. Some say that when developers die, they all hope to end up in this land of magic.

I will pick the RVM & bootstrap cookbooks.

Once you have those 2 cookbooks, this is a sample role which configures a server with rvm, latest Ruby 1.9.2 and sets up a new user under which we’ll deploy our Ruby app.

name          "ruby_apps"
description   "Ruby Apps"
run_list      "recipe[build-essential]",
              "recipe[git]",
              "recipe[ssh]",
              "recipe[sudo]",
              "recipe[bootstrap]",
              "recipe[bootstrap::users]",
              "recipe[rvm]",
              "recipe[rvm::users]",
              "recipe[bootstrap::ruby_apps]"

default_attributes(
  :ssh => {
    :password_authentication => "no",
    :permit_root_login => "no"
  },
  :sudo => {
    :groups => ["admin"],
    :users => ["ubuntu"] # the default sudo user on Ubuntu-based EC2 AMIs
  },
  :bootstrap=> {
    :users => {
      :gerhard => {
        :admin => true,
        :deploy => true,
        :keys => [
          "my-sha-is-longer-than-your-sha",
        ]
      }
    }
  },
  :rvm_rubies => ["1.9.2"],
  :ruby_apps => ["rubyapp"]
)

The other cookbooks are mostly taken from the gchef repository, everything else comes from opscode cookbooks.

Let’s make sysadmins drool over our deploys

Once you cook your server with the above role, it will be all set up for deploying. Before we go to the deployment files, pick some good hints from this abridged ~/.ssh/config file:

# to prevent SSH connections timing out. This will poll the server every 60".
ServerAliveInterval 60
ServerAliveCountMax 3
# use my local SSH key to authenticate on the remote hosts (when tunelling)
ForwardAgent yes

Host ec2-rubyapp
  Hostname ec2-50-19-201-63.compute-1.amazonaws.com # deploying to the cloud FTW!
  User gerhard # it will default to your username if omitted

Here’s a deploy.rb, taken from a production app.

# makes the output nice and colourful
# gem 'capistrano_colors'
require "capistrano_colors"

default_run_options[:pty] = true  # Must be set for the password prompt from git to work
ssh_options[:forward_agent] = true

set :application, "rubyapp"
set :repository,  "git@github.com:gerhard/#{application}.git"
set :user, application
set :use_sudo, false
set :default_shell, "/bin/bash" # required for rvm scripts to work properly

server "ec2-rubyapp", :app, :web

set :scm, :git
set :branch, "master"
set :deploy_via, :remote_cache
set :deploy_to, "/home/#{application}"
set :keep_releases, 10

set :myself, `whoami`.chomp # if more users have the same user on the server,
# with deploy privileges and ssh keys set up properly, they can deploy!
set :rvm_path, "/usr/local/rvm/scripts/rvm"

def close_sessions
  sessions.values.each { |session| session.close }
  sessions.clear
end

def with_user(new_user, &block)
  initial_user = user
  set :user, new_user
  close_sessions
  yield
  set :user, initial_user
  close_sessions
end

And now for the Capfile where all the magic happens. It assumes that you are using foreman for managing your app processes. If you’re not using it already, drop everything and get it set up straight away: David - the creator himself - introduces Foreman.

load 'deploy' if respond_to?(:namespace) # cap2 differentiator
load 'config/deploy' # remove this line to skip loading any of the default tasks

namespace :deploy do
  task :stop, :roles => :app, :except => { :no_release => true } do
    with_user myself do
      run [
        "sudo [ -f /etc/init/#{application}.conf ]",
        "[ $(sudo status #{application} | grep -c running) -eq 1 ]",
        "sudo stop #{application} || exit 0"
      ].join(" && ")
    end
  end

  task :generate_upstart, :roles => :app, :except => { :no_release => true } do
    with_user myself do
      run [
        "source ~/.profile",
        "cd #{current_release}",
        "bundle exec foreman export upstart /tmp -u #{application} -a #{application}",
        "sudo [ -f /etc/init/#{application}.conf ] || exit 0",
        "sudo rm /etc/init/#{application}*.conf",
      ].join(" && ")
      run "sudo mv /tmp/#{application}*.conf /etc/init/"
    end
  end

  task :start, :roles => :app, :except => { :no_release => true } do
    with_user myself do
      run [
        "sudo [ -f /etc/init/#{application}.conf ]",
        "[ $(sudo status #{application} | grep -c running) -eq 0 ]",
        "sudo start #{application}"
      ].join(" && ")
    end
  end

  task :restart, :roles => :app, :except => { :no_release => true } do
    stop
    generate_upstart
    start
  end
end

task :ensure_permissions do
  run "chmod g-w #{deploy_to}"
end
after 'deploy:setup', :ensure_permissions
after 'deploy:update', :ensure_permissions

task :bundle do
  run [
    "source #{rvm_path} && cd #{release_path} &&",
    "bundle install",
    "--gemfile #{release_path}/Gemfile",
    "--path #{shared_path}/bundle",
    "--deployment --quiet",
    "--without development test"
  ].join(" ")
end
after 'deploy:finalize_update', :bundle

after 'deploy:update', 'deploy:cleanup'

I’m making use of upstart (running on Ubuntu, aren’t you?!?), bundler & rvm, everything just works. Chef takes care of all the burden related to getting system-wide rvm to play nicely with user shell profiles. I’m using bash on the server and defining it explicitly in Capistrano. Trying to make the default Capistrano sh shell option work properly with rvm is guaranteed to make your blood boil, I prefer to play with unicorns and rainbows instead.

It might not be the most elegant solution, but it works. Everything is nice and straightforward, tasks are pretty self explanatory and can be tweaked to your own taste. Go on then, try some painless deploys, the feeling is awesome!

UPDATE

I came across Tom’s gem after getting this deploy strategy in place, his tomafro-deploy gem is definitely worth checking out.

2011 Resolutions

2011-01-01T00:00:00+00:00

Be more pragmatic

Take this blog for example. I had the new version done for January 2010. A year has gone by while I kept switching between Nesta, Toto, Padrino and Sinatra, yet the blog was still not live! Now I’m running on Jekyll, I can now focus on content and experiment with newer, better options as time permits.

Get shit done, then iterate

Lose 12kg (or 25lbs)

I cycle as much as 150 miles every week, daily commute being 30 miles, but my appetite rivals a hyppo’s. December is not a good month for exercising anyways, you tend to stay around the house more, shop more and indulge in all sorts of treats. A little baby limits your activities a lot, so it all adds up. I also have a sweet tooth, so when you combine that with a desk job, you gain lots of pounds, very quickly. That being said, I’m making this my top priority for 2011. After all, I have a honeymoon to look forward to.

Stop eating shit, work out more

Switch to vim

I have been using Textmate for years now, I’ve only played with vim briefly, most of it was in my gap between Macs. I had an older desktop which made a snazzy little Linux development machine, vim was the only serious text editor. There is gedit that comes close to Textmate, but vim is the benchmark on Linux. Yes emacs is the dog’s nuts also, but I just don’t get it, so there, flame war avoided.

Don’t take shit from your text editor, make it bend like a porn star

Become a British citizen

I have been living in the UK for over 7 years now, I can see myself living here for many more. I got married here, my baby boy was born here, I am working in London with awesome people, there’s no real reason to want to go anywhere else. It is said that it takes about 7 years for every single cell in the human body to get regenerated, so since all of them are British in mine, it’s time to get the paperwork done.

Don’t accept bullshit, do the honourable thing

Bring out glz_custom_fields v2.0

For those that don’t know, I’ve written this plugin for Textpattern in autumn 2007 and it had stellar success. It a nutshell, it turbo charges custom fields. If articles are regarded as objects, custom fields would be their properties. glz_custom_fields is a super stable plugin which powers a lot of websites. Adding new features to a mature, untested and fairly complex PHP script is not something I look forward to. I have been considering various approaches, I am planning to start work on the new major version early this year. With a bit of luck, it will be ready for TXP 5.

Reinvent some legacy code and make it shit hot

Take our honeymoon

Having a baby is a big deal, more so if you live away from all your family. We wanted to tackle the big task first, we knew that the honeymoon will be appreciated even more after the baby. We still have almost another year to go, but boy are we looking forward to it! The expectations are not high, we just want to get away from the daily baby ruled routines and soak in some sun and warm, white sand with turquoise, crystal clear water. Cuba, you cannot come soon enough!

Take a break from cleaning shit, go on an awesome holiday

Release my own Ruby app

Everything that I’ve been doing for the past 3 years will converge in this one app. I’ve been working on various aspects of it for over a year now, it should be all coming together nicely in 2011. I don’t want it to become a cash cow or my retirement fund. It’s something that I enjoy doing and would like to continue working on in my spare time. If others will find it useful, I’ll be glad, if not, my life will be better having it, and that’s all that matters.

Stop dreaming shit, this is it, make it work

That’s me for 2011, what about you?

Using Chef to manage Amazon EC2 instances, Part 2

2010-08-11T00:00:00+01:00

Now that Amazon AWS authentication is out of the way and Chef client is set up locally, let’s create our first EC2 instance.

I am only interested in Ubuntu AMIs. I’ll set up the smallest available instance for testing, I’ll choose US East & Lucid Lynx 10.04 LTS 32bit (all reflected in the AMI id). I am also going to choose my availability zone by using the -Z option. This comes in handy for when you get the Availability Zone Error.

$ knife ec2 server create -i ami-5ca44e35 -f m1.small -S ec2-keypair -Z us-east-1a

Small EC2 instances are 32bit only

I could have created a new EC2 instance with ec2-run-instances as well, but I will be using knife as much as I can because it holds the secret to eternal sysadmin nirvana.

Now that the server is created, let’s set up chef-client and register it with our Opscode platform user.

$ knife bootstrap <ec2-public-ip-address> -N my-first-ec2-instance -x ubuntu --sudo

You’ll be shown the public IP after the server is created. I’m giving my new instance a name my-first-ec2-instance, I’m logging in as the ubuntu user (root is disabled by default), and I’m running all commands as sudo.

After the above is finished, let’s see what we have:

$ knife node list
[
  "my-first-ec2-instance"
]

Congratulations, you’ve created your first EC2 instance without breaking a sweat!

Your chef cookbooks

We know that knife is the bees knees. But how do you use it with your EC2 instance and available cookbooks?

I am assuming that you have followed the Getting Started with Chef tutorial closely. If you have, you should have your chef-repo locally and knife configured. Let’s get some relatively simple cookbooks: MongoDB & Redis. Unpack and place them in your cookbooks folder, then upload them to your Opscode platform user:

$ knife cookbook upload redis
$ knife cookbook upload mongodb

Let’s create a new role for our EC2. We create a new file roles/master-db.json containing:

{
  "name": "master-db",
  "default_attributes": {
    "chef": {
      "server_url": "https://api.opscode.com/organizations/<your-organization>",
      "cache_path": "/var/chef/cache",
      "backup_path": "/var/chef/backup",
      "validation_client_name": "<your-validator>",
      "run_path": "/var/chef"
    }
  },
  "json_class": "Chef::Role",
  "run_list": [
    "recipe[redis::source]",
    "recipe[mongodb::source]"
  ],
  "description": "Master DB",
  "chef_type": "role",
  "override_attributes": {
  }
}

Upload the role, then assign it to your new EC2 instance:

$ knife role from file roles/master-db.json
$ knife node run_list add my-first-ec2-instance "role[master-db]"

Log into your EC2 instance, change your ubuntu user password and run chef-client as sudo:

$ ssh ubuntu@ec2-xxx.compute-1.amazonaws.com
$ sudo -i
$ passwd ubuntu
<mysupersecurepassword>
$ exit
$ sudo chef-client

And that’s it! Your new Amazon EC2 instance now has a very recent MongoDB & Redis set up.

If you want to modify any cookbooks, copy the entire cookbook into your site-cookbooks folder and get hacking.

Cleaning up

I don’t want to leave that EC2 instance running, it was only created for testing purposes after all:

$ knife ec2 server list # I need the server ID
$ knife ec2 server delete i-xxxxxx
$ knife node delete my-first-ec2-instance
$ knife node list
[

]

Using Chef to manage Amazon EC2 instances, Part 1

2010-08-10T00:00:00+01:00

So now that you have you Amazon AWS account, you must feel smug and think that you’re on the top of the cloud computing world. Not so quick tiger. Let’s say that you fire up an EC2 instance, what next? How are you going to configure it? Let’s tackle this step by step. I’m on OS X by the way, you should be too.

Keys, Certificates & authentication

Install brew if you don’t have it, then create an .ec2 folder:

$ brew install ec2-api-tools
$ mkdir ~/.ec2

Create a file in the newly created folder called e.g. gerhard_aws containing:

#!/bin/sh

# Setup Amazon EC2 Command-Line Tools
export EC2_ACCESS_KEY="<your-access-key>"
export EC2_SECRET_KEY="<your-secret-key>"
export JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Home"
export EC2_PRIVATE_KEY="$(/bin/ls $HOME/.ec2/pk-*.pem)"
export EC2_CERT="$(/bin/ls $HOME/.ec2/cert-*.pem)"
export EC2_HOME="/usr/local/Cellar/ec2-api-tools/<your-version>/jars" # ec2-version w/o date

The Access Key, Secret Key and Certificate should be easy to find in your Amazon AWS account. You’ll find them in Account > Security Credentials. You should be given the Private Key when creating your first keypair, but I’m not 100% sure. You need to move this Private Key into the same .ec2 folder.

In your shell’s configuration file add the following and reload the shell environment:

if [[ -s $HOME/.ec2/gerhard_aws ]] then
  source $HOME/.ec2/gerhard_aws
fi

After generating a new EC2 keypair, put everything starting with -----BEGIN RSA PRIVATE KEY----- into a new file called the same name as the keypair (ec2-keypair in my example). When done, let’s add this key to our agent so that we can ssh into our EC2 instances with no extra hassle:

$ ec2-add-keypair ec2-keypair
# this will output the new keypir contents
$ ssh-add ~/.ec2/ec2-keypair
$ ssh-add -l # make sure it was added!

As an extra tip, keep your .ec2 folder on Dropbox and just symlink it to your home folder:

$ ln -nfs ~/Dropbox/dotfiles/.ec2 ~/.ec2

Create Opscode account & install Chef locally

There is an excellent, up-to-date write-up on Getting Started with Chef, follow it!

Make sure that you install the following gems, they will save you a good ~~ba…~~ head scratch later:

$ gem install net-ssh-multi fog highline

At step 2, Create your Chef repository, I have created another folder on Dropbox and just symlinked it from there. I have also modified the cookbook path and added the AWS keys to .chef/knife.rb:

cookbook_path ["#{current_dir}/../site-cookbooks", "#{current_dir}/../cookbooks"]
# Amazon AWS
knife[:aws_access_key_id]     = ENV['EC2_ACCESS_KEY']
knife[:aws_secret_access_key] = ENV['EC2_SECRET_KEY']

We’ll create our first EC2 instance & configure it in the second part:
Using Chef to manage Amazon EC2 instances, Part 2