Cosmin Stejerean (offbytwo)

Audit your EC2 infrastructure with source control

2012-08-03T00:00:00-07:00

Audit your EC2 infrastructure with source control

03 August 2012 - Dallas, TX

You are performing a routine analysis of request logs on an internal web server when you notice a series of interesting requests from 10.191.12.13. A quick search determines that, as of this moment, this address does not belong to any of your servers. The requests happened 7 days ago, and much has changed during that time. Can you tell which of your instances had that IP address 7 days ago?

Just to be sure, you review the security groups for this instance to make sure only internal traffic is allowed. You discover a rule that explicitly allows traffic from 10.191.12.13. Can you tell how long this rule has been present? Can you find the time period during which traffic from 10.191.12.13 was allowed, and yet that address did not belong to you?

These questions, and many others about the historic state of your infrastructure, could be answered easily if this information was present in a source control repository. You could then easily see when changes happened, browse to a specific point in time, and even use your source control infrastructure for things like email alerts.

This is where ec2audit comes in. It can write the current state of your EC2 instances, security groups and ELB volumes to a series of JSON or YAML files that are suitable for version control.

In order to set up ec2audit you need IAM credentials, a source control repository, and some way to run it on a schedule. For example you can use Jenkins to schedule the runs and Git for source control. Things like git log -S make it easy to find when things changed.

To install ec2audit, use pip or easy_install. You can also download a tarball and run python setup.py install.

You can then run ec2audit as follows

ec2audit -I <access-key> -S <secret-key> us-east-1 -o outputdir

You can also supply AWS credentials via the standard environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

The AWS credentials must be granted read access to the EC2 APIs. You should create an IAM user with only the adequate permissions. If you are using the AWS Console, you can use the Amazon EC2 Read Only Access policy template for convenience. The following policy will also work:


{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "EC2:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "elasticloadbalancing:Describe*",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "autoscaling:Describe*",
      "Resource": "*"
    }
  ]
}

Remember to run ec2audit regularly and version control the output. You can create an empty git repository (or use your SCM of choice), and you can run it on a schedule using cron or your CI server.

Deploying Django applications on Heroku

2012-01-18T00:00:00-08:00

Deploying Django applications on Heroku

18 January 2012 – Melbourne, Australia

For a long time Ruby developers enjoyed painless deployment to Heroku

The Python landscape was limited to Google App Engine for quite some time (and I do mean limited but I’ll save that for another time). Once Heroku was acquired by Salesforce it seems that the market for cloud hosting of Python applications has exploded. Now we have plenty of choices such as epio gondor appHosted DjangoZoom (some of these are still in private beta). There is also dotCloud that seems to support just about everything. It’s a good time to be a Python developer.

After their acquisition Heroku has released new features faster than ever. Their Cedar stack now officially supports Ruby, Node.js, Clojure, Java, Python and Scala. Let’s take a look at how we can deploy a fairly typical, albeit simple, Django application to Heroku.

Prerequisites: pip and virtualenv

Before we get started we’ll need to install pip and virtualenv

If you have setuptools installed you should be able to install both using:

sudo easy_install pip
pip install virtualenv

If you need more detailed instructions please take a look at installing pip.

Create git repository

You should already know how to create a new git repository.

git init myawesomeproject
cd !$

In case you are wondering, !$ expands to the last argument to the last command (myawesomeproject in this case).

Create virtual-environment

If you have been using virtualenv for a while you might be used to creatting virtual environments in a folder called ve, env or similar. For the best experience when working
with Heroku you should however create the virtual environment directly at the root of your checkout.

virtualenv --no-site-packages .
source bin/activate

You have now created and activated your virtual environment. You will need to run bin/activate every time you are working on this project. While we’re at it let’s also ignore the virtualenv artifacts. Put the following in your .gitignore file

/bin
/include
/lib
/share

While you’re at it you should also consider adding *.pyc to .gitignore.

Install dependencies and freeze

For a simple Django application you will only need Django and psycopg2 (to talk to Postgres). Install them using pip and then freeze the exact versions used to a file called requirements.txt. Heroku will use requirements.txt to automatically install your dependencies when you push.

pip install Django psycopg2
pip freeze > requirements.txt

When you add new requirements to your project you can pip install them directly and regenerate requirements.txt with pip freeze.

Create Django application

Now you can create a Django project

django-admin.py startproject myproject

and make it awesome…

Handling database migrations

You are going to quickly need to migrate your database schema. Fortunately you can use Django South to handle data and schema migrations. Install it with pip

pip install south

and re-create your requirements file.

pip freeze > requirements.txt

Add it to your INSTALLED_APPS in settings.py and start converting your applications using convert_to_south

myawesomeproject/manage.py syncdb
myawesomeproject/manage.py convert_to_south your_application

Let’s also tell South that our current database schema is up to date, by fake applying the initial migration.

myawesomeproject/manage.py migrate --fake your_application

Now every time you make a change to your Django models, you can create new migrations and apply them.

myawesomeproject/manage.py schema_migration --auto your_application
myawesomeproject/manage.py schema_migration migrate your_application

You can learn more about South, including using it for data migrations, by checking out the tutorial and documentation

Handling static files

In a more traditional hosting setup you might use Apache or Nginx to handle serving static files. When deploying to Heroku though you should consider hosting your static files in S3. Luckily Django can easily support a variety of storage backends, and the django-storages package allows you to easily use S3.

First, create a bucket in S3, using either the AWS Console or your favorite tool. Then, modify your settings.py and add the following values:

import os

AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = '<YOUR BUCKET NAME>'

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

STATIC_URL = 'http://' + AWS_STORAGE_BUCKET_NAME + '.s3.amazonaws.com/'
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'

Notice that we are using environment variables to store the AWS access key and secret key. While we are on this topic, if you are planning to open source the Django application you are deploying, consider also storing your SECRET_KEY in an environment variable.

SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY')

We are now ready to Deploy to Heroku.

Creating an environment in Heroku

Let’s start by installing the Heroku gem

gem install heroku

followed by creating a new Cedar stack in Heroku.

heroku create --stack cedar

Optionally, you might want to map your own domain name to your Heroku stack.

heroku addons:add custom_domains
heroku domains:add www.example.com
heroku domains:add example.com

You can find information on managing custom domains in Heroku here

Let’s add the necessary environment variables (do the same for SECRET_KEY if necessar)

heroku config:add AWS_ACCESS_KEY_ID=youraswsaccesskey
heroku config:add AWS_SECRET_ACCESS_KEY=yourawssecretkey

For extra security you should use the Identity & Access Management (IAM) service to create a separate user account with the following policy

 {
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::BUCKETNAME",
        "arn:aws:s3:::BUCKETNAME/*"
      ]
    }
  ]
}

This way if the credentials stored in Heroku are ever compromised, the attacker will only have access to the files stored in the bucket of this application.

Deploying application to Heroku

This is as easy as running running git push.

git push heroku master

Your application is now deployed, but you still need to configure the database

heroku run manage.py syncdb
heroku run manage.py migrate

You should now have a working application, but we have not yet deployed our static files.

heroku run manage.py collectstatic

At this point you should have a fully functional Django application deployed to Heroku with static files hosted in S3. If you are having problems you can investigate the logs with heroku logs. You can also consider turning on DEBUG temporarily, but don’t forget to turn this off. To make it easier to turn DEBUG on and off consider adding the following to your settings.py

DEBUG = bool(os.environ.get('DJANGO_DEBUG', ''))
TEMPLATE_DEBUG = DEBUG

Now you can turn debug on and off using heroku config:add DJANGO_DEBUG=true and turning it off with heroku config:remove DJANGO_DEBUG

Emacs + paredit under terminal

2012-01-15T00:00:00-08:00

Emacs + paredit under terminal (Terminal.app, iTerm, iTerm2)

15 January 2012 – Melbourne, Australia

I prefer to use Emacs in a full-screen terminal window. One problem that has plagued me until today though has been the lack of proper Control and Meta arrow combinations when working at the terminal. Especially when working with paredit which frequently involves the use of C-left, C-right, C-M-left, C-M-right and less frequently of M-up and M-down. To get paredit to work properly I kept switching to Cocoa Emacs (in case you didn’t know, you can install Cocoa emacs with brew install emacs --cocoa if you are using homebrew )

Today I decided to get to the bottom of this problem at any cost. First, I suspected that Control arrow combinations were not being sent properly by my terminal, in my case iTerm2. You can also fix this however for Terminal.app or the original iTerm.

iTerm2 key bindings

Select Profiles > Open Profiles… from the menu bar, or press Command-O and take a look at the default profile. Click on the Keys section. While you are here verify you have Left Option and Right Option as +Esc. For the arrow key fixes though you will need to add a series of key shortcuts. The easiest way to get started is select Load Preset… > xterm Defaults.

This will map C-up, C-down, C-right, and C-left to send the following escape sequences:

C-up    : Esc-[1;5A
C-down  : Esc-[1;5B
C-right : Esc-[1;5C
C-left  : Esc-[1;5D

It will also define Shift arrows and Control-Shift arrows, but we don’t care about those at the moment. These are not quite sufficient, but before we go any futher, let’s make sure we can get these to work in Emacs.

Check Control-arrow bindings within Emacs

Open up a new terminal window and then open emacs at the terminal with emacs -nw. Now, with paredit turned off, try C-left and C-right, which should most likely move a word at a time left and right respectively. To verify Emacs is picking up the correct keys you can also try C-h k for Describe key followed by the key combination. For example, C-h k C-left should display

<C-left> runs the command backward-word, which is an interactive
compiled Lisp function in `simple.el'.

It is bound to <C-left>, <M-left>, M-b, ESC <left>.

(backward-word &optional ARG)

Move backward until encountering the beginning of a word.
With argument ARG, do this that many times.

[back]

As long as TERM is set to xterm the above bindings should work automatically in Emacs. Should you have to define your own bindings for these escape sequences you could do so with


(define-key input-decode-map "\e[1;5A" [C-up])
(define-key input-decode-map "\e[1;5B" [C-down])
(define-key input-decode-map "\e[1;5C" [C-right])
(define-key input-decode-map "\e[1;5D" [C-left])

Note that input-decode-map is only defined starting with Emacs 23.

Paredit

At this point you should have C-left working outside of paredit. Now turn on paredit mode (M-x paredit-mode) and try C-left and C-right again. Chances are you will see [1;5D and [1;5C. If this happens only in paredit mode, then the culprit is most likely the bidning of M-[. You can figure this out by trying describe key again. If you try C-h k C-left you will most likley see

M-[ runs the command paredit-bracket-wrap-sexp, which is an
interactive Lisp function in `paredit.el'.

It is bound to M-[.

(paredit-bracket-wrap-sexp &optional N)

Wrap a pair of bracket around a sexp

M-[

(foo |bar baz)
  ->
(foo [|bar] baz)

Huh?

Where does M-[ come from?

Each time you press Control + left arrow the terminal will send the following sequence as defined above: ESC [ 1 ; 5 D. Emacs starts interpreting this sequence, but it gets an early match on ESC [ which is the same as M-[ and invokes paredit-bracket-wrap-sexp. We need to turn off this behavior, which we can do by putting the following in ~/.emacs


(require 'paredit)
(define-key paredit-mode-map (kbd "M-[") nil)

Once you load the above code, try C-left again in paredit. If that works, you are ready for the next step.

Add Meta-arrow and Control-Meta-arrow to iTerm2

The xterm Defaults only provided us with certain key bindings. Go back to the profile key bindings under iTerm2 and add bindings for the following:

M-up      : Esc-[1;4A
M-down    : Esc-[1;4B
M-right   : Esc-[1;4C
M-left    : Esc-[1;4D

C-M-up    : Esc-[1;8A
C-M-down  : Esc-[1;8B
C-M-right : Esc-[1;8C
C-M-left  : Esc-[1;8D

To do this, click on the + sign, type the key sequence, then under Action: select Send Escape Sequence and type in the escape seequence starting with [1;. You can look as an example at the values for Control-left and friends that were added when you loaded the xterm Defaults map.

Why these values? I have no idea. I looked at the escape sequences for plain arrow keys, Shift-arrow keys and Control-arrow keys and I decided to experiment a little in the neighboring spaces, using C-h k to figure out which key sequence is bound to what I want. If you find a better explanation please let me know.

iTerm

If you are using iTerm you can add key bindings as follows:

Bookmarks > Manage Profiles > Keyboard Profiles > xterm and under Key map settings: add (by clicking on the + sign)

Key: cursor left
Modifier: Control
Action: send escape sequence

[1;5D

Add the remaining key bindings using the above format and the values from the iTerm2 table.

Terminal.app

In Terminal.app you will need to add a few key bindings by going to Preferences > Settings > Keyboard. The end result will be the same as iTerm2 but the interface is slightly different.

Key: cursor left
Modifier: Control
Action: send string to shell

\033[1;5D

where \033 represents Escape. For the other keys refer to the map for iTerm2 above.

Scripted installation of Java on Ubuntu

2011-07-20T00:00:00-07:00

Scripted installation of Java on Ubuntu (with Bash or Puppet)

20 July 2011 – Dallas

Every few months I need to script the installation of Java on Ubuntu, and I always seem to forget quite how to do it. I also seem to fail at finding anything useful on Google. Most of the posts either skip critical steps or involve manual steps. So I’m going to document this here for future reference.

Bash version (add sudo as necessary):


add-apt-repository "deb http://archive.canonical.com/ $(lsb_release -s -c) partner"
apt-get update

echo "sun-java6-jdk shared/accepted-sun-dlj-v1-1 select true" | debconf-set-selections
echo "sun-java6-jre shared/accepted-sun-dlj-v1-1 select true" | debconf-set-selections

DEBIAN_FRONTEND=noninteractive aptitude install -y -f sun-java6-jre sun-java6-bin sun-java6-jdk

Given that I do most of my automation with Puppet these days, here is a Puppet class that will accomplish the same.


class sun_java_6 {

  $release = regsubst(generate("/usr/bin/lsb_release", "-s", "-c"), '(\w+)\s', '\1')

  file { "partner.list":
    path => "/etc/apt/sources.list.d/partner.list",
    ensure => file,
    owner => "root",
    group => "root",
    content => "deb http://archive.canonical.com/ $release partner\ndeb-src http://archive.canonical.com/ $release partner\n",
    notify => Exec["apt-get-update"],
  }

  exec { "apt-get-update":
    command => "/usr/bin/apt-get update",
    refreshonly => true,
  }

  package { "debconf-utils":
    ensure => installed
  }

  exec { "agree-to-jdk-license":
    command => "/bin/echo -e sun-java6-jdk shared/accepted-sun-dlj-v1-1 select true | debconf-set-selections",
    unless => "debconf-get-selections | grep 'sun-java6-jdk.*shared/accepted-sun-dlj-v1-1.*true'",
    path => ["/bin", "/usr/bin"], require => Package["debconf-utils"],
  }

  exec { "agree-to-jre-license":
    command => "/bin/echo -e sun-java6-jre shared/accepted-sun-dlj-v1-1 select true | debconf-set-selections",
    unless => "debconf-get-selections | grep 'sun-java6-jre.*shared/accepted-sun-dlj-v1-1.*true'",
    path => ["/bin", "/usr/bin"], require => Package["debconf-utils"],
  }

  package { "sun-java6-jdk":
    ensure => latest,
    require => [ File["partner.list"], Exec["agree-to-jdk-license"], Exec["apt-get-update"] ],
  }

  package { "sun-java6-jre":
    ensure => latest,
    require => [ File["partner.list"], Exec["agree-to-jre-license"], Exec["apt-get-update"] ],
  }

}

include sun_java_6

Things you (probably) didn't know about xargs

2011-06-26T00:00:00-07:00

Things you (probably) didn’t know about xargs

26 June 2011 – Bangalore, India

If you’ve spent any amount of time at a Unix command line you’ve probably already seen xargs. In case you haven’t, xargs is a command used to execute commands based on arguments from standard input.

Common use cases

I often see xargs used in combination with find in order to do something with the list of files returned by find.

Pedantic note: As people have correctly pointed out on Twitter and on Hacker News, find is a very powerful command and it has built in flags such as -exec and -delete that you can often use instead of piping to xargs. However people either don’t know about the options to find, forget how to invoke -exec with it’s archaic syntax, or prefer the simplicity of xargs. There are also performance implications to the various choices. I should write a follow up post on find.

Contrived examples warning: I needed something simple examples that would not detract from the topic. This is the best I could do given the time I had. Patches are welcome :)

Recursively find all Python files and count the number of lines
find . -name '*.py' | xargs wc -l

Recursively find all Emacs backup files and remove them
find . -name '*~' | xargs rm

Recursively find all Python files and search them for the word ‘import’
find . -name '*.py' | xargs grep 'import'

Handling files or folders with spaces in the name

One problem with the above examples is that it does not correctly handle files or directories with a space in the name. This is because xargs by default will split on any white-space character. A quick solution to this is to tell find to delimit results with NUL (\0) characters (by supplying -print0 to find), and to tell xargs to split the input on NUL characters as well (-0).

Remove backup files recursively even if they contain spaces
find . -name '*~' -print0 | xargs -0 rm

Security note: filenames can often contain more than just spaces.

Placement of the arguments

In the examples above xargs reads all non-white-space elements from standard input and concatenates them into the given command line before executing it. This alone is very useful in many circumstances. Sometimes however you might want to insert the arguments into the middle of a command. The -I flag to xargs takes a string that will be replaced with the supplied input before the command is executed. A common choice is %.

Move all backup files somewhere else
find . -name '*~' -print 0 | xargs -0 -I % cp % ~/backups

Maximum command length

Sometimes the list of arguments piped to xargs would cause the resulting command line to exceed the maximum length allowed by the system. You can find this limit with

getconf ARG_MAX

In order to avoid hitting the system limit, xargs has its own limit to the maximum length of the resulting command. If the supplied arguments would cause the invoked command to exceed this built in limit, xargs will split the input and invoke the command repeatedly. This limit defaults to 4096, which can be significantly lower than ARG_MAX on modern systems. You can override xargs’s limit with the -s flag. This will be particularly important when you are dealing with a large source tree.

Operating on subset of arguments at a time

You might be dealing with commands that can only accept 1 or maybe 2 arguments at a time. For example the diff command operates on two files at a time. The -n flag to xargs specifies how many arguments at a time to supply to the given command. The command will be invoked repeatedly until all input is exhausted. Note that on the last invocation you might get less than the desired number of arguments if there is insufficient input. Let’s simply use xargs to break up the input into 2 arguments per line

$ echo {0..9} | xargs -n 2

0 1
2 3
4 5
6 7
8 9

In addition to running based on a specified number of arguments at time you can also invoke a command for each line of input at a time with -L 1. You can of course use an arbitrary number of lines a time, but 1 is most common. Here is how you might diff every git commit against its parent.

git log --format="%H %P" | xargs -L 1 git diff

Executing commands in parallel

You might be using xargs to invoke a compute intensive command for every line of input. Wouldn’t it be nice if xargs allowed you to take advantage of the multiple cores in your machine? That’s what -P is for. It allows xargs to invoke the specified command multiple times in parallel. You might use this for example to run multiple ffmpeg encodes in parallel. However I’m just going to show you yet another contrived example.

Parallel sleep

$ time echo {1..5} | xargs -n 1 -P 5 sleep

real    0m5.013s
user    0m0.003s
sys     0m0.014s

Sequential sleep

$ time echo {1..5} | xargs -n 1 sleep

real    0m15.022s
user    0m0.004s
sys     0m0.015s

If you are interested in using xargs for parallel computation also consider GNU parallel. xargs has the advantage of being installed by default on most systems, and easily available on BSD and OS X, but parallel has some really nice features.

Backing up MySQL using EBS snapshots

2011-03-28T00:00:00-07:00

Backing up MySQL using EBS snapshots

28 March 2011 – Dallas

For running applications that use MySQL on AWS I highly recommend taking a look at Amazon’s Relational Database Service (RDS). The smallest RDS instance however costs around $80 per month which is prohibitively expensive for small side projects. RDS might also not offer all the control necessary for more complicated database requirements. Not using RDS however comes with the overhead of having to adminster a MySQL installation. Specifically we are going to take a look at backing up a MySQL server on AWS.

In addition to MySQL’s built-in database dump commands, we can also leverage Elastic Block Store (EBS) snapshots in order to create a copy of the entire data volume. Amazon takes care of doing snapshots incrementally by only copying modified blocks. We are going to focus on how best to backup a MySQL database using EBS snapshots in this guide.

Consistent Snapshots

Before we take a snapshot of an EBS drive we need to ensure that we are getting a consistent view of the data at that point in time. In order to achieve this I recommend using the XFS filesystem which allows us to freeze writes to the filesystem. We also need to lock the MySQL database and flush all of its data to disk. Eric Hammond has written an excellent tool called ec2-consistent-snapshot that allows to freeze XFS, flush MySQL to disk and lock it, take an EBS snapshot and restore writes to XFS and MySQL.

Security considerations

We also need to provide ec2-consistent-snapshot with the AWS credentials needed to create the snapshot. Because this tool will be running on the database machine and because it should run automatically in an unattended fashion we need to upload our AWS access keys to the database host. This poses a security risk that we can mitigate by placing adequate access control on the file containing our credentials. We can also take advantage of Amazon’s Identity and Access Management (IAM) to create a sub-account that has more limited permissions. Specifically we can create a sub-account that is only allowed to make the CreateSnapshot API call. We can then place those credentials on the database host. This way if the credentials are compromized the attacker is only able to create snapshots rather than having full access to our account.

Boto

My favorite tool for interacting with Amazon’s API programatically is boto, a Python interface to AWS started by Mitch Garnaat. Boto can be installed using pip, easy_install or the OS’ package manager.

Using IAM policies

I recommend creating groups wih very narrow permissions and then creating users that can be added to one or more groups based on the capabilites needed for that user. We’ll start by creating a group for the create snapshot capability.


import boto

iam = boto.connect_iam(<access key>, <secret key>)
iam.create_group('snapshoters')

We then need to create a group policy granting access to the CreateSnapshot API call. Policies are represented with a JSON fragment that can be generated using Amazon’s AWS Policy Generator to create custom policies. Here is what the policy that allows CreateSnapshot looks like.


{
  "Statement": [
    {
      "Sid": "Stmt3121317131060",
      "Action": [
        "ec2:CreateSnapshot"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

In the past I have had difficulties copying and pasting this JSON fragment into a Python interpreter due to formatting issues. I was able to work around these issues by leveraging the fact that JSON is for the most part also valid Python syntax. This means that we can do the following.


create_snapshot_policy = {
  "Statement": [
    {
      "Sid": "Stmt3121317131060",
      "Action": [
        "ec2:CreateSnapshot"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

We can then use Python’s built-in JSON library to dump this policy back to JSON when making the API call. Here is how to add this new policy to the ‘snapshoters’ group.


import json

policy_txt = json.dumps(create_snapshot_policy)

iam.put_group_policy('snapshoters', 'CreateSnapshot', policy_txt)

We can then create a new user and add this new user to this group.


iam.create_user('dbbackup')
iam.add_user_to_group('snapshoters', 'dbbackup')

Finally we need to generate access keys for this new user that we can feed to ec2-consistent-snapshot.


iam.create_access_key('dbbackup')

Putting it all together

We can now create the following script at /sbin/backup-database-volume. Replace the values in angle brackets with the access key and secret key from previous step. Also use the id of the EBS vvolumen containing MySQL’s data. This script assumes that this volume is mounted at /vol.


#!/bin/sh

description="mysql-data-`date +%Y-%m-%d-%H-%M-%S`"

ec2-consistent-snapshot \
--aws-access-key-id=<ACCESS-KEY> \
--aws-secret-access-key=<SECRET-KEY> \
--description=$description  \
--mysql --freeze-filesystem='/vol' <VOLUME-ID>

We also need to restrict this file so that it can only be viewed, modified or executed by root. Run the following

$ sudo chown root /sbin/backup-database-volume

$ sudo chmod 700 /sbin/backup-database-volume

Credentials for connecting to MySQL either can be specified by using a .my.cnf or by adding —mysql-username, —mysql-password to the ec2-consistent-snapshot snapshot command. These and other options of ec2-consistent-snapshot can be found by running

$ man ec2-consistent-snapshot

Getting started (quickly) with OpenStack's Swift

2010-11-10T00:00:00-08:00

Getting started (quickly) with OpenStack’s Swift

10 November 2010 – San Antonio

I was at the OpenStack Design Summit this week and I wanted to check out the latest release of Swift, OpenStack’s blob storage technology. The Swift All In One guide contains everything necessary to get started with Swift on a development machine, but it involves far too much effort for my tastes.

I decided to quickly automate the entire process with a single bash script. So now you can get started with Swift with two commands on a new Ubuntu 10.10 VM

wget http://offbytwo.com/scripts/try-swift-on-ubuntu.sh
bash try-swift-on-ubuntu.sh devauth

There is actually a far better way to get Swift up and running in an automated fashion using Chef (and possibly Vagrant), but I’ll leave that for another post.

Working securely for multiple clients

2010-02-06T00:00:00-08:00

Working securely for multiple clients

06 February 2010 – Dallas

As a consultant I often end up working with sensitive client information on my laptop. Since laptops have a tendency to get misplaced or stolen, I need to keep this client information encrypted. At the same time however I need to be able to work with this data effectively without having to jump through too many hoops.

I believe that effective security requires a degree of convenience, since most people will inevitable circumvent security measures that interfere with their work. Therefore working securely on a laptop needs to be as convenient and natural as possible. I have a system I have developed over time that allows me to easily work with confidential client information on my laptop. I hope this guide will be useful to anyone in a similar situation. The specific examples used in this post involve OS X, TrueCrypt and Maven. It should be possible however to extrapolate and apply the same techniques to other technologies you might be using.

The ideal state

At this point you might be wondering what I consider to be secure and yet convenient, so I’ll go ahead and describe my ideal setup. When I fire up a new Terminal I would like to type a single command to start working on a particular client project, such as

$ work_on_project_a

This single command should mount my encrypted volume if not already mounted, change directory to the respective project, and alter my PATH and environment variables accordingly so that any tools I use will just work as expected for the given project. So let’s get started.

Mounting the encrypted volume on demand

For storing information securely I prefer using TrueCrypt because it is free, reliable and cross platform. Here is an example function that will check if our TrueCrypt volume is mounted, and attempt to mount it if not. In addition to easily mounting the encrypted volume, I want to also be able to quickly unmount it, since leaving a volume mounted unnecessarily increases the risk of compromise. Let’s also add a function to work_on_project_a that allows us to unmount from the command line.


function work_on_project_a() {
    TRUECRYPT="/Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt"
    SOURCE="/Volumes/someclient.tc"
    DESTINATION="/Volumes/SomeClient"
    
    function dismount() {
        $TRUECRYPT -d $SOURCE
    }
    
    if [ -z "`ls $DESTINATION`" ]; then
        echo "Trying to mount..."
        $TRUECRYPT --mount $SOURCE
    fi
    
    cd $DESTINATION/project_a
    
    # more stuff will go here
}

Configure the environment

I want any project specific scripts to automatically be in my PATH after activating a project. Let’s make that happen by adding the following to the work_on_project_a function.


    OLD_PATH="$PATH"
    export PATH="$DESTINATION/bin:$PATH"

In addition to configuring PATH, this might be a good place to configure other environment variables, such as JAVA_HOME if your project requires a specific version of JAVA, etc.

Configure maven to store artifacts securely

If you are using Maven, or a similar tool that automatically downloads and install artifacts, then I recommend configuring it to store all artifacts securely inside of the encrypted volume. In order to do so create a maven-settings.xml file under your encrypted volume and override localRepository setting.


    <localRepository>/Volumes/SomeClient/mavenRepo</localRepository>

You might also want to configure the mirrors in case you have a project specific repository.


    <mirrors>
        <mirror>
            <id>internal</id>
            <name>Internal Maven Repo</name>
            <url>http://internal.maven.repo/</url>
            <mirrorOf>central</mirrorOf>
        </mirror>
    </mirrors>

Now you can override Maven’s global settings file in order to pick up the appropriate mirrors and local repository. The best way to do this is to create a mvn script inside of your project’s bin folder that contains the following

</code>
#!/bin/sh

/usr/bin/mvn -s /Volumes/SomeClient/maven-settings.xml $*

Further configurations

If you need to perform further environment configuration you can do so in the work_on_project_a function. For example, if you have Nginx installed you might want to override the default Nginx configuration with the project specific one. I find this easier than trying to juggle multiple configurations in one Nginx file with vhosts. So for example I would use

</code>
    sudo rm $NGINX_DESTINATION
    sudo ln -sf $DESTINATION/nginx.conf $NGINX_DESTINATION

You can do something similar with Apache, etc.

Full example of work_on_project_a

Here is a full example that goes above and beyond what we described so far by adding a function to clean up after ourselves, as well as modifying PS1 to show the project we’re currently working on.


function work_on_project_a() {
    TRUECRYPT="/Applications/TrueCrypt.app/Contents/MacOS/TrueCrypt"
    SOURCE="/Volumes/someclient.tc"
    DESTINATION="/Volumes/SomeClient"
    NGINX_DESTIONAT="/usr/local/nginx/conf/nginx.conf"
    
    if [ -z "`ls $DESTINATION`" ]; then
        echo "Trying to mount..."
        $TRUECRYPT --mount $SOURCE
    fi
    
    OLD_PATH="$PATH"
    OLD_JAVA_HOME="$JAVA_HOME"
    OLD_PS1="$PS1"
    
    export PATH="$DESTINATION/bin:$PATH"
    export JAVA_HOME="/path/to/java/1.5"
    export PS1="\[\033[01;32m\]PROJ_A:\[\033[01;34m\]\W$\[\033[0m\] "
    
    sudo rm $NGINX_DESTINATION
    sudo ln -sf $DESTINATION/nginx.conf $NGINX_DESTINATION
    
    cd $DESTINATION/project_a
    
    function deactivate {
        export PATH=$OLD_PATH
        export JAVA_HOME=$OLD_JAVA_HOME
        export PS1=$OLD_PS1
        cd ~
    }
    
    function dismount() {
        $TRUECRYPT -d $SOURCE
    }
}

Note that this doesn’t clean up nginx’s configuration, but that’s OK, next project I activate will configure nginx properly.

Conclusion

The function we just developed allows me to conveniently start working on a project by mounting the encrypted volume on demand and setting up my environment accordingly. It also provides 2 functions for deactivating this environment to return to the default environment and for unmounting the encrypted volume when done. I hope you find this useful.

Running nosetests as a git pre-commit hook

2008-05-22T00:00:00-07:00

Running nosetests as a git pre-commit hook

22 May 2008 – Chicago

I’ve started using git for all my development recently (since it integrates so nicely with svn). I wanted to experiment with running my test as a pre-commit hook in git. In case you’re curious all the hooks in git live in the hooks folder inside of .git

Inside of this folder you will see various example scripts. The names should make it obvious when each hook is supposed to run. For example the pre-commit file will run before a commit (before you’re even asked for the commit message). There are also hooks that can intercept the commit message, run after updates happen, etc. By default none of these files are executable, so git doesn’t actually run them. If you would like to execute a hook simply put your code in the correct file and mark it executable.

In the case of the pre-commit hook git will abort the commit if the pre-commit file returns with a status code other than 0. By default this file contains some perl code that checks for lines with trailing spaces and lines that have a space before a tab at the beginning. You can safely remove this code (I found the trailing space to be an annoying check)).

So let’s say you want to run your unit tests before each commit (and abort the commit if they fail). I’m going to use nose (a Python unit testing framework) as an example. To run your nose tests you can simply issue the nosetests command. This will discover your tests, run them and exit with status code of 0 if everything passed. So you can simply put


    #!/usr/bin/env bash
    nosetests

in your pre-commit file and now your unit tests will run and your commit will abort if the are test failures. This works well, unless you have tests that you expect to fail but still have something you would like to commit. You have to choices: either remove the executable bit from the pre-commit file or adjust your script to give you some options. Here is a little script I put together to prompt you if you would like to commit anyway in the even of test failures. Keep in mind I know very little if any bash scripting so if there is a better way to do this please let me know.


    nosetests
    code=$?

    if [ "$code" == "0" ]; then
    exit 0
    fi

    echo -n "Not all tests pass. Commit (y/n): "
    read response
    if [ "$response" == "y" ]; then
    exit 0
    fi

    exit $code

Hope this helps.

Show IP address of VM as console pre-login message

2008-05-09T00:00:00-07:00

Show IP address of VM as console pre-login message

9 May 2008 – Chicago

In case you didn’t know the pre-login message you see at a Linux console typically comes from /etc/issue

You can customize this file to alter the message with some escape codes that will show things like the current date and time, machine name and domain, kernel version, etc. But one thing you can’t easily display is the IP address of a machine. Showing the IP address is especially useful when building a virtual machine that will use DHCP, like the Ubuntu development VM I use on my Macbook Pro. This way I can start VMware Fusion, see the IP address of the VM and then login over SSH.

In order to get the IP address to show in /etc/issue I needed to write a custom script that will rewrite /etc/issue with the IP address when the network interface is brought up. The first step was writing a simple script that will output the current IP address when run (by looking at the output of ifconfig).

/sbin/ifconfig | grep "inet addr" | grep -v "127.0.0.1" | awk '{ print $2 }' | awk -F: '{ print $2 }'

The above script will run ifconfig and print out the IP address (after filtering out the localhost interface). I saved this script to /usr/local/bin/get-ip-address. In order to get this into /etc/issue I decided to first copy /etc/issue to /etc/issue-standard, then create the following script that when run will overwrite /etc/issue with the contents of /etc/issue-standard + IP address.

Debian/Ubuntu

Save the following script as /etc/network/if-up.d/show-ip-address


#!/bin/sh
if [ "$METHOD" = loopback ]; then
    exit 0
fi

# Only run from ifup.
if [ "$MODE" != start ]; then
    exit 0
fi

cp /etc/issue-standard /etc/issue
/usr/local/bin/get-ip-address >> /etc/issue
echo "" >> /etc/issue

and don’t forget to mark it executable.

RedHat/CentOS

Save the following script as /sbin/ifup-local


#!/bin/sh

if [ "$1" = lo ]; then
    exit 0
fi

cp /etc/issue-standard /etc/issue
/usr/local/bin/get-ip-address >> /etc/issue
echo "" >> /etc/issue

and don’t forget to mark it executable.

Working with LTPA

2007-08-21T00:00:00-07:00

Working with Lightweight Third Party Authentication (LTPA)

21 August 2007 – Chicago

Lightweight Third-Party Authentication (LTPA), is an single sign-on technology used in IBM WebSphere and Lotus Domino products. A server that is configured to use the LTPA authentication will send a session cookie to the browser after sucessfuly authenticating a user. This cookie is only valid for one browsing session. This cookie contains the LTPA token.

A user with a valid LTPA cookie can access a server that is a member of the same authentication domain as the first server and will be automatically authenticated. The cookies themselves contain information about the user that has been authenticated, the realm the user was authenticated to (such as an LDAP server) and a timestamp. All of this information bis encrypted with a shared 3DES key and signed by a public/private key pair. This is all fine until you are trying to perform some troubleshooting and you realize you can’t look inside of these cookies.

One day I was in need of decrypting some tokens and couldn’t find any information on the subject so I spent some time studying the format of the cookie and wrote some code to decrypt them. You can check out the code from here

The LTPA cookie is encrypted with a 3DES key in DESede/ECB/PKCS5Padding mode. If you are extracting the key from a Websphere or other IBM server the key is likely protected with a password. The real key is encrypted also using 3DES in DESede/ECB/PKCS5Padding mode with the SHA hash of the supplied password padded with 0X0 up to 24 bytes. To decrypt the actual token you can take the password, generate a 3DES key, decrypt the encrypted key and then decrypt the cookie data. There is also a public/private key pair being used to sign the cookie. Since I had no intent in validating that the cookie is properly signed or in creating real cookies I did not spend anytime investigating the signature portion of the cookie. Drop me a note if you find the code useful or if you have some improvements you would like to share.

Keeping SSH sessions alive

2007-08-20T00:00:00-07:00

Keeping SSH sessions alive

20 August 2007 – Chicago

This is one of those things that I setup once a year when I get a new machine and then I always seem to forget the next time around so I’ll post it here as a reference to myself and perhaps also help the occasional Googler. If you are having problems with your SSH connection getting dropped after a certain amount of time (usually caused by NAT firewalls and home routers), you can use the following setting to keep your connection alive


Host *
    ServerAliveInterval 180

You can place this either in ~/.ssh/config for user level settings or in /etc/ssh/ssh_config for machine level settings. You may also replace * with a specific hostname or something like *.example.com to use on all machines within a domain.
This is the cleanest way of making sure your connections stay up and doesn’t require changes to the destination servers (over which you may not have control). I am not sure however how this plays with the IdleTimeout setting on the server. I am guessing that a server should be able to enforce its own policy about how long folks are remained to be idle for security reasons so you might still get disconnected after a certain amount of time.