Anomalous load spike

In my experience the order of likely causes of a problem goes like this: Buggy code recently deployed, large and expensive reports being run, or network (or firewall) problems causing processes to hang on IO operations. How to troubleshoot this problem on Passenger?

Checking the output of passenger-status we were seeing Rails processes not completing requests, the “Processed” column for certain Rails processes were not incrementing, indicating these processes were somehow stuck, thereby decreasing the pool of available processes for subsequent requests. Attaching to these stuck processes with strace revealed no activity in the process at all. Similarly, using gdb.rb revealed nothing immediately useful. Strangely we were seeing some Rails processes continuing to run even after they were no longer being actively managed by Passenger, at least according to the output of passenger-status, in which these processes no longer appeared.

First I upgraded Passenger (using Chef, of course) to the latest version on one server, to see if there was a Passenger bug causing this problem. The issue persisted. Making things a bit more complicated we had recently moved a memcached instance to a different part of the network, and all the stuck Rails processes had open network connections to this memcached instance, leading me to wonder if it was this network IO which was somehow causing the Rails process to hang. I could find no problems between the application servers and the memcached port and no logs of any network issues there. So what are the stuck Rails processes stuck on?

The way to troubleshoot this problem on Passenger is to  issue a kill -SIGABRT to the stuck Rails processes, and find their backtrace in the Rails log. Looking at the backtraces we were able to track the issue down to a bug in a Ruby gem which was being used to generate certain types of reports. Disabling this reporting prevented new occurrences of the problem until the underlying issue can be resolved. An interesting combination of two usual suspects: Reporting and code library bugs.

Brandon West’s post was very useful in getting started.

Summary: Download and install VodafoneMCInstaller then add the 3G card as a network interface and change the settings as mentioned in Brandon’s post.

To get online you’ll need to open the Vodafone application, and use the 3G card’s pin number to Activate your modem every time you connect. Once activated, you need to then dial up the modem using the Network panel in System Preferences.

Because your Vodacom 3G account is a metered account, and I’m not used to conserving bandwidth in any way, I found SurplusMeter very useful to watch my usage.

In Nagios I created 2 notify commands, one for host notifications and another for service notifications:

# 'notify-throttle-host' command definition
define command{
	command_name    notify-throttle-host
	command_line    /usr/local/bin/ruby $USER1$/throttle_alert.rb "$CONTACTEMAIL$" "[nagios] $NOTIFICATIONTYPE$ Host: $HOSTNAME$ $HOSTSTATE$" "at $LONGDATETIME$ \n\n $NOTIFICATIONTYPE$ : $HOSTNAME$ ($HOSTADDRESS$) is $HOSTSTATE$ \n\n: $HOSTOUTPUT$ \n"
}
 
# 'notify-throttle-service' command definition
define command{
	command_name    notify-throttle-service
	command_line    /usr/local/bin/ruby $USER1$/throttle_alert.rb "$CONTACTEMAIL$" "[nagios] $NOTIFICATIONTYPE$ Service: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$" "at $LONGDATETIME$ \n\n $NOTIFICATIONTYPE$ : Service $SERVICEDESC$ on $HOSTALIAS$ ($HOSTADDRESS$) is $SERVICESTATE$ \n\n: $SERVICEOUTPUT$ \n"
}

Then throttle_alert.rb uses a control file to determine how long it has been since the previous alert, and will redirect alerts to a secondary (non-SMS) email address (and note the throttling in the alert itself) if an alert occurs inside the throttled window (90 seconds in this example):

#!/usr/bin/env ruby -rubygems
############################################################
# throttle the notifications we send
# get 3 arguments
# "EMAIL" "SUBJECT" "BODY"
############################################################
 
unless ARGV.length == 3
  puts "usage: <> email subject body"
  exit
end
 
minimum_wait_time = 90   #seconds before we can alert again
alternative_email = "secondary-email@yourdomain.com" #send throttled alerts here when silenced
 
emailto = ARGV[0]
subject = ARGV[1]
details = ARGV[2]
 
email_p = emailto.gsub("@", "_").gsub(".", "_")
 
 
control_file = "/tmp/throttle_nagios_#{email_p}.txt"
mailer_cmd = "/usr/bin/mail"
 
if File.exists?(control_file)
  file_age = File.mtime(control_file)
  value = Time.now - file_age
  file_age_nice = sprintf("%1.1f", value)
 
  unless file_age_nice.to_f > minimum_wait_time.to_f
    system "/usr/bin/printf '%b' '#{details} (throttled due to a #{control_file} #{file_age_nice}s old' | #{mailer_cmd} -s '#{subject}' #{alternative_email}"
    Process.exit
  end
 
end
 
system "touch #{control_file}"
system "/usr/bin/printf '%b' '#{details}' | #{mailer_cmd} -s '#{subject}' #{emailto}"

Not rocket science and not perfect but it works to prevent draining my iPhone battery in a single incident.

Previously we installed Chef from the Opscode Debian packages and then replaced the Chef server URL in the Chef client configuration with sed so that we could register and validate the Chef client with the Chef server for the first time. This obviously won’t do, we need to manage all configuration files properly without little post-installation hacks in them. That is a core idea of the Chef. So let’s now take control of the Chef client configuration files themselves with Chef.

To do this we need to create a Chef cookbook called “chef” in our repo, in this cookbook we will create a recipe called “client” to handle the client stuff (as opposed to the Chef server stuff which can live happily side by side in the same cookbook but in different recipes), then we need to assign the Chef client recipe to a role (which we will call something like “base” because we are going to add it to all servers) and lastly we need to add that “base” role to our 2 nodes we have. Let’s make it work, people.

My workflow right now is to have my Chef Git repo cloned on my local Mac and to work in there, then commit to this repo and push it to our origin. Then I login to the Chef server, which has it’s own clone of the same repo on it, and then use Rake to pull the latest Git changes down and install them into the Chef server directories.

So on my local Mac, lets create the skeleton dir structure for this new cookbook of ours. You could use Rake for this (rake new_cookbook COOKBOOK=chef) but where’s the fun in letting someone else do all the menial labor for you?

git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo
mkdir -p cookbooks/chef/attributes
mkdir -p cookbooks/chef/templates/default
mkdir -p cookbooks/chef/recipes

Now let’s first define the attributes of our Chef client configuration. These are basically the values which we want in our config files. What we will now do is create some default values, which can be overridden in other places if we ever needed specific Chef client settings to be different on different servers for some reason.

Here is our barebones cookbooks/chef/attributes/chef.rb

set_unless[:chef][:server_fqdn]     = "chef.yourdomain.com"
set_unless[:chef][:server_port]     = "4000"
set_unless[:chef][:openid_port]     = "4001"
set_unless[:chef][:log]             = "/var/log/chef/client.log"
set_unless[:chef][:log_level]       = "debug"
set_unless[:chef][:filecache]       = "/var/cache/chef"
set_unless[:chef][:pid]             = "/var/run/chef/client.pid"

Now, we need a template file, which is an ERB template, which our recipe will use to make these dreamy attributes appear on the filesystem somewhere in a fully cooked configuration file.

This is what our cookbooks/chef/templates/default/client.erb looks like:

log_level          <%= @node[:chef][:log_level] %>
log_location       "<%= @node[:chef][:log] %>"
 
ssl_verify_mode    :verify_none
 
registration_url   "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
openid_url         "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:openid_port] %>"
template_url       "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
remotefile_url     "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
search_url         "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
role_url           "http://<%= @node[:chef][:server_fqdn] %>:<%= @node[:chef][:server_port] %>"
 
file_cache_path    "<%= @node[:chef][:filecache] %>"
pid_file           "<%= @node[:chef][:pid] %>"

OK, now let’s create a chef::client recipe which will take the attributes and the template and combine them into cold hard configuration files. This is our cookbooks/chef/recipes/client.rb

case node[:platform]
 when "debian","ubuntu"
 
 template "/etc/chef/client.rb" do
    source "client.erb"
    mode 0644
    owner "root"
    group "root"
    backup false
  end
 
end

OK, so now we have the attributes, the template and the recipe. Doesn’t do us much good unless we somehow tell the Chef server that our nodes need to execute the chef::client recipe when the Chef client runs. So let’s create our role, called “base”. Here is roles/base.rb

name "base"
description "All systems to get this role"
recipes "chef::client"
 
override_attributes "chef" => { 
    "server_fqdn" => "chef.yourdomain.com",
    "log_level" => "info"
}

Something to notice here. In our attributes file earlier, we used set_unless to define a few attributes of this cookbook. But here in the role we are overriding them (with the same values in this case). The utility here is that using the exact same chef cookbook, we could create a new role, called something like base_london and use override_attributes in the Role definition to give those nodes a different value for server_fqdn.

Now let’s commit all of this new code we have written to our Chef git repo.

git add *
git commit -a -m "adding our chef cookbook and role, first pass"
git push

Now, let’s login to our Chef server, pull down all these new changes in our git repo and actually get this stuff into our running Chef server.

ssh chef.yourdomain.com
git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo
rake install
sudo /etc/init.d/chef-server restart

(The last step, the restart of the Chef server, is needed because we have added a new role. Your role you defined in the Ruby file above gets compiled to JSON and then is stored in CouchDB, and we need to bounce the CouchDB server in the 0.7 Chef series for your changes to become effective).

OK. So, now we have everything in place, except one. We haven’t told the Chef server to actually apply the “base” role to our nodes. Now you would expect to use an elegant CLI tool which would use the Chef API to perform these types of operations on the Chef server. The good news is that Knife is that tool. The bad news is that I haven’t tested Knife yet, and because you are on my blog you are SOL. So we’ll be using the Chef web UI for this task until I get to try the Knife.

So, login to http://chef.yourdomain.com:4000 using your OpenID login. Go to Nodes and click on “edit” next to one of your nodes listed there if both chef.yourdomain.com and web1.yourdomain.com are registered on the server, or whichever node you have been working on. What you need to do now is drag “base” from the Available Roles into the Run List and Save Node.

So the final step you need to do is actually run chef-client on your server to actually fetch and apply the base role to it.

sudo chef-client -l debug

If all went according to plan, then you should see that Chef is now managing the contents of /etc/chef/client.rb.

This was a very simple quick example of how a Chef cookbook, recipe, role and runlist work together to Get Things Done. There is a whole lot more to a Chef cookbook than this. The next thing to do is to forget everything I just told you, thank the heavens for jtimberman and kallistec and all the other fine Chefs who hang out in #chef on freenode. Take a look at some real Cookbooks from these guys which really start to leverage the power of Chef.

In part 3 of this series, coming out to coincide with Duke Nukem Forever, I’ll talk about a more advanced cookbook, and hopefully Knife. Thanks for staying this long.

To start, create a directory structure like this (preferably in a Git or SVN repo)

base/
accounts/

Then in accounts/, create a subdirectory for each project/client/ec2 account, and add 3 things to each subdirectory: the EC2 X.509 certificate, the EC2 private key and a single SSH private key to use for SSH access. So end up with a directory structure something like this, separated by client/project/account:

base/
accounts/project1/cert-TYTYTHJHDJHDJHDJHDJH.pem
accounts/project1/pk-DNKJNFKDJNFFKJNDFKNSDFJ.pem
accounts/project1/id_dsa_project1
accounts/client55/cert-SASDASDASDASDASDASD.pem
accounts/client55/pk-UYWYWUYWUYNNSNSNS.pem
accounts/client55/mykey_client55
accounts/myappx/cert-XCMLKMLKMLKMLKM.pem
accounts/myappx/pk-CCJKJDKPOPOPOPPOP.pem
accounts/myappx/awskey_appx

Now we’ll download the latest EC2 API tools and expand them into base/, so we will have a directory structure like this:

base/THIRDPARTYLICENSE.TXT
base/bin/ec2-add-group
base/bin/ec2-add-group.cmd
base/bin/ec2-add-keypair
base/bin/ec2-add-keypair.cmd
-- ETC -- OMITTING THE REST OF bin/
base/lib/activation-1.1.jar
base/lib/bcprov.jar
-- ETC -- OMITTING THE REST OF lib/
base/license.txt
base/notice.txt
accounts/project1/cert-TYTYTHJHDJHDJHDJHDJH.pem
accounts/project1/pk-DNKJNFKDJNFFKJNDFKNSDFJ.pem
accounts/project1/id_dsa_project1
-- ETC -- OMITTING THE REST OF accounts/

Now, we will use a simple bash script to generate a bash alias for each EC2 operation for each EC2 account we have. It essentially maps each EC2 command to the appropriate key and cert for each of our projects/clients.

NOTE: This version sets up some environment variables which are specific to a Mac OS X > Leopard environment. Yours may need different JAVA_HOME, if you are on a different OS, etc.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#!/bin/bash
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/
export EC2_TOP=$( dirname $BASH_SOURCE )
export EC2_HOME=$EC2_TOP/base
 
ALL_ACCOUNTS=$( ls $EC2_TOP/accounts )
EC2_TOOLS=$( ls $EC2_HOME/bin | grep -v .cmd )
 
THIS_KEY=''
THIS_CERT=''
 
for a in $ALL_ACCOUNTS; do
        THIS_KEY=$( ls $EC2_TOP/accounts/$a/pk-*.pem )
        THIS_CERT=$( ls $EC2_TOP/accounts/$a/cert-*.pem )
        THIS_SSH=$( ls $EC2_TOP/accounts/$a/* | grep -v .pem )
 
        alias ssh-ec2-$a="ssh -i $THIS_SSH"
 
        for e in $EC2_TOOLS; do
                alias ec2-$a-$e="$EC2_HOME/bin/$e -K $THIS_KEY -C $THIS_CERT"
        done
 
        THIS_KEY=''
        THIS_CERT=''
 
done

This base script (I’ve called mine setup_env.sh) needs to live at the top level, so again, we have a structure like this:

base/
accounts/
setup_env.sh

Now, simply source setup_env.sh as part of your login procedure, by putting something like this in .bash_profile:

NOTE: My top level directory lives at ~/Tools/Amazon

source ~/Tools/Amazon/setup_env.sh

Now, when I login, I can create an instance on the correct EC2 account with something like:

ec2-project1-ec2run <options>

And I can SSH to a running EC2 instance with something like:

ssh-ec2-project1 root@myinstance.address.com

Simple, possibly inelegant, but very functional. Do you have anything which works better than this? Please share..

Idea 1: The Hosting Provider Incident Tracker (HPIT)

A real time database where we track major (and minor) incidents at hosting providers around the world. Too many times I have asked hosting providers about their history of incidents only to receive vague assertions about 100% uptime and heroic stories surviving the great blackouts of our time. This always fails to impress me. What would REALLY impress me is a provider who would give me a link to their incident history with detailed technical explanations around the root causes (ugh, hate that word) and mitigation steps. Sadly, providers still want to sell the snakeoil of 100% uptime. This isn’t the point, because there isn’t 100% uptime. There is only honest and timely response to inevitable outages. This is what matters. This, however, opens the door for an independent tracker of these incidents. This service will vet incident reports, seek and post responses from hosting providers, or simply link to their own responses, and be your source for evaluating how a hosting company responds to critical issues.

BTW: I don’t know why Pingdom isn’t providing this service today. It has most of the data required already.

I’m familiar with configuration management tools like Cfengine, Puppet and slack. I am also familiar with using homebrewed tools which leverage trees of shell scripts to apply and manage a bunch of configuration files stored in version control. I am going to teach myself to use Chef on Debian Lenny. Follow along.

Firstly, idempotent. Chef is idempotent. Wha? This means that if you run the chef client on a node multiple times, you should never get a gradually changing state. Given the same recipes, the system should always return to an identical state after the Chef tools run on the system. Think about those shell scripts you have written in the past. Sometimes they are appending strings to files, or removing strings from files, or moving files around based on some algorithm. Well, Chef doesn’t perform a single action out of context. It does every action, every time. And the actions Chef performs are defined in recipes. Collections of recipes are called cookbooks. These recipes are written in Ruby. Recipes can be grouped together in roles, so when you apply a role to a node that node will receive all the recipes in that role. Recipes can reference attributes in other recipes, and also apply other recipes.

The bulk of the Chef work is done by the chef-client which runs on the host you are managing with chef. It may or may not contact a chef-server. Using chef-solo you can do a bunch of useful things without needing a central chef-server.

There are quite a few different tools in use in the complete Chef stack. Wonderful tools like couchdb, Stomp, Merb. Luckily, thanks to great packages from Opscode, you can get all of this stuff installed without too much trouble.

Note on versions: Chef 0.8 branch is currently getting some very interesting sounding features. In fact, I remarked in IRC today that I fully expect Chef 0.8 to file my taxes and walk my dog. However, I am not going there yet. This post will be all about Chef 0.7 branch, specifically the version of Chef provided in the Opscode apt repositories. By the way, you should totally hang out in #chef on Freenode. The people in there are awesome, helpful and very very good looking.

This is it. Let’s do things.

Set up your Git repository

The various pieces which end up becoming files on the filesystem are stored in version control. You can apparently use SVN, but we’ll go with Git. You’ll need a nice fresh Git repo. To this repo you’ll add your Chef cookbooks and roles, etc. So you could either host your Git repo with GitHub (you’ll probably want to use a private repo, the repo will most likely contain some private things) or you could set up you own Git repo using Gitosis.

Once you have your Chef repo setup, get the contents of the Opscode Chef Repository into your own repo. I’m not going to go into the details of how to do this using Git, mostly because you don’t want Git advice from me. Nevertheless, you want to grab the contents of this repo, and merge them in your new repo:

git clone git://github.com/opscode/chef-repo.git

When all is said and done, your repo should have a structure similar to this:

1
2
3
4
5
6
7
8
9
10

certificates/
config/
config/client.rb.example
config/server.rb.example
config/rake.rb
config/solo.rb.example
cookbooks/
Rakefile
roles/
site-cookbooks/

More detailed info on the Chef Repository

Setup your 2 servers and your DNS

For the purposes of this tutorial, we’ll assume you have 2 servers. You could have 17 servers, that would be fine too. We’ll call server 1 chef.yourdomain.com and server 2 will be web1.yourdomain.com. Make sure these servers have Debian Lenny installed and you have created proper working DNS entries for the hostnames chef.yourdomain.com and web1.yourdomain.com

We’ll be installing Chef server AND a Chef client on chef.yourdomain.com
We’ll be installing only a Chef client on web1.yourdomain.com

Setup your Chef server

There are Chef cookbooks which will allow Chef to bootstrap itself, which is a fairly cool concept. This isn’t necessary, however, when using the Opscode chef packages for Debian, since they will provide everything we need for our Chef server in the packages. So we’ll simply install the packages and be on our way.

First, we’ll need these:

1
2

apt-get update
apt-get install -y curl git-core sudo

Now let’s set up the Opscode repo and install Chef:

1
2
3
4

echo "deb http://apt.opscode.com/ debian contrib" > /etc/apt/sources.list.d/opscode.list
curl http://apt.opscode.com/packages@opscode.com.gpg.key | apt-key add -
apt-get update
apt-get install -y rubygems libshadow-ruby1.8 ohai chef chef-server

Done. Buy Opscode some beer! OK, now let’s do some configuration of the Chef server. Firstly, Chef uses OpenID extensively for authentication. We need to have an OpenID provider, which we trust defined in the chef server config. If you don’t have an OpenID provider that you use, head over to myOpenID and sign up for an account. You’ll get an OpenID URL. Now we’ll put that into the chef config at /etc/chef/server.rb by adding the following line at the bottom of that file:

1

authorized_openid_providers [ "https://chef.localdomain", "https://chef", "https://whateveryourswas.myopenid.com"]

Now we are going to generate a validation token for our Chef clients to use to auto-register themselves against the Chef server. You need a nice random string. If you lack the creativity to make one up, try this:

1

thedate=$( date ); thehost=$( hostname ); echo $thedate$thehost | sha256sum

Now put the string you came up with above into /etc/chef/server.rb by finding the validation_token line, uncommenting it and giving it a value:

1

validation_token   "2d926b251e18cb39b00350bb956d44ef8639b4ed33809194a7e2ed60ac5d772c"

Restart Chef server:

1

/etc/init.d/chef-server restart

Setup your Chef Repository

Now we need our Git repository created earlier. We are going to use this repository to store our recipes, and we are going to use tools inside this repository to manage Chef itself. How meta. So on your newly installed chef server, in your home directory, or whereever else, clone your git repo:

1
2

git clone git@yourgitserver.yourdomain.com:your_chef_repo.git
cd your_chef_repo

We are now going to use Rake to install the initial (blank) Chef cookbooks into place, and also to set up the SSL certificates which Chef uses when communicating between components. Obviously you’ll want to use your Chef server’s proper FQDN here.

1
2

rake install
rake ssl_cert FQDN='chef.yourdomain.com'

Now, try logging into the Chef web UI, at http://chef.yourdomain.com:4000

In the login field, enter the openID URL you configured earlier, eg: https://whateveryourswas.myopenid.com you’ll be forwarded to the OpenID provider, asked to authenticate and sent back to Chef. You should now be logged in.

Set up your Chef client

OK, now that we have a working, albeit currently fairly useless, Chef server let’s install our first Chef client. On your other host, web1.yourdomain.com, you could use this script I quickly put together to go from a freshly installed server to a working Chef client. Obviously you should substitute your auth_token you created earlier and your chef’s DNS location.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/bash
### VARIABLES ##########################################################
chef_server="chef.yourdomain.com"
chef_client=/usr/bin/chef-client
chef_client_config="/etc/chef/client.rb"
chef_auth_token="2d926b251e18cb39b00350bb956d44ef8639b4ed33809194a7e2ed60ac5d772c"
 
opscode_apt="deb http://apt.opscode.com/ debian contrib"
 
### ACTIONS ############################################################
apt-get update
apt-get install -y curl git-core sudo
 
echo $opscode_apt > /etc/apt/sources.list.d/opscode.list
curl http://apt.opscode.com/packages@opscode.com.gpg.key | apt-key add -
apt-get update
apt-get install -y rubygems ohai chef libshadow-ruby1.8
 
### SETUP CHEF CLIENT AND REGISTER #####################################
sed -i "s/localhost/$chef_server/g" $chef_client_config
$chef_client -t $chef_auth_token
 
### REMOVE THYSELF #####################################################
rm $0

If all went according to plan, this should now be a working chef client which knows where the chef server is, and has auto registered itself with the Chef server. Have a look at http://chef.yourdomain.com:4000/registrations and see if web1.yourdomain.com is listed there. If so you are halfway to Nirvana. So now we actually want to start doing things by creating cookbooks, roles, ie: actually doing shit.

Stay tuned for part 2 in this enthralling series. Contact me for movie rights.

UPDATE: Part 2 now online