Hector Castro

Preseeding Ubuntu Server and Static IP Addresses

2011-11-18T00:00:00-08:00

Setting up a cluster of computers for any purpose usually requires installing an operating system. The installation process typically consists of several questions and identical answers for each node in the cluster. Automating the submission of answers to these questions is desirable — not only to prevent inconsistencies, but for general convenience.

Preseeding

I spent the last few days working to stand up a proof-of-concept Riak cluster. The first step involved installing Ubuntu Oneiric Ocelot (11.10) on four virtual machines. Luckily, Ubuntu/Debian has a process called preseeding to facilitate automated installations. Surprisingly, it also has limited support for Red Hat’s Kickstart. Playing it safe, I went with preseeding.

There are three methods that can be used for preseeding: initrd, file, and network. I wasn’t interested in re-authoring ISOs or setting up a TFTP server, so I went with a web-accessible preseed file. The pros of this approach are that the configuration file is easily modifiable, yet still accessible. The cons are that it doesn’t become available to the installer until the network is configured.

Assigning a Static IP Problem

Because web-accessible preseed files aren’t available until the network is configured, the step to assign a static IP address gets missed. Below are several approaches I found to assign a static IP address with preseeding.

Boot Parameters

The boot prompt is where you tell the installer how to locate your preseed file. It is also where you can pass a fixed number of preseed directives. In our example of assigning a static IP address, you’d pass things like IP address, hostname, domain, and netmask. Ultimately, I wasn’t too interested in this approach because it required a lot of typing without clipboard access.

Re-evaluating Network Configuration

The Ubuntu Help wiki has a suggested hack to trigger re-evaluation of preseeded network configuration settings by executing commands via preseed/run. Unfortunately, I was unable to get this to work successfully. In every combination I tried, it resulted in the installer failing. This related Ubuntu Forums post outlines the suggested steps pretty well.

Overwriting Network Configuration

Eventually this is the solution I used to assign a static IP address. It’s a hack, but in my eyes it was the lesser of three evils. Alongside each node’s preseed configuration file, I created a corresponding shell script. The shell script gets executed before the installer triggers a reboot and overwrites /etc/network/interfaces with a static IP configuration:

echo "auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
 address 192.168.1.10
 netmask 255.255.255.0
 gateway 192.168.1.1
" > /etc/network/interfaces

For completeness, I included my preseed configuration file as a Gist. If anyone has a better approach to setting a static IP address via preseeding or Kickstart, let me know!

Testing Command-line Applications with Aruba

2011-10-25T00:00:00-07:00

Cucumber is often used to test web applications. Many developers hook it into their Rails projects to integration test site features. Wouldn’t it be great if there were a way to test command-line applications in a similar fashion? You can with Aruba.

Aruba

Aruba is a Cucumber extension for testing command-line applications written in any language. Passing arguments, interacting with the file system, capturing exit codes, and mimicking interactive usage are all features provided out of the box. Below is a basic test for the mv command that passes:

Scenario: Backing up test.conf
  When I run `mv test.conf test.conf.bak`
  Then the output should contain:
  """
  mv: rename test.conf to test.conf.bak: No such file or directory
  """

Now let’s showoff a few of Aruba’s built-in steps to prevent the command from failing:

Scenario: Backing up test.conf
  Given an empty file named "test.conf"
  When I run `mv test.conf test.conf.bak`
  Then the exit status should be 0
  And the following files should exist:
    | test.conf.bak |
  And the following files should not exist:
    | test.conf     |

The first step creates an empty file and executes mv inside of Aruba’s sandbox directory. After the mv command is executed, its exit status is compared to 0 and the existence of test.conf.bak (and non-existence of test.conf) is confirmed.

It’s also worth noting that after each scenario Aruba clears out its sandbox — a temporary directory that becomes the current working directory for your command-line tool — unless you explicitly tag the scenario with @no-clobber. This tag preserves the previous scenario’s final state. Tying this back to the example above, the next scenario would begin with only test.conf.bak in the sandbox. Additional Aruba-specific tags can be found in the README.

Extending the Aruba API

As a command-line application evolves, other conditions not available in Aruba’s built-in API will require testing. For example, say you need to assert a file’s user and group attributes. Because Aruba’s API was built using Ruby modules, it can be reopened inside of Cucumber’s env.rb:

module Aruba
  module Api
    def check_file_owner_and_group(paths_and_users_and_groups)
      prep_for_fs_check do # Lower-level function provided by Aruba
        paths_and_users_and_groups.each do |path, user, group|
          stat = File.stat(path)

          Etc.getpwuid(stat.uid).name.should == user
          Etc.getgrgid(stat.gid).name.should == group
        end
      end
    end
  end
end

Then create a matcher:

Then /^the following files should have username "([^"]*)" and group "([^"]*)":$/ do |user, group, files|
  check_file_owner_and_group(files.raw.map { |file_row| (file_row << user) << group })
end

Now that step can be included to test the user and group attributes of files:

Scenario: Backing up test.conf
  Given an empty file named "test.conf"
  When I run `mv test.conf test.conf.bak`
  And the exit status should be 0
  And the following files should exist:
    | test.conf.bak |
  And the following files should not exist:
    | test.conf     |
  And the following files should have username "hector" and group "staff":
    | test.conf.bak |

Conclusion

Using a behavior-driven development approach for building command-line applications with Cucumber and Aruba was a pleasure. Aruba’s API covers a decent amount of ground and was easily expandable. The source code was straightforward and after skimming its internals, I was able to expand the API to meet my needs. Hopefully reading this will help you do the same.

Replacing Excel with the Eighteenth Letter of the Alphabet

2011-07-18T00:00:00-07:00

Every once in a while I have to graph data in order to better understand it. Most of the time, I use Microsoft Excel to generate graphs because it’s one of the easiest ways to produce them. Unfortunately, Excel’s ease of use quickly degrades once you move beyond pasting data into cells and clicking the graph button. I started looking for something to make the process of visualizing data more flexible. Something with pluggable libraries, helpful examples, and room for reproducibility — I ended up replacing Excel with R.

R

R is an open source programming language for statistical computing and publication-quality graphics. Different than a general purpose programming language, R’s core includes many features designed to empower statisticians. I’m no statistician, but I was intrigued by its approachable syntax and familiar data structures. Even more, it’s backed by a potent community that contributes numerous packages to solve all sorts common problems.

R code is typically interpreted in a REPL, but can also be captured in a file and executed as a script. To reduce my learning curve, I looked into an IDE for R. I settled on RStudio because it has a clean UI, was easy to install, and has its source code hosted on GitHub. If you’re going to explore R for the first time, I’d strongly encourage the use of an IDE. It makes searching documentation, viewing graphs, and inspecting output simple.

Graphing

My first task for R was to parse application logs and graph the frequency of specific user interactions. The application being logged provides a web interface to query financial data sets. The log records queries, so my goal was to plot each distinct data set and the number of times it was queried.

After reading through several examples of R’s standard graphing functions, I stumbled upon ggplot2. The ggplot2 package brands itself as the “Grammar of Graphics.” A graphing system that takes what’s good about R’s and omits the bad. In comparison to the base graphing library, ggplot2’s syntax is slightly more intuitive. This, and the fact that several answers on Stack Overflow recommend it, compelled me give it a try.

library(plyr)
library(ggplot2)

# Import a pipe-delimited file without a header row.
requests <- read.csv("requests.dat", header=FALSE, sep="|")

# Extract a subset of the requested data sets (column V8)
# with a frequency greater than 2000.
data_set_freq <- subset(count(requests, 'V8'), freq > 2000)

# Rename columns and rows.
colnames(data_set_freq) <- c('data_set', 'freq')
row.names(data_set_freq) <- data_set_freq$data_set

# Plot it.
ggplot(data_set_freq, aes(factor(data_set), freq)) + 
  geom_bar() + 
  labs(y = "Web Queries", x = "Data Sets")

The short snippet of code above produces the following graph:

Now, to take that a step further, I wanted to figure out which data set is associated with the most failures. Building on the code above, here’s how I accomplished that:

# Extract a subset of data set names (column V8) for
# failed requests (column V15).
data_set_errors_freq <- count(subset(requests,
  grepl('ERROR|killed', requests$V15), select = c(V8)), 'V8')

# Rename columns and rows.
colnames(data_set_errors_freq) <- c('data_set', 'freq')
row.names(data_set_errors_freq) <- data_set_errors_freq$data_set

# Merge the data set frequencies with errors and create
# a third column for percent error.
data_set_summary <- merge(data_set_freq, data_set_errors_freq,
  by.x = 'data_set', by.y = 'data_set')
data_set_summary <- ddply(data_set_summary, .(data_set), transform,
  percent_error = (freq.y / freq.x) * 100)

# Plot it.
ggplot(data_set_summary, aes(factor(data_set), freq.x, fill=percent_error)) + 
  geom_bar() + 
  labs(y = "Web Queries", x = "Data Sets", fill="Precent Error")

A similar graph to the one above, except that this one contains colors based on the percentage of errors:

Conclusion

Building graphs with R feels a lot like building servers with Chef. I can configure a server once manually, or I can write code that automates its deployment process forever. Likewise, I can paste data into Excel and point and click to build a graph, or I can write R code that reproduces a handful of steps with one command. There are a number of GUI tools that build graphs from data, but once you begin applying filters, merging data sets, or running calculations these tools break down. Writing code instead of clicking buttons has its downsides, but there is something comforting in knowing that as long as our log structure doesn’t change, I’ll be able to reproduce these graphs six months from now and immedietally know which data sets are most error prone.

Monday Morning System Administrator

2011-05-17T00:00:00-07:00

Not long ago I purchased a virtual private server from Linode. At the time, the most recent Ubuntu release was Maverick Meerkat (10.10). The first Sunday after purchasing, I sat down for a few hours and configured every aspect of the machine manually — it was going to be my personal web and application server. Feeling like I had accomplished something, I logged out and nodded in approval.

Fast-forward five months and Ubuntu’s most recent release is Natty Narwhal (11.04). Naturally, I wanted to upgrade, but a clean install would require another couple of hours on a Sunday afternoon. If I was going to invest time in configuring the server again, how could I configure it in a way that would apply for the next release? The answer I settled on was Chef.

Chef

Chef is an open source framework that allows you to build infrastructure from code. Explained another way, Chef allows you to checkout code from a repository and execute it against a base operating system install — producing a fully operational web server, database server, etc. No replaying steps off of a build document. No complex combinations of Perl and Bash scripts. No more manual configuration.

Chef comes in two flavors: server and solo. With Chef Server, a client (your laptop) sends configuration directives to a central server and that server propagates those directives to target nodes. The server can exist within your data center, or you can leverage Opscode’s Platform (Chef Server as a service). The alternative is Chef Solo. Chef Solo operates without Chef Server by sending directives directly to target nodes.

In the end I chose Chef Solo, but don’t let that impact your evaluation of Chef Server. Opscode makes it easy to test drive their platform by giving you up to five free target nodes.

Forward Progress

Coming back to the story I began in the section above, I opened up my laptop on a Saturday morning and launched VMware Fusion to run through the installation of Ubuntu 11.04. Following the installation, I rebooted the virtual machine and created a snapshot (restoring to the snapshot provided an easy way to rapidly reproduce a freshly installed operating system). After a few hours on Saturday and two to three on Sunday, I had put together several Chef cookbooks. When executed, the cookbooks turned a totally bare virtual machine into a fully functional web server.

On Monday morning I created a snapshot of my Linode VPS and wiped the virtual disks. Then, I instructed Linode to rebuild my VPS with its custom Ubuntu 11.04 image. When the machine came back up I replayed my Chef cookbooks against it — in a little under six minutes HTTP requests were being served.

Conclusion

With around six hours worth of work, I was able to write code that could successfully redeploy a machine in under six minutes. I had been exposed to Chef before, but actually going through the steps to build a server from start to finish enlightened me — I don’t think I’ll ever want to manually configure a server again. Build documents are a good starting point for consistently spinning up machines, but when a machine is configured with Chef I can be absolutely sure that its configuration matches its cookbooks. I have never been able to say the same for a manually configured machine based on build documents.

VMware: A Gift and a Curse

2011-03-20T00:00:00-07:00

I spent most of my free time over the past few days troubleshooting a puzzling combination of failures across databases, JVMs, operating systems, and virtual machine hypervisors. Because of the sheer amount of time I lost fiddling with settings, bugging colleagues, and reading white papers — I figured I would summarize my findings in a post to save others from the same misfortune.

The Setup

WRDS recently acquired several massive data sets from the New York Stock Exchange, and I’ve been looking at different storage engines to help accommodate it. One database that has proven it can handle over a hundred terabytes of data is Cassandra. Cassandra is a distributed database that is designed to scale horizontally across many servers in a tunable, highly consistent or highly available manner. Another strong contender is Hadoop, which shares some architectural similarities with Cassandra, but has a much broader scope. Hadoop could end up being a better solution, but I wanted to investigate Cassandra first.

After reading a book and several guides on Cassandra, I set out to build a cluster. Because Cassandra is a distributed database, it is recommended that clusters be at least 3-4 nodes. I took this recommendation and provisioned four identical Linux virtual machines in our local VMware environment.

This is where things began to get ugly.

Clear as Mud

Cassandra is written in Java and runs within a Java Virtual Machine. It comes bundled with scripts that dynamically assign JVM constraints based on physical machine resources — maximum heap size, parallel garbage collection, and the enabling of Java Management Extensions. I read through these scripts and agreed with most of the defaults. Now the only thing left to do was to spawn Cassandra instances.

Shortly after spawning one instance per node, Cassandra processes began dying abruptly without stacktraces. I inspected logs for exceptions but there were none. I restarted the Cassandra instances and connected to the JVMs remotely via JMX. This showed stable heap sizes, but also revealed that out-of-nowhere physical memory utilization was jumping to 99%. To alleviate the sudden memory pressure, oom-killer began killing processes — seemingly at random.

I restarted Cassandra instances again, this time connecting remotely via JMX and SSH — JMX to monitor the JVM and SSH to monitor per-process memory consumption. As JMX reported increased physical memory utilization, so did top) — the only problem was that Java held ~50% of all the memory (expected because of the maximum heap size settings), while all of the other machine’s processes held 2%. 52% does not equal 99%, so something else was consuming memory.

Because the operating system metrics weren’t adding up, I decided to investigate the virtual machine settings. After reading a VMware memory management white paper, it became clear that a hypervisor memory reclamation technique known as ballooning, combined with virtual machine cluster resource overcommitment, were at the root of my problems.

Ballooning

In the diagram below, a balloon driver is used by the hypervisor to reclaim free guest memory when it is under pressure. The pins in Figure (b) show memory that has been reserved by the driver for hypervisor reclamation. This chunk of memory is guaranteed not to page to disk under any circumstances, and cannot be used by the guest operating system. The combination of this balloon driver reservation, significant resource overcommitment, and a JVM that needs a sufficiently large heap size, all united to produce hard failures.

Conclusion

The tentative solution to this problem was to increase memory allocation reservation for virtual machines individually — this process is described in Section 4 of the white paper. Altering these settings ensures that virtual machines get all of the resources they ask for, regardless of ballooning or resource overcommitment. The real solution is to add more resources to the underlying virtual machine hosts so that hypervisor memory reclamation strategies don’t need to be triggered as often.

Virtualization is a tremendous tool for data center utilization and consolidation. Unfortunately, it also makes it easy to forget that there is still a dependance on physical hardware. Clicking around and adjusting sliders in a VMware management console is trivial, but being aware of the repercussions those changes have on guest operating systems is still essential to effective virtualization use. After all, virtualization is an intelligent computing strategy — not magic.

Ruby and Python Lightning Talks

2011-02-11T00:00:00-08:00

On Tuesday, I was lucky enough to participate in RedSnake 2011 — an event that included Philadelphia’s Ruby and Python user groups. Approximately 70 local developers and system administrations gathered to hear 11 lightning talks covering a wide range of topics:

Ruby

Python

virtualenv
IPython
Multiprocessing
System Administration
Interfacing to C
Zen of Python

Code that generated the slides that accompanied my presentation on RVM can be found here. I’m planning to update this post with links to other presentations as they become available. Also, Trevor Lalish Menagh was kind enough to take notes.

Books I Read in 2010

2010-12-31T00:00:00-08:00

I often get criticism for only reading technical books — this year was more of the same. Below are a list of books I read in 2010, and want to read in 2011.

Read

Want to Read

Barcamp Philly 2010 Notes

2010-11-14T00:00:00-08:00

How to Test Software

What is the number of people that need to be hit by a truck before your project is unrepairable?
Eliminate work from constructors.
“Singletons are globals with a haircut.”
SQLite has 600 times more test code than functional code.
Continuous integration requires culture change.
Builds should not fail – when they do, fixing them is the #1 priority.
“If you need a monkey, we can hit the zoo.”

Steps to Quality Code

Gather requirements
Write tests against requirements
Now you can write the code

Five W’s or Root Cause Analysis

American car makers refused to ever stop the assembly line. Doing so was a fireable offense. Because of this, even small problems that emerged became very expensive to fix. Toyota changed the game – anyone had the ability to stop the line. When it was stopped, everyone would drop what they were doing and focus on the issue at hand. The line would stop more often, but solving problems happened quicker and cost less.

Pair Programming

Pairing enables rapid learning.
Learn practically, not necessarily comprehensively (book learning).
It’s actually fun.
Driver + Navigator divide.
Pairing is hard work – exhausting.
Mix of work and research, instead of research on personal time.

Zombies In My Workplace

“Change your organization, or change your organization.”
Socialize at happy hours, lunches, coffee machine.
Bitching is fun – people relate.
Bitching gets old, and isn’t good for your health.
Do not diverge down the bitching road.
Code tasting (code review) – smells like legacy code from 1998.
Don’t ask for permission.
Weekly book club on a chapter by chapter basis.
Find ways to inject pair programming.

Test::Unit and Raw Request Body

2010-10-23T00:00:00-07:00

In testing a Rails 3 web API with Test::Unit, I found myself digging through documentation to POST raw JSON to a controller action. After a bit of searching, I decided to take a look at the source code for ActionDispatch::Request’s raw_post. Below is the end result:

test "should create joke" do
  @request.env['RAW_POST_DATA'] = {
    :body => 'Why did the chicken cross the road?'
  }.to_json
  @request.env['CONTENT_TYPE'] = 'application/json'
  post :create

  assert_response :success
  assert_not_nil @response.headers['Location']
end

RVM and Cron

2010-06-21T00:00:00-07:00

Recently I found myself having to execute a cronjob using specific rubies and gemsets contained within RVM. The following snippet is what I used as a solution:

@hourly bash -l -c 'rvm use rbx@gemset && rake cron'

The -l forces bash to act as if it had been invoked as a login shell. The -c tells it to read the string that follows.