devopsanywhere

SysAdmin Productivity and Chef

2013-02-10T22:39:00.001-08:00

Inspired by Nathen Harvey's "Learning to Automate" presentation, I came up with this graph of system administrator productivity. Enjoy

How to Write Reusable Chef Cookbooks, Gangnam Style

2012-11-10T11:53:00.002-08:00

I am at pains lately to explain to others how I think Chef cookbooks should be developed. There are only a few good blog posts on the subject, most notably Peter Donald's blog posts "Evolving Towards Cookbook Reusability" and Reusable Cookbooks Revisited and Jamie Winsor's "Authoring a Chef Cookbook as a Developer." These articles have influenced me significantly, as well Jamie Winsor's appearance on the FoodFightShow podcast. Both Jamie and Peter urge toward creating generic, attribute-driven cookbook wrapped with a policy-driven cookbook. That's a complex topic that we will come back to later. Jamie recommends something similar with his distinction between "library" and "application" cookbooks. Jamie has also convinced me that Chef's Roles are to be avoided. Some people alternately call this pattern "library + application" or "generic + wrapper." For lack of a better name, I call this way of cooking infrastructure Gangnam Style.

If you look closely at this image, you can see that Korean rapper PSY is subtly sending us a message about Chef cookbook development.

Why are roles bad? Why do you need to write two cookbooks rather than one? Well, this all boils down to cookbook development workflow. I start by developing locally on my workstation using Vagrant, Berkshelf, and Chef Solo. I find using Chef Solo with a local Vagrant VM much faster than working with a Chef Server. This has the added benefit that you will be able to share your library cookbooks with people outside of your immediate organization. This can help a lot when you look to the community for help fixing a thorny issue. Now you can use roles with Chef Solo, but in my opinion, it is a very bad idea. You will have two copies of a given role, one copy in a given cookbook and the main copy in your chef-repo code repository. Syncing the two will introduce errors and complexity unnecessarily. Roles are also a big fat global variable across your entire infrastructure. You can't version them. You can't roll out a new role incrementally. Changing a role instantly affects every node that uses it regardless of its environment.

It is critical to use a transient virtual machine for cookbook development. You can't really be sure that your cookbook works properly until your delete your virtual machine then reprovision it from scratch.

For the purpose of this article, let's say I am working on a PostgreSQL cookbook for Acme Corporation. I will fork the default PostgreSQL cookbook from Opscode and only make changes to that cookbook that I will be able to push upstream later. I will put all Acme-specific values and configuration into the acme-postgresql cookbook. In the past, I would have put the Acme-specific values into a role and then done my best not to tweak the PostgreSQL cookbook in any way that infected it with Acme-specific values that couldn't or shouldn't be upstreamed. Inevitably, Acme-specific logic and data would make its way into the library cookbook and the changes would never be upstreamed.

The Acme-PostgreSQL cookbook wraps the generic PostgreSQL cookbook with `include_recipe` and then overriding certain attributes and "editing" a few resources with chef-rewind (formerly named chef-edit). For instance, I may need to change the permissions of the main PostgreSQL configuration file because ACME's DBA team has a written policy that specifies the exact mode for that file. We can argue about whether or not the file mode is worth changing but that won't change the fact that ACME's DBA team is paying me an hourly rate on a short-term contract. If I don't fulfill their requirements to the letter, I won't get paid :). Next, I have to disable the PostgreSQL service because it is actually managed by Pacemaker.

Once I have most of the cookbook's elements in place, I begin testing it on my own laptop using Vagrant. Once I am happy with the state of the cookbook, I will deploy it to a server in the QA or Test environment. I am very strict about which cookbook versions are allowed by my environments. In case these two cookbooks are already in use, I make sure that I have incremented the cookbook versions in metadata.rb so they will not be loaded into a production environment accidentally. In the past, I would have to change a role used by both Production and QA servers. This was bad, very bad. Using a wrapper cookbook I can avoid this possibility entirely while keeping all of the benefits of a role.

Once I have deployed to a QA server, I monitor it for anything unexpected. In the case of my haproxy servers, I actually have written a set of integration tests using Rspec. If everything goes OK after a couple days, I will go increment the version of the wrapper cookbook allowed on the production environment to that of the new version and run chef-client on the target production servers.

Now what if we had chosen to use roles? Well then the deployment to QA would have been a Hell of a lot more nerve-wracking. Unfortunately, I have made the error in the past of adding an error to a base role. Luckily I caught it before it affected more than a couple servers.

So far I have explained how I distinguish between Library and Application Cookbooks but here is a table that I think will help you visualize those distinctions.

Gangnam Style Cookbook Pattern

Recently, there was an awesome Chef Hangout on Cookbook Reusability which you should definitely check out. Some participants said that editing resources with chef-rewind (it was called chef-edit at the time of the hangout) has a bad code smell, i.e. a pattern to be avoided. I argued then as I do now that this pattern is one of many tools we can use to make cookbooks reusable. I like it because it is the simplest thing that could possibly work. I could modify the underlying postgresql cookbook so that I can paremeterize those settings that I customize. In that case, I would still want to put the attributes for those parameters in the acme-postgresql recipe. For me personally, modifying a generic cookbook requires a lot of careful thought in order to avoid breaking functionality for users on other platforms. When I am pressed for time and doubt that I will get around to pushing my changes upstream, I prefer to use chef-rewind to override specific resources.

This bears more attention. In any open-source project, regardless of how well run it is, it is easier to pull upstream changes than push changes upstream. The team at Opscode does a great job of integrating changes in my humble opinion, but they have to make sure that any change you introduce doesn't break functionality for other users. For this reason, I recommend avoiding changes to the Opscode cookbooks unless there is a compelling reason to. This you way you will benefit from their ongoing maintenance and updates of the community cookbooks.

When should you only write one cookbook rather than two? Whenever you are writing something that is purely generic, you only need one cookbook. As soon as you start putting in application-specific data, I recommend splitting that application cookbook into a wrapper cookbook. I recommend this even if you never intend to open-source your generic cookbook. The flexibility you gain will save your ass later.

On the Cookbook Reusability Hangout there was a lot of talk of providing better abstractions and the need to provide general purpose LWRPs and DSLs for popular software applications. Adam Jacob made a great counterpoint with this argument "You could write a complete Ruby DSL for haproxy but would it be worth the engineering effort? I doubt it." Note: This is paraphrased. I tend to agree with Adam here, you could write a DSL for some configuration languages but sometimes it isn't worth the effort for the more complex ones. It especially isn't worth it if the DSL is more complex or harder to interpret than the original. Remember, the source configuration language may be a pain in the ass but at least it is documented and its problems known. The same likely cannot be said for your new DSL or LWRP. An LWRP has to be an order of magnitude simpler and more intuitive than the outputted source in order to be worth the effort.

I recently had the pleasure to work on a DSL for collectd plugins with Pierre Yves Ritschard. I really this DSL works because the collectd configuration language is ridiculously simple. Here is an example:

config = Collectd::DSL.parse do
  load_plugin :redis
  load_plugin :java
  load_plugin :disk

  plugin :disk do
    disk "/dev/sda"
    disk "/dev/sdb"
    ignore_selected "false"
  end
end

With collectd, the scope of configuring plugins is really quite narrow and provides an excellent opportunity for a DSL. This is where DSL's and LWRPs really shine: Providing conventions for either small problems or subsets of large problems. For example, I think it would be foolhardy to write a general purpose Ruby DSL for JBoss's XML configuration which covers tens of different J2EE service profiles from messaging to mail to clustered caching. I do believe, however, that it would be feasible to write a DSL for the most common set of configurations for JBoss's Web Profile.

Let's step back for a second and think about how we would use the collectd DSL from the above example. We could add a recipe to the collectd cookbook collectd::acme_plugins that uses that DSL for our custom-configured plugins. However, we would be far better off creating a new cookbook acme-collectd with the recipe named plugins. Whether or not you choose to use roles, wrapper cookbooks give you a lot of flexibility.

I too would love to parameterize all the cookbooks and LWRP all the things but we need to make sure that our abstractions are significantly easier to use than the source configuration languages. We also need an array of strategies to make cookbooks more reusable rather than one true path. In this post, I strongly advocate the Gangnam Style pattern to wrap a generic cookbook with one which holds application-specific logic and data. Whatever whacky patterns we come up with, we will never move forward as a community (and profession) if we do not succeed in making cookbooks reusable.

P.S. Nathen Harvey helped edit this post
P.P.S. Thanks to those who participated in the Chef Hangout on Cookbook Reusability which greatly stimulated my thinking on this topic

Rewinding Chef Resources

2012-11-08T01:01:00.002-08:00

Note: 11 Nov 2012, the chef-edit gem was renamed chef-rewind

Earlier this week I realized that I misunderstood how duplicate resources are handled. When you use a resource twice, it actually is actioned twice, not once. The second instance will inherit attributes from the first instance but not its action.

For example,


# jboss/recipes/default.rb


template “/usr/local/jboss/standalone/standalone.xml” do
  source “standalone.xml.erb”
  owner “jboss”
  mode “0755”
  notifies resources(“service[jboss]”), :restart, :immediately
end





# esb/recipes/default.rb

include_recipe “jboss”





template “/usr/local/jboss/standalone/standalone.xml” do

    source  “prod-standalone.xml.erb”

    owner “esb”

    notifies   
    resources(“service[jboss]”), :restart, :immediately

end

What actually happens here is that the second template inherits the “mode” of the 1^st template, but beyond that, itis a separate resourceservice restarted twice, on every chef run. This is a big problem.called “chef-rewind” and it adds an addition function to chef for rewinding an existing resource so it is only actioned once. I packaged it as a ruby gem and not a cookbook. You can install it using `chef_gem` When you run chef, the file will be template twice and the I wrote a ruby gem last night that should fix this.


# jboss/recipes/default.rb


template “/usr/local/jboss/standalone/standalone.xml” do

    source  “standalone.xml.erb”

    owner “jboss”

    mode   “0755”

    notifies   
    resources(“service[jboss]”), :restart, :immediately

end


# esb/recipes/default.rb

include_recipe “jboss”

chef_gem “chef-rewind”

require ‘chef/rewind’


rewind :template => "/usr/local/jboss/standalone/standalone.xml” do

    source  “prod-standalone.xml.erb”

    cookbook_name "esb"

    owner “esb”

end


Pay special attention the `cookbook_name` attribute. Without it, the resource will look for it in the jboss cookbook.

This is very alpha right now and I will be testing it today. Feedback would be very much appreciated.

A Year with Chef

2012-10-20T12:50:00.000-07:00

It literally snuck up on me the other day that I had written a year ago a blog post entitled "A Month with Chef." With the Opscode Community Summit happening this coming week, I think it might be time for a few reflections.

It is a little staggering to think about how use of Chef has impacted my work over the last 12 months. In large part because of Chef I was able to roll out over 130 virtual machines in a tiny fraction of the time that it took me to do with my bespoke python and bash scripts. I also rolled out comprehensive monitoring with collectd, statsd, logstash, and graphite for all of my VMs. I simply couldn't have accomplished that without Chef. I still don't have full JVM monitoring but I am working on that. Lastly, Chef has kept my job interesting. I love being a sysadmin but often this job can be tedious. Chef alleviated that tedium to a large extent.

Lessons Learned

So that's what Chef has done for me, but along the way there was a whole lot learning, mistakes, and learning from mistakes. There are many lessons that stick out most in my mind but a couple more than the others.

First off, follow Jamie Winsor on github and follow the workflow that RiotGames team has come up with. Jamie and company folded many of their best practices in the Berkshelf tool. I have experimented with a number of different workflow tools but The Berkshelf Way works best for me. Jamie wrote up an excellent guide on how to develop application cookbooks. Read it and live by it. I had already come to some of the same conclusions but many were new to me. I had the pleasure of interviewing Jamie for the Food Fight podcast. If you are into Chef or just curious, you should definitley give a listen.

The next most important lesson that you should really use pry in order to learn Ruby and Chef. Pry is an improved read-eval-print-loop for Ruby that will help you inspect, explore, and understand Chef. It is irb on mega-steroids. Pry isn't something you use once you are more advanced with Ruby but rather I recommend pry to learn Ruby from scratch. Go watch Josh Peek's 10-minute intro video _now_. Debugging a recipe is as easy as inserting require 'pry'; binding.pry . With pry-nav, you can easily step through your running Chef code.

Here are the rest of my lessons with the ones recommended by the _Berkshelf_Way_ denoted by (BW)

Roles are bad. Don't use them. Roles make it hard to collaborate on and test cookbooks in a distributed manner. Use a wrapper cookbook instead (BW).
Use wrapper cookbooks to add information and settings specific to your organization to a more generic cookbook. For example, use a acme-postgres cookbook to override the default values and add corporate-specific policies to the mainline postgresql cookbook (BW).
Use chef-solo when developing cookbooks even when you will use chef-server to deploy them. It will give you far faster feedback (BW).
Every cookbook must live in its own git repository.
Whenever possible, add tests to your cookbooks. It will save your sanity, seriously.
As a sysadmin, you won't be able to keep your cookbooks up-to-date without getting your developers involved in writing them.
You can't automate something you don't understand. You can't write a good postgresql cookbook when you know nothing about postgresql. You can write a lousy postgresql cookbook for a learning experience, but throw it away and start fresh once you know what the Hell you're doing.
Ruby is a lot of fun, see earlier point on pry.
The amount of instability in gems and the Ruby VM itself often drive me to the brink of insanity. The amount of effort that it takes to get Ruby 1.9.x installed on an arbitrary machine is ridiculous.
Update your cookbook's version in metadata.rb with every significant change.
open-source your cookbooks whenever possible. You will get valuable feedback that will improve their quality and make them more secure. Further, interacting with the larger open-source community can vastly improve your skills.
Follow Andrew Crump on Github. He is author of Chefspec, foodcritic, and a key contributor to minitest-chef and test-kitchen.

What is Needed Now

More than anything, Chef needs a comprehensive guide book. I know that Stephen Nelson-Smith is working on the Definitive Guide to Chef. Unfortunately, the release date for that book is uncertain. James Turnbull and Jeff McCune did a kickass job with Pro Puppet and Chef desperately needs the equivalent.

Here are some additional things that are needed:

We need a lightweight orchestration tool that works when managing several vagrant vms and to deploy a cluster on vmware. Rake tasks don't cut it and Apache Zookeeper is definitely not lightweight.
Some kind of maintainership program, culture, or outright cult to encourage people to maintain cookbooks for a variety of operating systems.
Better testing tooling. Test-kitchen is a good step but much more remains to be done such as better integration with Continuous Integration systems and support for Berkshelf.
Make it easier for newbies to get started w/ Ruby. This is the biggest hurdle for newbies. ruby 1.9 is easy to work w/ _once_ you get it installed. It really needs to be as easy as sudo apt-get install -y ruby19.

Chef Compared to Ansible, Pallet, and Salt Stack

I did write up a detailed comparison with Puppet but I have not given newer frameworks such as ansible, saltstack, and pallet the attention they deserve. Nor have I done a proper analysis of CFEngine 3. I have been remiss in that, mea culpa.

In Summary

The Chef tool and community are far more mature than they were 12 months ago. I expect to see exponential growth in the community and use of Chef itself. The hardest part of working with Chef is managing to keep up with the rapid pace of development and community contributions. Luckily, Nathen Harvey does a fantastic job of summing up the latest developments with his What's Cooking segment on the Food Fight Show.

To wrap up, the best thing about working with Chef for 12 months is that it has made me a better sysadmin.

Stash those logs! Set up Logstash quickly with Chef + Berkshelf

2012-07-02T00:25:00.004-07:00

I have been spending a lot of time lately with the excellent logstash tool. If you aren't familiar with logstash, it reads your logs files, can optionally filter and transform them, and can send them somewhere else for arbitrary purposes. Setting up logstash can be a bit complicated, especially once you add in elasticsearch as the backend. Luckily, there are some great chef cookbooks, from Karel Minarik and John E. Vincent, for both elasticsearch and logstash. I have extended both cookbooks for my purposes, with pull requests pending as of this writing.

I use logstash to monitor my haproxy front-end servers. Call me crazy but I like haproxy so much that I even use it do all of my front-end URL rewriting and redirects rather than using a more traditional tool like Apache or Nginx. At this time I only use Apache for my SSL termination. All actual HTTP serving is done by backend java application servers. Haproxy is in my humble opinion, more flexible, easier to configure, more performant than Apache except for one area. It is very easy to make apache send different kinds of logging information to separate log files. Haproxy sends all of its logging information to syslog and its up to you to break it apart. Enter logstash. As the great Lusis describes logstash.

Stuff comes in
Bits get Twiddled
Stuff comes out

For a good introduction to logstash, I recommend Lusis slides from Chefconf. I will only cover my immediate use case here. I need to add a logstash agent on each haproxy that sends them to a central logstash server that aggregates all the logs and makes them searchable by injecting them into elasticsearch.

Here is the basic configuration for the logstash agent on my haproxy servers

As you can problably intuit from shipper.conf, it reads from the haproxy.log file and sends it off to my main logstash_server using AMQP. I actually could read directly from syslog rather than the file. I just haven't gotten around to doing that yet. Then on my logstash_server I have the following configuration:

To actually install the agent on my haproxy server, I just add the Chef recipe logstash::agent to the run_list of my haproxy role, add a few attributes to the same haproxy role and voila! the agent is shipping logs. On the logstash server side, I add the following to my node's run_list plus a few odd attributes

    "role[elasticsearch_server]",
"recipe[logstash::server]",
"recipe[logstash::kibana]",
"recipe[logstash::haproxy]"

After my next node convergence you will see a pretty picture like this

Wow! This is the excellent Kibana interface created by Rashid Khan. You may have realized that it is a little more complicated to put this together than I have let on. Let me walk you down the steps of how I actually put it together. But first let's be clear what is happening here

1. Logstash agent on each haproxy server reads the log files and ships each individual entry over AMQP to the logstash_server
2. A rabbitmq-server on logstash_server receives the amqp messages and passes them to the logstash server
3. The logstash_server `groks` each entry, i.e. it parses the entry and labels named fields according to a pattern, in this case the HAPROXYHTTP
4. logstash_server puts the grokked log entries into elasticsearch so that they are searchable
5. You can use the pretty kibana interface to watch a live stream of log entries or query by any `grokked` field. Kibana queries elasticsearch directly using Lucene query syntax. It does not talk directly to the logstash_server.

As you can see, that's a lot of moving parts! Each moving part requires its own Chef cookbook. This ends up being quite a lot of cookbooks to manage. Enter Berkshelf, an excellent tool for managing cookbooks, written by the great team at Riot Games. We will use Berkshelf to do much of the heavy lifting for us. I will walk you through how to use berkshelf together with chef-solo to set up logstash.

Before we move on, you should know about the other "moving part" that everybody talks about, the awesomeGraphite. Many, many people use logstash to glean metrics from log files and send information on to Graphite where it becomes pretty graphs. This is also a large part of Lusis aforementioned presentation. Unfortunately, I haven't started with Graphite yet but once I do I will have another juicy blog post to share with you.

Setting up Berkshelf

It used to be a bit difficult to install Berkshelf, but now it couldn't be easier

   $ gem install berkshelf

How Berkshelf Works

Simply insert the following at the top of your Vagrantfile and Berkshelf will ensure that your cookbooks are uploaded to your vagrant VM

require 'berkshelf/vagrant'

To upload your cookbooks specified in your Berksfile to your chef-server, do the following

$ berks upload

This Berksfile instantly gives your team a mechanism to work off the same set of cookbooks. Back in the dark ages of Chef, Oh 9 months ago, we all kept our cookbooks bunched together in a single git repository. This caused all kinds of headaches if you actually wanted to push your changes to an individual cookbook back to Opscode or to someone else. Some months ago, Joshua Timberman split the monolithic https://github.com/opscode/cookbooks repository out into individual git repositories. This has made it a million times easier to collaborate on cookbooks but much harder to work on a common set within a team.

This scenario shows how Berkshelf can help a team collaborate on common set of cookbooks but we will use berkshelf for a different purpose. When working on a cookbook or set of cookbooks for an application, we don't need all of the cookbooks in the our top-level Berksfile at ~/chef-repo/Berksfile. Further, we may be collaborating with people outside your organization who don't need or want to see all your corporate cookbooks. For ease of development, each cookbook needs a description of its dependencies. For this reason we will also have a Berksfile for the logstash cookbook that describes its dependencies.

Here is my Berksfile for logstash.

Loading ....

Let's Get this Show on the Road!

Too much talking, let's get some work done. First, clone my chef-logstash repository.

$ cd ~/projects/
$ git clone git://github.com/lusis/chef-logstash.git logstash
$ cd logstash

It is critical that you clone the repository as logstash and not the default chef-logstash. Otherwise you will get a cryptic error message later on.

Next, let's fire up a vagrant VM using the Vagrant definitions that come with the logstash cookbook. I am assuming here that you are already familiar with Vagrant but if you are not, check it out here.

$ vagrant up lucid32

I also have included vagrant definitions for 64-bit Lucid and CentOS 32-bit.

This will take a while. Vagrant will fetch a vagrant box for 32-bit Ubuntu Lucid and then apply a set the set of Chef recipes to set up logstash. You should also see the result of some integration tests I have written for the elasticsearch cookbook using the minitest-handler.

You should see at the end of the run the output from the integration tests.

Finished tests in 8.395664s, 0.4764 tests/s, 0.4764 assertions/s.

4 tests, 4 assertions, 0 failures, 0 errors, 0 skips
[Mon, 02 Jul 2012 08:17:03 +0200] INFO: Report handlers complete
[Mon, 02 Jul 2012 08:17:03 +0200] DEBUG: Exiting

You may also see the a bunch of invocations of the `curl` command in red. The integration tests use curl extensively and curl outputs its progress to stderr. You can disregard those.

Next, fire up a web browser with the using the IP address of your vagrant VM, something like http://192.168.2.x. POW! There is Kibana, the wonderful web interface for Logstash created by Rashid Khan. No logs are displayed as we haven't yet put any data into logstash.

Lets go into more detail about how the logstash cookbook works. The main logstash configuration file consists of what are essentially ruby hashes.

Here is a sample logstash server configuration

Loading ....

Instead of writing an LWRP for logstash. I make it very easy to template the the logstash configuration using attributes provided by a role or a node. Here is a sample role for configuring a logstash_server that matches the previous logstash.conf file.

Loading ....

All you have to do is provide a ruby hash and the logstash templates will render them them into logstash configurations. There are more details in the README.

There is a lot more to the awesome logstash and its underlying dependencies like elasticsearch. Even if you are not looking to implement logstash any time soon, you should seriously take a look at using Berkshelf for managing your cookbook dependencies. Let's hope I find time in the next month to write about Logstash + Graphite.

P.S. Thanks to Jamie Winsor (Berkshelf) and Karel Minarik (elasticsearch) for their assistance in working with their respective projects.

Real Tests, Real Simple with Minitest, Vagrant, and Toft

2012-04-30T00:18:00.002-07:00

I have put a lot of work into my Ark cookbook lately and it has grown to such complexity that it my existing set of workflow and tools were not enough. I needed some way to sanity check my code. I needed tests.

You can't refactor with confidence without tests. I have fully rewritten ark several times already. After making serious changes, I then had to spend roughly 20 minutes manually testing the cookbook to ensure that it still worked the way I intended it to. This manual testing was getting tedious, really tedious.

Looking at my various options, I first took a look at Cucumber. There are some nice tools out there like simple_cuke, cuken, and ironfan-ci. I made some attempts at using simple_cuke but they failed fairly miserably. That's because I thought I could use Cucumber without having to actually learn it. Cucumber is a powerful and somewhat complex tool. To benefit from using Cucumber, I was really going to have to read the The Cucumber Book, something I don't have time for right now. I needed the simplest thing that could possibly work. Further, the real benefit of Cucumber appears to be the communication it facilitates between customer and engineer. In this case I am the client and the extra layer of communication is unnecessary.

Enter MiniTest

Luckily for me, BTM and Andrew Crump have been doing great work adapting David Calavera's chef-minitest-handler https://github.com/calavera/minitest-chef-handler. MiniTest is the simplest thing that could possibly work. It works by gathering all the files that match the path files/default/tests/minitest/*_test.rb for the cookbooks in your run_list into /var/chef/minitest/. Once your regular chef run completes, the minitest-handler cookbook executes each of those tests and displays the results.

Note: As of 30 April, 2012 there is a known bug in minitest-handler that requires you run chef-client twice in a row for the tests to actually execute.
Update: Set the network type to bridged and you won't have this problem config.vm.network :bridged

However, running in bridged mode may require sudo.

My ark cookbook is really just an LWRP. The default recipe does nothing more than install the unzip package on those *nix distributions that don't have it by default. I created an ark::test recipe that exercises all of the features of the ark resource. Here is a snippet of it.

ark/recipes/test.rb

Loading ....

Next, I have my actual MiniTest tests classes, scoped according to my recipe "test"

ark/files/default/tests/minitest/ark_test.rb

Loading ....

To run these tests I have to first add chef_handler and minitest-handler to the beginning of my run_list. Note that the minitest-handler-cookbook doesn't yet support chef-solo, but I patched it over the weekend so that it does.

Here is what the output of the tests looks like

I originally wrote my tests using MiniTest::Unit test cases. However, just a few days ago I discovered the awesomeness that Andrew Crump has built into MiniTest::Spec and a whole set of helper modules that really simplify the code you have to write.

Here is a snippet to test whether a file exists and has the correct owner

Loading ....

I especially love all of the resource matchers that let you connect to the actual resource objects created during your Chef run and to check whether their configuration matches the actual system state.

For some detailed examples, check out this example.

The Not so Simple Part

Ok, that was the easy part. The harder part is to run your tests on a clean environment. You need a relatively fresh VM to ensure that the tests are actually testing changes made by your cookbook and not some change made eons ago for another purpose. The simplest tool for this is Vagrant. Here is the Vagrantfile I use to test the ark cookbook.

Loading ....

Here are the commands you run to provision your new server using vagrant

Loading ....

Note when using Vagrant that bridging your VM's network to your wireless network connection is not a straightforward process, at least not on my linux laptop. If you run linux as your primary OS, I recommend you spare yourself the pain of trying to bridge your VM to wlan0 and just stick an Ethernet cable into your laptop.

While Vagrant is an awesome tool it is not the best tool for all situations. One issue I have is that my corporate workstation is a very locked down Windows Vista installation. I do virtually all of my work within an Ubuntu virtual machine running on top of Virtualbox. Vagrant uses virtualbox to create new virtual machines and there is no mechanism to run virtualbox within virtualbox. Earlier this year I attempted to get ruby and vagrant running on Windows Vista but after 4 hours I didn't even have ruby running on Vista.

Another issue with Virtualbox is that it's flexibility comes at a cost. Virtualbox runs entirely in userspace which means it is less performant than other virtualization tools such as KVM and linux containers (LXC). This performance lag could become very noticeable once you try to spin up a 4-node postgresql cluster. Of the three virtualization technologies mentioned here, LXC adds the least overhead as each container shares the same copy of the running kernel. There may be other optimizations that LXC provides which I am not aware of.

Enter LXC and Toft

As mentioned earlier, LXC has the least overhead of the other virtualization technologies available. Toft is a ruby library for testing chef and puppet cookbooks using LXC containers. Toft has a beautiful interface for working with containers as you can see in the following code.

I can easily see spinning up a multi-node cluster with toft, running my tests, then destroying the whole setup.

While toft has an excellent interface it is a young project with some rough edges, some of which have more to do with LXC than the toft library itself. To get started with toft, there are some nice guides on the project wiki

I very unpleasantly discovered that the version of the cgroup-lite package for Ubuntu 11.10 installed with LXC will make your system unbootable. The solution is to install cgroup-lite from the oneiric-proposed repository.

Loading ....

Another pitfall is that the bind9 and dhcpd servers installed along with LXC will report erroneous information after initial installation. The fix is easy. Run the `lxc-prepare-host` command and you will be in business.

Finally, at this time Toft only works with Ruby 1.8.7 and does not support Ruby 1.9.*

As noted in the Vagrant section, I highly recommend connecting your laptop to Ethernet if your primary OS is Linux.

Summary

Testing is an exciting and increasing critical part of developing infrastructure. While the existing tools need some polish, you can get a lot done with them. The linkbait title of this post is a bit disingenous. Testing w/ minitest is a breeze, but using vagrant and toft are not straightforward affairs. Toft, in particular, is extremely promising and worthy of your attention. It is extremely well-suited to many of the usecases that Vagrant is not.

Unpacking and Installing from Archives with the Ark Resource

2012-03-21T01:36:00.001-07:00

Updated 14 April 2012: The ark API has changed!

The ark_put, ark_dump, ark_cherry_pick have been rolled into the main ark library as the actions :put, :dump, and :cherry_pick. Please see the updated README

There are many wheels that sysadmins reinvent but the one I see reinvented most frequently is the unpacking of a tarball and placing its contents in an arbitrary location. This task seems trivial but there are many slight variations but critical variations to handle. Most chef users resorts to a combination of remote_file, bash, and link resources. This gets the job done but it is a lot of boilerplate for an essentially simple task and it is ridiculously easy to inject errors. I first wrote the java_ark Lightweight Resource Provider to handle this. Then I discovered Infochimps excellent install_from_release LWRP.

Unfortunately, install_from_release did not do everything I need so I had to extend it further. In the process of extending install_from_release, I mangled Infochimps beautiful code into an unworkable mess. I have now refactored that mess into something that is more maintainable and extensible. Please meet the ark resource. Think of ark as an archive but Kewler. You can thank Joshua Timberman for the name as he didn't care for my original name for it, `cpr` for Crappy Package Resource. CPR was a play on the medical term CardioPulmonary Resuscitation. Ark does not yet handle all of the functionality that install_from_release does. Install_from_release not not only unpacks tarballs but can build with make, ant, and more. Ark does not currently support ant, make, or setup.py but it wouldn't be too much effort to add that support.

Let's get back to the problem that ark solves. For many of us, the packages available from your given *nix distribution are too old for our needs. This is especially true in the Java ecosystem. The latest stable version of Tomcat is 7.0.26 yet only 7.0.21 is available on ubuntu 11.10. Tomcat 7 isn't even in the main RHEL 6 channels. My customers insist on having the latest security fixes, so installation from rpm/deb isn't an option. Installing the Oracle JDK is an even thornier issue as it is no longer available in Ubuntu nor RHEL software channels.

Here is an example of how I would install the Oracle JDK in the bad old days.

Here is how I accomplish the same with ark.

Wow! That's a lot easier, and frankly more intuitive. Pretty soon I found myself repeating several variations on the ark resource. At first I accomplished this by adding more attributes to the ark resource. This made the resource more verbose, intuitive, and made the code harder to maintain. A few weeks ago I read Nikolay Sturm's blog post on Cookbook Reusability and then interviewed him for the Food Fight Show. Through Nikolay, I discovered the Interface Segregation Principle. Rather than adding more options to the ark resource, I totally refactored ark from an LWRP into a Provider and Resource classes that subclass Chef::Provider::ArkBase and Chef::Resource::ArkBase.

Here is the ark_put resource which handles the simplest case:

ark_put simply unpacks ivy to the default location /usr/local/ivy, it does not create any symbolic links
I often have to download arbitrary collections of dependencies and place them in a lib/ directory. I do this frequently for tomcat-based applications. I just need to "dump" the dependencies in a specified directory that likely already holds a number of library files. This situation is a little more complicated as I need a way to determine whether or not the archive has previously been unpacked. With the ark and ark_put resources I simply check if the destination directory is empty. This logic won't work for a directory like tomcat/lib that holds library files from the initial tomcat installation. For this reason, I have to specify a "creates" attribute which specifies the file whose presence indicates that the ark has already been unpacked. Additionally, I need to discard any file hierarchy inside the archive.

ark_dump will place all the *.jar files in the archive into /usr/local/tomcat/lib/. Note: ark_dump currently only supports *.zip archives.

My friends at Oracle seem bent on making me work for my salary. I have encountered on several occasions the need to extract a single file from a tarball and place it in a arbitrary location. In this case that single file will be specified by the "creates" attribute. In this case I extract the JDBC connector for MySQL.

That's the state of the ark resource at the moment. If you are interested to use it and/or extend it, check it out on github. You should note that it is not a collection of LWRPs but actual Provider and Resource classes. LWRPs are great to get started with but don't provide the same facility for code reuse that pure Ruby does. Thanks to Nikolay Sturm for inspiring me to venture beyond the LWRP DSL and to MrFlip (Philip Kromer) for creating the awesome install_from cookbook. I hope that ark will eventually support all of the functionality that install_from_release does.

FOSDEM Summary

2012-02-06T22:50:00.000-08:00

I thoroughly enjoyed FOSDEM. The weather sucked, the beer was great, and the people were wonderful. Thanks to the FOSDEM organizers for putting on such a great conference. What follows is a very rough summary of my experience.

I spent most of my time in the Virtualization room. There were a number of interesting talks there. I didn't spend much time in the Configuration Management Devroom, that is not because the content there was lacking but because I felt more knowledegeable about the subjects there than in the Network/IO,
virtualization, and hypervisor talks where I was pretty much ignorant. From what I could tell, the
organizers did a fine job organizing the Systems and Configuration Management Devroom.

One of the best parts of going to a conference like this is meeting in person that you have collaborated with via e-mail and IRC. I met up w/ Cheffolk Andrea Campi, Juan Jesus Croissier (juanje), Nicholas
Szalay (nico), Stefan Jourdan, and Greg Karekinian (sp?). I have interacted with these guys quite extensively over the last four months, so it was awesome to finally meet them. Juanje has a very different use case for Chef than most users. He manages thousands of desktop machines running GNOME. He must take care not to overwrite the users custom settings, a challenge I have never had to deal with. For this reason he created an LWRP to manage portions of a file. Nico has written a nagios check that will report failing chef runs. I need to start using that.

The best moment of FOSDEM for me was a very personal one. I ran into Sulochan Acharya, a friend of mine from Nepal. We worked together to build the One Laptop Per Child project in Nepal. I didn't even know he was in Europe nor still working in IT. Now he works at Rackspace on Openstack-related stuff. It is a small, small open-source world.

Best talks I attended were Anil Madhavapeddy's "Wild West of UNIX I/O" and Charles Nutter's JRuby talk. Anil talked about the unpredictable behavior that we see in UNIX I/O in virtualized environments. In a
multicore world, hypervisors don't know enough about the underlying architecture to make intelligent decisions about I/O between CPU cores. This leads to very unpredictable I/O results. According to Anil, this requires a better I/O abstraction that is aware of the underlying microarchitecture--CPU layout, interconnects, NUMA--and can make intelligent choices for the developer. Anil calls his next-generation I/O framework FABLE, which is still in very early development. Charles Nutter is a great speaker though his
talk was a bit over my head. He focused on the "impossible" challenges of implementing Ruby on the JVM and how his team overcame them.

I missed the ZIO and CERN talks. Both talked about next-generation I/O frameworks. Also sadly missed Jonathan Ellis's talk on "Overcoming JVM limitations in Cassandra". I spent no time w/ the Mozilla track, which is a shame as I love how they manage their community and the kinds of projects they work on.

I was struck by the large number of projects that featured web frontends written in Ruby on Rails. Besides Horizon, the Openstack web UI, it seemed that virtually every other web UI I saw was Rails. Ironically, there were no talks specifically about Ruby nor Rails at FOSDEM.

Red Hat is doing a lot of activity in the DevOps space. oVirt a web UI for managing VMs. Matahari a system administration API which looks interesting but seems like it is still in early development. I had the pleasure of talking w/ Ohad Levy, a very sharp fellow who develops Foreman, a front-end to Puppet and is employed by Red Hat.

I found out about the meta-project Katello from Red Hat that contains many components to automate system administration. To my limited knowledge, it includes Matahari for low-level administration, Puppet for configuration management, Foreman as a nice GUI to manage your configurations and server provisioning, oVirt to manage your KVM hosts a la VMware's vSphere, and pacemaker-cloud for high-availability across different cloud providers. I believe the combined stack is at alpha level. I would
love to know if Chef could be used in place of Puppet.

Met the very smart Marek Goldmann who is doing a lot of work packaging JBOSS for Fedora in addition to his boxgrinder project. It seems like there is a fair amount amount of overlap between what he and are trying to do in this regard. I need to spend some time hanging out on #fedora-java to see how we can avoid duplicating efforts.

My greatest regret is that I myself didn't give a presentation. I didn't think I had anything to say that others would be interested, but that view was wholly wrong. There was a lot of interest in Chef but unfortunately not much content.

I met a couple of people who had listened to my podcast, Food Fight, but they told me it was "unlistenable" due to the recording quality. Guess I will have to improve the audio quality :(

I was very lucky to have dinner with a group that included the DevOps gurus Kris Buytaert and Patrick DeBois. After following both of them on Twitter for a number of months it was really interesting to see what they are orking on. These guys really are the "Belgian Braintrust" of DevOps not to mention that Patrick actually invented the term. If I recall correctly, both Patrick and Kris are spending a lot of time
lately working on monitoring and logging. Not only are Kris and Patrick DevOps heroes but they are also genuinely nice guys.

Beer

Belgians make the best beer in the world and they enjoy it at all hours and in all places. Quite a number of people enjoys fine brews during presentations and meetups. The Opensuse stand actually gave away opensuse beer. Note to self, avoid the 10 % alchohol beer when trying to stay alert. Stick to the delicious 4% blonde beers.

Final Thoughts

Openstack is the real deal, not just a cool idea. It will really be ready for production use this year. I had my doubts about an open-source project led by an industry consortium but what I saw made me put them
aside. Particularly impressive was Ryan Lane's explanation of how Wikipedia is using Openstack for its infrastructure.

There is awesome stuff happening with KVM, both in terms of performance and tooling.

There was a lot of interest in Chef but I didn't meet that many people who had actually used it. This must be remedied. I really should have given a talk and put more effort into planning a Chef meetup.

I will definitely be back next year and hopefully as a presenter.

Announcing the Food Fight Podcast

2012-01-15T22:23:00.000-08:00

I just published the inaugural episode of Food Fight, the Chef community podcast that I host together with Matt Ray. The podcast will cover not only the developments within the Chef community but also larger world of DevOps. Enjoy!

Chef in 2012

2012-01-04T14:08:00.000-08:00

image by Simon Griffee

Note: Petey King has written a great response to this post. Definitely worth a look.

I discovered Chef in the last last quarter of 2011 and turned my world upside down, work-wise at least, in a really awesome way. Over the holiday season I had some time to reflect about where Chef as a piece of software and as a community of people might go in 2012.

Packaging Cookbooks

Cookbooks are starting to more and more look like packages, with complex dependencies between them. I am not aware of any cookbooks that only work with Ruby 1.9.* but I expect we will see a number of them in 2012. I personally believe that we need to use an existing package management system rather than creating yet another one. RPM, DEB, dmg are all out of the question because they are operating system specific. Ruby Gems seems to be the best choice though I am not very knowledgeable about ruby gems. If you think this a terrible idea, please do let me know what problems ruby gems pose. I envision the scenario where your cookbook_path in knife.rb

As you can see from my example, you still would be use cookbooks as they currently are, but also allow chef to look for additional cookbooks in your GEM_PATH. Cookbook gems would have to be prefixed with something like "chef-ckbk-" to indicate to knife which gems are cookbooks.

Using ruby gems does present a couple challenges, particularly with rvm, which completely compartmentalizes gems from one ruby version to another. This means that changing your ruby interpreter requires you install anew a large portion of your cookbooks. Using gems may also require a lot of people to change the structure of their cookbooks. Perhaps, moving moving all the directories to under gem_name/lib/ subdirectory.

Testing 1, 2, 3

There still isn't a broad consensus on how to automate the testing of chef cookbooks. There was some nice progress during 2011 with foodcritic (lint for cookbooks), chef-minitest (unit-testing?), and cucumber-chef but those tools are all still early days. Testing is a particular pain point as testing a minor change to a cookbook may require testing against 5 different linux distributions. I really hope we see more progress on this front though I can't point to any particular initiatives. I personally hope to spend some quality time with minitest in the next couple weeks.

Other Miscellaneous Technical Needs

Selinux support is currently very limited. Some people feel that selinux shouldn't be used in the first place, however, many sysadmins don't have the luxury to make this choice themselves but are required by an external corporate or legal body to use selinux. As Andrea Campi has pointed out on the Chef mailing list, there are a number of consulting firms, such as ZephirWorks, that have the capacity to implement better selinux support if someone steps forward with funding.

Image by Simon Griffee

The java application stack also needs a lot of love. The current set of tomcat, jetty, maven cookbooks only support minimal configurations. I am reorganizing these cookbooks around the java_ark LWRP. In a nutshell, the java_ark LWRP extracts a tarball, adds any included executables in the path, and runs update-alternatives. Along with the jboss cookbook I have already started, I expect to put a lot of work into the java stack. If this is something you are interested in, suggestions, code criticism, and patches are most welcome.

While chef-client and chef-server generally perform at acceptable levels, there are some pain points. For chef-client, memory usage can sometimes be an issue due to the node object ballooning in size during chef-client runs. If I recall correctly, Opscode is working hard on reducing memory usage so we should see some progress this year. On the server side, we can expect a lot of performance improvements as chef-server moves from ruby + couchdb to erlang + mysql. This may happen sooner than one might normally expect as Opscode has already converted part of Hosted Chef to Erlang and MySQL, it just hasn't open-sourced that code yet.

Growing Community

2012 should be the year that the myths "Chef is for programmers" and "Chef is hard" are demolished. These myths sadly linger and inhibit adoption in my opinion. Perhaps part of the reason that some sysadmins think Chef is hard is that it is pure Ruby. Now Ruby is not a difficult language but it is unfamiliar to most sysadmins. Advocating Chef (and Puppet to a lesser extent) also means advocating ruby amongst sysadmins. How to do that? Here are a couple suggestions.

Get ruby 1.9 into the default server installs for RHEL, Ubuntu, ArchLinux, etc. This could be quite a challenge as according to this blog post, there is almost nobody in the Ubuntu world doing any Ruby work. With the exception of the awesome Sergio Rubio, the same could be said for RHEL/CentOS/SL
More blog articles on using Ruby to solve system administration problems and that sysadmins should consider using Ruby as their default scripting language

All that talk about Ruby advocacy aside, Sysadmins will need to know less Ruby in 2012 in order to write Chef recipes. This is because we will see more and more LWRPs pop up. I love LWRPs. I think they are the best thing since dynamically loadable kernel modules.

Chef is growing by leaps and bounds because it solves an important problem in configuration management. The greatest constraint to that growth is the supply of sysadmins willing to make the investment to learn it. At this point, the majority of the community are self-taught linux geeks who found Chef on their own. At the next stage of growth, Chef needs to be attractive to the group I call "the Professionals". Professionals are talented engineers who make system administration their job but not their life. They don't read blogs during their lunch hour and they don't hang out on IRC in the evenings. The Professionals will typically learn Chef because their bosses tell them to. We need to make that learning process as smooth as possible.

The Chef wiki is an incredible reference but we need a guided tour to accompany it. We really need the Chef equivalent to Pro Git and the Git Community Book site. These resources hold your hand and walk you through all the important parts, whispering comforting words when you get frustrated. I know there is work being done a Definitive Guide to Chef from a major publisher, but I don't know its status.

Lastly on the subject of community, we need some mechanisms to summarize community activity.
The Chef project has a high velocity and I find it increasingly hard to keep up with. The Opscode monthly newsletter is a good step in this direction. Additionally, I am starting a Chef podcast together with Opscode Technical Evangelist Matt Ray, focused on summarizing community activity and exploring important topics.

Dangers in 2012

I sometimes worry that Chef could become a victim of its own success. As I mentioned earlier, the greatest constraint to the growth of Chef is the supply of sysadmins willing and able to learn it. If enough well-meaning software architects and aggressive CIOs force Chef on resistant ops teams, we will start hearing about implementations gone horribly wrong. Top-down adoption can only work if the ground below is fertile. That's my two cents.

In Summary

Chef will be a success when the default installation instructions for application X point to a Chef cookbook. I think we will make solid progress towards this goal in 2012 but to really get there, a lot of things have to happen first. I couldn't be more excited. Thanks to everyone in the Chef community for giving sysadmins like me such a great toy to play with. Happy Hacking!

PS: Let me know if you like the Chef icons. My good buddy Simon Griffee created them after a brainstorming session fueled by Sagrantino Di Montefalco.

Emacs for Sysadmins

2011-11-20T05:14:00.001-08:00

I am not going to tell you that emacs is superior to vi, because it is not. For me, vi is far better for basic text editing. However, emacs has such powerful modes that I find myself using it more often than vi these days. Emacs has historically been favored by programmers and vi by sysadmins. I have written in the past that sysadmins are becoming less like ninjas and more like pirates because of this crazy thing called devops.

As a professional system administrator you can't choose between vi and emacs; you have to learn vi. There are just too many times you will be working on an arbitrary *nix that only has vi installed. You can choose to learn emacs in addition to vi. Vi lets you jump in and around files quickly. Nothing can match the speed of hjkl, g, G, etc. for movement or v + y for copying text. However, we sysadmins are moving from managing individual servers to managing configurations. In the last several couple months I have spent at least 50% of my work time entirely within my chef-repo/ directory working on data bags, roles, and cookbooks. Inspired by the devotion of a number of Chef community members to emacs--notably Matt Ray, Stephen Nelson-Smith, Josh Timberman, among others--I decided to give emacs a try. On the whole, it has been a very positive experience. That is why I recommend sysadmins try emacs.

Ninja	Pirate

Bill Joy Vi Creator Hand placement conceals poison dart	Richard Stallman Emacs Creator Note beard

from philosecurity blog ninjas or pirates, emacs or vi

Why Emacs?

Emacs has some truly awesome modes. The default ruby, python, bash, xml, html seem to do more than their vim counterparts. I think this is large part because emacs' extension language (elisp) is more powerful than vimscript. I made a stab at vimscript some years ago but came away quite frustrated. While I am not a lisp convert, it is a full-featured language that doesn't hold you back. Working with multiple buffers (10+) is extremely easy. Org-mode though, is what got me hooked.

Why Not Emacs?

Using vi is like being in a relationship, you are involved. Using Emacs is like a marriage, you are committed, in large part because there is so much to learn. This is one of the reasons I don't recommend it for everyone.

Further, vi is built for touch typists. You rarely, if ever, have to take your fingers off the home row keys. You can keep your fingers on the home row with emacs but that often means completing a number of key chords entirely with your left hand. Take for example opening a file, writing its contents to another file, and then closing emacs.

C-x C-f + C-x C-w + C-x C-k

If you keep your fingers on the home row, you will only use your left-hand for the above commands, except for the final "k". You could start to feel some discomfort in your left hand if you did that key sequence several times in a row. For this reason a number of emacs addicts will hit the Control key with their right hand, the "x" with their left, Control again with the right, "f" with the left, and so on. This leads to a more "hunt and peck" typing style but saves your hands.

First, Learn How to be a Pirate

Before you even start playing with emacs you should make sure to swap your Caps Lock and left Control key. Emacs commands use the Control key heavily and leaving it in the default position will soon leave you angry, crippled, and ready to kill Richard Stallman.

Next, I recommend you organize your dotfiles on github or another source code repository. Being a pirate means you have booty and treasure to carry around. You need a ship to carry around all that treasure and for me that ship is github. The core of my dotfiles repo is the Rakefile, copied from Ryan Bates' master, that makes it dead simple to install your dotfiles on a new machine. I don't install my dotfiles on every machine I work on, only the 2-3 I use most frequently.

create your own github/bitbucket/private github repo called dotfiles
copy your existing dotfiles -- .bashrc, .vimrc, .emacs, etc -- to ~/dotfiles but without the leading "." !
download Ryan Bates Rakefile and put it in ~/dotfiles
to test, rename ~/.bashrc to to ~/.bashrc_bak then run $ cd ~/dotfiles/ && rake install. Your .bashrc should be back
Push to your repository

Whenever you make a change to your .emacs or .bashrc on your ubuntu laptop at home, remember to push your changes and then run "rake install" on your work desktop.

Learning Emacs

OK, finally emacs. I recommend you buy Peepcode's "Meet Emacs" tutorial. It is only $12 and it is the best and most convincing emacs tutorial. It is well worth the money. If this tutorial doesn't convince you that emacs is worth your time stop, turn around, and go back to vi. One of the first things covered in "Meet Emacs" is how to install emacs 24 and the emacs-starter-kit. The emacs starter kit will set of bunch of defaults that help you avoid many of emacs' warts and show off some awesomeness such as ido-mode.

Org-Mode, Emacs Killer Feature

Ok, I have to admit that I have been working with emacs for longer than a few months. I have been using emacs' Org-Mode to manage my life for the last several years, ever since I happened upon Abhijeet Chavan's great article in Linux Journal. Soon after that, I came across Scott Jaderholm's excellent screencast for Org-mode. At the time I read the article, I was the Chief Technology Officer for a non-profit technology startup in Kathmandu, Nepal. I was completely overwhelmed by the sheer volume of different tasks I had to accomplish and all the attendant details, especially the details. I no longer have all that stress now that I am a regular sysadmin, but I do have a large number of very detailed tasks. Org-mode is in many ways my memory. It is also how I manage my attention span.

Many people use Org-mode to implement the Getting Things Done methodology, pioneered by David Allen. I have bastardized it into my own system.

Today is a list of general things that I want to get done today. This week, a list of tasks to complete this week. And of course, right now is a list of specific next actions, that I will complete starting at 1.

When you are finished with a task, don't delete it. Archive it so you have a record of when you are done. Org-mode handily records the timestamp of when you completed the item.

C-c $ # archives a subtree to your archive file,

If I finish reading the O'Reilly Emacs Manual and then archive it, here is what the entry looks like.

I love that I can move items in a numbered list almost effortlessly. M-<up> moves an item up and adjusts the numbering and M-<down> moves an item down in the list.

I typically keep the following org-mode files and their associated archive files.

mygtd.org -- my todo list, for today, this week, this month, and right now!
meetings.org -- all meeting notes and preparation
notes.org -- this is where I keep notes while I work on a problem

The notes.org file deserves special attention. This is my running work log. As I diagnose a problem, I write up my possible hypotheses, then tests for these hypotheses. Finally, I record the results of testing those hypotheses, pass or fail.

The true beauty of using emacs to manage your todo list is that you can grep through it to find a specific piece of information. This has saved my ass more times than I can count.

Org-mode protip: put your *.org files in a Dropbox folder so you have access to them on your various machines. It may be tempting to put them on github but you should realize that those org files will quickly fill up with personal and corporate information.

Working with Buffers

I never found it that easy to work with multiple buffers or tabs in vim without doing extensive key rebinding. Buffers are something that emacs really gets right, especially when you have ido-mode turned on. I have found working with multiple buffers really effective when working with several chef cookbooks at once.

Let's say I am working in the sudo/recipes/default.rb recipe but want to jump back to the previous recipe I was working on, in this case iptables/recipes/default.rb. I type C-x b and ido-mode shows me a list of open buffers. To get the last buffer I was in before this, simply hit [RET] or I can start typing the name of the file and ido-mode will dynamically filter that list.

C-x b [RET] to get to the last visited buffer

Dired-Mode

I have always heard that Midnight Commander is an awesome but I have never taken the time to learn it. I believe that Dired-Mode offers many of the features of Midnight Commander but lets you leverage the emacs skills you have picked up.

Here are commands that I love

Key Binding	Command
v	View a file or directory in read-only mode
d	mark a file or directory for deletion
C	Copy a file
x	execute changes marked, i.e. delete files marked for deletion
q	close an open dired buffer
[RET]	when u hit [RET] on a file, emacs opens it for editing
R	Rename or move a file or directory
M-x find-grep-dired	Grep a directory for a regex and then view the matching files in a Dired Buffer
Q	Find and replace a regex across multiple files

While Dired-Mode is very useful for managing files I find it most useful for browsing code. You can hit "v" to view as read-only then q to close that buffer when you are done. Viewing files in emacs means you get all that juicy syntax highlighting. This helps me make sense of those butt-ugly XML configuration files I encounter on jboss and tomcat installations.

Miscellaneous Stuff I find Useful in Emacs

Shell-Mode - This is very handy for moving text back and forth from a shell and code file. M-x shell-command-region or M-! lets you execute a shell command on a region of selected text
Inf-Ruby-mode - lets you select a bunch of ruby code and immediately execute it in an irb session. I would love to see this work with shef
Modes for Languages you rarely touch - I rarely code in java but occasionally I have to read java code. java-mode helps me parse java without having to deal with Eclipse. Have to change a config file written in Erlang? There's a mode for that.
Properly indenting text - select a region then C-M-\
Completion - M-/ will guess what you are trying to type based on open buffers
Commenting out multiple lines of code -- select a region, M-;

Emacs Shortcuts are Everywhere

In case you haven't already noticed, virtually every unix shell and programming REPL (irb, python, perl) supports Emacs keyboard shortcuts. This is because all those shells are built on top of libreadline, which implements a subset of emacs shortcuts. Even REPL's for newer JVM languages like scala, groovy, and clojure use a clone of libreadline, jline, that implements those same shortcuts.

Emacs Stuff I Haven't Learned Yet but Want to

Tramp-Mode -- editing files over SSH, this would be pretty sweet for configuring a bunch of Virtual Box VM's running on my desktop
Calendar Integration with gmail - just haven't gotten around to this yet
ERC - I debate whether to move from Xchat to irrsi or ERC (Emacs irc client). jury is still out
gist - This is a mode that allows you to select a region and create a github gist from it

Wrapping Up

Emacs, despite conventional wisdom, can be very useful for systems administrators and not solely the domain of Lisp Hackers and not Exclusively for Middle-Aged Computer Scientists. While it may be One True Program for some people, it is not for everyone. The highlights of Emacs for me are:

Org-mode -- time management for busy sysadmins
Dired Mode
Buffer Management

If you have some tricks that you find especially useful for sysadmin work, please share them! Here are some additional resources you may want to look at:

Why Emacs? Borizhdar Batisov's excellent, more developer-focussed ode to emacs
Hack Emacs Screencasts
Mastering Emacs blog
Emacswiki
#emacs on freenode.net

Chef for Developers

2011-11-20T00:26:00.001-08:00

Last Friday I gave a presentation to a group of developers on how they can get started writing Chef cookbooks. I put about 5 hours into writing this presentation. To my chagrin, Warwick Poole released a superior presentation that same afternoon that does a better job of explaining the basic concepts of Chef. I strongly recommend checking out Warwick's "LearnChef" if you haven't already. It is the single best introduction to Chef that I have seen so far. In more good news, Warwick informs me that he intends to create a 5 part series building on "LearnChef." I very much look forward to it.

My Chef presentation doesn't do as good a job explaining the core concepts of Chef but it does walk developers through some scenarios on how they can work with chef on a project. You can check it out below and even reuse my slide material by downloading the .ppt file here. My slides borrow heavily from Joshua Timberman's excellent Chef 101 presentation. The slides are under the Creative Commons 3.0 Unported CC-BY-SA. You can find the full list of publicly available Chef presentations on the Chef wiki.

Chef, Devops, and You

View more presentations from Bryan Berry.

A Month with Chef

2011-10-30T06:14:00.000-07:00

It is hard to believe that I started using Chef only one month ago. The experience has hugely changed how I view the practice of system administration and started to unseat the deepest of ingrained habits. It has been mostly a smooth ride but there have been some significant bumps along the way. Here are my reflections on learning Chef, the community around it, putting it to work, and my hopes for the future of the project.

Learning Curve

There are roughly three technology components to learning chef, each with their own learning curve 1) the Chef software itself 2) git and 3) ruby. I found understanding Chef to be quite easy, in large part because of prior experience with Puppet. If you have never used a configuration management tool before, this can be a stumbling block. However, the biggest stumbling block of the three is probably git, not ruby and not the Chef software itself. Again, I had previous experience with git but even I ran into occasional git-induced hurdles.

When you treat your "infrastructure as code," you have to spend a fair amount of time managing that code. Git is the tool you will use to manage your chef repository, your cookbooks, pull updates the mainline opscode/cookbooks and push your patches to others. If you are a n00b to both Chef and git, do yourself a favor and reach for a good git tutorial before you get too deep into Chef.

The fact that Chef recipes are written in Ruby has turned out to be far less of an impediment than I thought. You can learn enough Ruby to write a recipe in less than twenty minutes. Further, with copy-and-paste plus playing with irb you can guess your way to getting a lot accomplished--though this isn't a good strategy for the long-term. For my own education, I read the pickaxe book and Flanagan's The Ruby Programming Language. Both were good but I prefer the Ruby Programming Language. I have heard from others that Ruby Koans is an excellent way to learn Ruby though I haven't tried them myself.

My first impressions of Ruby the language are shaped in part by my experiences working in the Python world. Python is clean and neat and you can see the hand of Guido guiding the language and standard library at every turn. Conversely, Ruby is creative, chaotic, and messy. It has bitten me in the ass several times that Ruby has hundreds of top-level methods and operators. To say that the namespaces are "cluttered" is an understatement. You have to be quite careful when creating new variables and method names. For instance, In one recipe I created the variables "alias" and "rand", which are coincidentally top-level Ruby methods. Overwriting them caused some nasty side effects that took me a while to track down. If Ruby were a human being it would have a beautiful body, a terrible case of acne, and noticeable digestive problems. Worse yet, its clothes wouldn't match.

My biggest single problem with Ruby is that shit breaks. Last week the main json gem was broken, causing pain throughout the Ruby world. With my first install of Chef-Server, the code-ray gem broke the chef-server-webui. As I am not a rubyist, it took me a non-trivial amount of time to figure out how to downgrade to an older version with the following command $ gem install code-ray --version '< 0.9.8'. I have since discovered the concept of gem sets, which define dependencies to specific gems per specific versions. Why would you have to do this? Because you can't count that the latest gem release will be stable. Working with Python I rarely had these same kinds of problems. On the plus side, the Ruby community moves fast. For whatever reason(s), new Ruby projects pop up quickly and continue to evolve at an astonishing pace. Tools like thor, sinatra, capistrano, and fog all seemed to appear before their Pythonic equivalents*.

* software historians, please correct me if I am mistaken

It is easy to overlook that Chef assumes you already thoroughly understand your operating system and application stack. Chef makes experienced system administrators more productive. It also makes inexperienced system administrators more dangerous and can even hamper their learning. It is for this reason that I don't recommend that those new to *nix pick up Chef. You can't automate something you don't understand.

Using Chef

My experience with Chef has not been all roses. Like Ruby, Chef moves fast. For a FOSS project, it is ridiculously young, first released in January 2009 and has not even reached version 1.0 yet. It still has significant bugs and Linux distribution support is still quite uneven, especially compared to Puppet. To their credit, the Opscoders acknowledge these issues. It must be noted, Opscode did not officially support Enterprise Linux until recently. Further, despite its youth, I found the Chef-Server to be quite stable and easy to setup.

I had a nasty time with the initial installation. It took me over three days to complete, mostly due to a bug in Ohai that misreported the linux distribution. However, that was early October. I did a fresh install last week and it took me only 30 minutes, largely due to documentation improvements, bug fixes, and updated rpms for both EL 5 and 6.

The opscode cookbooks, like the chef software, have spotty support for Enterprise Linux. One of the first cookbooks I tried to use was the selinux cookbook. I couldn't even upload that cookbook to my chef-server as it has a syntax error. I have sent a patch, logged a ticket, and sent a pull request to fix the error. That issue was not fixed the last time I checked. My pull request isn't lonely either. Last time I checked on github.com and I saw that there were 99 open pull requests in their queue. My impression is that the Opscoders are not intentionally ignoring such requests but are struggling to keep up with the rapid growth of Chef community and Opscode's paying customer base. I hope this improves in the months ahead.

Once I got Chef-server up and running, I felt a bit lost. Chef is a very flexible tool and not nearly as opinionated as Puppet. It is hard to decide where to get started. One can easily get lost among Chef's powerful abstractions such as definitions, providers, resources, and libraries. Worse, Chef doesn't have a definitive technical book to hold your hand through the entire process as James Turnbull's excellent "Pro Puppet" does for Puppet.

After my initial bout of confusion I set to work on converting a 400-line RHEL 5 server provisioning shell script into a set of cookbooks. Amazingly, I was able to convert it to a set of Chef cookbooks in less than a week and a half. Even better, these cookbooks could be applied to an existing server at any time in its life, not just at installation time. Bonus, my bash script handled RHEL/CentOS 5-6 on x86_64 and i386 architectures whereas the original bash script only handled RHEL 5 x86_64. I must confess that part of my productivity gains stem from the fact that I am far more proficient in RHEL than I was when I originally wrote the Bash script. That said, I believe that using Chef for the server provisioning task took me 30-40% of the time it would have required to rewrite the Bash script to handle RHEL 6 and the i386 architecture.

I have made a lot of progress with Chef in this first month but I have yet to automate an entire application stack. Here are the tasks I hope to implement with Chef in the next few months:

Consistently test my cookbooks with rspec or another testing tool
Create a Jboss cookbook
Cluster Tomcat instances

Best Practices thus Far

So here is a collection of the Chef "best practices" I have picked up in this first month. Given that it has only been a month, please take them with a grain of salt. Master Chefs, please don't hesitate to correct me if there is any incorrect information here.

If you want to get started quickly, i.e. learn chef in a weekend, use hosted chef and micro instances on EC2. For me it was important to install the entire stack before using it because that is the kind of geek I am. It also sucked up a lot of time.
Separate your ingredients (data) from your recipes. Don't hardcode your site-specific information like IP addresses or host names into the recipes. Put them in Data Bags, role attributes, or node attributes.
Data Bags are for complex or global data like users or samba shares
Role attributes and node attributes are best suited for simple values like host names for NTP servers, Nagios server, or whether to run X service as a independent daemon or under Xinted
If you aren't already familiar with git, you should make learning the basics of git your #2 priority after completing the Chef Getting Started Guide. Since we're talking about "Infrastructure as Code" here, you have to become as serious about version control as a software developer is.
After you clone the main chef repository, create the sub-folder site-cookbooks/ at the same level as cookbooks/ . Put your new cookbooks in site-cookbooks/ and make each one an individual git repository
In .gitignore, add ".chef" and "site-cookbooks". You store all authentication keys in your .chef/ so it is a directory that you really don't want to accidentally push to a public repository.
Sign the CLA for Chef. You can't contribute fixes without doing so.
Don't use $ knife role edit <role-name>; use the Ruby DSL instead. That way you have your roles under version control.
Check, double-check, then ask on IRC before starting a cookbook from scratch. I wrote four different cookbooks before realizing that they duplicated functionality already present in the main opscode/cookbooks
Use shef, it is a wonderful tool to debug and explore how Chef works. Note that it does not load roles attributes at this time.
Fat cookbooks, skinny recipes. You should put the recipes for both the client and server parts of an application in the same cookbook. This makes them easier to keep track of and forces you to make the recipes more modular.

Here are some Ruby tips that I have picked up

Put require 'irb/completion' in your ~/.irbrc, then you have auto completion in irb and in shef as it uses irb

For learning ruby, I actually prefer pry over irb, which reminds me of my beloved IPython. I haven't yet figured out yet how to get shef to invoke pry rather than irb.

To install pry

$ gem install pry pry-doc

You also have to put require 'irb/completion' in your ~/.pryrc or type it in at the prompt. Here is an example of how tab completion works

The next step is to actually check the docs for what particular method does. The command for that is
pry> show-doc Object.method

You can also read ri docs by simply typing pry> ri Class#method like you would at the normal shell prompt.

There are a couple tools that I intend to use in the near future, specifically libarian-chef by Jay Feldblum, but haven't yet gotten the opportunity.

More Pirate, Less Ninja

Chef has made a big impact on my work style. You definitely start working more like a developer, spending more time on your workstation and less SSH'd into a random machine. It is hard to justify heavily customizing your own dotfiles when you spend most of your time on random machines where the only text editor you can count on being present is vi. This article describes vi users as ninjas and emacs users as pirates but I think many of the same comparisons apply to sysadmins and developers. Ninjas can't carry much around with them on their dark forays just like sysadmins can't lug around a bloated .vimrc or .emacs file. Pirates, on the other hand, stay on their ships where they have plenty of room for whiskey, stolen treasure, ( .emacs.d/ || .vim.d/ ), .screenrc, .zshrc, etc.

With Chef, I spend a lot more time working on cookbooks, roles, and data bags in localhost:~/chef-repo/. Knife makes it very easy to manipulate your chef-server and chef-clients with fewer ssh console sessions. I think this work style may explain why there are many more emacs users in the chef-community than in your typical group of sysadmins. Historically, system administration has forced you to be a vi user but as a "chef" you can spend more time in emacs, sam, joe, or even *gasp* nano. I personally have heavily customized my .vimrc over the last month and even dabbled in zsh. I am not suggesting that Chef will turn you into an emacs user, but it will turn you into a pirate.

Selling it to the Team

Managers love chef. They get the idea in the first five minutes and ask you weren't using Chef five years ago. They love it because 1) big, respected companies are using it 2) it makes system configuration far more transparent, i.e. you can see how systems are actually configured and not how you hope they are 3) it will help them deal with the never-ending requests for new Virtual Machines coming from the developers. It can be harder to sell it to those sysadmins who may know *nix like the back of their hands but are used to doing everything by hand and have never gotten into the habit of automating tasks with their own shell scripts. It can also be hard to sell to those sysadmins who have already invested heavily in their own custom bash/Perl/Python/Tcl infrastructure framework.

Here is my pitch for Chef in a nutshell.

A few years ago we had a ratio of n machines per sysadmin. Now we have 3n machines per admin. Next year we will likely have between 6-9n machines per sysadmin. We are struggling mightily to support 3n now and our current coping mechanisms will break down entirely when we move to 6n. The only way we can cope is by moving from home-brewed infrastructure to a real configuration management system, such as Chef.

If you aren't comfortable with writing your own automation scripts, you are at least comfortable using the scripts of others. I don't expect everybody here to be able write their own cookbooks from scratch but I believe everyone here can learn how to reuse and modify existing cookbooks to meet our needs.

Frankly, I don't yet have a way to sell Chef to the NotInventedHere crowd. Whatever benefits you bring up, Chef still wasn't their idea. One option would be to convince them that Chef just does what they have been doing all along and using Chef will free them from maintenance work for their current tools. Then they can focus on what really needs reinventing. Unfortunately, implementing Chef may require a substantial amount of their help and not leave them so much time to start that new pet project.

Thoughts on the Community

I feared that the Chef community would be populated with developers who were "slumming" as sysadmins. I found quite the opposite. The vast majority of the community are Unix-lovin', bash-spittin' geeks who make their careers as sysadmins and not as a side project. In terms of geography, the community is overwhelmingly concentrated in North America. It can be tough to get a response on IRC during the morning in Europe while you will get tens of responses during the US West Coast's working hours. For all hours outside of the European morning, you can find a large numbers of core developers on IRC and they are extremely helpful.

The vast majority of chef users seem to work on Ubuntu, with Enterprise Linux a distant second. Surprisingly, there is a relatively large presence of OpenSolaris/Ilumos and I am not sure exactly why that is. Perhaps Solaris users are disproportionately drawn to automation tools?

When I can't get a quick answer on IRC, the two best channels are the 1) chef-users mailing list and 2) twitter. For some reason, the devops world has really, really gathered around twitter. I can get a quick answer at any hour by using the hashtag #opschef. However, this only works for questions that entirely fit into 110 characters. It is bad form to link to a mailing list post or extend the question across multiple tweets. For complicated questions, the mailing list is always the best place to go for help.

The community is extremely friendly and so far there are no trolls. However, this is typical of a young project. Early adopters by their nature tend to be more positive and self-sufficient. Most people on #chef are there because they discovered chef on their own and not because their boss mandated they use it.

My Short-Term Chef Wishlist

Here is my wishlist for the next 3-6 months

Clear guidelines on how to collaborate on cookbooks and on how to push changes upstream to opscode/cookbooks (something which jtimberman is already working on)
Lower Memory Usage. Chef can really, really be a memory hog when you have several hundred nodes
Better support for Enterprise Linux, both in core chef and the cookbooks
A few weeks ago I would have said better documentation but they have already made huge strides in this regard
A stronger connection to the FOSS community worldwide

Chef will achieve world domination when the installation instructions for the average software application start not by pointing to a .rpm, .deb, or tarball but to a Chef cookbook. Key to achieving that are 1) making Chef awesome (of course) and 2) better engagement with the communities that manage software packaging in the public sphere--the FOSS geeks. These guys are the guerilla foot soldiers of software packaging. They write the man pages, manage a FOSS project's build servers, maintain wikis, etc. The phenomena of "cloud computing" couldn't exist today if it weren't for all their hard work at every layer of the GNU/Linux/*BSD stacks.

How can Chef win over the FOSS geeks? First, having a presence at conferences like FOSDEM, OSI, FUDCON, FISL, etc. would be relatively inexpensive and give them a lot of bang for their buck. Secondly, offering some version of free hosted chef for up to n nodes for open-source projects and foundations like Software Freedom Law Center, Python Software Foundation, Software Freedom Conservancy, etc.

Wrapping up

So far, Chef has lived up to the hype. It has made me more productive in a relatively short amount of time. I can't imagine my future as a sysadmin without it. Configuration management--whether using chef, puppet, or other--will soon be an essential skill for a professional system administrator. Now how does Chef affect my work in the larger context of Devops? Where does it fit in my devops-toolchain? Well, those are subjects for another blog post entirely.

Comments for this post are on Hacker News

Why Chef?

2011-10-16T02:22:00.000-07:00

Being a system administrator is alternately thrilling and tedious. The thrilling part is setting up an awesome, rock-solid system. The tedious part is doing that 10 or even 100 times more. Further, throughout your career you will configure certain applications again and again. You know the ones, sendmail, apache, samba, etc. Well, there have been many attempts to build a framework, an abstraction, for automating the process of configuring these well-worn tools and others. Today there is one such framework that finally gets it right, Chef. There are many, many reasons to use Chef but the main reason you should use chef is this: You, System Administrator, are too damn smart to spend the rest of your professional life reinventing the wheel.

Let me tell you a little more about how wonderful Chef is, then I will show you briefly some technical highlights, and finally I will clear up some Chef myths.

The Top 5 Reasons to use Chef

Writing reams of documentation sucks. Chef drastically reduces the amount of documentation you have to write.
Bash doesn't scale. Seriously.
Technical Awesomeness
Chef grows with you
You can stop reinventing the wheel

Now let's talk about that awesomeness in more detail.

Write Less Documentation

Less docs? You're kidding me right? Actually, I'm not. If you're like me, you'r sick of annotating the your entire bash history. If you're not like me, you don't even bother. With Chef, your infrastructure literally is code rather than a collection of unique artifacts. Top-level readmes and a few well-placed code comments should be sufficient. Take a look at this recipe for a basic sendmail configuration.

basic sendmail configuration with chef

Loading ....

Bash Doesn't Scale

Bash is a wonderful thing, but like all UNIX tools, it is fundamentally limited by design. Bash doesn't have a code reuse mechanism more powerful than functions. It uses a single global namespace. These and other limitations have made it hard for us sysadmins to reuse and generalize our shell scripts across different distributions, let alone different versions of *nix. Lastly, it is quite difficult to write shell scripts that are idempotent, that is, they have the same effect whether run once or 100 times. This is particularly true when manipulating configuration files. Let's take a look at how you would configure /etc/sudoers with chef.

Loading ....

All the variables in this template are passed in from this recipe. If you think you can replicate this template using sed, please submit it as a comment so we can compare results.

Last but not least, Bash can be just as unreadable and obfuscated as the darkest perl code. With Chef you have a high level, intuitive DSL for describing your configuration. Remember what I said about needing less documentation?

Technical Awesomeness

NOSQL FTW

One of the virtues that many *nix tools share is that they store their configurations in text files rather than binary formats or in a database. Chef stores your system configurations in text and in a database. It accomplishes this by using the document-oriented database, CouchDB. This makes the configuration searchable and higher performance. One of my annoyances with nagios is that I have to restart it every time I change any service or host. There are no annoying restarts with Chef. Each Configuration change you make takes effect instantly. Further you can use your favorite text editor and the command line to make configuration changes. Finally, using a database means that Chef is easily accessed through a GUI. We sysadmins often look down on GUIs as crutches for the Windows crowd. The Chef web UI is excellent for visualizing your own infrastructure, something that is hard to do while staring at plain text.

Knowing is Half the Battle

Chef uses Ohai to collect data about your system. Your recipes can access these attributes and make decisions based on them. For example, you can determine which version of Red Hat you are using simply by looking up the value of node['platform_version']. You don't have to cat | grep | awk to find out which release you are on.

Ohai makes cookbooks more dynamic and able to support different distributions. As we will see later, this is one of the reasons there are so many high-quality cookbooks available.

Search

Search is a feature in Chef Server that allows you to query the configuration information of all other servers and of globally-defined databags. This allows you to do things like configure clusters where a member of cluster needs to know not only about its own configuration but about the configurations of the other members of the cluster.

Example of using search with data bags. For the full recipe go here.

Loading ....

Knife
Knife is one of the truly great command line tools. It is your primary mechanism for interacting with the chef-server. Knife shares many usage patterns with git. If you love git, you'll love knife.

shef

shef works the way you work, in an iterative manner. Most of us system administrators are self-taught and we learn best by doing. Fire up shef and you can on the fly play with attributes and create recipes. Further, you can connect to your server and download the cookbooks.

Chef Grows with You

Chef uses pure Ruby as its configuration language, not a shackled subset of ruby, nor yet another custom configuration language. You only have to learn a small amount of ruby to get started with chef. Once you get beyond the basics of Chef, you can go farther with Ruby. Many of you are grumbling now because ruby sucks to compared to Perl/Python/TCL/<insert-interpeted-language-here>. Well, Ruby may pale in comparison to Python but it is still a powerful, full-featured language.

Just like Perl, there is a lot of dark magic inside Ruby. It can't be used carelessly or it will bite you in the ass. Unlike Python, Ruby does not make it difficult to do the wrong thing.

You can stop reinventing the wheel

Until Chef, we sysadmins did not have a truly modular way to abstract and share our system configurations. Please stop reading and browse http://community.opscode.com/cookbooks. Later on, you should look at the code in https://github.com/opscode/cookbooks. You will discover that some turncoat sysadmins are giving away our trade secrets. Now is the time for you to do the same. In fact, you can rip out a large chunk of your shell scripts and replace them with Chef cookbooks. You will find that many existing recipes meet your requirements and you can easily add new recipes to existing cookbooks for your unmet requirements.

Recently, I replaced a 500 line shell script with 7 cookbooks, 5 reused from community.opscode.com and 2 new ones written from scratch. In the process of replacing the shell script I actually wrote 4 cookbooks that unwittingly duplicated the functionality of existing cookbooks.

The emergence of the cloud-computing, whether on the public cloud such as EC2 or internally with openstack, means that we systems administrators will have to manage at least 10 times more systems. We have to become 10 times more productive. We have to shift from "managing servers" to "managing configurations." Chef is one of key tools to accomplishing this.

One Big Fat Chef Caveat

Chef makes good sysadmins more productive. It does not turn junior sysadmins into experts. In fact, it makes them more dangerous. As I overheard on IRC one day, "Chef is to sysadmins what C++ is to software engineering." This is very true. You can automate system configurations with off-the-shelf cookbooks but you have to understand exactly what they do and test them very rigorously.

Chef Myths

There are a few myths about Chef floating around the intertubes that need to be exploded. The biggest one is that you have to be a professional programmer to use Chef. This is untrue for a couple of reasons, including the assumption that system administrators don't program already. Writing complex shell scripts is programming. Creating Apache rewrite rules is quite close to programming. Further, Chef doesn't require you to be a Rubyist to get started, just as you don't have to be a Bash Hacker to use the command line. Take a look at this tutorial and you will see what I mean.

I hear murmurs now and then that Chef is "insecure" because the recipes are executed on the chef-client rather than on the chef-server. I argue that chef is more secure because of this. If recipes were executed on the chef-server, there would be a risk that a single malignant recipe modified other recipes to turn your entire network into a botnet. This would turn your chef-server into a single point of failure and make it a powerful attack vector.

Finally, some will argue that Chef sucks because every server you configure should be hand-crafted and beautifully unique in its own way. If you believe that, perhaps you should go into the virtual performance art and get out of the business of making the Internet awesome.

Wrapping Up

Chef is a truly powerful tool that every serious system administrator should consider using. It offers immense personal and professional benefits.

Please post comments on Hacker News

Marcus van Dam (m4rcu5) helped edit this post

Note: The title of this post is a reference to Eric S. Raymond's great post "Why Python?" which inspired me to learn Python, a long, long time ago.

Puppet vs. Chef, Fight!

2011-10-09T07:46:00.000-07:00

Update: see Puppet vs. Chef en francais, not a translation but an original article inspired by this one

If you aren't yet using Puppet or Chef for managing your *nix infrastructure, you should seriously consider it. The benefits are enormous. Frankly, I don't think you __can__ manage a modern infrastructure without using Puppet, Chef, or reinventing a configuration management framework with your own custom scripts. However, you can't use both Chef and Puppet together, so which should you choose?

If you're like me, you obsess over tool choices. You agonize for weeks over these decisions. Why? Because you will end up putting a huge personal investment in the chosen tool. There isn't all that much on the relative benefits of Puppet vs. Chef besides this quora thread. Below you will find my completely subjective, non-comprehensive study of the matter. TL;DR, Chef wins but by only a narrow margin.

The Criteria

Community Strength
Leadership
Corporate Adoption
Technical Merits
Hands-on Experience

Community Strength

Community Size

Number of people on IRC on Sunday Oct 9th, 2011 10 AM GMT
#puppet 541 users
#chef 233 users

Mailing Lists
puppet-users 3363 members, 5961 topics
chef: 600 members, # of topics unknown

Regarding IRC, Chef seems much more U.S.-centric. It can be hard to get answers during the European workday. The situation is probably better for Asian users as they can catch the North American community when it is early evening on the West Coast of the United States. That is not true for Puppet, where you can get an answer at any time of day or night. R.I. Pienaar is based in London and seems to be on IRC at all hours. Somehow he manages to kick out awesome code and be continuously available. Further, James Turnbull doesn't seem to sleep either and is incredibly helpful.

Quality of Cookbook Sites

http://community.opscode.com/cookbooks
http://forge.puppetlabs.com/

Github Repositories:
puppet: 1622 repositories
chef: 1299 repositories

While Chef is much younger project than Puppet, it is really impressive how many cookbooks there are and their respective quality, especially when Chef is less than half the age of Puppet. From what I can tell, most people bypass the community sites and go straight to github to find their Chef cookbooks or puppet modules.

Documentation, Official and Informal

Both Puppet and Chef have excellent reference documentation. In my opinion, the Puppet docs help you get up and running much faster. Pro Puppet, by Puppet Labs engineer James Turnbull, is an awesome introduction to Puppet. I found it immensely helpful. There is a 50-page O'Reilly publication on Chef, Test-Driven Infrastructure with Chef, but it cannot match the breadth of Pro Puppet. I have heard that O'Reilly are going to be publishing a feature-length definitive guide, scheduled to be out by the Summer of 2012.

The Chef community definitely tends more towards programmers and rubyists than systems administrators. I think that puppet meets the needs of the majority of its users so well that they never try Chef.

Leadership

We all know that leaders can make or break open-source projects. The BDFL's personality shapes the project more than the actual code. What would linux be without Linus Torvalds' stewardship? I am wary to bet against Puppet founder Luke Kanies. He is both a genius and a visionary. He reminds me a little of another Reed College alumnus. I highly recommend listening to Luke Kanies interview on DevOps Cafe. He shows himself to be very intelligent and very opinionated. My understanding is that his original motivation for creating Puppet was to make life better for the average sysadmin.

The resumes of the Opscode team are just amazing. CTO Chris Brown and co-founder Jesse Robbins were core infrastructure engineers at Amazon, the company that perhaps more than any other understands how to manage infrastructure at scale. Listen to co-founder Jesse Robbin's Interview and you will find him to be very pragmatic about infrastructure and less opinionated than Luke. His motivation was to create an open-source tool that solved the kinds of infrastructure problems he dealt with while working at Amazon.com.

On the funding side, both companies have closed substantial Series B funding rounds. We're talking more that 10 million dollars each. Both of these companies can throw a number of talented software engineers at the devops problem domain for a while to come.

Corporate Adoption

I believe that both Puppet and Opscode corporations will become extremely successful. Jesse Robbins explains this best in Virtual Strategy Magazine.

Our primary competitor is the in-house, home-grown tooling that every systems engineer has had to cobble together to deal with their legacy environments. That’s our principal competitor.

That's right. The biggest competitor to Chef and Puppet isn't each other but that hairball nest of Bash and Perl that holds your data center together.

Still, Puppet has an amazing customer list but Chef is also racking up wins. If Puppet Enterprise 2.0 delivers half of what its press release promises, it will be a huge success. Puppet Enterprise's integration with mcollective is particularly exciting.

I am particularly impressed that a number of companies have built their own internal provisioning tools on top of Chef, notably Rackspace Cloud Builders and Dell's Openstack team.

Technical Merits

Puppet is designed to be approachable by systems administrators. It is not necessarily written to empower rubyists. Writing Puppet manifests, at least initially, feels much like writing nagios or apache configurations. However, it starts feeling much like programming once you get beyond simple resource specification.

Chef is designed to help engineers manage the configuration of their infrastructure at scale. It does not have the same focus on easy-of-use, nor is it as opinionated as Puppet. Now let's look at a few salient features.

Syntax

Here are examples of how to create a user in first Puppet and then chef

Loading ....

They don't look terribly different, but now let's throw some Ruby into the mix. In the following example, I iterate through a list of users stored in a data bag

Loading ....

Now you should have a better sense of the difference between Puppet and Chef. Some may have a valid criticism that Chef lets you do too much by allowing you use regular Ruby. I have to admit that undisciplined use of Ruby is as dangerous if not more dangerous as perl or bash abuse. At the same time, it is quite empowering. The Chef community really needs some kind of chef-lint tool to tell you when you're doing something you shouldn't.Update: Foodcritic now excellently fills that need.

Debugging

I miss noop mode in puppet, though this is on the way for Chef. It is a wonderful way to see how your puppet configuration differs from what is actually on disk without actually changing anything. I have heard some refer to noop mode as "Lie to me mode" though I have yet to have it bite me in the ass. I might feel very differently if I had experienced that personally. I also liked how puppet spit out the text of rendered templates to the console. Maybe you can do this with Chef as well, but I haven't figured out how to yet.Update:Chef now has the equivalent of noop mode with its why-run mode.

Chef's Secret Sauce

Chef's Data Bags are incredibly useful. They are a very nice way to feed lists of users to multiple cookbooks. Think of them as global variables for your chef-server. One the benefits of data bags is that they allow you to separate your corporate information from your cookbooks. Thus, making it easier for Chef users to open-source their cookbooks. Puppet has extlookup but that is quite a bit more limited than data bags. It looks like Puppet maybe getting something similar to data bags with Hiera.

I also have to praise the Search function. It is an extemely intuitive way to to tie together dependencies between nodes, such as between a nagios server and its clients. Puppet has a way to "export" resources but I could never get my head around how that worked.

In my experience, Chef has a more powerful read-eval-print-loop (shef) than Puppet does. I use shef quite frequently and I cannot understate its helfulness. Puppet does not have a similar repl as far as I am aware.

Server-side, Chef is much more complex but also seems to be better thought out. I love the fact that Chef uses a document-oriented database. This combines the benefits of using files to store your configuration (easy to understand) with the benefits of using a database (speed).

Hands-On

I don't believe there are many in the Puppet community that have extensively experimented with Chef before choosing Puppet. There are many in the Chef community that have worked previously with Puppet and found that it did not meet their needs.

I started working with Puppet in June of this year and quickly fell in love. I read Pro Puppet and did all the exercises. Next I set to rewriting my 400 line linux provisioning script in puppet manifests. At first it went well but then things got more complicated. I kept tripping over the DSL's syntax and was surprised that there weren't more puppet modules in puppetforge or github. I felt that I had to learn both the Puppet DSL and Ruby to really to be effective with Puppet. I started thinking that the grass might be greener on the other side.

Two things convinced me to try out Chef. The first was the announcements by Dell and Rackspace that they had built there Openstack provisioners on top of Chef. If the people pioneering devops chose to build their platform on top of Chef, then it definitely deserved my attention. The next was Jesse Robbins' interview on DevopsCafe. He made all the action in the Chef community sound so exciting that I had to go check it out for myself.

It took me three days to get Chef Server up and running on RHEL 5.7. Next, I started writing my own cookbooks. Now I have nearly converted that linux provisioning script to a cookbook. Not counting the time it took me to set up the server, it took me roughly half the time it took with Puppet.

It was easy to get started with Puppet, but things became more complicated with time. With Chef I am having the opposite experience. Again, the single biggest drawback to Chef is that it has a steeper learning curve than Puppet. Further, most of the existing tutorials focus on the deeper concepts in Chef rather than giving novices quick gratification. To this end, I have started the "Dirty Chef" series of tutorials.

Summary

Final Scorecard
Community -- Puppet wins
Leadership -- Draw
Corporate Adoption -- Draw
Technical Merits -- Chef
Hands-On -- Chef

According to this non-scientific criteria, Chef is the better tool. In my humble opinion, Chef is technically superior. Don't buy the argument "Chef requires you to be a programmer, Puppet doesn't." While Puppet is geared towards non-programmers, it becomes regular programming once you do anything moderately complicated. That's when Puppet's DSL becomes more of a liability than an asset and where Chef really shines. Now while chef is the winner of my little bake-off, it is not without faults. Puppet has far better support for RHEL/Centos and it does have a larger, more responsive community.

Dear Puppet community, please know that I write this blog post with great ambivalence and I still love you. Jame Turnbull and R.I. Pienaar, thanks for all your help via IRC, Twitter, and the mailing list. Puppet Labs is developing a toolchain that is much larger than just the puppetmaster. This toolchain may turn out to be superior to what Opscode and its respective partners come up with.

On a final note, please know that both of these tools are good enough for configuration management and are only one part of a larger toolchain.

P.S. I will delete any comments that trash either Puppet or Chef. They are both excellent tools, built by
visionaries.

Most of the discussion related this article seems to be happening on Hacker News.

A Stupid, Simple Chef Tutorial for Dirty, Lazy Fry Cooks (Like Me) Part I

2011-10-07T00:55:00.000-07:00

disclaimer: I am not a chef expert. To really learn chef, you need to go to the wiki

I really like the chef documentation, I really do. It is an excellent reference. It is the devops equivalent of the manual they must use at the Le Cordon Bleu cooking school. But I have to admit that sometimes I am just a dirty, lazy fry cook, especially when playing with a new technology. I want instant gratification or at least in under 30 minutes. What I love about the Puppet documentation, is that it gets you started quickly, before you begin optimizing and modularizing and all that other crap that good devops chefs do. To that end, I have created a chef repository that is exactly what you shouldn't do in production but is hopefully useful for learning.

In this tutorial, I am going to use chef-solo for the purpose of the tutorial. Chef-solo simply means running chef with out connecting to central chef server that provides you with cookbooks and some other functionality. Chef-server is awesome but currently a pain in the butt to set up. I am also going to use RHEL 6.0 but you can use CentOS.

Why Chef?

If you aren't yet using configuration management with Chef, Puppet, or similar tool you're just plain doing it wrong. I'm doing it wrong. I cobble together my infrastructure with a combination of kickstart, bash scripts, fabfiles, and random python. But the diversity of applications I support and the sheer number of virtual machines are increasing rapidly. My agglomeration of virtual glue and duct tape is starting to look more like a house of cards and less like modern infrastructure.

In the immortal words of Puppet founder Luke Kanies:

"SSH in a for loop just doesn't cut it anymore."

To build Luke's statement, our imperative bash/perl/python scripts are lousy for customizing server templates to specific node attributes and they suck at maintaining system state. I have tried both Puppet and Chef and I prefer Chef for a number of reasons. The main ones are that it has a more elegant and easier to use DSL and that chef-server has the awesome search functionality.

Installation

Install my chef repository somewhere on your system. I recommend /opt.

git clone git://github.com/bryanwb/dirty-chef.git

Next install chef. Here is the main install guide

This should work on either CENTOS/RHEL 5 or 6

For RHEL 5

$ rpm -Uvh http://rbel.co/rbel5

For RHEL 6

rpm -Uvh http://rbel.co/rbel6

Finally, on either


$ yum install -y rubygem-chef ruby-shadow

Now edit solo.rb, you have to edit these paths to reflect the actual location of dirty-chef otherwise chef-solo will not work

file:solo.rb


file_cache_path  "/opt/dirty-chef/"

cookbook_path "/opt/dirty-chef/cookbooks"

data_bag_path  "/opt/dirty-chef/databags"

If you are using RHEL 5 see the install link above, it has steps for RHEL 5 but they are more complicated Run the following command

$ ohai | less

This shows you all the info that chef knows about your system. You can do reasoning in your recipes based on these values.

Start Cooking

Here is a stupid hello world

$ cd dirty-chef
$ chef-solo -c solo.rb -j node.json
ohai!

What the heck did we just do? The solo.rb tells chef where the cookbooks are, i.e. where the recipes live, and node.json tells chef-solo which recipe to actually apply.

file:node.json


{

  "run_list": [ "recipe[dirty]" ]

}

This loads the recipe in cookbooks/dirty/default.rb

If you get any errors here, you should make sure that you changed the paths in solo.rb to reflect the location of dirty-chef/

Now let us do something more significant. You can use chef to declare "resources" in this case a package. Let's install subversion. In chef recipes, you spend most of your time declaring resources, in this case I declare that I want subversion installed and chef takes care of the rest.

file:cookbooks/dirty/recipes/example1.rb
package "subversion"

$ chef-solo -c solo.rb -j example1.json

file:example1.json

{
  "run_list": [ "recipe[dirty::example1]" ]
}

This loads the recipe in cookbooks/dirty/example1.rb

Here is an example of installing multiple packages, take a look at cookbooks/dirty/recipes/example2.rb

$ chef-solo -c solo.rb -j example2.json

What is that syntax of the configuration language? Why it is pure ruby. If you like perl, you will like ruby as it is the lovechild of perl and smalltalk. Here is a good quick reference

You still are not impressed? True you could do all these things in a kickstart more easily and without having to go to so much trouble. There are several reasons chef is better than kickstart. You can use chef to maintain the state of your server while you can only use kickstart to initially set up your server. Once you install your server from a kickstart, you cannot rerun it w/out reformatting it. Chef on the other hand is "idempotent" Meaning it has the same effect whether run once or 100 times. Say for some reason subversion is removed from your machine. The above chef recipe will reinstall it w/out needing your intervention. You can seamlessly handle different platform versions. You can write one recipe that handles both RHEL 5 and RHEL 6 by doing reasoning on node attributes. Maybe you are like me and love nothing better than a good REPL (Read-Eval Print Loop), i.e. console. Chef comes with an awesome REPL, shef. Let us use it to play node attributes and even load recipes interactively

[root@woof-chef recipes]# shef

loading configuration: none (standalone shef session)

Session type: standalone
Ohai2u root@woof-chef!
chef > if node["platform_version"] =~ /6./
chef ?> puts "hi rhel 6!"
chef ?> end
hi rhel 6!
[ type ctl-D to exit ]

In shef, you have access to all the node attributes and cookbooks on your system. Now let us put the same logic in a recipe.

$ chef-solo -c solo.rb -j example3.json

Now we still haven't actually done anything useful. We will get to that in Part II

What is devopsanywhere?

2011-10-03T01:47:00.000-07:00

Why DevopsAnywhere? Because our profession is undergoing some profound changes. On the positive side, the best of us are in increasing demand. On the other hand, it will take increasing sophistication on our parts to keep up w/ the blistering pace of change in our industry. I believe that to really keep up-to-date, you have be an autodidact, that is someone who is primarily self-taught.

In my experience, autodidacts are randomly distributed around the planet. Linux obsessing, python hacking, ruby scripting geeks are even more randomly distributed. Should companies really want to recruit us, they will have to hire us where we are, be that Kathmandu, Sicily, or Utah. Gone are the days where system administrators lived at the datacenter, as I explain in greater detail later.

As system administrators we were tied to the datacenter. Even if we worked remotely, we had to be within easy driving distance. We were software/hardware hybrids that understood all the various permutations of RAID 0/1/5/6/10. We looked on with envy at the alpha programmers who could do their jobs while lying on the beach in Thailand or while huddled in a cozy cabin in the French Alps.

Our profession is transforming from System Administrators to Devops Engineers. Devops is a bit tricky as it is as vague a term as agile. Still, it points to a new class of engineers that are focused on operations and work seamlessly with software developers. We are self-taught engineers who have the luxury to work remotely. This blog will be entirely focused on the concerns and interests of devops engineers working remotely. Further, I won't talk about expensive proprietary software or hardware. Actually, I will rarely talk about hardware at all.

Some, like John Willis and Damon Edwards of the awesome Devops Cafe, will argue that process is more important tooling. I agree with this 100%. However, as system administrators we often aren't in the position to reorganize our team's management structure. We are in a position to introduce new tools and lobby for better ways of working.

Here is a comparison of the requirements for being a SysAdmin and that for Devops Engineer

A new Set of Skills

System Administration today requires:

deep knowledge of *nix, including linux, solaris, and even some proprietary unices like HP-UX
you really __know__ bash
some basic version control w/ cvs/svn
simple scripting w/ bash/perl/python
understanding of virtualization
You can troubleshoot hardware, firmware issues, even flip a dip switch or change SCSI IDs if necessary

Devops requires:

deep knowledge of Linux -- generally not Solaris, AIX or HP-UX
you __know__ bash
you are very proficient in svn and/or git
You are a practicioner of puppet or chef
system scripting with ruby
System packaging -- ruby gems, rpms, debian packages
You can setup package repositiories on a whim
you understand unit testing and behavior-driven testing, hopefully you practice it w/ Cucumber or something like it
AWS, Rackspace, cloud provisioning -- but of course

Hardware is now (mostly) irrelevant

The only NIC cards a devops engineer cares about are the Intel e1000 or whatever paravirtualized driver your hypervisor of choice offers. We know that working on hardware issues only lowers our hourly billing rate. Now, more than ever, the power is in software and hardware-oriented solutions are to be avoided.

I must confess that I have an ulterior motive for starting this blog. I am a linux system administrator who is following his wife's career around the world. Previously we were in Nepal, now we are in Italy, next up who knows! But I want to work with top-notch teams wherever I end up and not be tied to whatever opportunities are locally available. If you are a devops guy and you feel the same, let me know!

How Ruby is beating Python in the battle for the Soul of System Administration

2011-09-16T11:12:00.000-07:00

Roughly three years ago I purchased the book "Python for Unix and Linux System Administration" and it convinced me that Python would be the default scripting language for linux system administrators for the foreseeable future. I was working on the One Laptop Per Child project at the time, where Python was the lingua franca. It seemed that Red Hat was using Python for almost every new project. Further, Unladen Swallow was making rapid progress. I couldn't have been more wrong.

Through a combination of historical accident and some seemingly minor feature differences, ruby is inexorably becoming the dominant scripting language for linux system administration.

Before I go any farther you die-hard sysadmins are probably screaming "We already have a scripting language, Bash!" While we love bash and commandlinefu, Bash becomes a liability not an asset once your script exceeds 100 lines and a total nightmare if you need to parse or output HTML, CSV, XML, JSON, etc.

Historical Accidents

In 2004, Luke Kanies set about building Puppet, a configuration management system to improve upon the deficiencies he saw in CFEngine. From the excellent On Ruby blog

I tried to implement my idea in perl, but I just couldn’t get the class relationships to work (the attributes and resource types each needed to be classes, according to the design in my head). This was back when Python was the shiznit, so I naturally tried it, but Python just makes my eyes bleed (and no, it wasn’t the whitespace, it was things like the fact that ‘print’ was a statement instead of a function, and ‘len’ was a function instead of a method).

I had a friend who had heard Ruby was cool but hadn’t actually tried it himself. Since I was just messing around at the time, I figured I’d give it a go. Four hours in, never having seen a line of Ruby previously, I had a functional prototype.

A couple years, a number of Puppet developers felt that Puppet didn't meet their needs. They started their own configuration management tool, Chef, heavily inspired by Puppet. The biggest outward difference between Puppet and Chef is that Chef uses pure ruby as its "recipes" whereas Puppet uses its own configuration language based on ruby.

Both Puppet and Chef are seeing rapid adoption by big companies according to BusinessWeek. If you aren't yet using Puppet or Chef, you should be planning on it in the near future. Whether you choose Chef or Puppet, you are effectively scripting your infrastructure with ruby. After spending 25% of your time working with Puppet, you will be much more likely to reach for ruby for your next scripting task.

Popular Projects

Off-hand here is a non-definitive list of sysadmin/devops related projects in ruby

puppet
chef
vagrant
mcollective
cucumber (behavior-driven testing)
capistrano
rake (ruby Make)
aeolus project/openshift
cloud foundry
graylog2
logstash
travis-ci

and in Python:

What's important here isn't the length of the respective lists but the importance of the individual projects. Chef, Puppet, and Vagrant are the new hammers and screwdrivers of system administration. If you are a sysadmin and aren't yet using these tools, don't worry, you will be sooner or later.

Openstack deserves special attention as it is a very exciting but early stage project. It could play a big role in your future as a system administrator.

Please notify me of significant devops projects that are missing from this list.

What a ~~Girl~~ Sysadmin Wants

Here are the features in a scripting language that a sysadmin wants

A DSL for the problem domain
High productivity, i.e. concise and expressive syntax
Easy to interaction with shell commands
Regular Expressions
powerful one-liners

Notice that performance is not on this list of requirements. Ruby is significantly slower than Python and this was particularly true for the 1.8.* series of Matz Ruby Interpreter (MRI). However, performance just isn't critical for 90% of our work as sysadmins. We care about productivity more than we care about performance. Readability is nice but it is a distant second to productivity.

Python doesn't have regular expressions embedded in the language, probably to improve readability. It also limits the number of top-level built-in global variables. From a language design perspective, this is much cleaner. From the perspective of your average street-trained, stress-laden, vi-addicted sysadmin, it is annoying.

These top-level variables also make one-liners more concise. Here are just a few

$_ last line read from STDIN

$? last exit code from a child process

$stdin reference to stdin

$stderr reference to stdout

$stdout reference to stdout

Here is an irb session to show you these values in action

hitman@hiroko:~/pr$ irb

irb(main):001:0> %x[ ls -a ]

=> ".\n..\nfoo1\nfoo2\nfoo3\n"

irb(main):002:0> puts $?

0

=> nil

irb(main):003:0> %x[ ls xys ]

ls: cannot access xys: No such file or directory

=> ""

irb(main):004:0> puts $?

512

=> nil

"IRB?" you scoff "You will have to tear IPython from my cold dead hands."

I love IPython. I have used IPython for 4+ years and IRB doesn't hold a candle to it. That said, ruby's shortcuts are really useful, enough to compensate for IRB's shortcomings. There is something called wirble that I haven't tried yet that may make irb a lot more productive.

Here some python code to detect whether a machine is a VMware VM.

# edit: fixed python code tks to kstrauser

  import os
    if 'vmware' in os.popen('dmidecode').upper():
        print 'this is a vmware vm'
    else:
        print 'this is not a vmware vm'

Here the same code in ruby

`dmidecode`
if $_ ~= /vmware/i
puts 'this is a vmware vm'
else
puts 'this is not a vmware vm'

Frankly, this kind of code makes my eyes bleed. The python example is far more readable and maintainable. That said, a lot of sysadmins would appreciate the terseness of the ruby example.

Let's look at writing one-liners in both languages.

This code prints out the first 10 lines in a file using Python

python -c "import sys; sys.stdout.write(''.join(sys.stdin.readlines()[:10]))" < /path/to/your/file

here is the same in Ruby


ruby -ne 'puts $_ if $. <= 10 ' < /path/to/your/file

Compare for yourself between these collections of oneliners in python and ruby. If ruby reminds of perl, your eyes do not deceive you. In many ways it is the love child of perl and smalltalk.

DSLs FTW

Some time ago, I tried to watch all the Structure and Interpretation of Computing lectures. This endeavor failed miserably but I recall Harold Abelson saying that every large program should have its own internal DSL suited to the problem space. This is debatable in regards to the larger world of programming but I think it is very apt for system administration. We spend our entire careers with n different DSLs. Each different configuration file format is its own DSL.

If you compare rake (Ruby Make) to rails code, they look quite different, almost like different languages. If you compare fabric code to a django class, they look quite similar. This is both a strength and a liability. I am not a language lawyer but it seems much easier to create DSL's (Domain Specific Languages). Ruby certainly spawns DSLs with much greater frequency than python. No single pythonic build tool dominates the problem space like rake does in the ruby community. Most python projects seems to use setup.py for administrative tasks even though that is not its explicit purpose.

Both puppet and chef are DSLs for system administration. Capistrano is a DSL for application deployment. Ruby's operator overriding and blocks lend themselves to the easy creation of DSLs.

In Summary

Ruby's greatest strength is its amazing flexibility. There is a lot of "magic" in ruby and sometimes it is dark magic. Python intentionally has minimal magic. It's greatest strengths are the best practices it enforces across its community. These practices make Python very readable across different projects; they ensure high quality documentation; they make the standard library kick ass. But the fact is that we sysadmins need flexibility more than we need raw power or consistency. Still, these are not the real reasons that ruby is overtaking python.

Ruby is fast becoming the default scripting language for sysadmins because back in 2004, Luke Kanies looked at Python and felt ill (I had the opposite reaction). As a sysadmin you either currently are or soon will be using Puppet or Chef on a daily basis, spending a heck of a lot of time essentially coding in ruby. Personally, I much prefer writing python but am shifting to writing my scripts in ruby because I spend so much time with Chef.