devopsanywhere

FOSDEM Summary

2012-02-06T22:50:00.000-08:00

I thoroughly enjoyed FOSDEM. The weather sucked, the beer was great, and the people were wonderful. Thanks to the FOSDEM organizers for putting on such a great conference. What follows is a very rough summary of my experience.

I spent most of my time in the Virtualization room. There were a number of interesting talks there. I didn't spend much time in the Configuration Management Devroom, that is not because the content there was lacking but because I felt more knowledegeable about the subjects there than in the Network/IO,
virtualization, and hypervisor talks where I was pretty much ignorant. From what I could tell, the
organizers did a fine job organizing the Systems and Configuration Management Devroom.

One of the best parts of going to a conference like this is meeting in person that you have collaborated with via e-mail and IRC. I met up w/ Cheffolk Andrea Campi, Juan Jesus Croissier (juanje), Nicholas
Szalay (nico), Stefan Jourdan, and Greg Karekinian (sp?). I have interacted with these guys quite extensively over the last four months, so it was awesome to finally meet them. Juanje has a very different use case for Chef than most users. He manages thousands of desktop machines running GNOME. He must take care not to overwrite the users custom settings, a challenge I have never had to deal with. For this reason he created an LWRP to manage portions of a file. Nico has written a nagios check that will report failing chef runs. I need to start using that.

The best moment of FOSDEM for me was a very personal one. I ran into Sulochan Acharya, a friend of mine from Nepal. We worked together to build the One Laptop Per Child project in Nepal. I didn't even know he was in Europe nor still working in IT. Now he works at Rackspace on Openstack-related stuff. It is a small, small open-source world.

Best talks I attended were Anil Madhavapeddy's "Wild West of UNIX I/O" and Charles Nutter's JRuby talk. Anil talked about the unpredictable behavior that we see in UNIX I/O in virtualized environments. In a
multicore world, hypervisors don't know enough about the underlying architecture to make intelligent decisions about I/O between CPU cores. This leads to very unpredictable I/O results. According to Anil, this requires a better I/O abstraction that is aware of the underlying microarchitecture--CPU layout, interconnects, NUMA--and can make intelligent choices for the developer. Anil calls his next-generation I/O framework FABLE, which is still in very early development. Charles Nutter is a great speaker though his
talk was a bit over my head. He focused on the "impossible" challenges of implementing Ruby on the JVM and how his team overcame them.

I missed the ZIO and CERN talks. Both talked about next-generation I/O frameworks. Also sadly missed Jonathan Ellis's talk on "Overcoming JVM limitations in Cassandra". I spent no time w/ the Mozilla track, which is a shame as I love how they manage their community and the kinds of projects they work on.

I was struck by the large number of projects that featured web frontends written in Ruby on Rails. Besides Horizon, the Openstack web UI, it seemed that virtually every other web UI I saw was Rails. Ironically, there were no talks specifically about Ruby nor Rails at FOSDEM.

Red Hat is doing a lot of activity in the DevOps space. oVirt a web UI for managing VMs. Matahari a system administration API which looks interesting but seems like it is still in early development. I had the pleasure of talking w/ Ohad Levy, a very sharp fellow who develops Foreman, a front-end to Puppet and is employed by Red Hat.

I found out about the meta-project Katello from Red Hat that contains many components to automate system administration. To my limited knowledge, it includes Matahari for low-level administration, Puppet for configuration management, Foreman as a nice GUI to manage your configurations and server provisioning, oVirt to manage your KVM hosts a la VMware's vSphere, and pacemaker-cloud for high-availability across different cloud providers. I believe the combined stack is at alpha level. I would
love to know if Chef could be used in place of Puppet.

Met the very smart Marek Goldmann who is doing a lot of work packaging JBOSS for Fedora in addition to his boxgrinder project. It seems like there is a fair amount amount of overlap between what he and are trying to do in this regard. I need to spend some time hanging out on #fedora-java to see how we can avoid duplicating efforts.

My greatest regret is that I myself didn't give a presentation. I didn't think I had anything to say that others would be interested, but that view was wholly wrong. There was a lot of interest in Chef but unfortunately not much content.

I met a couple of people who had listened to my podcast, Food Fight, but they told me it was "unlistenable" due to the recording quality. Guess I will have to improve the audio quality :(

I was very lucky to have dinner with a group that included the DevOps gurus Kris Buytaert and Patrick DeBois. After following both of them on Twitter for a number of months it was really interesting to see what they are orking on. These guys really are the "Belgian Braintrust" of DevOps not to mention that Patrick actually invented the term. If I recall correctly, both Patrick and Kris are spending a lot of time
lately working on monitoring and logging. Not only are Kris and Patrick DevOps heroes but they are also genuinely nice guys.

Beer

Belgians make the best beer in the world and they enjoy it at all hours and in all places. Quite a number of people enjoys fine brews during presentations and meetups. The Opensuse stand actually gave away opensuse beer. Note to self, avoid the 10 % alchohol beer when trying to stay alert. Stick to the delicious 4% blonde beers.

Final Thoughts

Openstack is the real deal, not just a cool idea. It will really be ready for production use this year. I had my doubts about an open-source project led by an industry consortium but what I saw made me put them
aside. Particularly impressive was Ryan Lane's explanation of how Wikipedia is using Openstack for its infrastructure.

There is awesome stuff happening with KVM, both in terms of performance and tooling.

There was a lot of interest in Chef but I didn't meet that many people who had actually used it. This must be remedied. I really should have given a talk and put more effort into planning a Chef meetup.

I will definitely be back next year and hopefully as a presenter.

Announcing the Food Fight Podcast

2012-01-15T22:23:00.000-08:00

I just published the inaugural episode of Food Fight, the Chef community podcast that I host together with Matt Ray. The podcast will cover not only the developments within the Chef community but also larger world of DevOps. Enjoy!

Chef in 2012

2012-01-04T14:08:00.000-08:00

image by Simon Griffee

Note: Petey King has written a great response to this post. Definitely worth a look.

I discovered Chef in the last last quarter of 2011 and turned my world upside down, work-wise at least, in a really awesome way. Over the holiday season I had some time to reflect about where Chef as a piece of software and as a community of people might go in 2012.

Packaging Cookbooks

Cookbooks are starting to more and more look like packages, with complex dependencies between them. I am not aware of any cookbooks that only work with Ruby 1.9.* but I expect we will see a number of them in 2012. I personally believe that we need to use an existing package management system rather than creating yet another one. RPM, DEB, dmg are all out of the question because they are operating system specific. Ruby Gems seems to be the best choice though I am not very knowledgeable about ruby gems. If you think this a terrible idea, please do let me know what problems ruby gems pose. I envision the scenario where your cookbook_path in knife.rb

As you can see from my example, you still would be use cookbooks as they currently are, but also allow chef to look for additional cookbooks in your GEM_PATH. Cookbook gems would have to be prefixed with something like "chef-ckbk-" to indicate to knife which gems are cookbooks.

Using ruby gems does present a couple challenges, particularly with rvm, which completely compartmentalizes gems from one ruby version to another. This means that changing your ruby interpreter requires you install anew a large portion of your cookbooks. Using gems may also require a lot of people to change the structure of their cookbooks. Perhaps, moving moving all the directories to under gem_name/lib/ subdirectory.

Testing 1, 2, 3

There still isn't a broad consensus on how to automate the testing of chef cookbooks. There was some nice progress during 2011 with foodcritic (lint for cookbooks), chef-minitest (unit-testing?), and cucumber-chef but those tools are all still early days. Testing is a particular pain point as testing a minor change to a cookbook may require testing against 5 different linux distributions. I really hope we see more progress on this front though I can't point to any particular initiatives. I personally hope to spend some quality time with minitest in the next couple weeks.

Other Miscellaneous Technical Needs

Selinux support is currently very limited. Some people feel that selinux shouldn't be used in the first place, however, many sysadmins don't have the luxury to make this choice themselves but are required by an external corporate or legal body to use selinux. As Andrea Campi has pointed out on the Chef mailing list, there are a number of consulting firms, such as ZephirWorks, that have the capacity to implement better selinux support if someone steps forward with funding.

Image by Simon Griffee

The java application stack also needs a lot of love. The current set of tomcat, jetty, maven cookbooks only support minimal configurations. I am reorganizing these cookbooks around the java_ark LWRP. In a nutshell, the java_ark LWRP extracts a tarball, adds any included executables in the path, and runs update-alternatives. Along with the jboss cookbook I have already started, I expect to put a lot of work into the java stack. If this is something you are interested in, suggestions, code criticism, and patches are most welcome.

While chef-client and chef-server generally perform at acceptable levels, there are some pain points. For chef-client, memory usage can sometimes be an issue due to the node object ballooning in size during chef-client runs. If I recall correctly, Opscode is working hard on reducing memory usage so we should see some progress this year. On the server side, we can expect a lot of performance improvements as chef-server moves from ruby + couchdb to erlang + mysql. This may happen sooner than one might normally expect as Opscode has already converted part of Hosted Chef to Erlang and MySQL, it just hasn't open-sourced that code yet.

Growing Community

2012 should be the year that the myths "Chef is for programmers" and "Chef is hard" are demolished. These myths sadly linger and inhibit adoption in my opinion. Perhaps part of the reason that some sysadmins think Chef is hard is that it is pure Ruby. Now Ruby is not a difficult language but it is unfamiliar to most sysadmins. Advocating Chef (and Puppet to a lesser extent) also means advocating ruby amongst sysadmins. How to do that? Here are a couple suggestions.

Get ruby 1.9 into the default server installs for RHEL, Ubuntu, ArchLinux, etc. This could be quite a challenge as according to this blog post, there is almost nobody in the Ubuntu world doing any Ruby work. With the exception of the awesome Sergio Rubio, the same could be said for RHEL/CentOS/SL
More blog articles on using Ruby to solve system administration problems and that sysadmins should consider using Ruby as their default scripting language

All that talk about Ruby advocacy aside, Sysadmins will need to know less Ruby in 2012 in order to write Chef recipes. This is because we will see more and more LWRPs pop up. I love LWRPs. I think they are the best thing since dynamically loadable kernel modules.

Chef is growing by leaps and bounds because it solves an important problem in configuration management. The greatest constraint to that growth is the supply of sysadmins willing to make the investment to learn it. At this point, the majority of the community are self-taught linux geeks who found Chef on their own. At the next stage of growth, Chef needs to be attractive to the group I call "the Professionals". Professionals are talented engineers who make system administration their job but not their life. They don't read blogs during their lunch hour and they don't hang out on IRC in the evenings. The Professionals will typically learn Chef because their bosses tell them to. We need to make that learning process as smooth as possible.

The Chef wiki is an incredible reference but we need a guided tour to accompany it. We really need the Chef equivalent to Pro Git and the Git Community Book site. These resources hold your hand and walk you through all the important parts, whispering comforting words when you get frustrated. I know there is work being done a Definitive Guide to Chef from a major publisher, but I don't know its status.

Lastly on the subject of community, we need some mechanisms to summarize community activity.
The Chef project has a high velocity and I find it increasingly hard to keep up with. The Opscode monthly newsletter is a good step in this direction. Additionally, I am starting a Chef podcast together with Opscode Technical Evangelist Matt Ray, focused on summarizing community activity and exploring important topics.

Dangers in 2012

I sometimes worry that Chef could become a victim of its own success. As I mentioned earlier, the greatest constraint to the growth of Chef is the supply of sysadmins willing and able to learn it. If enough well-meaning software architects and aggressive CIOs force Chef on resistant ops teams, we will start hearing about implementations gone horribly wrong. Top-down adoption can only work if the ground below is fertile. That's my two cents.

In Summary

Chef will be a success when the default installation instructions for application X point to a Chef cookbook. I think we will make solid progress towards this goal in 2012 but to really get there, a lot of things have to happen first. I couldn't be more excited. Thanks to everyone in the Chef community for giving sysadmins like me such a great toy to play with. Happy Hacking!

PS: Let me know if you like the Chef icons. My good buddy Simon Griffee created them after a brainstorming session fueled by Sagrantino Di Montefalco.

Emacs for Sysadmins

2011-11-20T05:14:00.001-08:00

I am not going to tell you that emacs is superior to vi, because it is not. For me, vi is far better for basic text editing. However, emacs has such powerful modes that I find myself using it more often than vi these days. Emacs has historically been favored by programmers and vi by sysadmins. I have written in the past that sysadmins are becoming less like ninjas and more like pirates because of this crazy thing called devops.

As a professional system administrator you can't choose between vi and emacs; you have to learn vi. There are just too many times you will be working on an arbitrary *nix that only has vi installed. You can choose to learn emacs in addition to vi. Vi lets you jump in and around files quickly. Nothing can match the speed of hjkl, g, G, etc. for movement or v + y for copying text. However, we sysadmins are moving from managing individual servers to managing configurations. In the last several couple months I have spent at least 50% of my work time entirely within my chef-repo/ directory working on data bags, roles, and cookbooks. Inspired by the devotion of a number of Chef community members to emacs--notably Matt Ray, Stephen Nelson-Smith, Josh Timberman, among others--I decided to give emacs a try. On the whole, it has been a very positive experience. That is why I recommend sysadmins try emacs.

Ninja	Pirate

Bill Joy Vi Creator Hand placement conceals poison dart	Richard Stallman Emacs Creator Note beard

from philosecurity blog ninjas or pirates, emacs or vi

Why Emacs?

Emacs has some truly awesome modes. The default ruby, python, bash, xml, html seem to do more than their vim counterparts. I think this is large part because emacs' extension language (elisp) is more powerful than vimscript. I made a stab at vimscript some years ago but came away quite frustrated. While I am not a lisp convert, it is a full-featured language that doesn't hold you back. Working with multiple buffers (10+) is extremely easy. Org-mode though, is what got me hooked.

Why Not Emacs?

Using vi is like being in a relationship, you are involved. Using Emacs is like a marriage, you are committed, in large part because there is so much to learn. This is one of the reasons I don't recommend it for everyone.

Further, vi is built for touch typists. You rarely, if ever, have to take your fingers off the home row keys. You can keep your fingers on the home row with emacs but that often means completing a number of key chords entirely with your left hand. Take for example opening a file, writing its contents to another file, and then closing emacs.

C-x C-f + C-x C-w + C-x C-k

If you keep your fingers on the home row, you will only use your left-hand for the above commands, except for the final "k". You could start to feel some discomfort in your left hand if you did that key sequence several times in a row. For this reason a number of emacs addicts will hit the Control key with their right hand, the "x" with their left, Control again with the right, "f" with the left, and so on. This leads to a more "hunt and peck" typing style but saves your hands.

First, Learn How to be a Pirate

Before you even start playing with emacs you should make sure to swap your Caps Lock and left Control key. Emacs commands use the Control key heavily and leaving it in the default position will soon leave you angry, crippled, and ready to kill Richard Stallman.

Next, I recommend you organize your dotfiles on github or another source code repository. Being a pirate means you have booty and treasure to carry around. You need a ship to carry around all that treasure and for me that ship is github. The core of my dotfiles repo is the Rakefile, copied from Ryan Bates' master, that makes it dead simple to install your dotfiles on a new machine. I don't install my dotfiles on every machine I work on, only the 2-3 I use most frequently.

create your own github/bitbucket/private github repo called dotfiles
copy your existing dotfiles -- .bashrc, .vimrc, .emacs, etc -- to ~/dotfiles but without the leading "." !
download Ryan Bates Rakefile and put it in ~/dotfiles
to test, rename ~/.bashrc to to ~/.bashrc_bak then run $ cd ~/dotfiles/ && rake install. Your .bashrc should be back
Push to your repository

Whenever you make a change to your .emacs or .bashrc on your ubuntu laptop at home, remember to push your changes and then run "rake install" on your work desktop.

Learning Emacs

OK, finally emacs. I recommend you buy Peepcode's "Meet Emacs" tutorial. It is only $12 and it is the best and most convincing emacs tutorial. It is well worth the money. If this tutorial doesn't convince you that emacs is worth your time stop, turn around, and go back to vi. One of the first things covered in "Meet Emacs" is how to install emacs 24 and the emacs-starter-kit. The emacs starter kit will set of bunch of defaults that help you avoid many of emacs' warts and show off some awesomeness such as ido-mode.

Org-Mode, Emacs Killer Feature

Ok, I have to admit that I have been working with emacs for longer than a few months. I have been using emacs' Org-Mode to manage my life for the last several years, ever since I happened upon Abhijeet Chavan's great article in Linux Journal. Soon after that, I came across Scott Jaderholm's excellent screencast for Org-mode. At the time I read the article, I was the Chief Technology Officer for a non-profit technology startup in Kathmandu, Nepal. I was completely overwhelmed by the sheer volume of different tasks I had to accomplish and all the attendant details, especially the details. I no longer have all that stress now that I am a regular sysadmin, but I do have a large number of very detailed tasks. Org-mode is in many ways my memory. It is also how I manage my attention span.

Many people use Org-mode to implement the Getting Things Done methodology, pioneered by David Allen. I have bastardized it into my own system.

Today is a list of general things that I want to get done today. This week, a list of tasks to complete this week. And of course, right now is a list of specific next actions, that I will complete starting at 1.

When you are finished with a task, don't delete it. Archive it so you have a record of when you are done. Org-mode handily records the timestamp of when you completed the item.

C-c $ # archives a subtree to your archive file,

If I finish reading the O'Reilly Emacs Manual and then archive it, here is what the entry looks like.

I love that I can move items in a numbered list almost effortlessly. M-<up> moves an item up and adjusts the numbering and M-<down> moves an item down in the list.

I typically keep the following org-mode files and their associated archive files.

mygtd.org -- my todo list, for today, this week, this month, and right now!
meetings.org -- all meeting notes and preparation
notes.org -- this is where I keep notes while I work on a problem

The notes.org file deserves special attention. This is my running work log. As I diagnose a problem, I write up my possible hypotheses, then tests for these hypotheses. Finally, I record the results of testing those hypotheses, pass or fail.

The true beauty of using emacs to manage your todo list is that you can grep through it to find a specific piece of information. This has saved my ass more times than I can count.

Org-mode protip: put your *.org files in a Dropbox folder so you have access to them on your various machines. It may be tempting to put them on github but you should realize that those org files will quickly fill up with personal and corporate information.

Working with Buffers

I never found it that easy to work with multiple buffers or tabs in vim without doing extensive key rebinding. Buffers are something that emacs really gets right, especially when you have ido-mode turned on. I have found working with multiple buffers really effective when working with several chef cookbooks at once.

Let's say I am working in the sudo/recipes/default.rb recipe but want to jump back to the previous recipe I was working on, in this case iptables/recipes/default.rb. I type C-x b and ido-mode shows me a list of open buffers. To get the last buffer I was in before this, simply hit [RET] or I can start typing the name of the file and ido-mode will dynamically filter that list.

C-x b [RET] to get to the last visited buffer

Dired-Mode

I have always heard that Midnight Commander is an awesome but I have never taken the time to learn it. I believe that Dired-Mode offers many of the features of Midnight Commander but lets you leverage the emacs skills you have picked up.

Here are commands that I love

Key Binding	Command
v	View a file or directory in read-only mode
d	mark a file or directory for deletion
C	Copy a file
x	execute changes marked, i.e. delete files marked for deletion
q	close an open dired buffer
[RET]	when u hit [RET] on a file, emacs opens it for editing
R	Rename or move a file or directory
M-x find-grep-dired	Grep a directory for a regex and then view the matching files in a Dired Buffer
Q	Find and replace a regex across multiple files

While Dired-Mode is very useful for managing files I find it most useful for browsing code. You can hit "v" to view as read-only then q to close that buffer when you are done. Viewing files in emacs means you get all that juicy syntax highlighting. This helps me make sense of those butt-ugly XML configuration files I encounter on jboss and tomcat installations.

Miscellaneous Stuff I find Useful in Emacs

Shell-Mode - This is very handy for moving text back and forth from a shell and code file. M-x shell-command-region or M-! lets you execute a shell command on a region of selected text
Inf-Ruby-mode - lets you select a bunch of ruby code and immediately execute it in an irb session. I would love to see this work with shef
Modes for Languages you rarely touch - I rarely code in java but occasionally I have to read java code. java-mode helps me parse java without having to deal with Eclipse. Have to change a config file written in Erlang? There's a mode for that.
Properly indenting text - select a region then C-M-\
Completion - M-/ will guess what you are trying to type based on open buffers
Commenting out multiple lines of code -- select a region, M-;

Emacs Shortcuts are Everywhere

In case you haven't already noticed, virtually every unix shell and programming REPL (irb, python, perl) supports Emacs keyboard shortcuts. This is because all those shells are built on top of libreadline, which implements a subset of emacs shortcuts. Even REPL's for newer JVM languages like scala, groovy, and clojure use a clone of libreadline, jline, that implements those same shortcuts.

Emacs Stuff I Haven't Learned Yet but Want to

Tramp-Mode -- editing files over SSH, this would be pretty sweet for configuring a bunch of Virtual Box VM's running on my desktop
Calendar Integration with gmail - just haven't gotten around to this yet
ERC - I debate whether to move from Xchat to irrsi or ERC (Emacs irc client). jury is still out
gist - This is a mode that allows you to select a region and create a github gist from it

Wrapping Up

Emacs, despite conventional wisdom, can be very useful for systems administrators and not solely the domain of Lisp Hackers and not Exclusively for Middle-Aged Computer Scientists. While it may be One True Program for some people, it is not for everyone. The highlights of Emacs for me are:

Org-mode -- time management for busy sysadmins
Dired Mode
Buffer Management

If you have some tricks that you find especially useful for sysadmin work, please share them! Here are some additional resources you may want to look at:

Why Emacs? Borizhdar Batisov's excellent, more developer-focussed ode to emacs
Hack Emacs Screencasts
Mastering Emacs blog
Emacswiki
#emacs on freenode.net

Chef for Developers

2011-11-20T00:26:00.001-08:00

Last Friday I gave a presentation to a group of developers on how they can get started writing Chef cookbooks. I put about 5 hours into writing this presentation. To my chagrin, Warwick Poole released a superior presentation that same afternoon that does a better job of explaining the basic concepts of Chef. I strongly recommend checking out Warwick's "LearnChef" if you haven't already. It is the single best introduction to Chef that I have seen so far. In more good news, Warwick informs me that he intends to create a 5 part series building on "LearnChef." I very much look forward to it.

My Chef presentation doesn't do as good a job explaining the core concepts of Chef but it does walk developers through some scenarios on how they can work with chef on a project. You can check it out below and even reuse my slide material by downloading the .ppt file here. My slides borrow heavily from Joshua Timberman's excellent Chef 101 presentation. The slides are under the Creative Commons 3.0 Unported CC-BY-SA. You can find the full list of publicly available Chef presentations on the Chef wiki.

Chef, Devops, and You

View more presentations from Bryan Berry.

A Month with Chef

2011-10-30T06:14:00.000-07:00

It is hard to believe that I started using Chef only one month ago. The experience has hugely changed how I view the practice of system administration and started to unseat the deepest of ingrained habits. It has been mostly a smooth ride but there have been some significant bumps along the way. Here are my reflections on learning Chef, the community around it, putting it to work, and my hopes for the future of the project.

Learning Curve

There are roughly three technology components to learning chef, each with their own learning curve 1) the Chef software itself 2) git and 3) ruby. I found understanding Chef to be quite easy, in large part because of prior experience with Puppet. If you have never used a configuration management tool before, this can be a stumbling block. However, the biggest stumbling block of the three is probably git, not ruby and not the Chef software itself. Again, I had previous experience with git but even I ran into occasional git-induced hurdles.

When you treat your "infrastructure as code," you have to spend a fair amount of time managing that code. Git is the tool you will use to manage your chef repository, your cookbooks, pull updates the mainline opscode/cookbooks and push your patches to others. If you are a n00b to both Chef and git, do yourself a favor and reach for a good git tutorial before you get too deep into Chef.

The fact that Chef recipes are written in Ruby has turned out to be far less of an impediment than I thought. You can learn enough Ruby to write a recipe in less than twenty minutes. Further, with copy-and-paste plus playing with irb you can guess your way to getting a lot accomplished--though this isn't a good strategy for the long-term. For my own education, I read the pickaxe book and Flanagan's The Ruby Programming Language. Both were good but I prefer the Ruby Programming Language. I have heard from others that Ruby Koans is an excellent way to learn Ruby though I haven't tried them myself.

My first impressions of Ruby the language are shaped in part by my experiences working in the Python world. Python is clean and neat and you can see the hand of Guido guiding the language and standard library at every turn. Conversely, Ruby is creative, chaotic, and messy. It has bitten me in the ass several times that Ruby has hundreds of top-level methods and operators. To say that the namespaces are "cluttered" is an understatement. You have to be quite careful when creating new variables and method names. For instance, In one recipe I created the variables "alias" and "rand", which are coincidentally top-level Ruby methods. Overwriting them caused some nasty side effects that took me a while to track down. If Ruby were a human being it would have a beautiful body, a terrible case of acne, and noticeable digestive problems. Worse yet, its clothes wouldn't match.

My biggest single problem with Ruby is that shit breaks. Last week the main json gem was broken, causing pain throughout the Ruby world. With my first install of Chef-Server, the code-ray gem broke the chef-server-webui. As I am not a rubyist, it took me a non-trivial amount of time to figure out how to downgrade to an older version with the following command $ gem install code-ray --version '< 0.9.8'. I have since discovered the concept of gem sets, which define dependencies to specific gems per specific versions. Why would you have to do this? Because you can't count that the latest gem release will be stable. Working with Python I rarely had these same kinds of problems. On the plus side, the Ruby community moves fast. For whatever reason(s), new Ruby projects pop up quickly and continue to evolve at an astonishing pace. Tools like thor, sinatra, capistrano, and fog all seemed to appear before their Pythonic equivalents*.

* software historians, please correct me if I am mistaken

It is easy to overlook that Chef assumes you already thoroughly understand your operating system and application stack. Chef makes experienced system administrators more productive. It also makes inexperienced system administrators more dangerous and can even hamper their learning. It is for this reason that I don't recommend that those new to *nix pick up Chef. You can't automate something you don't understand.

Using Chef

My experience with Chef has not been all roses. Like Ruby, Chef moves fast. For a FOSS project, it is ridiculously young, first released in January 2009 and has not even reached version 1.0 yet. It still has significant bugs and Linux distribution support is still quite uneven, especially compared to Puppet. To their credit, the Opscoders acknowledge these issues. It must be noted, Opscode did not officially support Enterprise Linux until recently. Further, despite its youth, I found the Chef-Server to be quite stable and easy to setup.

I had a nasty time with the initial installation. It took me over three days to complete, mostly due to a bug in Ohai that misreported the linux distribution. However, that was early October. I did a fresh install last week and it took me only 30 minutes, largely due to documentation improvements, bug fixes, and updated rpms for both EL 5 and 6.

The opscode cookbooks, like the chef software, have spotty support for Enterprise Linux. One of the first cookbooks I tried to use was the selinux cookbook. I couldn't even upload that cookbook to my chef-server as it has a syntax error. I have sent a patch, logged a ticket, and sent a pull request to fix the error. That issue was not fixed the last time I checked. My pull request isn't lonely either. Last time I checked on github.com and I saw that there were 99 open pull requests in their queue. My impression is that the Opscoders are not intentionally ignoring such requests but are struggling to keep up with the rapid growth of Chef community and Opscode's paying customer base. I hope this improves in the months ahead.

Once I got Chef-server up and running, I felt a bit lost. Chef is a very flexible tool and not nearly as opinionated as Puppet. It is hard to decide where to get started. One can easily get lost among Chef's powerful abstractions such as definitions, providers, resources, and libraries. Worse, Chef doesn't have a definitive technical book to hold your hand through the entire process as James Turnbull's excellent "Pro Puppet" does for Puppet.

After my initial bout of confusion I set to work on converting a 400-line RHEL 5 server provisioning shell script into a set of cookbooks. Amazingly, I was able to convert it to a set of Chef cookbooks in less than a week and a half. Even better, these cookbooks could be applied to an existing server at any time in its life, not just at installation time. Bonus, my bash script handled RHEL/CentOS 5-6 on x86_64 and i386 architectures whereas the original bash script only handled RHEL 5 x86_64. I must confess that part of my productivity gains stem from the fact that I am far more proficient in RHEL than I was when I originally wrote the Bash script. That said, I believe that using Chef for the server provisioning task took me 30-40% of the time it would have required to rewrite the Bash script to handle RHEL 6 and the i386 architecture.

I have made a lot of progress with Chef in this first month but I have yet to automate an entire application stack. Here are the tasks I hope to implement with Chef in the next few months:

Consistently test my cookbooks with rspec or another testing tool
Create a Jboss cookbook
Cluster Tomcat instances

Best Practices thus Far

So here is a collection of the Chef "best practices" I have picked up in this first month. Given that it has only been a month, please take them with a grain of salt. Master Chefs, please don't hesitate to correct me if there is any incorrect information here.

If you want to get started quickly, i.e. learn chef in a weekend, use hosted chef and micro instances on EC2. For me it was important to install the entire stack before using it because that is the kind of geek I am. It also sucked up a lot of time.
Separate your ingredients (data) from your recipes. Don't hardcode your site-specific information like IP addresses or host names into the recipes. Put them in Data Bags, role attributes, or node attributes.
Data Bags are for complex or global data like users or samba shares
Role attributes and node attributes are best suited for simple values like host names for NTP servers, Nagios server, or whether to run X service as a independent daemon or under Xinted
If you aren't already familiar with git, you should make learning the basics of git your #2 priority after completing the Chef Getting Started Guide. Since we're talking about "Infrastructure as Code" here, you have to become as serious about version control as a software developer is.
After you clone the main chef repository, create the sub-folder site-cookbooks/ at the same level as cookbooks/ . Put your new cookbooks in site-cookbooks/ and make each one an individual git repository
In .gitignore, add ".chef" and "site-cookbooks". You store all authentication keys in your .chef/ so it is a directory that you really don't want to accidentally push to a public repository.
Sign the CLA for Chef. You can't contribute fixes without doing so.
Don't use $ knife role edit <role-name>; use the Ruby DSL instead. That way you have your roles under version control.
Check, double-check, then ask on IRC before starting a cookbook from scratch. I wrote four different cookbooks before realizing that they duplicated functionality already present in the main opscode/cookbooks
Use shef, it is a wonderful tool to debug and explore how Chef works. Note that it does not load roles attributes at this time.
Fat cookbooks, skinny recipes. You should put the recipes for both the client and server parts of an application in the same cookbook. This makes them easier to keep track of and forces you to make the recipes more modular.

Here are some Ruby tips that I have picked up

Put require 'irb/completion' in your ~/.irbrc, then you have auto completion in irb and in shef as it uses irb

For learning ruby, I actually prefer pry over irb, which reminds me of my beloved IPython. I haven't yet figured out yet how to get shef to invoke pry rather than irb.

To install pry

$ gem install pry pry-doc

You also have to put require 'irb/completion' in your ~/.pryrc or type it in at the prompt. Here is an example of how tab completion works

The next step is to actually check the docs for what particular method does. The command for that is
pry> show-doc Object.method

You can also read ri docs by simply typing pry> ri Class#method like you would at the normal shell prompt.

There are a couple tools that I intend to use in the near future, specifically libarian-chef by Jay Feldblum, but haven't yet gotten the opportunity.

More Pirate, Less Ninja

Chef has made a big impact on my work style. You definitely start working more like a developer, spending more time on your workstation and less SSH'd into a random machine. It is hard to justify heavily customizing your own dotfiles when you spend most of your time on random machines where the only text editor you can count on being present is vi. This article describes vi users as ninjas and emacs users as pirates but I think many of the same comparisons apply to sysadmins and developers. Ninjas can't carry much around with them on their dark forays just like sysadmins can't lug around a bloated .vimrc or .emacs file. Pirates, on the other hand, stay on their ships where they have plenty of room for whiskey, stolen treasure, ( .emacs.d/ || .vim.d/ ), .screenrc, .zshrc, etc.

With Chef, I spend a lot more time working on cookbooks, roles, and data bags in localhost:~/chef-repo/. Knife makes it very easy to manipulate your chef-server and chef-clients with fewer ssh console sessions. I think this work style may explain why there are many more emacs users in the chef-community than in your typical group of sysadmins. Historically, system administration has forced you to be a vi user but as a "chef" you can spend more time in emacs, sam, joe, or even *gasp* nano. I personally have heavily customized my .vimrc over the last month and even dabbled in zsh. I am not suggesting that Chef will turn you into an emacs user, but it will turn you into a pirate.

Selling it to the Team

Managers love chef. They get the idea in the first five minutes and ask you weren't using Chef five years ago. They love it because 1) big, respected companies are using it 2) it makes system configuration far more transparent, i.e. you can see how systems are actually configured and not how you hope they are 3) it will help them deal with the never-ending requests for new Virtual Machines coming from the developers. It can be harder to sell it to those sysadmins who may know *nix like the back of their hands but are used to doing everything by hand and have never gotten into the habit of automating tasks with their own shell scripts. It can also be hard to sell to those sysadmins who have already invested heavily in their own custom bash/Perl/Python/Tcl infrastructure framework.

Here is my pitch for Chef in a nutshell.

A few years ago we had a ratio of n machines per sysadmin. Now we have 3n machines per admin. Next year we will likely have between 6-9n machines per sysadmin. We are struggling mightily to support 3n now and our current coping mechanisms will break down entirely when we move to 6n. The only way we can cope is by moving from home-brewed infrastructure to a real configuration management system, such as Chef.

If you aren't comfortable with writing your own automation scripts, you are at least comfortable using the scripts of others. I don't expect everybody here to be able write their own cookbooks from scratch but I believe everyone here can learn how to reuse and modify existing cookbooks to meet our needs.

Frankly, I don't yet have a way to sell Chef to the NotInventedHere crowd. Whatever benefits you bring up, Chef still wasn't their idea. One option would be to convince them that Chef just does what they have been doing all along and using Chef will free them from maintenance work for their current tools. Then they can focus on what really needs reinventing. Unfortunately, implementing Chef may require a substantial amount of their help and not leave them so much time to start that new pet project.

Thoughts on the Community

I feared that the Chef community would be populated with developers who were "slumming" as sysadmins. I found quite the opposite. The vast majority of the community are Unix-lovin', bash-spittin' geeks who make their careers as sysadmins and not as a side project. In terms of geography, the community is overwhelmingly concentrated in North America. It can be tough to get a response on IRC during the morning in Europe while you will get tens of responses during the US West Coast's working hours. For all hours outside of the European morning, you can find a large numbers of core developers on IRC and they are extremely helpful.

The vast majority of chef users seem to work on Ubuntu, with Enterprise Linux a distant second. Surprisingly, there is a relatively large presence of OpenSolaris/Ilumos and I am not sure exactly why that is. Perhaps Solaris users are disproportionately drawn to automation tools?

When I can't get a quick answer on IRC, the two best channels are the 1) chef-users mailing list and 2) twitter. For some reason, the devops world has really, really gathered around twitter. I can get a quick answer at any hour by using the hashtag #opschef. However, this only works for questions that entirely fit into 110 characters. It is bad form to link to a mailing list post or extend the question across multiple tweets. For complicated questions, the mailing list is always the best place to go for help.

The community is extremely friendly and so far there are no trolls. However, this is typical of a young project. Early adopters by their nature tend to be more positive and self-sufficient. Most people on #chef are there because they discovered chef on their own and not because their boss mandated they use it.

My Short-Term Chef Wishlist

Here is my wishlist for the next 3-6 months

Clear guidelines on how to collaborate on cookbooks and on how to push changes upstream to opscode/cookbooks (something which jtimberman is already working on)
Lower Memory Usage. Chef can really, really be a memory hog when you have several hundred nodes
Better support for Enterprise Linux, both in core chef and the cookbooks
A few weeks ago I would have said better documentation but they have already made huge strides in this regard
A stronger connection to the FOSS community worldwide

Chef will achieve world domination when the installation instructions for the average software application start not by pointing to a .rpm, .deb, or tarball but to a Chef cookbook. Key to achieving that are 1) making Chef awesome (of course) and 2) better engagement with the communities that manage software packaging in the public sphere--the FOSS geeks. These guys are the guerilla foot soldiers of software packaging. They write the man pages, manage a FOSS project's build servers, maintain wikis, etc. The phenomena of "cloud computing" couldn't exist today if it weren't for all their hard work at every layer of the GNU/Linux/*BSD stacks.

How can Chef win over the FOSS geeks? First, having a presence at conferences like FOSDEM, OSI, FUDCON, FISL, etc. would be relatively inexpensive and give them a lot of bang for their buck. Secondly, offering some version of free hosted chef for up to n nodes for open-source projects and foundations like Software Freedom Law Center, Python Software Foundation, Software Freedom Conservancy, etc.

Wrapping up

So far, Chef has lived up to the hype. It has made me more productive in a relatively short amount of time. I can't imagine my future as a sysadmin without it. Configuration management--whether using chef, puppet, or other--will soon be an essential skill for a professional system administrator. Now how does Chef affect my work in the larger context of Devops? Where does it fit in my devops-toolchain? Well, those are subjects for another blog post entirely.

Comments for this post are on Hacker News

Why Chef?

2011-10-16T02:22:00.000-07:00

Being a system administrator is alternately thrilling and tedious. The thrilling part is setting up an awesome, rock-solid system. The tedious part is doing that 10 or even 100 times more. Further, throughout your career you will configure certain applications again and again. You know the ones, sendmail, apache, samba, etc. Well, there have been many attempts to build a framework, an abstraction, for automating the process of configuring these well-worn tools and others. Today there is one such framework that finally gets it right, Chef. There are many, many reasons to use Chef but the main reason you should use chef is this: You, System Administrator, are too damn smart to spend the rest of your professional life reinventing the wheel.

Let me tell you a little more about how wonderful Chef is, then I will show you briefly some technical highlights, and finally I will clear up some Chef myths.

The Top 5 Reasons to use Chef

Writing reams of documentation sucks. Chef drastically reduces the amount of documentation you have to write.
Bash doesn't scale. Seriously.
Technical Awesomeness
Chef grows with you
You can stop reinventing the wheel

Now let's talk about that awesomeness in more detail.

Write Less Documentation

Less docs? You're kidding me right? Actually, I'm not. If you're like me, you'r sick of annotating the your entire bash history. If you're not like me, you don't even bother. With Chef, your infrastructure literally is code rather than a collection of unique artifacts. Top-level readmes and a few well-placed code comments should be sufficient. Take a look at this recipe for a basic sendmail configuration.

basic sendmail configuration with chef

Bash Doesn't Scale

Bash is a wonderful thing, but like all UNIX tools, it is fundamentally limited by design. Bash doesn't have a code reuse mechanism more powerful than functions. It uses a single global namespace. These and other limitations have made it hard for us sysadmins to reuse and generalize our shell scripts across different distributions, let alone different versions of *nix. Lastly, it is quite difficult to write shell scripts that are idempotent, that is, they have the same effect whether run once or 100 times. This is particularly true when manipulating configuration files. Let's take a look at how you would configure /etc/sudoers with chef.

All the variables in this template are passed in from this recipe. If you think you can replicate this template using sed, please submit it as a comment so we can compare results.

Last but not least, Bash can be just as unreadable and obfuscated as the darkest perl code. With Chef you have a high level, intuitive DSL for describing your configuration. Remember what I said about needing less documentation?

Technical Awesomeness

NOSQL FTW

One of the virtues that many *nix tools share is that they store their configurations in text files rather than binary formats or in a database. Chef stores your system configurations in text and in a database. It accomplishes this by using the document-oriented database, CouchDB. This makes the configuration searchable and higher performance. One of my annoyances with nagios is that I have to restart it every time I change any service or host. There are no annoying restarts with Chef. Each Configuration change you make takes effect instantly. Further you can use your favorite text editor and the command line to make configuration changes. Finally, using a database means that Chef is easily accessed through a GUI. We sysadmins often look down on GUIs as crutches for the Windows crowd. The Chef web UI is excellent for visualizing your own infrastructure, something that is hard to do while staring at plain text.

Knowing is Half the Battle

Chef uses Ohai to collect data about your system. Your recipes can access these attributes and make decisions based on them. For example, you can determine which version of Red Hat you are using simply by looking up the value of node['platform_version']. You don't have to cat | grep | awk to find out which release you are on.

Ohai makes cookbooks more dynamic and able to support different distributions. As we will see later, this is one of the reasons there are so many high-quality cookbooks available.

Search

Search is a feature in Chef Server that allows you to query the configuration information of all other servers and of globally-defined databags. This allows you to do things like configure clusters where a member of cluster needs to know not only about its own configuration but about the configurations of the other members of the cluster.

Example of using search with data bags. For the full recipe go here.

Knife
Knife is one of the truly great command line tools. It is your primary mechanism for interacting with the chef-server. Knife shares many usage patterns with git. If you love git, you'll love knife.

shef

shef works the way you work, in an iterative manner. Most of us system administrators are self-taught and we learn best by doing. Fire up shef and you can on the fly play with attributes and create recipes. Further, you can connect to your server and download the cookbooks.

Chef Grows with You

Chef uses pure Ruby as its configuration language, not a shackled subset of ruby, nor yet another custom configuration language. You only have to learn a small amount of ruby to get started with chef. Once you get beyond the basics of Chef, you can go farther with Ruby. Many of you are grumbling now because ruby sucks to compared to Perl/Python/TCL/<insert-interpeted-language-here>. Well, Ruby may pale in comparison to Python but it is still a powerful, full-featured language.

Just like Perl, there is a lot of dark magic inside Ruby. It can't be used carelessly or it will bite you in the ass. Unlike Python, Ruby does not make it difficult to do the wrong thing.

You can stop reinventing the wheel

Until Chef, we sysadmins did not have a truly modular way to abstract and share our system configurations. Please stop reading and browse http://community.opscode.com/cookbooks. Later on, you should look at the code in https://github.com/opscode/cookbooks. You will discover that some turncoat sysadmins are giving away our trade secrets. Now is the time for you to do the same. In fact, you can rip out a large chunk of your shell scripts and replace them with Chef cookbooks. You will find that many existing recipes meet your requirements and you can easily add new recipes to existing cookbooks for your unmet requirements.

Recently, I replaced a 500 line shell script with 7 cookbooks, 5 reused from community.opscode.com and 2 new ones written from scratch. In the process of replacing the shell script I actually wrote 4 cookbooks that unwittingly duplicated the functionality of existing cookbooks.

The emergence of the cloud-computing, whether on the public cloud such as EC2 or internally with openstack, means that we systems administrators will have to manage at least 10 times more systems. We have to become 10 times more productive. We have to shift from "managing servers" to "managing configurations." Chef is one of key tools to accomplishing this.

One Big Fat Chef Caveat

Chef makes good sysadmins more productive. It does not turn junior sysadmins into experts. In fact, it makes them more dangerous. As I overheard on IRC one day, "Chef is to sysadmins what C++ is to software engineering." This is very true. You can automate system configurations with off-the-shelf cookbooks but you have to understand exactly what they do and test them very rigorously.

Chef Myths

There are a few myths about Chef floating around the intertubes that need to be exploded. The biggest one is that you have to be a professional programmer to use Chef. This is untrue for a couple of reasons, including the assumption that system administrators don't program already. Writing complex shell scripts is programming. Creating Apache rewrite rules is quite close to programming. Further, Chef doesn't require you to be a Rubyist to get started, just as you don't have to be a Bash Hacker to use the command line. Take a look at this tutorial and you will see what I mean.

I hear murmurs now and then that Chef is "insecure" because the recipes are executed on the chef-client rather than on the chef-server. I argue that chef is more secure because of this. If recipes were executed on the chef-server, there would be a risk that a single malignant recipe modified other recipes to turn your entire network into a botnet. This would turn your chef-server into a single point of failure and make it a powerful attack vector.

Finally, some will argue that Chef sucks because every server you configure should be hand-crafted and beautifully unique in its own way. If you believe that, perhaps you should go into the virtual performance art and get out of the business of making the Internet awesome.

Wrapping Up

Chef is a truly powerful tool that every serious system administrator should consider using. It offers immense personal and professional benefits.

Please post comments on Hacker News

Marcus van Dam (m4rcu5) helped edit this post

Note: The title of this post is a reference to Eric S. Raymond's great post "Why Python?" which inspired me to learn Python, a long, long time ago.

Puppet vs. Chef, Fight!

2011-10-09T07:46:00.000-07:00

Update: see Puppet vs. Chef en francais, not a translation but an original article inspired by this one

If you aren't yet using Puppet or Chef for managing your *nix infrastructure, you should seriously consider it. The benefits are enormous. Frankly, I don't think you __can__ manage a modern infrastructure without using Puppet, Chef, or reinventing a configuration management framework with your own custom scripts. However, you can't use both Chef and Puppet together, so which should you choose?

If you're like me, you obsess over tool choices. You agonize for weeks over these decisions. Why? Because you will end up putting a huge personal investment in the chosen tool. There isn't all that much on the relative benefits of Puppet vs. Chef besides this quora thread. Below you will find my completely subjective, non-comprehensive study of the matter. TL;DR, Chef wins but by only a narrow margin.

The Criteria

Community Strength
Leadership
Corporate Adoption
Technical Merits
Hands-on Experience

Community Strength

Community Size

Number of people on IRC on Sunday Oct 9th, 2011 10 AM GMT
#puppet 541 users
#chef 233 users

Mailing Lists
puppet-users 3363 members, 5961 topics
chef: 600 members, # of topics unknown

Regarding IRC, Chef seems much more U.S.-centric. It can be hard to get answers during the European workday. The situation is probably better for Asian users as they can catch the North American community when it is early evening on the West Coast of the United States. That is not true for Puppet, where you can get an answer at any time of day or night. R.I. Pienaar is based in London and seems to be on IRC at all hours. Somehow he manages to kick out awesome code and be continuously available. Further, James Turnbull doesn't seem to sleep either and is incredibly helpful.

Quality of Cookbook Sites

http://community.opscode.com/cookbooks
http://forge.puppetlabs.com/

Github Repositories:
puppet: 1622 repositories
chef: 1299 repositories

While Chef is much younger project than Puppet, it is really impressive how many cookbooks there are and their respective quality, especially when Chef is less than half the age of Puppet. From what I can tell, most people bypass the community sites and go straight to github to find their Chef cookbooks or puppet modules.

Documentation, Official and Informal

Both Puppet and Chef have excellent reference documentation. In my opinion, the Puppet docs help you get up and running much faster. Pro Puppet, by Puppet Labs engineer James Turnbull, is an awesome introduction to Puppet. I found it immensely helpful. There is a 50-page O'Reilly publication on Chef, Test-Driven Infrastructure with Chef, but it cannot match the breadth of Pro Puppet. I have heard that O'Reilly are going to be publishing a feature-length definitive guide, scheduled to be out by the Summer of 2012.

The Chef community definitely tends more towards programmers and rubyists than systems administrators. I think that puppet meets the needs of the majority of its users so well that they never try Chef.

Leadership

We all know that leaders can make or break open-source projects. The BDFL's personality shapes the project more than the actual code. What would linux be without Linus Torvalds' stewardship? I am wary to bet against Puppet founder Luke Kanies. He is both a genius and a visionary. He reminds me a little of another Reed College alumnus. I highly recommend listening to Luke Kanies interview on DevOps Cafe. He shows himself to be very intelligent and very opinionated. My understanding is that his original motivation for creating Puppet was to make life better for the average sysadmin.

The resumes of the Opscode team are just amazing. CTO Chris Brown and co-founder Jesse Robbins were core infrastructure engineers at Amazon, the company that perhaps more than any other understands how to manage infrastructure at scale. Listen to co-founder Jesse Robbin's Interview and you will find him to be very pragmatic about infrastructure and less opinionated than Luke. His motivation was to create an open-source tool that solved the kinds of infrastructure problems he dealt with while working at Amazon.com.

On the funding side, both companies have closed substantial Series B funding rounds. We're talking more that 10 million dollars each. Both of these companies can throw a number of talented software engineers at the devops problem domain for a while to come.

Corporate Adoption

I believe that both Puppet and Opscode corporations will become extremely successful. Jesse Robbins explains this best in Virtual Strategy Magazine.

Our primary competitor is the in-house, home-grown tooling that every systems engineer has had to cobble together to deal with their legacy environments. That’s our principal competitor.

That's right. The biggest competitor to Chef and Puppet isn't each other but that hairball nest of Bash and Perl that holds your data center together.

Still, Puppet has an amazing customer list but Chef is also racking up wins. If Puppet Enterprise 2.0 delivers half of what its press release promises, it will be a huge success. Puppet Enterprise's integration with mcollective is particularly exciting.

I am particularly impressed that a number of companies have built their own internal provisioning tools on top of Chef, notably Rackspace Cloud Builders and Dell's Openstack team.

Technical Merits

Puppet is designed to be approachable by systems administrators. It is not necessarily written to empower rubyists. Writing Puppet manifests, at least initially, feels much like writing nagios or apache configurations. However, it starts feeling much like programming once you get beyond simple resource specification.

Chef is designed to help engineers manage the configuration of their infrastructure at scale. It does not have the same focus on easy-of-use, nor is it as opinionated as Puppet. Now let's look at a few salient features.

Syntax

Here are examples of how to create a user in first Puppet and then chef

They don't look terribly different, but now let's throw some Ruby into the mix. In the following example, I iterate through a list of users stored in a data bag

Now you should have a better sense of the difference between Puppet and Chef. Some may have a valid criticism that Chef lets you do too much by allowing you use regular Ruby. I have to admit that undisciplined use of Ruby is as dangerous if not more dangerous as perl or bash abuse. At the same time, it is quite empowering. The Chef community really needs some kind of chef-lint tool to tell you when you're doing something you shouldn't.

Debugging

I miss noop mode in puppet, though this is on the way for Chef. It is a wonderful way to see how your puppet configuration differs from what is actually on disk without actually changing anything. I have heard some refer to noop mode as "Lie to me mode" though I have yet to have it bite me in the ass. I might feel very differently if I had experienced that personally. I also liked how puppet spit out the text of rendered templates to the console. Maybe you can do this with Chef as well, but I haven't figured out how to yet.

Chef's Secret Sauce

Chef's Data Bags are incredibly useful. They are a very nice way to feed lists of users to multiple cookbooks. Think of them as global variables for your chef-server. One the benefits of data bags is that they allow you to separate your corporate information from your cookbooks. Thus, making it easier for Chef users to open-source their cookbooks. Puppet has extlookup but that is quite a bit more limited than data bags. It looks like Puppet maybe getting something similar to data bags with Hiera.

I also have to praise the Search function. It is an extemely intuitive way to to tie together dependencies between nodes, such as between a nagios server and its clients. Puppet has a way to "export" resources but I could never get my head around how that worked.

In my experience, Chef has a more powerful read-eval-print-loop (shef) than Puppet does. I use shef quite frequently and I cannot understate its helfulness. Puppet does not have a similar repl as far as I am aware.

Server-side, Chef is much more complex but also seems to be better thought out. I love the fact that Chef uses a document-oriented database. This combines the benefits of using files to store your configuration (easy to understand) with the benefits of using a database (speed).

Hands-On

I don't believe there are many in the Puppet community that have extensively experimented with Chef before choosing Puppet. There are many in the Chef community that have worked previously with Puppet and found that it did not meet their needs.

I started working with Puppet in June of this year and quickly fell in love. I read Pro Puppet and did all the exercises. Next I set to rewriting my 400 line linux provisioning script in puppet manifests. At first it went well but then things got more complicated. I kept tripping over the DSL's syntax and was surprised that there weren't more puppet modules in puppetforge or github. I felt that I had to learn both the Puppet DSL and Ruby to really to be effective with Puppet. I started thinking that the grass might be greener on the other side.

Two things convinced me to try out Chef. The first was the announcements by Dell and Rackspace that they had built there Openstack provisioners on top of Chef. If the people pioneering devops chose to build their platform on top of Chef, then it definitely deserved my attention. The next was Jesse Robbins' interview on DevopsCafe. He made all the action in the Chef community sound so exciting that I had to go check it out for myself.

It took me three days to get Chef Server up and running on RHEL 5.7. Next, I started writing my own cookbooks. Now I have nearly converted that linux provisioning script to a cookbook. Not counting the time it took me to set up the server, it took me roughly half the time it took with Puppet.

It was easy to get started with Puppet, but things became more complicated with time. With Chef I am having the opposite experience. Again, the single biggest drawback to Chef is that it has a steeper learning curve than Puppet. Further, most of the existing tutorials focus on the deeper concepts in Chef rather than giving novices quick gratification. To this end, I have started the "Dirty Chef" series of tutorials.

Summary

Final Scorecard
Community -- Puppet wins
Leadership -- Draw
Corporate Adoption -- Draw
Technical Merits -- Chef
Hands-On -- Chef

According to this non-scientific criteria, Chef is the better tool. In my humble opinion, Chef is technically superior. Don't buy the argument "Chef requires you to be a programmer, Puppet doesn't." While Puppet is geared towards non-programmers, it becomes regular programming once you do anything moderately complicated. That's when Puppet's DSL becomes more of a liability than an asset and where Chef really shines. Now while chef is the winner of my little bake-off, it is not without faults. Puppet has far better support for RHEL/Centos and it does have a larger, more responsive community.

Dear Puppet community, please know that I write this blog post with great ambivalence and I still love you. Jame Turnbull and R.I. Pienaar, thanks for all your help via IRC, Twitter, and the mailing list. Puppet Labs is developing a toolchain that is much larger than just the puppetmaster. This toolchain may turn out to be superior to what Opscode and its respective partners come up with.

On a final note, please know that both of these tools are good enough for configuration management and are only one part of a larger toolchain.

P.S. I will delete any comments that trash either Puppet or Chef. They are both excellent tools, built by
visionaries.

Most of the discussion related this article seems to be happening on Hacker News.

A Stupid, Simple Chef Tutorial for Dirty, Lazy Fry Cooks (Like Me) Part I

2011-10-07T00:55:00.000-07:00

disclaimer: I am not a chef expert. To really learn chef, you need to go to the wiki

I really like the chef documentation, I really do. It is an excellent reference. It is the devops equivalent of the manual they must use at the Le Cordon Bleu cooking school. But I have to admit that sometimes I am just a dirty, lazy fry cook, especially when playing with a new technology. I want instant gratification or at least in under 30 minutes. What I love about the Puppet documentation, is that it gets you started quickly, before you begin optimizing and modularizing and all that other crap that good devops chefs do. To that end, I have created a chef repository that is exactly what you shouldn't do in production but is hopefully useful for learning.

In this tutorial, I am going to use chef-solo for the purpose of the tutorial. Chef-solo simply means running chef with out connecting to central chef server that provides you with cookbooks and some other functionality. Chef-server is awesome but currently a pain in the butt to set up. I am also going to use RHEL 6.0 but you can use CentOS.

Why Chef?

If you aren't yet using configuration management with Chef, Puppet, or similar tool you're just plain doing it wrong. I'm doing it wrong. I cobble together my infrastructure with a combination of kickstart, bash scripts, fabfiles, and random python. But the diversity of applications I support and the sheer number of virtual machines are increasing rapidly. My agglomeration of virtual glue and duct tape is starting to look more like a house of cards and less like modern infrastructure.

In the immortal words of Puppet founder Luke Kanies:

"SSH in a for loop just doesn't cut it anymore."

To build Luke's statement, our imperative bash/perl/python scripts are lousy for customizing server templates to specific node attributes and they suck at maintaining system state. I have tried both Puppet and Chef and I prefer Chef for a number of reasons. The main ones are that it has a more elegant and easier to use DSL and that chef-server has the awesome search functionality.

Installation

Install my chef repository somewhere on your system. I recommend /opt.

git clone git://github.com/bryanwb/dirty-chef.git

Next install chef. Here is the main install guide

This should work on either CENTOS/RHEL 5 or 6

For RHEL 5

$ rpm -Uvh http://rbel.co/rbel5

For RHEL 6

rpm -Uvh http://rbel.co/rbel6

Finally, on either


$ yum install -y rubygem-chef ruby-shadow

Now edit solo.rb, you have to edit these paths to reflect the actual location of dirty-chef otherwise chef-solo will not work

file:solo.rb


file_cache_path  "/opt/dirty-chef/"

cookbook_path "/opt/dirty-chef/cookbooks"

data_bag_path  "/opt/dirty-chef/databags"

If you are using RHEL 5 see the install link above, it has steps for RHEL 5 but they are more complicated Run the following command

$ ohai | less

This shows you all the info that chef knows about your system. You can do reasoning in your recipes based on these values.

Start Cooking

Here is a stupid hello world

$ cd dirty-chef
$ chef-solo -c solo.rb -j node.json
ohai!

What the heck did we just do? The solo.rb tells chef where the cookbooks are, i.e. where the recipes live, and node.json tells chef-solo which recipe to actually apply.

file:node.json


{

  "run_list": [ "recipe[dirty]" ]

}

This loads the recipe in cookbooks/dirty/default.rb

If you get any errors here, you should make sure that you changed the paths in solo.rb to reflect the location of dirty-chef/

Now let us do something more significant. You can use chef to declare "resources" in this case a package. Let's install subversion. In chef recipes, you spend most of your time declaring resources, in this case I declare that I want subversion installed and chef takes care of the rest.

file:cookbooks/dirty/recipes/example1.rb
package "subversion"

$ chef-solo -c solo.rb -j example1.json

file:example1.json

{
  "run_list": [ "recipe[dirty::example1]" ]
}

This loads the recipe in cookbooks/dirty/example1.rb

Here is an example of installing multiple packages, take a look at cookbooks/dirty/recipes/example2.rb

$ chef-solo -c solo.rb -j example2.json

What is that syntax of the configuration language? Why it is pure ruby. If you like perl, you will like ruby as it is the lovechild of perl and smalltalk. Here is a good quick reference

You still are not impressed? True you could do all these things in a kickstart more easily and without having to go to so much trouble. There are several reasons chef is better than kickstart. You can use chef to maintain the state of your server while you can only use kickstart to initially set up your server. Once you install your server from a kickstart, you cannot rerun it w/out reformatting it. Chef on the other hand is "idempotent" Meaning it has the same effect whether run once or 100 times. Say for some reason subversion is removed from your machine. The above chef recipe will reinstall it w/out needing your intervention. You can seamlessly handle different platform versions. You can write one recipe that handles both RHEL 5 and RHEL 6 by doing reasoning on node attributes. Maybe you are like me and love nothing better than a good REPL (Read-Eval Print Loop), i.e. console. Chef comes with an awesome REPL, shef. Let us use it to play node attributes and even load recipes interactively

[root@woof-chef recipes]# shef

loading configuration: none (standalone shef session)

Session type: standalone
Ohai2u root@woof-chef!
chef > if node["platform_version"] =~ /6./
chef ?> puts "hi rhel 6!"
chef ?> end
hi rhel 6!
[ type ctl-D to exit ]

In shef, you have access to all the node attributes and cookbooks on your system. Now let us put the same logic in a recipe.

$ chef-solo -c solo.rb -j example3.json

Now we still haven't actually done anything useful. We will get to that in Part II

What is devopsanywhere?

2011-10-03T01:47:00.000-07:00

Why DevopsAnywhere? Because our profession is undergoing some profound changes. On the positive side, the best of us are in increasing demand. On the other hand, it will take increasing sophistication on our parts to keep up w/ the blistering pace of change in our industry. I believe that to really keep up-to-date, you have be an autodidact, that is someone who is primarily self-taught.

In my experience, autodidacts are randomly distributed around the planet. Linux obsessing, python hacking, ruby scripting geeks are even more randomly distributed. Should companies really want to recruit us, they will have to hire us where we are, be that Kathmandu, Sicily, or Utah. Gone are the days where system administrators lived at the datacenter, as I explain in greater detail later.

As system administrators we were tied to the datacenter. Even if we worked remotely, we had to be within easy driving distance. We were software/hardware hybrids that understood all the various permutations of RAID 0/1/5/6/10. We looked on with envy at the alpha programmers who could do their jobs while lying on the beach in Thailand or while huddled in a cozy cabin in the French Alps.

Our profession is transforming from System Administrators to Devops Engineers. Devops is a bit tricky as it is as vague a term as agile. Still, it points to a new class of engineers that are focused on operations and work seamlessly with software developers. We are self-taught engineers who have the luxury to work remotely. This blog will be entirely focused on the concerns and interests of devops engineers working remotely. Further, I won't talk about expensive proprietary software or hardware. Actually, I will rarely talk about hardware at all.

Some, like John Willis and Damon Edwards of the awesome Devops Cafe, will argue that process is more important tooling. I agree with this 100%. However, as system administrators we often aren't in the position to reorganize our team's management structure. We are in a position to introduce new tools and lobby for better ways of working.

Here is a comparison of the requirements for being a SysAdmin and that for Devops Engineer

A new Set of Skills

System Administration today requires:

deep knowledge of *nix, including linux, solaris, and even some proprietary unices like HP-UX
you really __know__ bash
some basic version control w/ cvs/svn
simple scripting w/ bash/perl/python
understanding of virtualization
You can troubleshoot hardware, firmware issues, even flip a dip switch or change SCSI IDs if necessary

Devops requires:

deep knowledge of Linux -- generally not Solaris, AIX or HP-UX
you __know__ bash
you are very proficient in svn and/or git
You are a practicioner of puppet or chef
system scripting with ruby
System packaging -- ruby gems, rpms, debian packages
You can setup package repositiories on a whim
you understand unit testing and behavior-driven testing, hopefully you practice it w/ Cucumber or something like it
AWS, Rackspace, cloud provisioning -- but of course

Hardware is now (mostly) irrelevant

The only NIC cards a devops engineer cares about are the Intel e1000 or whatever paravirtualized driver your hypervisor of choice offers. We know that working on hardware issues only lowers our hourly billing rate. Now, more than ever, the power is in software and hardware-oriented solutions are to be avoided.

I must confess that I have an ulterior motive for starting this blog. I am a linux system administrator who is following his wife's career around the world. Previously we were in Nepal, now we are in Italy, next up who knows! But I want to work with top-notch teams wherever I end up and not be tied to whatever opportunities are locally available. If you are a devops guy and you feel the same, let me know!

How Ruby is beating Python in the battle for the Soul of System Administration

2011-09-16T11:12:00.000-07:00

Roughly three years ago I purchased the book "Python for Unix and Linux System Administration" and it convinced me that Python would be the default scripting language for linux system administrators for the foreseeable future. I was working on the One Laptop Per Child project at the time, where Python was the lingua franca. It seemed that Red Hat was using Python for almost every new project. Further, Unladen Swallow was making rapid progress. I couldn't have been more wrong.

Through a combination of historical accident and some seemingly minor feature differences, ruby is inexorably becoming the dominant scripting language for linux system administration.

Before I go any farther you die-hard sysadmins are probably screaming "We already have a scripting language, Bash!" While we love bash and commandlinefu, Bash becomes a liability not an asset once your script exceeds 100 lines and a total nightmare if you need to parse or output HTML, CSV, XML, JSON, etc.

Historical Accidents

In 2004, Luke Kanies set about building Puppet, a configuration management system to improve upon the deficiencies he saw in CFEngine. From the excellent On Ruby blog

I tried to implement my idea in perl, but I just couldn’t get the class relationships to work (the attributes and resource types each needed to be classes, according to the design in my head). This was back when Python was the shiznit, so I naturally tried it, but Python just makes my eyes bleed (and no, it wasn’t the whitespace, it was things like the fact that ‘print’ was a statement instead of a function, and ‘len’ was a function instead of a method).

I had a friend who had heard Ruby was cool but hadn’t actually tried it himself. Since I was just messing around at the time, I figured I’d give it a go. Four hours in, never having seen a line of Ruby previously, I had a functional prototype.

A couple years, a number of Puppet developers felt that Puppet didn't meet their needs. They started their own configuration management tool, Chef, heavily inspired by Puppet. The biggest outward difference between Puppet and Chef is that Chef uses pure ruby as its "recipes" whereas Puppet uses its own configuration language based on ruby.

Both Puppet and Chef are seeing rapid adoption by big companies according to BusinessWeek. If you aren't yet using Puppet or Chef, you should be planning on it in the near future. Whether you choose Chef or Puppet, you are effectively scripting your infrastructure with ruby. After spending 25% of your time working with Puppet, you will be much more likely to reach for ruby for your next scripting task.

Popular Projects

Off-hand here is a non-definitive list of sysadmin/devops related projects in ruby

puppet
chef
vagrant
mcollective
cucumber (behavior-driven testing)
capistrano
rake (ruby Make)
aeolus project/openshift
cloud foundry
graylog2
logstash
travis-ci

and in Python:

What's important here isn't the length of the respective lists but the importance of the individual projects. Chef, Puppet, and Vagrant are the new hammers and screwdrivers of system administration. If you are a sysadmin and aren't yet using these tools, don't worry, you will be sooner or later.

Openstack deserves special attention as it is a very exciting but early stage project. It could play a big role in your future as a system administrator.

Please notify me of significant devops projects that are missing from this list.

What a ~~Girl~~ Sysadmin Wants

Here are the features in a scripting language that a sysadmin wants

A DSL for the problem domain
High productivity, i.e. concise and expressive syntax
Easy to interaction with shell commands
Regular Expressions
powerful one-liners

Notice that performance is not on this list of requirements. Ruby is significantly slower than Python and this was particularly true for the 1.8.* series of Matz Ruby Interpreter (MRI). However, performance just isn't critical for 90% of our work as sysadmins. We care about productivity more than we care about performance. Readability is nice but it is a distant second to productivity.

Python doesn't have regular expressions embedded in the language, probably to improve readability. It also limits the number of top-level built-in global variables. From a language design perspective, this is much cleaner. From the perspective of your average street-trained, stress-laden, vi-addicted sysadmin, it is annoying.

These top-level variables also make one-liners more concise. Here are just a few

$_ last line read from STDIN

$? last exit code from a child process

$stdin reference to stdin

$stderr reference to stdout

$stdout reference to stdout

Here is an irb session to show you these values in action

hitman@hiroko:~/pr$ irb

irb(main):001:0> %x[ ls -a ]

=> ".\n..\nfoo1\nfoo2\nfoo3\n"

irb(main):002:0> puts $?

0

=> nil

irb(main):003:0> %x[ ls xys ]

ls: cannot access xys: No such file or directory

=> ""

irb(main):004:0> puts $?

512

=> nil

"IRB?" you scoff "You will have to tear IPython from my cold dead hands."

I love IPython. I have used IPython for 4+ years and IRB doesn't hold a candle to it. That said, ruby's shortcuts are really useful, enough to compensate for IRB's shortcomings. There is something called wirble that I haven't tried yet that may make irb a lot more productive.

Here some python code to detect whether a machine is a VMware VM.

# edit: fixed python code tks to kstrauser

  import os
    if 'vmware' in os.popen('dmidecode').upper():
        print 'this is a vmware vm'
    else:
        print 'this is not a vmware vm'

Here the same code in ruby

`dmidecode`
if $_ ~= /vmware/i
puts 'this is a vmware vm'
else
puts 'this is not a vmware vm'

Frankly, this kind of code makes my eyes bleed. The python example is far more readable and maintainable. That said, a lot of sysadmins would appreciate the terseness of the ruby example.

Let's look at writing one-liners in both languages.

This code prints out the first 10 lines in a file using Python

python -c "import sys; sys.stdout.write(''.join(sys.stdin.readlines()[:10]))" < /path/to/your/file

here is the same in Ruby


ruby -ne 'puts $_ if $. <= 10 ' < /path/to/your/file

Compare for yourself between these collections of oneliners in python and ruby. If ruby reminds of perl, your eyes do not deceive you. In many ways it is the love child of perl and smalltalk.

DSLs FTW

Some time ago, I tried to watch all the Structure and Interpretation of Computing lectures. This endeavor failed miserably but I recall Harold Abelson saying that every large program should have its own internal DSL suited to the problem space. This is debatable in regards to the larger world of programming but I think it is very apt for system administration. We spend our entire careers with n different DSLs. Each different configuration file format is its own DSL.

If you compare rake (Ruby Make) to rails code, they look quite different, almost like different languages. If you compare fabric code to a django class, they look quite similar. This is both a strength and a liability. I am not a language lawyer but it seems much easier to create DSL's (Domain Specific Languages). Ruby certainly spawns DSLs with much greater frequency than python. No single pythonic build tool dominates the problem space like rake does in the ruby community. Most python projects seems to use setup.py for administrative tasks even though that is not its explicit purpose.

Both puppet and chef are DSLs for system administration. Capistrano is a DSL for application deployment. Ruby's operator overriding and blocks lend themselves to the easy creation of DSLs.

In Summary

Ruby's greatest strength is its amazing flexibility. There is a lot of "magic" in ruby and sometimes it is dark magic. Python intentionally has minimal magic. It's greatest strengths are the best practices it enforces across its community. These practices make Python very readable across different projects; they ensure high quality documentation; they make the standard library kick ass. But the fact is that we sysadmins need flexibility more than we need raw power or consistency. Still, these are not the real reasons that ruby is overtaking python.

Ruby is fast becoming the default scripting language for sysadmins because back in 2004, Luke Kanies looked at Python and felt ill (I had the opposite reaction). As a sysadmin you either currently are or soon will be using Puppet or Chef on a daily basis, spending a heck of a lot of time essentially coding in ruby. Personally, I much prefer writing python but am shifting to writing my scripts in ruby because I spend so much time with Chef.