devopsanywhere

Let's make CLI Tools less Shitty

2016-08-01T07:55:00.001-07:00

Alternate Title: Bash Programmers Shall Inherit the Earth

I have recently come to the conclusion that the future of configuration management is not Ruby, Python, or a putrified YAML DSL but it is Bash and PowerShell. My thinking largely stems from the recent release of Habitat from Chef Inc. and several years of experience writing automation tools for Sysadmins with varying levels of sophistication. I have written tons of Chef cookbooks, Puppet manifests, and Ansible Playbooks over the last 7 years. My greatest frustration with all of these tools has been that it is very difficult for users not already immersed in those tools to reuse and modify their respective cookbook/manifest/playbook. I have written previously about how it might work if we gave CLI tools Restful APIs and this post builds on some of those ideas.

My friend Milos Gajdos asked if this proposal supports the spawning of long-lived processes that can be interacted with asynchronously like Restful services. That is an idea I proposed In my earlier post. I am no longer sure how that would best work so it is not part of this proposal.

Bash is the lowest common denominator for automation scripting. Habitat puts enough structure around Bash that it *might* be able to replace Chef entirely for automation tasks. Note that while Bash is a very primitive environment, Powershell is not. Powershell is a very sophisticated tool but Bash sets the minimum capabilities that we have to expect.

As noted earlier, I have spent a lot of time writing code for Chef, Puppet, and Ansible. I want to move away from writing sophisticated tooling inside these frameworks to writing more flexible command-line tools that work well with Bash, Powershell, and higher-level frameworks.

What I have in mind is a standard for CLI tools that I call CREST or Command REST. The similarity to ReST is intentional as it takes some inspiration from Representational State Transfer

Before we get into what CREST might look like, let's start with a meaningful example command line invocation and show how it might take advantage of crest. At acme Corp our users need a cli tool to create amazon ec2 instances and returns a list of instance ids for the newly created instances. For the sake of this argument, please ignore that there are many tools and libraries that already handle this. This command should have the option to be idempotent. Sysadmin Bob in Connecticut wants to use the command in their bespoke bash scripting config mgt system. The devoops team in San Jose will use the tool within their existing chef cookbooks. There is even a team of Windows devs in Utah that want to invoke our new tool from their Windows-based CI server that is just a pile of Powershell. The developers in a new acquisition use Ansible. Acme corp users will use this tool on Windows, OS X, and Linux machines.

Let's call or new command hamilton after James Hamilton, a distinguished engineer at Amazon who gives great presentations. For the common case, hamilton does not need any arguments. It creates a single EC2 instance using a bunch of built in defaults.

$ hamilton

stdout:
instance_id: i-2345345
public_ip: 1.1.1.1
region: us-east1
Name: acme-rand12345
machine_type: t2.small

stderr: [empty]

$ echo $?
0

That was easy! However, if you have ever worked with EC2 instances you know that the API supports many, many different parameters. Let's create 3 ec2 instances with the following properties

region: southeast-ap1
image_id ami-123456,
instance_type: c2.xlarge
termination_protection: on
ephemeral volumes: 1 50GB
enhanced IOPS
tags: ["Name: foo_instance#{count}", "environment: testing"]
and even more . . .

Wow! That's a lot of options. So many that they are infeasible to supply on the command-line. We could allow 100+ command-line options. That would make our Bash users happy but frustrate our Chef and Ansible users to no end. This is especially true when you consider what a pain it is to generate command-line invocations for Powershell. Putting together long sets of strings for Powershell is a bloody fucking nightmare. At this point many people would write a Python (or Ruby) library that can be manipulated to create ec2 instances with the correct options. Congratulations! You have just required everyone on your distributed team to learn Python! This is not entirely true as some of your users will never learn Python but simply ask you to create scripts for them that they can call from Bash/Powershell/Ansible.

Instead of writing a Python library or creating a CLI w/ a bazillion options. I would give hamilton the most common options on the CLI and allow it to ingest a big ball of JSON for the more complex cases. Our Bash users can grudgingly write JSON documents and our Ansible and Chef users will happily generate them from dictionaries.

$ hamilton input.json

contents of input.json

{ "region": "southeast-ap1",
"image_id": "ami-123456",
"instance_type": "c2.xlarge",
"termination_protection": "on",
"ephemeral_volumes": { [ "size": "50GB" ] },
"enhanced_iops": true,
"tags": ["Name: foo_instance#{count}", "environment: testing"]
}

Here is the first part of CREST. The Request, should support a JSON input for easier automation.

Now let's look at the Response we want to get back.

stdout:

instance_id: i-2345345
public_ip: 1.1.1.1
region: us-east1
Name: acme-rand12345
machine_type: t2.small

stderr: empty

Alright this looks fine. We got back the details we needed and we can go about our work. That said, what should hamilton return if there are no c2.xlarge instances available? What if the EC2 API calls failed and returned a long error message? How would we expose that to the user?

I believe that we should separate the "data" of the response from the metadata about it. The UNIX standard does have mechanisms to do this but it is not consistently implemented or even clear how to do it. The convention of using an exit status to report command failure to report command status is too coarse in my opinion. I am a big fan of the Unofficial Bash Strict Mode. However, one unpleasant side effect of using strict mode is that it leads you to ignore exit codes in your scripting or to turn strict mode on and off in error-prone ways.

I propose that the Response should support outputting on stderr a message in either of two formats, newline-delimited or JSON.

stderr:
exit_status: 403 # A descriptive status about the API that matches a common convention, just like HTTP
error_message: A human-readable error message
verbose: stack trace, log messages, anything else to print to user

Why put the exit status in stderr? Use of the `set -e` option in Bash will cause our script to exit as soon as a non-zero exit status is generated. However, we may not want to exit the script in all error cases. To get around this, many scripters write lines like this `cat some_file | grep "some_string" || true`. In this case, a non-zero exit status is ignored so our script dpesn't exit prematurely but we have also lost some information.

To summarize here is what I have in mind for CREST. I would love your input!

CREST - Command REST

A CRESTful tool:

is available as a standalone binary
accepts arguments as command-line args or a JSON document, similar to an HTTP Request
The "data" output by the tool goes to stdout and can be in any format. This maps to the data part of an HTTP response
The metadata for the result of the command is output to stderr and must support both newline-delimited and JSON output formats. This maps to Response Headers in HTTP.
The exit code should only be 0 to indicate success and 1 to indicate failure
If the invocation succeeds, an exit_status of 2xx is returned
If invocation fails, an exit status other than 2xx is returned

The standalone binary is critical as a CRESTful tool and its dependencies must avoid making assumptions whenever possible about what common libraries might be installed. Note that this can be hard to avoid entirely, see GLIBC for example.

Notes: It is critical that metadata can be output in either newline-delimited or JSON formats. We cannot assume that our users have a JSON parser like jq at their disposal.

Here is a possible format for Standard Error Output. Note that only exit_status is required.

exit_status: a commonly understood status code like HTTP uses [required]
error_message: human-readable error message [optional]
verbose: stack trace, log messages, anything else we wish to print to user in case of failure [optional]
arbitrary additional headers . . .

I prefer http status codes over C exit statuses as there is no standard meaning to C exit statuses while HTTP has relatively well understood codes that even I can grok without searching for a man page that might not exist.

There is a giant hole in this specification. I have not specified how a CRESTful tool manages internal state transitions and reflects that status to users. This is a hard but important problem. It will take some trial and error to suss this out. Thanks to Adam Jacob for highlighting the importance of this.

I would love your thoughts and input on this!

Update August 1, 2016: Thanks to Milos Gajdos for his input.

Update August 2, 2016: Thanks to Adam Jacob for his input. My vision of CREST is less ambitious than the issues and use cases that he brought up. Adam thinks it is critical that CRESTful tools support idempotency. Doing that, however, requires each CRESTful tool manage state transitions. There is a lot "devil" in those details. He also pointed out that the example tool I use in this post, "hamilton," is perhaps the poorest demonstration of the utility of CREST. Better examples are user management, sysctl, and firewall rules.

Why I am excited about Habitat

2016-06-19T20:02:00.001-07:00

Last week, Chef Inc. announced Habitat a new tool for packaging and deploying applications. It is hard for me to overstate how excited I am about this new project. Let's start w/ a little backstory. I have been involved with the Chef community since ~ 2010 but not actively involved over the last couple years. The primary reason for this was that the Chef tooling was less and less a good fit for my daily work.

My primary client over the last couple years needed a highly reliable, deterministic system where the provenance of all code deployed to production had to be verified and tracked. Let's call this Project X to protect the innocent. That is, we had to know where all the code came from and prove that it had not been tampered with. This meant that pulling gems off of rubygems.org or pypi was a non-starter. Even pulling these resources from private mirrors was problematic as it we still had to put in place a mechanism that documented the hashes of all rubygems, pip packages, jars, tarballs anything else to be used prior to actually deploying anything. We needed a deployable, accountable unit of software. "Docker!" you say. "Nay!" I reply. Docker is a great tool but it only provides a storage unit. It has no mechanisms for code provenance or authentication. A Docker image gives you is an opaque blob opaque blog of bits without any information about where they came from.

Needless to say, Chef was a poor fit for Project X. Chef leaves all configuration up to the very last moment possible, the time of deployment. This gives you a lot of flexibility that should not be taken for granted. Combine this with the infinite flexibility of Ruby and you have a wonderful tool for gluing together systems. This flexibility comes at a cost however, it makes the system overall fairly non-deterministic and a really bad fit for clients that require code provenance and tracking. Back on Project X, I spent some time researching the Nix Package Manager. If you are not familiar with the Nix package manager, it has a number of neat properties that address Project X's requirements and more. The tooling around Nix, however, left a lot to be desired as did concerns about long-term adoption of Nix. Sadly, I cannot remember the specific concerns the team had about Nix tooling.

Rewind to last week's announcement of Habitat. Habitat takes a lot of inspiration from Nix as far as I can tell, and from Consul as well. There are a few nix features missing from Habitat. The big one is that Nix packages use cryptographic hashes to verify the current package and all of it dependencies. This gives you a nice way to track not only the current package but everything it depends on. As far as I can tell, Habitat does not give you a way to say "I want core/glibc/2.22/RELEASE/CHECKSUM". You can only specify "core/glibc/2.22/RELEASE". Further, you can use the use the cryptographic hash of a deployed package to determine all of its exact dependencies.

One thing I really like about Habitat is that it provides many of the advantages of containers w/out forcing you to use them. You can get many of the touted advantages of containers minus the complexity with a simple Linux chroot. A Habitat package, once installed, is essentially a chrooted application. Docker et al make most sense to me when you are working on top of a system like Kubernetes or Mesos. W/out that tooling in place, you can be in for a lot of pain and needless complexity. However, if you are on the Docker train Habitat supports exporting packages to Docker and the ACI format.

I still have a lot of open questions.

How will Habitat work on Windows? Will it be possible to isolate packages w/out the chroot facilities that Linux has?
Does Habitat obsolete Chef or should they be used in concert? I suspect the former.
How does one set global settings like firewall rules and sysctls?

On this last point, Adam Jacob told me that global settings might best be handled by a separate Habitat service.

In conclusion, I am very excited about Habitat and I think it is worth your time to take a look.

P.S. I am very happy to see Habitat is written in Rust. Having spent part of the last year learning Go, it is very exciting to read source code that has dangerous features like pattern matching, generics, and monads.

P.P.S. After some time away from the Chef community, it is great to see many of my old friends in the Habitat Slack channel. Ohai!

Give your CLI Tool a ReSTful API

2016-04-16T08:45:00.001-07:00

I have some rough ideas about how to use some of the concepts of ReSTful APIs to improve command-line tools. I do not have a lot of expertise in building ReSTful APIs. I would love your feedback!

Lately I have been working with Azure's new Resource Management (ARM) API and been quite impressed by its behavior. Mind you, many other ReSTful APIs may behave similarly but Azure's is the first I have had similar experience with. A neat aspect of Azure's ARM API is that it is mostly asynchronous and how it lets you monitor the status of a long-running operation.
Let's say I want to create or update a VirtualMachine (azure parlance for an EC2 instance). I would take the following steps:

Build up an object describing the virtual machine to be created
Submit HTTP PUT request, with the serialized virtual machine object in the body of the request
Receive back a response w/ a status code and json document representing the virtual machine if the initial request was successful.

This response comes back more or less asynchronously. Azure sends back a 204 response if the request was accepted, i.e. passed validation and can be fulfilled. The only blocking part here is the validation of supplied parameters. The returned json document contains a property 'status' that indicates the provisioning status of the new virtual machine. This status will initially be "InProgress" and will eventually become "Succeeded" once the virtual machine has started. I can send Azure GET requests for the VM to check for the "Succeeded" status. HTTP's PUT method is particularly interesting because it is an idempotent operation.

Now let's look at deleting a VM.

Submit a DELETE request for a given VM
Receive a response that indicates whether the request was accepted and a URL to query for the status of the operation
Query the status url and receive a json document that indicates the status of the delete operation. The status will be "Succeeded" once the deletion completes.

The status URL is only valid for a limited period of time after which GET operations will return a 400.

What I really like about this ReST API is that I can execute fairly complex operations that complete (mostly) asynchronously. Could we create CLI tools that operated in a similar fashion?

A common task that I automate for cloud instances is the formatting and mounting mounting of ephemeral and longer-lived nfs volumes. This task can be fairly complex as it can involve LVM volumes, software RAID (mdadm), and possibly an encrypted filesystem. This operation task can also take some time to complete depending on the sizes of disks involved and the filesystem used. For example, it is not uncommon for this task to take several minutes when using ext4. I currently use a fairly complex Chef recipe to handle this task. It would be very interesting if we could use an external cli tool to handle this operation for us. Let's call this imaginary tool "cvol" for cloud volume. There are a lot of knobs and switches when it comes to managing filesystems so it is nigh impossible that I would be able to write all the possible options nor want compose a shell command that uses them. Rather, let's just pass cvol a json document describing the operation and including all the relevant parameters.

This document describes a volume mounted at /mnt/data that uses all ephemeral volumes in a RAID 0 configuration. Additionally, the volume is encrypted and formatted w/ xfs.

{ "mountpoint": "/mnt/data",
   "use_all_ephemeral": true,
   "raid": 0,
   "encrypted": true,
   "encryption": {
       "password": "123456789",
   "format": "luks"
   },
   "format": "xfs"
}

$ cvol create-or-update -i format_my_disks.json

cvol would respond immediately w/ a json document. That doesn't mean that cvol is finished. It would have to fork itself and detach from the session to actually do the work.

{ "status": 204,
   "operationId": "123456",
   "port": 9999
   "state": "started"
}

To check the status of the operation

$ cvol get --id 123456 --port 9999

{ "status": 200,
   "operationId": "123456",
   "state": "succeeded"
}

`cvol get` would tell us the status of the operation. For this to actually work, our first invocation of cvol would have to be still running in the background with and embedded http server for some finite period after the operation has completed. cvol would not run indefinitely in the background as that would be a crazy waste of resources. Let's say it stays in memory for 3 minutes after the operation has completed. What gets tricky is if the cli tool dies mid-operation. How would we know? I don't have a good solution for this.

Let's look at cvol's shred operation. This operation not only unmounts a volume but also securely
deletes all files stored there. Shredding a volume can take quite while.

$ cvol shred /mnt/data --iterations=10

returns

{"status": 204,
"operationId": "0913256",
"port": 8777,
"state": "started"
}

To check the current status:

$ cvol get --id 0913256 --port 8777

After the shred operation completes, `cvol get --id 0913256` returns a response with the property
"succeeded".

Making our cli tool ReSTful adds a significant amount of overhead. It requires embedding an http server into the tool and forcing the cli to use at least two threads; one thread to answer requests about the status of an operation and another to do the actual work. Why go to all this trouble? The primary reason is that it might make it easier to create higher-level building blocks for managing systems.

Chef is Awesome but Dependencies Suck

I have been a happy Chef user for over six years and I still think it is the best configuration management tool out there. I do not, however, think it is the one true configuration management system for all time, forever. There were many before it and there will be many after it. Further, ruby dependencies drive me nuts (as do Python and Node.js dependencies). Having to pull dependencies down from Ruby gems (or a mirror) and the compiler toolchain in place to build them is a continual headache. Having a terrible deployment story is somewhat endemic to dynamic languages. This is frustrating because my configuration management tool should be easier to deploy than my application code.

"But, c'mon just use Ansible," you say. "No server-side deps!" Ansible definitely is an easy tool to get started with but I would never, ever provision 1,000 EC2 instances with it for reasons of performance. I have done that with Chef and it has worked flawlessly. Further, you hit some pretty hard limits on code reuse early on. Writing the equivalent of cvol in Ansible would be many times longer and bug-prone. And again, that code would live and die w/ Ansible.

With some discipline, you can write a Chef LWRP to the handle complex tasks like the kind cvol would w/out using requiring gems external to Chef. My concern here is that this Ruby code will live and die with Chef. We won't reuse it when the new CTO decides to dump Chef for a hot new configuration management tool written in Haskell.

ReSTful cli tools would be easy to integrate with Chef, Ansible, and whatever other CM tool comes down the pike.

Implementation Ideas:

A ReSTful CLI tool would ideally be a single static binary. Obviously, Go would be a good choice for this but Python tools compiled w/ PEX or PyInstaller would also be viable options. The CLI tool would require an embedded HTTP/RPC server to respond to GET requests. The tool would also require multiple threads in order to both respond to status requests and get actual work done. Supporting TLS would be a big mistake as it would add a lot of overhead. Instead, a ReSTful CLI application should only listen on 127.0.0.1.

Note that HTTP isn't essential here. We could easily use another RPC protocol. The important parts that I want to adopt are:

ability to describe an arbitrarily complex operation using a document rather than cobbling together an endless chain of command-line arguments and flags
Ability to execute idempotent operations with a cli like the HTTP PUT operation
Option to execute an operation synchronously or asynchronously

I have to admit that #3 could be pretty challenging. We could get a lot of value out of just #1 and #2.

Closing Thoughts

These are some very, very rough ideas and I would love your feedback on them.

Thanks to Milos Gajdos for reviewing a draft of this.

Notes:

For the Docker heads out there who believe we will all have immutable servers. Our application instances may be immutable however we will always have to configure the cloud instances or bare metal that host those docker images. That problem will not go away.

Scala meet DevOps, DevOps meet Scala

2015-03-01T11:49:00.001-08:00

Alternate Title: Scala Shines Where Go Stumbles

It's embarrassing that I haven't blogged in over 2 years. Let's hope I can change. In the last time we spoke, I had begun learning Java. Things haven't gone exactly as planned. I got diverted from that plan and spent quite a bit of time becoming reacquainted with Python and JavaScript. In the last four months however I have spent a lot of time on the JVM, just not with Java. In my $DAYJOB, I have had the opportunity to learn scala. While the learning curve was more than a little steep, I am really enjoying working in it and I think it is a perfect fit for many of the problems we in devops-land work on.

Scala the Language

Getting started with a Scala was a little rough. First off, I found the official scala documentation dense and bewildering. It is nothing near as straightforward as the Python documentation nor the official Java tutorials from Oracle. It reminds me more than a little bit of the Ruby documentation landscape. However, most Ruby newbs starts with the wonderful PickAxe book, freely available online. There is a wonderful counterpart to the PickAxe in Scala for the impatient. I have really, really enjoyed this book. Sadly, only the first 100 pages are available for free. The remaining 300 pages are well worth $30.00. Scala for the Impatient does have one ginormous fault, it doesn't cover akka! We'll see why akka is important later. I found a nice, concise intro to Akka and Futures in the Scala Cookbook from O'Reilly. It kind of sucks that you need to spend ~$60 for a decent introduction to the language.

While the documentation story isn't great, I was very happy to find that the Scala community embraces the REPL. When I code Ruby and Python, I rely heavily on Pry and IPython to explore APIs and libraries. Often as not, I use boto from IPython to interact with AWS rather than using the AWS console. I have learned HTTP and ReST basics using Faraday and Requests. Unlike Java or golang (last I checked), Scala works really well with the REPL in large part because everything is an expression, i.e. every line returns a value. Further, scala's build tool sbt has a wonderful command `sbt console` that starts a repl with all your dependencies available, including your own code. It works the same as running `bundle exec pry` or `bundle exec irb`

One of the complaints I had heard about scala is that the type system is very complex and hard to use. I haven't found that to be the case. In fact I have found the type system to be quite helpful in diagnosing problems and as a baseline form of documentation. That said I am still learning how to use the type system. I still haven't figured out what contravariants, existential types, structured types are. I do find Generics immensely useful as an end user. I love that I can use map, filter, fold, sum, etc. on virtually any type. To my knowledge this is not possible in Golang, and this saddens me.

Dear Gophers, is it true that to map over an collection of ints I need MapInt, for doubles MapDouble, for bytes, MapBytes, and so on? Please let me know if I am mistaken.

Here is a Python script that I frequently invoke in Python

conn = boto.connect_ec2()
reservations = conn.get_all_instances(instance_ids=instances) # instances in cmd line arg
for res in reservations:
     for instance in res.instances:
            print("Terminating instance %s" % instance.id)
            try:
                instance.terminate()
            except EC2ResponseError as e:
                print("Termination of instance %s encountered error
                      EC2ResponseError with "
                      "message %s" % (instance.id, e.message))

I assume the golang equivalent would look similar except that you would have to manually check for `nils`.

Here is how I would write the same in scala.

Ec2.instances(instancesIds).foreach { i =>
try {
      EC2.terminate(i)
    } catch { e: EC2ResponseError => s"killing instance _ failed"}
}

Buyer-beware, I haven't tested out this code yet. For any Scalaists, that may be reading, is there a scala-environment/tool that I could use for writing scripts like this?

I like type-checking. Sure, a good test suite will keep you safe as long you run it each time before you make your commit. That said, you know how many times I have visited a file after running the tests and accidentally injected an errant 'w', 's', or before committing? Far more times than I care to admit. Sure these things are caught in CI but I don't want to wait that long.

While I may like static typing, I LOVE regular expressions and internal DSLs.

I recently discovered that pattern matching and regular expressions are a match made in Heaven. check this out.

val pattern = "([0-9]+) ([a-z]+)".r
val pattern(num, item) = "99 bottles"
# result    num: String = 99 item: String = bottles

Fanboyism aside, what I do find challenging about the scala language is the sheer amount of goddamn syntax. There is just so much language to learn, so many features to grok. Like Ruby and the lisps, successful scala teams need to limit the scope of how much of scala they use to keep their projects from becoming unknowable messes. For example, in the Ruby world there are a lot of conventions on when and how to use monkey patches.

Sadly, Emacs support for Scala didn't work out of the box for me. There is the popular Ensime mode but I couldn't get it working in 4 hours. I hope to revisit it at a later time. I have been using the community edition of Intellij and it has worked better than I care to admit. You probably can code scala w/ bare Emacs or vim but it doesn't compare to my workflow w/ ELPy or the wealth of ruby modes.

The Community

My experiences with the community so far have been quite positive. I have gotten a lot of assistance from the fine people on the #scala IRC channel, particularly lmm, Tpolecat, kueller, and others. It not as warm and as fuzzy as the Ruby community, but well few communities are.

Where Scala fits in to DevOps?

Increasingly in DevOps-land, I find myself writing operations-related webservices and command-line utilities. I have been writing Chef cookbooks for quite some time now and too often I find myself adding log rotation via logrotate, exernal monitoring via collectd, metrics gathering via statsd, etc. One day I came upon the wonderful DropWizard framework which incorporated healthchecks, metrics publishing, powerful logging features into the application itself. When you think about, it really makes sense for the application to report its own health, gather its own metrics, rather than external applications trying to reach into its guts to figure out if its screwed up. Now that I am writing my own webservicces, I'll be damned if some poor sap has to pry it apart to see if it's working properly.

To put it succinctly, scala is awesome for building operations-friendly services. You absolutely can write great services in Dropwizard with Java, but with a heck of a lot more boilerplate.

On a recent weekend I created an Akka Actor that periodically wakes up, invokes a long running task. If it wakes up and the task is still running, the task is not invoked again. What I love about this this actor is that it responds to healthchecks! The magic here is that since the long operation runs in a Future, it doesn't block the actor from responding to health checks. I can deploy this app with just the JVM. To do the same with Python would require Celery + Redis or RabbitMQ. Also I don't know if there is a way for Celery workers to respond to healthchecks. If there is, let me know! I doubt it's possible because of uhh, you know, the GIL.

class BryansActor extends Actor with RegisteredHealthCheckActor {

  import context.dispatcher
  val id = number
  val period = 4000
  val tick = context.system.scheduler.schedule(400 millis, period millis, self, "tick")
  case class LongOperationResult(result: Double)
  var longOperationRunning = false

  override def postStop() = tick.cancel()

  def receive = {
    case "tick" =>
      if (longOperationRunning == false) {
        println("start long operation")
        Future[LongOperationResult] {
          val start = System.currentTimeMillis()
          Thread.sleep(10000)
          new longResult(System.currentTimeMillis() - start)
        } pipeTo self
        longOperationRunning = true
      } else {
        println("future still running, nothing to do")
      }

    case longOperationResult(runtime) =>
      longOperationRunning = false
      println(s"Runtime was $runtime")
        // Determine our health
    case GetHealth => sender ! HealthInfo("bryans-actor", HealthState.OK, "doing great!")
  }


}

There is actually a lot more to a great service, things like backpressure and circuit breakers but that's a whole other blog post. Dear Gophers, is there a golang library that supports actors? The closest I found in my brief research was go-kite.

Friends of mine know that I have long been an admirer of Erlang and more specifically the OTP framework. As far as I can tell, Akka has done a decent job stealing many of the great ideas from Erlang so that JVM schmucks like me can take advantage of them. Actors really fit how my brain works. I have written a fair amount of asynchronous code in javascript, w/ and w/out Promises and I never found them as easy to work with as actors.

In addition to the Akka DSL, I quite like the Spray DSL

import spray.routing.SimpleRoutingApp

object Main extends App with SimpleRoutingApp {
  implicit val system = ActorSystem("my-system")

  startServer(interface = "localhost", port = 8080) {
    path("hello") {
      get {
        complete {
          <h1>Say hello to spray</h1>
        }
      }
    }
  }
}

I am also itching to try out the Slick DSL which looks a little bit like the JOOQ library in Java.

Remember that how I said I love DSLs? I should have said that I love internal DSLs. I spend 90% of my day working in DSLs, some of which I'll decline to name. I love Chef for its internal DSL that gives me the full power of Ruby. I'll decline to name some other tools which I use that have external DSLs in YAML, JSON, or XML. I find these tools exceptionally easy to get started with but inevitably I find them very unexpressive and constraining. Often as not, I end up writing plugins in python/ruby/powershell for tasks as simple as list comprehensions.

So what sucks about Scala for devops? As an aside, at FOSDEM 2012 I asked a group of Rubyists why Ruby what 5 different popular testing libraries. Gareth Rushgrove answered "Because in Ruby there are 5 different libraries for everything!" So far this also rings true in scala-land. There are 5-6 different popular libraries working with JSON, for instance. Sadly, I have yet to find a single sane one for working with YAML after wasting a morning trying to get SnakeYAML to do what I want. Dear reader, if you can point me to easy path for working with YAML and Scala, I would much appreciate it.

Conclusion

On the whole, scala is harder but simpler. Rich Hickey of Clojure fame gave a great presentation entitled "Simple made Easy." The gist is that what's easy right now is just what's most familiar to you. Simpler and more maintainable solutions may take more initial investment. That mess of for loops may be easy to write, but it's more complex and thus expensive to maintain over time.

I feel that too many people bypass Scala and languages like it because they perceive them as too hard. I call this The Tyranny of 15 minutes. Too often in Software-land, people focus on technologies that they can get comfortable with in the course of their lunch hour. This pisses me off. It's one thing if you love Golang because it fits your problem space and your head but another if you chose it for your next project because you got through the Tour of Go in an afternoon. We're engineers folks, "it's too hard" is not a valid excuse. You'll be maintaining that code - or someone else will - 100x longer than it took for you to learn the language.

I have no idea how scala will fare in the programming language popularity wars. My crystal ball is very cloudy. I do think it is worth your time to learn. I hope to spend a lot more time with Scala over the next few months. I'll let you know how it goes.

P.S. This post may sound a little harsh on Golang. I believe that Go is a very productive and powerful tool. The virtues of Golang are also tradeoffs. I just hope people are of aware of them. Also, I am not an expert on Golang, so please correct any incorrect statements that I may have made!

P.S.S. I didn't mention Clojure once in this post and that's because I have yet to spend any time with it. If you have code examples for similar problems, I would love to see them.

You Should Start a Technical Podcast

2014-01-10T14:48:00.001-08:00

In summer of 2010, my wife and I moved from a Kathmandu, Nepal, where I had worked for a kickass technology startup to Rome, Italy, a city with practically no technology scene at all. We had moved for the sake of my wife's career with the United Nations. Prior to the move, I had worked in a variety of IT positions but once in Rome I found a job as Linux system administrator. While I was lucky to find to a job relevant to my career I felt very isolated. I stared listening to the amazing DevOps Cafe podcast, and my sense of isolation only got worse. In a quiet, cry for help, I started my own podcast FoodFightShow, focused on Devops, linux, and general software geekery.

That cry for help was heard! FoodFightShow now has over 100 episodes and over 125,000 downloads. I have to credit my cohost, Nathen Harvey, for a large part of the show's, success. I like to think that FoodFightShow has benefited my professional community of linux system administrators but I know that it has benefited me. The show has really expanded my professional network and given me an amazing peer group. Too often people discount the importance of a peer group, but I think it is very helpful for maintaining one's motivation and focus. Additionally, for whatever reason, I have found that a number of people accord my opinions with greater weight simply because I cohost a podcast. One of my show's sponsored paid for me to attend a conference in San Francisco where I met my current, super awesome employer. Most important of all, I have learned a lot by doing the show and really, really enjoyed it. I think that you (yes you!) should start a technical podcast of your own, be it about IOS development, FPGA design, robotics, or advanced HVAC.

There is a deep dark secret that all podcast hosts share. Hosting a podcast is the best way to get the smartest people in your industry to answer your questions for 40-60 minutes. Good luck getting them to do that at a conference, in a mailing list, or on IRC.

Podcasts are Super Fun

I love podcasts. I think they are a ridiculously easy way for people to connect in a personal way through a conversation or monologue, and share that discussion with a larger group of people. If you work at top-flight technology company, you are lucky enough to engage in these times a day. However, we don't all live in Silicon Valley nor work for Google or Netflix. Podcasts provide a more intimate conversation than you can get on IRC. In case, you do work at Google or Facebook, podcasts provide a great way to hear from people who work in your profession but are not in your immediate zone of corporate influence.

It's Easy

Back in the dark ages of 2010, it was pain in the ass to produce a podcast. One had to connect an audio editor to Skype, tweak it just right, and reshare that audio. Today, we have Hangouts on Air, which make it ridiculously easy to record a multi-person podcast and even engage a live audience. If you are too lazy to synthesize the audio portion into an MP3 files, the youtube video stream can suffice.

How to Do It

Let's assume that I have convinced you to start your new podcast on Windows Kernel Development, school psychology, or .. whatever! I have got some highly-subjective tips for you. Your Mileage May Vary.

Publish early and often
Content is King
Pick a format and stick to it
Use Audacity
Hire an editor
Recording happens when you and the guest show up
Reach beyond the "usual suspects"
Avoid "Inside baseball"
Keep it family-friendly
Don't talk trash about groups within your professional community

The hardest part of doing a podcast is to keep it going. Most podcasts don't make it past a few episodes. Don't put a lot of time upfront into developing into learning Audacity and developing an awesome logo and theme song. Focus on the recording episodes and iteratively improving the audio quality as you go along. The audio-quality of the first 10 FoodFight episodes was absolutely, or "unlistenable" as Greg Karekinian told me. We have steadily improved over time. I think it's absolutely fine to compromise on audio quality. Publish early and often, make it a habit.

While the audio-quality of our show hasn't always been greatest, I have always paid a lot attention to the technical content of our show. From the very beginning I drafted fairly detailed script for each episode to ensure we didn't descend into random banter. I never started a show without a pretty good idea what I wanted to cover. As John Willis of DevOps Cafe told me, "Content is King." Nerds won't care if you stutter or aren't ridiculously eloquent. They will lose interest if you only talk about last night NFLBAMLB game or about your children. They will lose interest if the show if you don't consistently talk about the opportunities and challenges in your technical domain.

Damon Edwards of DevOps Cafe recommends that you pick a format for your show and stick to it. I strongly second this. Damon elaborates

Decide upfront (or as early as possible) on a format and stick to it. Short discussion segments? Quick hit news recap? News discussion? Short form interview? Long form interview? Tip of the day/week? Technical Q&A? Whatever it is, decide now on work to that. It will drive everything from content decisions to editing (if at all). If you want a new format, make a new show.

I recommend you use Audacity to edit your episode, not GarageBand. Garageband ate my recordings multiple time and wasted my time in a multitude of other ways. Audacity may have a bit more of a learning curve but it is reliable.

In order to better focus on your technical domain, I recommend you pay someone else to edit your shows. Editing audio tracks is not much fun and distraction from recording or planning another awesome episode. I highly recommend TheRubyRep who does great work for a very reasonable price.

Scheduling a podcast that involves multiple people can be challenging, particularly if you have a panel of guests, as the FoodFightShow does. We use doodle.com to plan our shows but the prevailing rule is that the show happens whenever the organizing host and primary guest can make it. You can easily chew up a lot of in scheduling coordination that would be better spent actually recording a podcast.

In any professional community, there are a number of who present themselves very well and appear frequently in a number of community forums. I call these people "the usual suspects." I strongly encourage you to recruit anyone who you think has a lot to say, regardless of how much experience they have speaking publicly. I had to twist the arm of Miah Johnson to appear on our show. She had little prior experience speaking publicly and didn't feel very comfortable doing it. But she did have a wealth of experience to share. After her first appearance on the show, one listener wrote in "Thanks for having Miah on the show, she is the kind of engineer I aspire to be."

Another thing I like to avoid is what is referred to as "Inside baseball". While it's great to discuss the minutiae of your profession it's not so exciting to discuss the minutiae of the more well-know people and their personalities or activities. It is very easy for a panel-oriented podcast degenerate into a group of friends talking about their friends. While this can be fun if you happen to be a member of that small group of friends, it's not every interesting otherwise.

Another thing to avoid is cursing, sexist language, or discussion of politics. In any given technical profession, the common thing that binds together your listeners together is their shared professional interest. I know that our podcast has listeners of all religious faiths, political stripes, and genders.

On the off-chance that there happen to already be a significant number of podcasts in your professional area, I still recommend you start a podcast. In my own area, there are 6-7 great podcasts yet there is still so much that doesn't get discussed. Just as importantly, you have opinions and the Internet needs to hear them!

I have left out a crucial piece of information. How much time do I spend on my bi-monthly podcast? Roughly 5 hours per month. That's a small investment when you think about it. That's all for now. Happy Podcasting!

Why I'm Learning Java

2013-11-17T13:51:00.002-08:00

I spend most of my days writing either Python or Ruby. I enjoy programming in these languages but am painfully aware of their faults. The single process concurrency limitation in both sucks ass. I love that Python is straightforward and predictable but have to admit that I prefer Ruby because I can always bend Ruby to my needs while I sometimes have to change my thinking to fit Python. There are several things about both Ruby and Python that annoy me and make them a poor fit for some of the problems I deal with. Problem #1 is that I need serious concurrency. Python's multiprocessing module doesn't cut it. Problem #2 is that neither is a great fit for writing small predictably performing daemons, though Python is a little better at this task. Problem #3 is that I am sick of spelling errors causing my code to fail. I need static typing or something to reduce the need to run my code just to know if it is syntactically correct.

It is high time for me to expand my skills and learn a new programming language, a process which I don't take lightly. It may take only a few hours to learn the basic syntax of a new language but in my experience it can take quite a while to comfortably program idiomatically in a new language. Adapting one's development environment to a new language, learning the standard library, becoming familiar with certain popular libraries, etc. can take a good chunk of time. I believe it took me a year from starting with Ruby to actually programming idiomatically in it.

So how would one select which language to learn? For me the most important factors are, in order of importance:

Which am i most likely to use at my day job?
Which is well-suited to kinds of problems I am interested in and work on professionally?
Which language are many of the open-source projects I work with regularly written in?
Which language is key to the open-source projects that I want to work with but currently don't?
Which language is a joy to work in?

Based on these factors, I have decided to learn Java. Wait, WAT? If I had written this post a few years ago, this list would have been inverted. If I were to pick a programming language solely on point #5, here is the list I would come up with, listed in order of interest.

elixir
erlang
Go Language
Scala
Clojure

Notice that Java does not appear on that list. However, the language that I am currently learning is Java and not the cooler, sexier languages in that list. Elixir is freaking awesome because it promises much of the developer happiness that I get from developing Ruby but with all the power of the Erlang VM and OTP library. Erlang is brilliant for a number of reasons, including the aforementioned OTP. I love Golang for developing small services and command-line utilities though it doesn't offer the me the flexibility and freedom that elixir or Ruby do. Finally, Scala and Clojure are both ridiculously interesting with their functional approaches.

Now back to the present day. In his great post Why Sysadmins Can't Code, Ben Rockwood advises that one of the best ways to learn a programming language is to attack a problem that you personally have. Since I have a very demanding 4 month old baby and a pretty darn intense day job, there is preciously little time for hobby programming in my life. I barely even have time for my podcast FoodFightShow. We don't use Erlang, elixir, Golang, etc. in my dayjob nor is there any immediate plan to. On a daily basis I do wrangle with java-based tools such as Cassandra, Tomcat, ZooKeeper, ElasticSearch, and a few others. Per Ben's advice, the problems I personally have and can actually allocate time to on a consistent basis are related to my day job, which primarily revolves around java-related tools.

I had fought learning Java for a long-time because it is ugly, verbose, and the Emacs support has historically been pretty lousy. Did I mention that I hate Eclipse? That XML-based DSLs suck ass?Lack of closures and mixins? However, these aren't good enough reasons not to learn Java. Luckily, Java 8 finally includes closures with Project Lambda and mixins with extension methods.

While there are a number ops-related applications written in Java, Golang is coming on strong. Derek Collison, one of the key developers of CloudFoundry, famously tweeted "Golang will become the dominant language in Iaas, Paas, and orchestration in 24 months." He wrote that roughly 14 months ago and arguably Golang has made impressive progress toward that goal in the meantime. So why not learn Golang hoping that day job eventually sees the light? I feel that despite all the golang excitement, the Java behemoth has incredible momentum. Projects such as the Berkeley analytics stack, Hadoop, and Cassandra keep innovating at a breakneck pace. Companies such as Twitter, Netflix, LinkedIn and others keep kicking out awesome tools, all based on the JVM. Further, a new language like Golang is most likely to be first applied to the areas where it is best-suited. We don't really know how well golang will serve the needs of application servers, large dataprocessing apps, and other niches. It could serve those use cases incredibly well . . . or not. We'll have to see.

NetflixOSS - The Quiet Tide of JavaOps

While HackerNews keeps talking up the virtues of Golang (which are many), Netflix has done an incredible job of releasing kickass operational tools that mostly happen to be JVM-specific. We did an awesome Foodfightshow episode with Adrian Cockcroft and Jeremy Edberg. The venerable ShipShow also did a great episode with Netflix that you should listen to. The NetflixOSS stack deftly integrates metrics, monitoring, backpressure, and dynamic configuration in a way that makes the resulting applications very easy to support. Their libraries reuse common Java tooling such as JMX, tomcat, slf4j, and others rather than reinventing everything from scratch. JMX, slf4j probably couldn't work on a run-time like Ruby MRI or Python's CPython as they require robust threading support that doesn't block the main thread of execution.

You could possibly implement the kind of monitoring, dynamic configuration, and other properties that NetflixOSS runtime utilizes in Golang, however you probably would have to implement these features from scratch in Golang while they are stable and production-ready in java. Go users, please let me know if I am mistaken.

The Runtime's the Thing

What really makes java truly compelling is the Java Virtual Machine. There are many elements to a language runtime but the one's I deal with on a daily basis are the thread scheduler and garbage collection algorithms. Effectively managing parallelism in a programming language requires a sophisticated scheduler inside the language itself. Such a scheduler must be able to efficiently allocate threads while maximizing CPU-level cache efficiency. That's a tall order especially across ARM, x86_64, and NUMA architectures. A dumb scheduler will simply place your threads on whatever CPU is available, regardless of where that thread was last running and irrespective of NUMA requirements. This 'cache thrashing' can actually make multithreaded programs run slower on multicore machines than on single core ones. You could just run tons of node.js daemons and push this problem down to the operating system. This does work to some extent, however you have a ton of extra overhead switching process-level contexts. And don't get me started on what a nightmare it would be to share some minimal state between 1000s of node.js daemons. While you may not need to share state between stateless webapps, one does need to do that if you are coding up a new database or complex batch process.

The programming language scheduler needs to make a number of optimizations to accommodate different cpu architectures. Developing such a scheduler is probably a very expensive and time-consuming process. I guess conservatively that such a scheduler requires at least several million dollars worth of software engineering to develop. Even if Ruby and Python removed their global interpreter locks tomorrow (which would break most C extensions), I doubt any company will step up and invest the money needed for a proper thread scheduler in those languages.

So this leaves us with the few programming languages for which corporations are willing to sponsor the continued development of the language runtime. To the best of my knowledge that leaves us with Java, Erlang, maybe Haskell, and the languages that can run on top of those runtimes.

Another important aspect of a language runtime is the garbage collection algorithms that it supports. I find it ridiculous that it's already 2013 and there isn't an open-source pauseless garbage collection algorithm for OpenJDK yet though it is heartening that RedHat is working on one. It really sucks having to limit your Java heap to 8 GB on a 64 GB box. Projects like HBase and Cassandra get around this by using off-heap caches, but it adds no small amount of complexity and significant runtime overhead. This presentation from Neil Ferguson gives a great overview of how many Java projects maximize RAM usage. I am extremely curious if there is work on a pauseless GC implementation for Golang. Go users, please chime in!

Applications and Libraries Trump Language Features

Java doesn't have pattern matching like scala, elixir, or erlang. If you are unfamiliar with pattern matching, you should become so as it is fucking awesome. Nor does Java have hygienic macros like Clojure or Elixir. However it is the source language for ZooKeeper, Metrics, and Cassandra, all applications which I really, really enjoy working with.

Next Steps

I am slowly working through Cay Horstmann's Core Java series. I have set up my Eclipse environment and customized it with the excellent Emacs+ plugin that makes it bearable. I intend to first become comfortable with Eclipse as it is the most popular one in my office so I have plenty of people to ask questions of. Longer term I would really like to use something like eclim with emacs. Eventually, I hope that my java knowledge could be a useful springboard to learning some actually fun languages such as Clojure and Scala.

Summing up, I am learning Java because I have the opportunity to learn it on my day job during a period in my life where I have minimal free time AND because it is the source language for some really compelling open-source projects that I would like to contribute to. In a perfect world we would all be hacking on Elixir or Haskell but alas, the stars are not currently so aligned.

Please don't read this blog post as my statement on which programming language is *best*. That is a meaningless measure. Obviously the coolest programming language on the planet is Elixir but the best programming language for Bryan Willson Berry, originally of Whittier, CA (the nicest part of East Los Angeles), to learn at this moment in time is Java.

SysAdmin Productivity and Chef

2013-02-10T22:39:00.001-08:00

Inspired by Nathen Harvey's "Learning to Automate" presentation, I came up with this graph of system administrator productivity. Enjoy

How to Write Reusable Chef Cookbooks, Gangnam Style

2012-11-10T11:53:00.002-08:00

I am at pains lately to explain to others how I think Chef cookbooks should be developed. There are only a few good blog posts on the subject, most notably Peter Donald's blog posts "Evolving Towards Cookbook Reusability" and Reusable Cookbooks Revisited and Jamie Winsor's "Authoring a Chef Cookbook as a Developer." These articles have influenced me significantly, as well Jamie Winsor's appearance on the FoodFightShow podcast. Both Jamie and Peter urge toward creating generic, attribute-driven cookbook wrapped with a policy-driven cookbook. That's a complex topic that we will come back to later. Jamie recommends something similar with his distinction between "library" and "application" cookbooks. Jamie has also convinced me that Chef's Roles are to be avoided. Some people alternately call this pattern "library + application" or "generic + wrapper." For lack of a better name, I call this way of cooking infrastructure Gangnam Style.

If you look closely at this image, you can see that Korean rapper PSY is subtly sending us a message about Chef cookbook development.

Why are roles bad? Why do you need to write two cookbooks rather than one? Well, this all boils down to cookbook development workflow. I start by developing locally on my workstation using Vagrant, Berkshelf, and Chef Solo. I find using Chef Solo with a local Vagrant VM much faster than working with a Chef Server. This has the added benefit that you will be able to share your library cookbooks with people outside of your immediate organization. This can help a lot when you look to the community for help fixing a thorny issue. Now you can use roles with Chef Solo, but in my opinion, it is a very bad idea. You will have two copies of a given role, one copy in a given cookbook and the main copy in your chef-repo code repository. Syncing the two will introduce errors and complexity unnecessarily. Roles are also a big fat global variable across your entire infrastructure. You can't version them. You can't roll out a new role incrementally. Changing a role instantly affects every node that uses it regardless of its environment.

It is critical to use a transient virtual machine for cookbook development. You can't really be sure that your cookbook works properly until your delete your virtual machine then reprovision it from scratch.

For the purpose of this article, let's say I am working on a PostgreSQL cookbook for Acme Corporation. I will fork the default PostgreSQL cookbook from Opscode and only make changes to that cookbook that I will be able to push upstream later. I will put all Acme-specific values and configuration into the acme-postgresql cookbook. In the past, I would have put the Acme-specific values into a role and then done my best not to tweak the PostgreSQL cookbook in any way that infected it with Acme-specific values that couldn't or shouldn't be upstreamed. Inevitably, Acme-specific logic and data would make its way into the library cookbook and the changes would never be upstreamed.

The Acme-PostgreSQL cookbook wraps the generic PostgreSQL cookbook with `include_recipe` and then overriding certain attributes and "editing" a few resources with chef-rewind (formerly named chef-edit). For instance, I may need to change the permissions of the main PostgreSQL configuration file because ACME's DBA team has a written policy that specifies the exact mode for that file. We can argue about whether or not the file mode is worth changing but that won't change the fact that ACME's DBA team is paying me an hourly rate on a short-term contract. If I don't fulfill their requirements to the letter, I won't get paid :). Next, I have to disable the PostgreSQL service because it is actually managed by Pacemaker.

Once I have most of the cookbook's elements in place, I begin testing it on my own laptop using Vagrant. Once I am happy with the state of the cookbook, I will deploy it to a server in the QA or Test environment. I am very strict about which cookbook versions are allowed by my environments. In case these two cookbooks are already in use, I make sure that I have incremented the cookbook versions in metadata.rb so they will not be loaded into a production environment accidentally. In the past, I would have to change a role used by both Production and QA servers. This was bad, very bad. Using a wrapper cookbook I can avoid this possibility entirely while keeping all of the benefits of a role.

Once I have deployed to a QA server, I monitor it for anything unexpected. In the case of my haproxy servers, I actually have written a set of integration tests using Rspec. If everything goes OK after a couple days, I will go increment the version of the wrapper cookbook allowed on the production environment to that of the new version and run chef-client on the target production servers.

Now what if we had chosen to use roles? Well then the deployment to QA would have been a Hell of a lot more nerve-wracking. Unfortunately, I have made the error in the past of adding an error to a base role. Luckily I caught it before it affected more than a couple servers.

So far I have explained how I distinguish between Library and Application Cookbooks but here is a table that I think will help you visualize those distinctions.

Gangnam Style Cookbook Pattern

Recently, there was an awesome Chef Hangout on Cookbook Reusability which you should definitely check out. Some participants said that editing resources with chef-rewind (it was called chef-edit at the time of the hangout) has a bad code smell, i.e. a pattern to be avoided. I argued then as I do now that this pattern is one of many tools we can use to make cookbooks reusable. I like it because it is the simplest thing that could possibly work. I could modify the underlying postgresql cookbook so that I can paremeterize those settings that I customize. In that case, I would still want to put the attributes for those parameters in the acme-postgresql recipe. For me personally, modifying a generic cookbook requires a lot of careful thought in order to avoid breaking functionality for users on other platforms. When I am pressed for time and doubt that I will get around to pushing my changes upstream, I prefer to use chef-rewind to override specific resources.

This bears more attention. In any open-source project, regardless of how well run it is, it is easier to pull upstream changes than push changes upstream. The team at Opscode does a great job of integrating changes in my humble opinion, but they have to make sure that any change you introduce doesn't break functionality for other users. For this reason, I recommend avoiding changes to the Opscode cookbooks unless there is a compelling reason to. This you way you will benefit from their ongoing maintenance and updates of the community cookbooks.

When should you only write one cookbook rather than two? Whenever you are writing something that is purely generic, you only need one cookbook. As soon as you start putting in application-specific data, I recommend splitting that application cookbook into a wrapper cookbook. I recommend this even if you never intend to open-source your generic cookbook. The flexibility you gain will save your ass later.

On the Cookbook Reusability Hangout there was a lot of talk of providing better abstractions and the need to provide general purpose LWRPs and DSLs for popular software applications. Adam Jacob made a great counterpoint with this argument "You could write a complete Ruby DSL for haproxy but would it be worth the engineering effort? I doubt it." Note: This is paraphrased. I tend to agree with Adam here, you could write a DSL for some configuration languages but sometimes it isn't worth the effort for the more complex ones. It especially isn't worth it if the DSL is more complex or harder to interpret than the original. Remember, the source configuration language may be a pain in the ass but at least it is documented and its problems known. The same likely cannot be said for your new DSL or LWRP. An LWRP has to be an order of magnitude simpler and more intuitive than the outputted source in order to be worth the effort.

I recently had the pleasure to work on a DSL for collectd plugins with Pierre Yves Ritschard. I really this DSL works because the collectd configuration language is ridiculously simple. Here is an example:

config = Collectd::DSL.parse do
  load_plugin :redis
  load_plugin :java
  load_plugin :disk

  plugin :disk do
    disk "/dev/sda"
    disk "/dev/sdb"
    ignore_selected "false"
  end
end

With collectd, the scope of configuring plugins is really quite narrow and provides an excellent opportunity for a DSL. This is where DSL's and LWRPs really shine: Providing conventions for either small problems or subsets of large problems. For example, I think it would be foolhardy to write a general purpose Ruby DSL for JBoss's XML configuration which covers tens of different J2EE service profiles from messaging to mail to clustered caching. I do believe, however, that it would be feasible to write a DSL for the most common set of configurations for JBoss's Web Profile.

Let's step back for a second and think about how we would use the collectd DSL from the above example. We could add a recipe to the collectd cookbook collectd::acme_plugins that uses that DSL for our custom-configured plugins. However, we would be far better off creating a new cookbook acme-collectd with the recipe named plugins. Whether or not you choose to use roles, wrapper cookbooks give you a lot of flexibility.

I too would love to parameterize all the cookbooks and LWRP all the things but we need to make sure that our abstractions are significantly easier to use than the source configuration languages. We also need an array of strategies to make cookbooks more reusable rather than one true path. In this post, I strongly advocate the Gangnam Style pattern to wrap a generic cookbook with one which holds application-specific logic and data. Whatever whacky patterns we come up with, we will never move forward as a community (and profession) if we do not succeed in making cookbooks reusable.

P.S. Nathen Harvey helped edit this post
P.P.S. Thanks to those who participated in the Chef Hangout on Cookbook Reusability which greatly stimulated my thinking on this topic

Rewinding Chef Resources

2012-11-08T01:01:00.002-08:00

Note: 11 Nov 2012, the chef-edit gem was renamed chef-rewind

Earlier this week I realized that I misunderstood how duplicate resources are handled. When you use a resource twice, it actually is actioned twice, not once. The second instance will inherit attributes from the first instance but not its action.

For example,


# jboss/recipes/default.rb


template “/usr/local/jboss/standalone/standalone.xml” do
  source “standalone.xml.erb”
  owner “jboss”
  mode “0755”
  notifies resources(“service[jboss]”), :restart, :immediately
end





# esb/recipes/default.rb

include_recipe “jboss”





template “/usr/local/jboss/standalone/standalone.xml” do

    source  “prod-standalone.xml.erb”

    owner “esb”

    notifies   
    resources(“service[jboss]”), :restart, :immediately

end

What actually happens here is that the second template inherits the “mode” of the 1^st template, but beyond that, itis a separate resourceservice restarted twice, on every chef run. This is a big problem.called “chef-rewind” and it adds an addition function to chef for rewinding an existing resource so it is only actioned once. I packaged it as a ruby gem and not a cookbook. You can install it using `chef_gem` When you run chef, the file will be template twice and the I wrote a ruby gem last night that should fix this.


# jboss/recipes/default.rb


template “/usr/local/jboss/standalone/standalone.xml” do

    source  “standalone.xml.erb”

    owner “jboss”

    mode   “0755”

    notifies   
    resources(“service[jboss]”), :restart, :immediately

end


# esb/recipes/default.rb

include_recipe “jboss”

chef_gem “chef-rewind”

require ‘chef/rewind’


rewind :template => "/usr/local/jboss/standalone/standalone.xml” do

    source  “prod-standalone.xml.erb”

    cookbook_name "esb"

    owner “esb”

end


Pay special attention the `cookbook_name` attribute. Without it, the resource will look for it in the jboss cookbook.

This is very alpha right now and I will be testing it today. Feedback would be very much appreciated.

A Year with Chef

2012-10-20T12:50:00.000-07:00

It literally snuck up on me the other day that I had written a year ago a blog post entitled "A Month with Chef." With the Opscode Community Summit happening this coming week, I think it might be time for a few reflections.

It is a little staggering to think about how use of Chef has impacted my work over the last 12 months. In large part because of Chef I was able to roll out over 130 virtual machines in a tiny fraction of the time that it took me to do with my bespoke python and bash scripts. I also rolled out comprehensive monitoring with collectd, statsd, logstash, and graphite for all of my VMs. I simply couldn't have accomplished that without Chef. I still don't have full JVM monitoring but I am working on that. Lastly, Chef has kept my job interesting. I love being a sysadmin but often this job can be tedious. Chef alleviated that tedium to a large extent.

Lessons Learned

So that's what Chef has done for me, but along the way there was a whole lot learning, mistakes, and learning from mistakes. There are many lessons that stick out most in my mind but a couple more than the others.

First off, follow Jamie Winsor on github and follow the workflow that RiotGames team has come up with. Jamie and company folded many of their best practices in the Berkshelf tool. I have experimented with a number of different workflow tools but The Berkshelf Way works best for me. Jamie wrote up an excellent guide on how to develop application cookbooks. Read it and live by it. I had already come to some of the same conclusions but many were new to me. I had the pleasure of interviewing Jamie for the Food Fight podcast. If you are into Chef or just curious, you should definitley give a listen.

The next most important lesson that you should really use pry in order to learn Ruby and Chef. Pry is an improved read-eval-print-loop for Ruby that will help you inspect, explore, and understand Chef. It is irb on mega-steroids. Pry isn't something you use once you are more advanced with Ruby but rather I recommend pry to learn Ruby from scratch. Go watch Josh Peek's 10-minute intro video _now_. Debugging a recipe is as easy as inserting require 'pry'; binding.pry . With pry-nav, you can easily step through your running Chef code.

Here are the rest of my lessons with the ones recommended by the _Berkshelf_Way_ denoted by (BW)

Roles are bad. Don't use them. Roles make it hard to collaborate on and test cookbooks in a distributed manner. Use a wrapper cookbook instead (BW).
Use wrapper cookbooks to add information and settings specific to your organization to a more generic cookbook. For example, use a acme-postgres cookbook to override the default values and add corporate-specific policies to the mainline postgresql cookbook (BW).
Use chef-solo when developing cookbooks even when you will use chef-server to deploy them. It will give you far faster feedback (BW).
Every cookbook must live in its own git repository.
Whenever possible, add tests to your cookbooks. It will save your sanity, seriously.
As a sysadmin, you won't be able to keep your cookbooks up-to-date without getting your developers involved in writing them.
You can't automate something you don't understand. You can't write a good postgresql cookbook when you know nothing about postgresql. You can write a lousy postgresql cookbook for a learning experience, but throw it away and start fresh once you know what the Hell you're doing.
Ruby is a lot of fun, see earlier point on pry.
The amount of instability in gems and the Ruby VM itself often drive me to the brink of insanity. The amount of effort that it takes to get Ruby 1.9.x installed on an arbitrary machine is ridiculous.
Update your cookbook's version in metadata.rb with every significant change.
open-source your cookbooks whenever possible. You will get valuable feedback that will improve their quality and make them more secure. Further, interacting with the larger open-source community can vastly improve your skills.
Follow Andrew Crump on Github. He is author of Chefspec, foodcritic, and a key contributor to minitest-chef and test-kitchen.

What is Needed Now

More than anything, Chef needs a comprehensive guide book. I know that Stephen Nelson-Smith is working on the Definitive Guide to Chef. Unfortunately, the release date for that book is uncertain. James Turnbull and Jeff McCune did a kickass job with Pro Puppet and Chef desperately needs the equivalent.

Here are some additional things that are needed:

We need a lightweight orchestration tool that works when managing several vagrant vms and to deploy a cluster on vmware. Rake tasks don't cut it and Apache Zookeeper is definitely not lightweight.
Some kind of maintainership program, culture, or outright cult to encourage people to maintain cookbooks for a variety of operating systems.
Better testing tooling. Test-kitchen is a good step but much more remains to be done such as better integration with Continuous Integration systems and support for Berkshelf.
Make it easier for newbies to get started w/ Ruby. This is the biggest hurdle for newbies. ruby 1.9 is easy to work w/ _once_ you get it installed. It really needs to be as easy as sudo apt-get install -y ruby19.

Chef Compared to Ansible, Pallet, and Salt Stack

I did write up a detailed comparison with Puppet but I have not given newer frameworks such as ansible, saltstack, and pallet the attention they deserve. Nor have I done a proper analysis of CFEngine 3. I have been remiss in that, mea culpa.

In Summary

The Chef tool and community are far more mature than they were 12 months ago. I expect to see exponential growth in the community and use of Chef itself. The hardest part of working with Chef is managing to keep up with the rapid pace of development and community contributions. Luckily, Nathen Harvey does a fantastic job of summing up the latest developments with his What's Cooking segment on the Food Fight Show.

To wrap up, the best thing about working with Chef for 12 months is that it has made me a better sysadmin.

Stash those logs! Set up Logstash quickly with Chef + Berkshelf

2012-07-02T00:25:00.004-07:00

I have been spending a lot of time lately with the excellent logstash tool. If you aren't familiar with logstash, it reads your logs files, can optionally filter and transform them, and can send them somewhere else for arbitrary purposes. Setting up logstash can be a bit complicated, especially once you add in elasticsearch as the backend. Luckily, there are some great chef cookbooks, from Karel Minarik and John E. Vincent, for both elasticsearch and logstash. I have extended both cookbooks for my purposes, with pull requests pending as of this writing.

I use logstash to monitor my haproxy front-end servers. Call me crazy but I like haproxy so much that I even use it do all of my front-end URL rewriting and redirects rather than using a more traditional tool like Apache or Nginx. At this time I only use Apache for my SSL termination. All actual HTTP serving is done by backend java application servers. Haproxy is in my humble opinion, more flexible, easier to configure, more performant than Apache except for one area. It is very easy to make apache send different kinds of logging information to separate log files. Haproxy sends all of its logging information to syslog and its up to you to break it apart. Enter logstash. As the great Lusis describes logstash.

Stuff comes in
Bits get Twiddled
Stuff comes out

For a good introduction to logstash, I recommend Lusis slides from Chefconf. I will only cover my immediate use case here. I need to add a logstash agent on each haproxy that sends them to a central logstash server that aggregates all the logs and makes them searchable by injecting them into elasticsearch.

Here is the basic configuration for the logstash agent on my haproxy servers

As you can problably intuit from shipper.conf, it reads from the haproxy.log file and sends it off to my main logstash_server using AMQP. I actually could read directly from syslog rather than the file. I just haven't gotten around to doing that yet. Then on my logstash_server I have the following configuration:

To actually install the agent on my haproxy server, I just add the Chef recipe logstash::agent to the run_list of my haproxy role, add a few attributes to the same haproxy role and voila! the agent is shipping logs. On the logstash server side, I add the following to my node's run_list plus a few odd attributes

    "role[elasticsearch_server]",
"recipe[logstash::server]",
"recipe[logstash::kibana]",
"recipe[logstash::haproxy]"

After my next node convergence you will see a pretty picture like this

Wow! This is the excellent Kibana interface created by Rashid Khan. You may have realized that it is a little more complicated to put this together than I have let on. Let me walk you down the steps of how I actually put it together. But first let's be clear what is happening here

1. Logstash agent on each haproxy server reads the log files and ships each individual entry over AMQP to the logstash_server
2. A rabbitmq-server on logstash_server receives the amqp messages and passes them to the logstash server
3. The logstash_server `groks` each entry, i.e. it parses the entry and labels named fields according to a pattern, in this case the HAPROXYHTTP
4. logstash_server puts the grokked log entries into elasticsearch so that they are searchable
5. You can use the pretty kibana interface to watch a live stream of log entries or query by any `grokked` field. Kibana queries elasticsearch directly using Lucene query syntax. It does not talk directly to the logstash_server.

As you can see, that's a lot of moving parts! Each moving part requires its own Chef cookbook. This ends up being quite a lot of cookbooks to manage. Enter Berkshelf, an excellent tool for managing cookbooks, written by the great team at Riot Games. We will use Berkshelf to do much of the heavy lifting for us. I will walk you through how to use berkshelf together with chef-solo to set up logstash.

Before we move on, you should know about the other "moving part" that everybody talks about, the awesomeGraphite. Many, many people use logstash to glean metrics from log files and send information on to Graphite where it becomes pretty graphs. This is also a large part of Lusis aforementioned presentation. Unfortunately, I haven't started with Graphite yet but once I do I will have another juicy blog post to share with you.

Setting up Berkshelf

It used to be a bit difficult to install Berkshelf, but now it couldn't be easier

   $ gem install berkshelf

How Berkshelf Works

Simply insert the following at the top of your Vagrantfile and Berkshelf will ensure that your cookbooks are uploaded to your vagrant VM

require 'berkshelf/vagrant'

To upload your cookbooks specified in your Berksfile to your chef-server, do the following

$ berks upload

This Berksfile instantly gives your team a mechanism to work off the same set of cookbooks. Back in the dark ages of Chef, Oh 9 months ago, we all kept our cookbooks bunched together in a single git repository. This caused all kinds of headaches if you actually wanted to push your changes to an individual cookbook back to Opscode or to someone else. Some months ago, Joshua Timberman split the monolithic https://github.com/opscode/cookbooks repository out into individual git repositories. This has made it a million times easier to collaborate on cookbooks but much harder to work on a common set within a team.

This scenario shows how Berkshelf can help a team collaborate on common set of cookbooks but we will use berkshelf for a different purpose. When working on a cookbook or set of cookbooks for an application, we don't need all of the cookbooks in the our top-level Berksfile at ~/chef-repo/Berksfile. Further, we may be collaborating with people outside your organization who don't need or want to see all your corporate cookbooks. For ease of development, each cookbook needs a description of its dependencies. For this reason we will also have a Berksfile for the logstash cookbook that describes its dependencies.

Here is my Berksfile for logstash.

Loading ....

Let's Get this Show on the Road!

Too much talking, let's get some work done. First, clone my chef-logstash repository.

$ cd ~/projects/
$ git clone git://github.com/lusis/chef-logstash.git logstash
$ cd logstash

It is critical that you clone the repository as logstash and not the default chef-logstash. Otherwise you will get a cryptic error message later on.

Next, let's fire up a vagrant VM using the Vagrant definitions that come with the logstash cookbook. I am assuming here that you are already familiar with Vagrant but if you are not, check it out here.

$ vagrant up lucid32

I also have included vagrant definitions for 64-bit Lucid and CentOS 32-bit.

This will take a while. Vagrant will fetch a vagrant box for 32-bit Ubuntu Lucid and then apply a set the set of Chef recipes to set up logstash. You should also see the result of some integration tests I have written for the elasticsearch cookbook using the minitest-handler.

You should see at the end of the run the output from the integration tests.

Finished tests in 8.395664s, 0.4764 tests/s, 0.4764 assertions/s.

4 tests, 4 assertions, 0 failures, 0 errors, 0 skips
[Mon, 02 Jul 2012 08:17:03 +0200] INFO: Report handlers complete
[Mon, 02 Jul 2012 08:17:03 +0200] DEBUG: Exiting

You may also see the a bunch of invocations of the `curl` command in red. The integration tests use curl extensively and curl outputs its progress to stderr. You can disregard those.

Next, fire up a web browser with the using the IP address of your vagrant VM, something like http://192.168.2.x. POW! There is Kibana, the wonderful web interface for Logstash created by Rashid Khan. No logs are displayed as we haven't yet put any data into logstash.

Lets go into more detail about how the logstash cookbook works. The main logstash configuration file consists of what are essentially ruby hashes.

Here is a sample logstash server configuration

Loading ....

Instead of writing an LWRP for logstash. I make it very easy to template the the logstash configuration using attributes provided by a role or a node. Here is a sample role for configuring a logstash_server that matches the previous logstash.conf file.

Loading ....

All you have to do is provide a ruby hash and the logstash templates will render them them into logstash configurations. There are more details in the README.

There is a lot more to the awesome logstash and its underlying dependencies like elasticsearch. Even if you are not looking to implement logstash any time soon, you should seriously take a look at using Berkshelf for managing your cookbook dependencies. Let's hope I find time in the next month to write about Logstash + Graphite.

P.S. Thanks to Jamie Winsor (Berkshelf) and Karel Minarik (elasticsearch) for their assistance in working with their respective projects.

Real Tests, Real Simple with Minitest, Vagrant, and Toft

2012-04-30T00:18:00.002-07:00

I have put a lot of work into my Ark cookbook lately and it has grown to such complexity that it my existing set of workflow and tools were not enough. I needed some way to sanity check my code. I needed tests.

You can't refactor with confidence without tests. I have fully rewritten ark several times already. After making serious changes, I then had to spend roughly 20 minutes manually testing the cookbook to ensure that it still worked the way I intended it to. This manual testing was getting tedious, really tedious.

Looking at my various options, I first took a look at Cucumber. There are some nice tools out there like simple_cuke, cuken, and ironfan-ci. I made some attempts at using simple_cuke but they failed fairly miserably. That's because I thought I could use Cucumber without having to actually learn it. Cucumber is a powerful and somewhat complex tool. To benefit from using Cucumber, I was really going to have to read the The Cucumber Book, something I don't have time for right now. I needed the simplest thing that could possibly work. Further, the real benefit of Cucumber appears to be the communication it facilitates between customer and engineer. In this case I am the client and the extra layer of communication is unnecessary.

Enter MiniTest

Luckily for me, BTM and Andrew Crump have been doing great work adapting David Calavera's chef-minitest-handler https://github.com/calavera/minitest-chef-handler. MiniTest is the simplest thing that could possibly work. It works by gathering all the files that match the path files/default/tests/minitest/*_test.rb for the cookbooks in your run_list into /var/chef/minitest/. Once your regular chef run completes, the minitest-handler cookbook executes each of those tests and displays the results.

Note: As of 30 April, 2012 there is a known bug in minitest-handler that requires you run chef-client twice in a row for the tests to actually execute.
Update: Set the network type to bridged and you won't have this problem config.vm.network :bridged

However, running in bridged mode may require sudo.

My ark cookbook is really just an LWRP. The default recipe does nothing more than install the unzip package on those *nix distributions that don't have it by default. I created an ark::test recipe that exercises all of the features of the ark resource. Here is a snippet of it.

ark/recipes/test.rb

Loading ....

Next, I have my actual MiniTest tests classes, scoped according to my recipe "test"

ark/files/default/tests/minitest/ark_test.rb

Loading ....

To run these tests I have to first add chef_handler and minitest-handler to the beginning of my run_list. Note that the minitest-handler-cookbook doesn't yet support chef-solo, but I patched it over the weekend so that it does.

Here is what the output of the tests looks like

I originally wrote my tests using MiniTest::Unit test cases. However, just a few days ago I discovered the awesomeness that Andrew Crump has built into MiniTest::Spec and a whole set of helper modules that really simplify the code you have to write.

Here is a snippet to test whether a file exists and has the correct owner

Loading ....

I especially love all of the resource matchers that let you connect to the actual resource objects created during your Chef run and to check whether their configuration matches the actual system state.

For some detailed examples, check out this example.

The Not so Simple Part

Ok, that was the easy part. The harder part is to run your tests on a clean environment. You need a relatively fresh VM to ensure that the tests are actually testing changes made by your cookbook and not some change made eons ago for another purpose. The simplest tool for this is Vagrant. Here is the Vagrantfile I use to test the ark cookbook.

Loading ....

Here are the commands you run to provision your new server using vagrant

Loading ....

Note when using Vagrant that bridging your VM's network to your wireless network connection is not a straightforward process, at least not on my linux laptop. If you run linux as your primary OS, I recommend you spare yourself the pain of trying to bridge your VM to wlan0 and just stick an Ethernet cable into your laptop.

While Vagrant is an awesome tool it is not the best tool for all situations. One issue I have is that my corporate workstation is a very locked down Windows Vista installation. I do virtually all of my work within an Ubuntu virtual machine running on top of Virtualbox. Vagrant uses virtualbox to create new virtual machines and there is no mechanism to run virtualbox within virtualbox. Earlier this year I attempted to get ruby and vagrant running on Windows Vista but after 4 hours I didn't even have ruby running on Vista.

Another issue with Virtualbox is that it's flexibility comes at a cost. Virtualbox runs entirely in userspace which means it is less performant than other virtualization tools such as KVM and linux containers (LXC). This performance lag could become very noticeable once you try to spin up a 4-node postgresql cluster. Of the three virtualization technologies mentioned here, LXC adds the least overhead as each container shares the same copy of the running kernel. There may be other optimizations that LXC provides which I am not aware of.

Enter LXC and Toft

As mentioned earlier, LXC has the least overhead of the other virtualization technologies available. Toft is a ruby library for testing chef and puppet cookbooks using LXC containers. Toft has a beautiful interface for working with containers as you can see in the following code.

I can easily see spinning up a multi-node cluster with toft, running my tests, then destroying the whole setup.

While toft has an excellent interface it is a young project with some rough edges, some of which have more to do with LXC than the toft library itself. To get started with toft, there are some nice guides on the project wiki

I very unpleasantly discovered that the version of the cgroup-lite package for Ubuntu 11.10 installed with LXC will make your system unbootable. The solution is to install cgroup-lite from the oneiric-proposed repository.

Loading ....

Another pitfall is that the bind9 and dhcpd servers installed along with LXC will report erroneous information after initial installation. The fix is easy. Run the `lxc-prepare-host` command and you will be in business.

Finally, at this time Toft only works with Ruby 1.8.7 and does not support Ruby 1.9.*

As noted in the Vagrant section, I highly recommend connecting your laptop to Ethernet if your primary OS is Linux.

Summary

Testing is an exciting and increasing critical part of developing infrastructure. While the existing tools need some polish, you can get a lot done with them. The linkbait title of this post is a bit disingenous. Testing w/ minitest is a breeze, but using vagrant and toft are not straightforward affairs. Toft, in particular, is extremely promising and worthy of your attention. It is extremely well-suited to many of the usecases that Vagrant is not.

Unpacking and Installing from Archives with the Ark Resource

2012-03-21T01:36:00.001-07:00

Updated 14 April 2012: The ark API has changed!

The ark_put, ark_dump, ark_cherry_pick have been rolled into the main ark library as the actions :put, :dump, and :cherry_pick. Please see the updated README

There are many wheels that sysadmins reinvent but the one I see reinvented most frequently is the unpacking of a tarball and placing its contents in an arbitrary location. This task seems trivial but there are many slight variations but critical variations to handle. Most chef users resorts to a combination of remote_file, bash, and link resources. This gets the job done but it is a lot of boilerplate for an essentially simple task and it is ridiculously easy to inject errors. I first wrote the java_ark Lightweight Resource Provider to handle this. Then I discovered Infochimps excellent install_from_release LWRP.

Unfortunately, install_from_release did not do everything I need so I had to extend it further. In the process of extending install_from_release, I mangled Infochimps beautiful code into an unworkable mess. I have now refactored that mess into something that is more maintainable and extensible. Please meet the ark resource. Think of ark as an archive but Kewler. You can thank Joshua Timberman for the name as he didn't care for my original name for it, `cpr` for Crappy Package Resource. CPR was a play on the medical term CardioPulmonary Resuscitation. Ark does not yet handle all of the functionality that install_from_release does. Install_from_release not not only unpacks tarballs but can build with make, ant, and more. Ark does not currently support ant, make, or setup.py but it wouldn't be too much effort to add that support.

Let's get back to the problem that ark solves. For many of us, the packages available from your given *nix distribution are too old for our needs. This is especially true in the Java ecosystem. The latest stable version of Tomcat is 7.0.26 yet only 7.0.21 is available on ubuntu 11.10. Tomcat 7 isn't even in the main RHEL 6 channels. My customers insist on having the latest security fixes, so installation from rpm/deb isn't an option. Installing the Oracle JDK is an even thornier issue as it is no longer available in Ubuntu nor RHEL software channels.

Here is an example of how I would install the Oracle JDK in the bad old days.

Here is how I accomplish the same with ark.

Wow! That's a lot easier, and frankly more intuitive. Pretty soon I found myself repeating several variations on the ark resource. At first I accomplished this by adding more attributes to the ark resource. This made the resource more verbose, intuitive, and made the code harder to maintain. A few weeks ago I read Nikolay Sturm's blog post on Cookbook Reusability and then interviewed him for the Food Fight Show. Through Nikolay, I discovered the Interface Segregation Principle. Rather than adding more options to the ark resource, I totally refactored ark from an LWRP into a Provider and Resource classes that subclass Chef::Provider::ArkBase and Chef::Resource::ArkBase.

Here is the ark_put resource which handles the simplest case:

ark_put simply unpacks ivy to the default location /usr/local/ivy, it does not create any symbolic links
I often have to download arbitrary collections of dependencies and place them in a lib/ directory. I do this frequently for tomcat-based applications. I just need to "dump" the dependencies in a specified directory that likely already holds a number of library files. This situation is a little more complicated as I need a way to determine whether or not the archive has previously been unpacked. With the ark and ark_put resources I simply check if the destination directory is empty. This logic won't work for a directory like tomcat/lib that holds library files from the initial tomcat installation. For this reason, I have to specify a "creates" attribute which specifies the file whose presence indicates that the ark has already been unpacked. Additionally, I need to discard any file hierarchy inside the archive.

ark_dump will place all the *.jar files in the archive into /usr/local/tomcat/lib/. Note: ark_dump currently only supports *.zip archives.

My friends at Oracle seem bent on making me work for my salary. I have encountered on several occasions the need to extract a single file from a tarball and place it in a arbitrary location. In this case that single file will be specified by the "creates" attribute. In this case I extract the JDBC connector for MySQL.

That's the state of the ark resource at the moment. If you are interested to use it and/or extend it, check it out on github. You should note that it is not a collection of LWRPs but actual Provider and Resource classes. LWRPs are great to get started with but don't provide the same facility for code reuse that pure Ruby does. Thanks to Nikolay Sturm for inspiring me to venture beyond the LWRP DSL and to MrFlip (Philip Kromer) for creating the awesome install_from cookbook. I hope that ark will eventually support all of the functionality that install_from_release does.

FOSDEM Summary

2012-02-06T22:50:00.000-08:00

I thoroughly enjoyed FOSDEM. The weather sucked, the beer was great, and the people were wonderful. Thanks to the FOSDEM organizers for putting on such a great conference. What follows is a very rough summary of my experience.

I spent most of my time in the Virtualization room. There were a number of interesting talks there. I didn't spend much time in the Configuration Management Devroom, that is not because the content there was lacking but because I felt more knowledegeable about the subjects there than in the Network/IO,
virtualization, and hypervisor talks where I was pretty much ignorant. From what I could tell, the
organizers did a fine job organizing the Systems and Configuration Management Devroom.

One of the best parts of going to a conference like this is meeting in person that you have collaborated with via e-mail and IRC. I met up w/ Cheffolk Andrea Campi, Juan Jesus Croissier (juanje), Nicholas
Szalay (nico), Stefan Jourdan, and Greg Karekinian (sp?). I have interacted with these guys quite extensively over the last four months, so it was awesome to finally meet them. Juanje has a very different use case for Chef than most users. He manages thousands of desktop machines running GNOME. He must take care not to overwrite the users custom settings, a challenge I have never had to deal with. For this reason he created an LWRP to manage portions of a file. Nico has written a nagios check that will report failing chef runs. I need to start using that.

The best moment of FOSDEM for me was a very personal one. I ran into Sulochan Acharya, a friend of mine from Nepal. We worked together to build the One Laptop Per Child project in Nepal. I didn't even know he was in Europe nor still working in IT. Now he works at Rackspace on Openstack-related stuff. It is a small, small open-source world.

Best talks I attended were Anil Madhavapeddy's "Wild West of UNIX I/O" and Charles Nutter's JRuby talk. Anil talked about the unpredictable behavior that we see in UNIX I/O in virtualized environments. In a
multicore world, hypervisors don't know enough about the underlying architecture to make intelligent decisions about I/O between CPU cores. This leads to very unpredictable I/O results. According to Anil, this requires a better I/O abstraction that is aware of the underlying microarchitecture--CPU layout, interconnects, NUMA--and can make intelligent choices for the developer. Anil calls his next-generation I/O framework FABLE, which is still in very early development. Charles Nutter is a great speaker though his
talk was a bit over my head. He focused on the "impossible" challenges of implementing Ruby on the JVM and how his team overcame them.

I missed the ZIO and CERN talks. Both talked about next-generation I/O frameworks. Also sadly missed Jonathan Ellis's talk on "Overcoming JVM limitations in Cassandra". I spent no time w/ the Mozilla track, which is a shame as I love how they manage their community and the kinds of projects they work on.

I was struck by the large number of projects that featured web frontends written in Ruby on Rails. Besides Horizon, the Openstack web UI, it seemed that virtually every other web UI I saw was Rails. Ironically, there were no talks specifically about Ruby nor Rails at FOSDEM.

Red Hat is doing a lot of activity in the DevOps space. oVirt a web UI for managing VMs. Matahari a system administration API which looks interesting but seems like it is still in early development. I had the pleasure of talking w/ Ohad Levy, a very sharp fellow who develops Foreman, a front-end to Puppet and is employed by Red Hat.

I found out about the meta-project Katello from Red Hat that contains many components to automate system administration. To my limited knowledge, it includes Matahari for low-level administration, Puppet for configuration management, Foreman as a nice GUI to manage your configurations and server provisioning, oVirt to manage your KVM hosts a la VMware's vSphere, and pacemaker-cloud for high-availability across different cloud providers. I believe the combined stack is at alpha level. I would
love to know if Chef could be used in place of Puppet.

Met the very smart Marek Goldmann who is doing a lot of work packaging JBOSS for Fedora in addition to his boxgrinder project. It seems like there is a fair amount amount of overlap between what he and are trying to do in this regard. I need to spend some time hanging out on #fedora-java to see how we can avoid duplicating efforts.

My greatest regret is that I myself didn't give a presentation. I didn't think I had anything to say that others would be interested, but that view was wholly wrong. There was a lot of interest in Chef but unfortunately not much content.

I met a couple of people who had listened to my podcast, Food Fight, but they told me it was "unlistenable" due to the recording quality. Guess I will have to improve the audio quality :(

I was very lucky to have dinner with a group that included the DevOps gurus Kris Buytaert and Patrick DeBois. After following both of them on Twitter for a number of months it was really interesting to see what they are orking on. These guys really are the "Belgian Braintrust" of DevOps not to mention that Patrick actually invented the term. If I recall correctly, both Patrick and Kris are spending a lot of time
lately working on monitoring and logging. Not only are Kris and Patrick DevOps heroes but they are also genuinely nice guys.

Beer

Belgians make the best beer in the world and they enjoy it at all hours and in all places. Quite a number of people enjoys fine brews during presentations and meetups. The Opensuse stand actually gave away opensuse beer. Note to self, avoid the 10 % alchohol beer when trying to stay alert. Stick to the delicious 4% blonde beers.

Final Thoughts

Openstack is the real deal, not just a cool idea. It will really be ready for production use this year. I had my doubts about an open-source project led by an industry consortium but what I saw made me put them
aside. Particularly impressive was Ryan Lane's explanation of how Wikipedia is using Openstack for its infrastructure.

There is awesome stuff happening with KVM, both in terms of performance and tooling.

There was a lot of interest in Chef but I didn't meet that many people who had actually used it. This must be remedied. I really should have given a talk and put more effort into planning a Chef meetup.

I will definitely be back next year and hopefully as a presenter.

Announcing the Food Fight Podcast

2012-01-15T22:23:00.000-08:00

I just published the inaugural episode of Food Fight, the Chef community podcast that I host together with Matt Ray. The podcast will cover not only the developments within the Chef community but also larger world of DevOps. Enjoy!

Chef in 2012

2012-01-04T14:08:00.000-08:00

image by Simon Griffee

Note: Petey King has written a great response to this post. Definitely worth a look.

I discovered Chef in the last last quarter of 2011 and turned my world upside down, work-wise at least, in a really awesome way. Over the holiday season I had some time to reflect about where Chef as a piece of software and as a community of people might go in 2012.

Packaging Cookbooks

Cookbooks are starting to more and more look like packages, with complex dependencies between them. I am not aware of any cookbooks that only work with Ruby 1.9.* but I expect we will see a number of them in 2012. I personally believe that we need to use an existing package management system rather than creating yet another one. RPM, DEB, dmg are all out of the question because they are operating system specific. Ruby Gems seems to be the best choice though I am not very knowledgeable about ruby gems. If you think this a terrible idea, please do let me know what problems ruby gems pose. I envision the scenario where your cookbook_path in knife.rb

As you can see from my example, you still would be use cookbooks as they currently are, but also allow chef to look for additional cookbooks in your GEM_PATH. Cookbook gems would have to be prefixed with something like "chef-ckbk-" to indicate to knife which gems are cookbooks.

Using ruby gems does present a couple challenges, particularly with rvm, which completely compartmentalizes gems from one ruby version to another. This means that changing your ruby interpreter requires you install anew a large portion of your cookbooks. Using gems may also require a lot of people to change the structure of their cookbooks. Perhaps, moving moving all the directories to under gem_name/lib/ subdirectory.

Testing 1, 2, 3

There still isn't a broad consensus on how to automate the testing of chef cookbooks. There was some nice progress during 2011 with foodcritic (lint for cookbooks), chef-minitest (unit-testing?), and cucumber-chef but those tools are all still early days. Testing is a particular pain point as testing a minor change to a cookbook may require testing against 5 different linux distributions. I really hope we see more progress on this front though I can't point to any particular initiatives. I personally hope to spend some quality time with minitest in the next couple weeks.

Other Miscellaneous Technical Needs

Selinux support is currently very limited. Some people feel that selinux shouldn't be used in the first place, however, many sysadmins don't have the luxury to make this choice themselves but are required by an external corporate or legal body to use selinux. As Andrea Campi has pointed out on the Chef mailing list, there are a number of consulting firms, such as ZephirWorks, that have the capacity to implement better selinux support if someone steps forward with funding.

Image by Simon Griffee

The java application stack also needs a lot of love. The current set of tomcat, jetty, maven cookbooks only support minimal configurations. I am reorganizing these cookbooks around the java_ark LWRP. In a nutshell, the java_ark LWRP extracts a tarball, adds any included executables in the path, and runs update-alternatives. Along with the jboss cookbook I have already started, I expect to put a lot of work into the java stack. If this is something you are interested in, suggestions, code criticism, and patches are most welcome.

While chef-client and chef-server generally perform at acceptable levels, there are some pain points. For chef-client, memory usage can sometimes be an issue due to the node object ballooning in size during chef-client runs. If I recall correctly, Opscode is working hard on reducing memory usage so we should see some progress this year. On the server side, we can expect a lot of performance improvements as chef-server moves from ruby + couchdb to erlang + mysql. This may happen sooner than one might normally expect as Opscode has already converted part of Hosted Chef to Erlang and MySQL, it just hasn't open-sourced that code yet.

Growing Community

2012 should be the year that the myths "Chef is for programmers" and "Chef is hard" are demolished. These myths sadly linger and inhibit adoption in my opinion. Perhaps part of the reason that some sysadmins think Chef is hard is that it is pure Ruby. Now Ruby is not a difficult language but it is unfamiliar to most sysadmins. Advocating Chef (and Puppet to a lesser extent) also means advocating ruby amongst sysadmins. How to do that? Here are a couple suggestions.

Get ruby 1.9 into the default server installs for RHEL, Ubuntu, ArchLinux, etc. This could be quite a challenge as according to this blog post, there is almost nobody in the Ubuntu world doing any Ruby work. With the exception of the awesome Sergio Rubio, the same could be said for RHEL/CentOS/SL
More blog articles on using Ruby to solve system administration problems and that sysadmins should consider using Ruby as their default scripting language

All that talk about Ruby advocacy aside, Sysadmins will need to know less Ruby in 2012 in order to write Chef recipes. This is because we will see more and more LWRPs pop up. I love LWRPs. I think they are the best thing since dynamically loadable kernel modules.

Chef is growing by leaps and bounds because it solves an important problem in configuration management. The greatest constraint to that growth is the supply of sysadmins willing to make the investment to learn it. At this point, the majority of the community are self-taught linux geeks who found Chef on their own. At the next stage of growth, Chef needs to be attractive to the group I call "the Professionals". Professionals are talented engineers who make system administration their job but not their life. They don't read blogs during their lunch hour and they don't hang out on IRC in the evenings. The Professionals will typically learn Chef because their bosses tell them to. We need to make that learning process as smooth as possible.

The Chef wiki is an incredible reference but we need a guided tour to accompany it. We really need the Chef equivalent to Pro Git and the Git Community Book site. These resources hold your hand and walk you through all the important parts, whispering comforting words when you get frustrated. I know there is work being done a Definitive Guide to Chef from a major publisher, but I don't know its status.

Lastly on the subject of community, we need some mechanisms to summarize community activity.
The Chef project has a high velocity and I find it increasingly hard to keep up with. The Opscode monthly newsletter is a good step in this direction. Additionally, I am starting a Chef podcast together with Opscode Technical Evangelist Matt Ray, focused on summarizing community activity and exploring important topics.

Dangers in 2012

I sometimes worry that Chef could become a victim of its own success. As I mentioned earlier, the greatest constraint to the growth of Chef is the supply of sysadmins willing and able to learn it. If enough well-meaning software architects and aggressive CIOs force Chef on resistant ops teams, we will start hearing about implementations gone horribly wrong. Top-down adoption can only work if the ground below is fertile. That's my two cents.

In Summary

Chef will be a success when the default installation instructions for application X point to a Chef cookbook. I think we will make solid progress towards this goal in 2012 but to really get there, a lot of things have to happen first. I couldn't be more excited. Thanks to everyone in the Chef community for giving sysadmins like me such a great toy to play with. Happy Hacking!

PS: Let me know if you like the Chef icons. My good buddy Simon Griffee created them after a brainstorming session fueled by Sagrantino Di Montefalco.

Emacs for Sysadmins

2011-11-20T05:14:00.001-08:00

I am not going to tell you that emacs is superior to vi, because it is not. For me, vi is far better for basic text editing. However, emacs has such powerful modes that I find myself using it more often than vi these days. Emacs has historically been favored by programmers and vi by sysadmins. I have written in the past that sysadmins are becoming less like ninjas and more like pirates because of this crazy thing called devops.

As a professional system administrator you can't choose between vi and emacs; you have to learn vi. There are just too many times you will be working on an arbitrary *nix that only has vi installed. You can choose to learn emacs in addition to vi. Vi lets you jump in and around files quickly. Nothing can match the speed of hjkl, g, G, etc. for movement or v + y for copying text. However, we sysadmins are moving from managing individual servers to managing configurations. In the last several couple months I have spent at least 50% of my work time entirely within my chef-repo/ directory working on data bags, roles, and cookbooks. Inspired by the devotion of a number of Chef community members to emacs--notably Matt Ray, Stephen Nelson-Smith, Josh Timberman, among others--I decided to give emacs a try. On the whole, it has been a very positive experience. That is why I recommend sysadmins try emacs.

Ninja	Pirate

Bill Joy Vi Creator Hand placement conceals poison dart	Richard Stallman Emacs Creator Note beard

from philosecurity blog ninjas or pirates, emacs or vi

Why Emacs?

Emacs has some truly awesome modes. The default ruby, python, bash, xml, html seem to do more than their vim counterparts. I think this is large part because emacs' extension language (elisp) is more powerful than vimscript. I made a stab at vimscript some years ago but came away quite frustrated. While I am not a lisp convert, it is a full-featured language that doesn't hold you back. Working with multiple buffers (10+) is extremely easy. Org-mode though, is what got me hooked.

Why Not Emacs?

Using vi is like being in a relationship, you are involved. Using Emacs is like a marriage, you are committed, in large part because there is so much to learn. This is one of the reasons I don't recommend it for everyone.

Further, vi is built for touch typists. You rarely, if ever, have to take your fingers off the home row keys. You can keep your fingers on the home row with emacs but that often means completing a number of key chords entirely with your left hand. Take for example opening a file, writing its contents to another file, and then closing emacs.

C-x C-f + C-x C-w + C-x C-k

If you keep your fingers on the home row, you will only use your left-hand for the above commands, except for the final "k". You could start to feel some discomfort in your left hand if you did that key sequence several times in a row. For this reason a number of emacs addicts will hit the Control key with their right hand, the "x" with their left, Control again with the right, "f" with the left, and so on. This leads to a more "hunt and peck" typing style but saves your hands.

First, Learn How to be a Pirate

Before you even start playing with emacs you should make sure to swap your Caps Lock and left Control key. Emacs commands use the Control key heavily and leaving it in the default position will soon leave you angry, crippled, and ready to kill Richard Stallman.

Next, I recommend you organize your dotfiles on github or another source code repository. Being a pirate means you have booty and treasure to carry around. You need a ship to carry around all that treasure and for me that ship is github. The core of my dotfiles repo is the Rakefile, copied from Ryan Bates' master, that makes it dead simple to install your dotfiles on a new machine. I don't install my dotfiles on every machine I work on, only the 2-3 I use most frequently.

create your own github/bitbucket/private github repo called dotfiles
copy your existing dotfiles -- .bashrc, .vimrc, .emacs, etc -- to ~/dotfiles but without the leading "." !
download Ryan Bates Rakefile and put it in ~/dotfiles
to test, rename ~/.bashrc to to ~/.bashrc_bak then run $ cd ~/dotfiles/ && rake install. Your .bashrc should be back
Push to your repository

Whenever you make a change to your .emacs or .bashrc on your ubuntu laptop at home, remember to push your changes and then run "rake install" on your work desktop.

Learning Emacs

OK, finally emacs. I recommend you buy Peepcode's "Meet Emacs" tutorial. It is only $12 and it is the best and most convincing emacs tutorial. It is well worth the money. If this tutorial doesn't convince you that emacs is worth your time stop, turn around, and go back to vi. One of the first things covered in "Meet Emacs" is how to install emacs 24 and the emacs-starter-kit. The emacs starter kit will set of bunch of defaults that help you avoid many of emacs' warts and show off some awesomeness such as ido-mode.

Org-Mode, Emacs Killer Feature

Ok, I have to admit that I have been working with emacs for longer than a few months. I have been using emacs' Org-Mode to manage my life for the last several years, ever since I happened upon Abhijeet Chavan's great article in Linux Journal. Soon after that, I came across Scott Jaderholm's excellent screencast for Org-mode. At the time I read the article, I was the Chief Technology Officer for a non-profit technology startup in Kathmandu, Nepal. I was completely overwhelmed by the sheer volume of different tasks I had to accomplish and all the attendant details, especially the details. I no longer have all that stress now that I am a regular sysadmin, but I do have a large number of very detailed tasks. Org-mode is in many ways my memory. It is also how I manage my attention span.

Many people use Org-mode to implement the Getting Things Done methodology, pioneered by David Allen. I have bastardized it into my own system.

Today is a list of general things that I want to get done today. This week, a list of tasks to complete this week. And of course, right now is a list of specific next actions, that I will complete starting at 1.

When you are finished with a task, don't delete it. Archive it so you have a record of when you are done. Org-mode handily records the timestamp of when you completed the item.

C-c $ # archives a subtree to your archive file,

If I finish reading the O'Reilly Emacs Manual and then archive it, here is what the entry looks like.

I love that I can move items in a numbered list almost effortlessly. M-<up> moves an item up and adjusts the numbering and M-<down> moves an item down in the list.

I typically keep the following org-mode files and their associated archive files.

mygtd.org -- my todo list, for today, this week, this month, and right now!
meetings.org -- all meeting notes and preparation
notes.org -- this is where I keep notes while I work on a problem

The notes.org file deserves special attention. This is my running work log. As I diagnose a problem, I write up my possible hypotheses, then tests for these hypotheses. Finally, I record the results of testing those hypotheses, pass or fail.

The true beauty of using emacs to manage your todo list is that you can grep through it to find a specific piece of information. This has saved my ass more times than I can count.

Org-mode protip: put your *.org files in a Dropbox folder so you have access to them on your various machines. It may be tempting to put them on github but you should realize that those org files will quickly fill up with personal and corporate information.

Working with Buffers

I never found it that easy to work with multiple buffers or tabs in vim without doing extensive key rebinding. Buffers are something that emacs really gets right, especially when you have ido-mode turned on. I have found working with multiple buffers really effective when working with several chef cookbooks at once.

Let's say I am working in the sudo/recipes/default.rb recipe but want to jump back to the previous recipe I was working on, in this case iptables/recipes/default.rb. I type C-x b and ido-mode shows me a list of open buffers. To get the last buffer I was in before this, simply hit [RET] or I can start typing the name of the file and ido-mode will dynamically filter that list.

C-x b [RET] to get to the last visited buffer

Dired-Mode

I have always heard that Midnight Commander is an awesome but I have never taken the time to learn it. I believe that Dired-Mode offers many of the features of Midnight Commander but lets you leverage the emacs skills you have picked up.

Here are commands that I love

Key Binding	Command
v	View a file or directory in read-only mode
d	mark a file or directory for deletion
C	Copy a file
x	execute changes marked, i.e. delete files marked for deletion
q	close an open dired buffer
[RET]	when u hit [RET] on a file, emacs opens it for editing
R	Rename or move a file or directory
M-x find-grep-dired	Grep a directory for a regex and then view the matching files in a Dired Buffer
Q	Find and replace a regex across multiple files

While Dired-Mode is very useful for managing files I find it most useful for browsing code. You can hit "v" to view as read-only then q to close that buffer when you are done. Viewing files in emacs means you get all that juicy syntax highlighting. This helps me make sense of those butt-ugly XML configuration files I encounter on jboss and tomcat installations.

Miscellaneous Stuff I find Useful in Emacs

Shell-Mode - This is very handy for moving text back and forth from a shell and code file. M-x shell-command-region or M-! lets you execute a shell command on a region of selected text
Inf-Ruby-mode - lets you select a bunch of ruby code and immediately execute it in an irb session. I would love to see this work with shef
Modes for Languages you rarely touch - I rarely code in java but occasionally I have to read java code. java-mode helps me parse java without having to deal with Eclipse. Have to change a config file written in Erlang? There's a mode for that.
Properly indenting text - select a region then C-M-\
Completion - M-/ will guess what you are trying to type based on open buffers
Commenting out multiple lines of code -- select a region, M-;

Emacs Shortcuts are Everywhere

In case you haven't already noticed, virtually every unix shell and programming REPL (irb, python, perl) supports Emacs keyboard shortcuts. This is because all those shells are built on top of libreadline, which implements a subset of emacs shortcuts. Even REPL's for newer JVM languages like scala, groovy, and clojure use a clone of libreadline, jline, that implements those same shortcuts.

Emacs Stuff I Haven't Learned Yet but Want to

Tramp-Mode -- editing files over SSH, this would be pretty sweet for configuring a bunch of Virtual Box VM's running on my desktop
Calendar Integration with gmail - just haven't gotten around to this yet
ERC - I debate whether to move from Xchat to irrsi or ERC (Emacs irc client). jury is still out
gist - This is a mode that allows you to select a region and create a github gist from it

Wrapping Up

Emacs, despite conventional wisdom, can be very useful for systems administrators and not solely the domain of Lisp Hackers and not Exclusively for Middle-Aged Computer Scientists. While it may be One True Program for some people, it is not for everyone. The highlights of Emacs for me are:

Org-mode -- time management for busy sysadmins
Dired Mode
Buffer Management

If you have some tricks that you find especially useful for sysadmin work, please share them! Here are some additional resources you may want to look at:

Why Emacs? Borizhdar Batisov's excellent, more developer-focussed ode to emacs
Hack Emacs Screencasts
Mastering Emacs blog
Emacswiki
#emacs on freenode.net

Chef for Developers

2011-11-20T00:26:00.001-08:00

Last Friday I gave a presentation to a group of developers on how they can get started writing Chef cookbooks. I put about 5 hours into writing this presentation. To my chagrin, Warwick Poole released a superior presentation that same afternoon that does a better job of explaining the basic concepts of Chef. I strongly recommend checking out Warwick's "LearnChef" if you haven't already. It is the single best introduction to Chef that I have seen so far. In more good news, Warwick informs me that he intends to create a 5 part series building on "LearnChef." I very much look forward to it.

My Chef presentation doesn't do as good a job explaining the core concepts of Chef but it does walk developers through some scenarios on how they can work with chef on a project. You can check it out below and even reuse my slide material by downloading the .ppt file here. My slides borrow heavily from Joshua Timberman's excellent Chef 101 presentation. The slides are under the Creative Commons 3.0 Unported CC-BY-SA. You can find the full list of publicly available Chef presentations on the Chef wiki.

Chef, Devops, and You

View more presentations from Bryan Berry.

A Month with Chef

2011-10-30T06:14:00.000-07:00

It is hard to believe that I started using Chef only one month ago. The experience has hugely changed how I view the practice of system administration and started to unseat the deepest of ingrained habits. It has been mostly a smooth ride but there have been some significant bumps along the way. Here are my reflections on learning Chef, the community around it, putting it to work, and my hopes for the future of the project.

Learning Curve

There are roughly three technology components to learning chef, each with their own learning curve 1) the Chef software itself 2) git and 3) ruby. I found understanding Chef to be quite easy, in large part because of prior experience with Puppet. If you have never used a configuration management tool before, this can be a stumbling block. However, the biggest stumbling block of the three is probably git, not ruby and not the Chef software itself. Again, I had previous experience with git but even I ran into occasional git-induced hurdles.

When you treat your "infrastructure as code," you have to spend a fair amount of time managing that code. Git is the tool you will use to manage your chef repository, your cookbooks, pull updates the mainline opscode/cookbooks and push your patches to others. If you are a n00b to both Chef and git, do yourself a favor and reach for a good git tutorial before you get too deep into Chef.

The fact that Chef recipes are written in Ruby has turned out to be far less of an impediment than I thought. You can learn enough Ruby to write a recipe in less than twenty minutes. Further, with copy-and-paste plus playing with irb you can guess your way to getting a lot accomplished--though this isn't a good strategy for the long-term. For my own education, I read the pickaxe book and Flanagan's The Ruby Programming Language. Both were good but I prefer the Ruby Programming Language. I have heard from others that Ruby Koans is an excellent way to learn Ruby though I haven't tried them myself.

My first impressions of Ruby the language are shaped in part by my experiences working in the Python world. Python is clean and neat and you can see the hand of Guido guiding the language and standard library at every turn. Conversely, Ruby is creative, chaotic, and messy. It has bitten me in the ass several times that Ruby has hundreds of top-level methods and operators. To say that the namespaces are "cluttered" is an understatement. You have to be quite careful when creating new variables and method names. For instance, In one recipe I created the variables "alias" and "rand", which are coincidentally top-level Ruby methods. Overwriting them caused some nasty side effects that took me a while to track down. If Ruby were a human being it would have a beautiful body, a terrible case of acne, and noticeable digestive problems. Worse yet, its clothes wouldn't match.

My biggest single problem with Ruby is that shit breaks. Last week the main json gem was broken, causing pain throughout the Ruby world. With my first install of Chef-Server, the code-ray gem broke the chef-server-webui. As I am not a rubyist, it took me a non-trivial amount of time to figure out how to downgrade to an older version with the following command $ gem install code-ray --version '< 0.9.8'. I have since discovered the concept of gem sets, which define dependencies to specific gems per specific versions. Why would you have to do this? Because you can't count that the latest gem release will be stable. Working with Python I rarely had these same kinds of problems. On the plus side, the Ruby community moves fast. For whatever reason(s), new Ruby projects pop up quickly and continue to evolve at an astonishing pace. Tools like thor, sinatra, capistrano, and fog all seemed to appear before their Pythonic equivalents*.

* software historians, please correct me if I am mistaken

It is easy to overlook that Chef assumes you already thoroughly understand your operating system and application stack. Chef makes experienced system administrators more productive. It also makes inexperienced system administrators more dangerous and can even hamper their learning. It is for this reason that I don't recommend that those new to *nix pick up Chef. You can't automate something you don't understand.

Using Chef

My experience with Chef has not been all roses. Like Ruby, Chef moves fast. For a FOSS project, it is ridiculously young, first released in January 2009 and has not even reached version 1.0 yet. It still has significant bugs and Linux distribution support is still quite uneven, especially compared to Puppet. To their credit, the Opscoders acknowledge these issues. It must be noted, Opscode did not officially support Enterprise Linux until recently. Further, despite its youth, I found the Chef-Server to be quite stable and easy to setup.

I had a nasty time with the initial installation. It took me over three days to complete, mostly due to a bug in Ohai that misreported the linux distribution. However, that was early October. I did a fresh install last week and it took me only 30 minutes, largely due to documentation improvements, bug fixes, and updated rpms for both EL 5 and 6.

The opscode cookbooks, like the chef software, have spotty support for Enterprise Linux. One of the first cookbooks I tried to use was the selinux cookbook. I couldn't even upload that cookbook to my chef-server as it has a syntax error. I have sent a patch, logged a ticket, and sent a pull request to fix the error. That issue was not fixed the last time I checked. My pull request isn't lonely either. Last time I checked on github.com and I saw that there were 99 open pull requests in their queue. My impression is that the Opscoders are not intentionally ignoring such requests but are struggling to keep up with the rapid growth of Chef community and Opscode's paying customer base. I hope this improves in the months ahead.

Once I got Chef-server up and running, I felt a bit lost. Chef is a very flexible tool and not nearly as opinionated as Puppet. It is hard to decide where to get started. One can easily get lost among Chef's powerful abstractions such as definitions, providers, resources, and libraries. Worse, Chef doesn't have a definitive technical book to hold your hand through the entire process as James Turnbull's excellent "Pro Puppet" does for Puppet.

After my initial bout of confusion I set to work on converting a 400-line RHEL 5 server provisioning shell script into a set of cookbooks. Amazingly, I was able to convert it to a set of Chef cookbooks in less than a week and a half. Even better, these cookbooks could be applied to an existing server at any time in its life, not just at installation time. Bonus, my bash script handled RHEL/CentOS 5-6 on x86_64 and i386 architectures whereas the original bash script only handled RHEL 5 x86_64. I must confess that part of my productivity gains stem from the fact that I am far more proficient in RHEL than I was when I originally wrote the Bash script. That said, I believe that using Chef for the server provisioning task took me 30-40% of the time it would have required to rewrite the Bash script to handle RHEL 6 and the i386 architecture.

I have made a lot of progress with Chef in this first month but I have yet to automate an entire application stack. Here are the tasks I hope to implement with Chef in the next few months:

Consistently test my cookbooks with rspec or another testing tool
Create a Jboss cookbook
Cluster Tomcat instances

Best Practices thus Far

So here is a collection of the Chef "best practices" I have picked up in this first month. Given that it has only been a month, please take them with a grain of salt. Master Chefs, please don't hesitate to correct me if there is any incorrect information here.

If you want to get started quickly, i.e. learn chef in a weekend, use hosted chef and micro instances on EC2. For me it was important to install the entire stack before using it because that is the kind of geek I am. It also sucked up a lot of time.
Separate your ingredients (data) from your recipes. Don't hardcode your site-specific information like IP addresses or host names into the recipes. Put them in Data Bags, role attributes, or node attributes.
Data Bags are for complex or global data like users or samba shares
Role attributes and node attributes are best suited for simple values like host names for NTP servers, Nagios server, or whether to run X service as a independent daemon or under Xinted
If you aren't already familiar with git, you should make learning the basics of git your #2 priority after completing the Chef Getting Started Guide. Since we're talking about "Infrastructure as Code" here, you have to become as serious about version control as a software developer is.
After you clone the main chef repository, create the sub-folder site-cookbooks/ at the same level as cookbooks/ . Put your new cookbooks in site-cookbooks/ and make each one an individual git repository
In .gitignore, add ".chef" and "site-cookbooks". You store all authentication keys in your .chef/ so it is a directory that you really don't want to accidentally push to a public repository.
Sign the CLA for Chef. You can't contribute fixes without doing so.
Don't use $ knife role edit <role-name>; use the Ruby DSL instead. That way you have your roles under version control.
Check, double-check, then ask on IRC before starting a cookbook from scratch. I wrote four different cookbooks before realizing that they duplicated functionality already present in the main opscode/cookbooks
Use shef, it is a wonderful tool to debug and explore how Chef works. Note that it does not load roles attributes at this time.
Fat cookbooks, skinny recipes. You should put the recipes for both the client and server parts of an application in the same cookbook. This makes them easier to keep track of and forces you to make the recipes more modular.

Here are some Ruby tips that I have picked up

Put require 'irb/completion' in your ~/.irbrc, then you have auto completion in irb and in shef as it uses irb

For learning ruby, I actually prefer pry over irb, which reminds me of my beloved IPython. I haven't yet figured out yet how to get shef to invoke pry rather than irb.

To install pry

$ gem install pry pry-doc

You also have to put require 'irb/completion' in your ~/.pryrc or type it in at the prompt. Here is an example of how tab completion works

The next step is to actually check the docs for what particular method does. The command for that is
pry> show-doc Object.method

You can also read ri docs by simply typing pry> ri Class#method like you would at the normal shell prompt.

There are a couple tools that I intend to use in the near future, specifically libarian-chef by Jay Feldblum, but haven't yet gotten the opportunity.

More Pirate, Less Ninja

Chef has made a big impact on my work style. You definitely start working more like a developer, spending more time on your workstation and less SSH'd into a random machine. It is hard to justify heavily customizing your own dotfiles when you spend most of your time on random machines where the only text editor you can count on being present is vi. This article describes vi users as ninjas and emacs users as pirates but I think many of the same comparisons apply to sysadmins and developers. Ninjas can't carry much around with them on their dark forays just like sysadmins can't lug around a bloated .vimrc or .emacs file. Pirates, on the other hand, stay on their ships where they have plenty of room for whiskey, stolen treasure, ( .emacs.d/ || .vim.d/ ), .screenrc, .zshrc, etc.

With Chef, I spend a lot more time working on cookbooks, roles, and data bags in localhost:~/chef-repo/. Knife makes it very easy to manipulate your chef-server and chef-clients with fewer ssh console sessions. I think this work style may explain why there are many more emacs users in the chef-community than in your typical group of sysadmins. Historically, system administration has forced you to be a vi user but as a "chef" you can spend more time in emacs, sam, joe, or even *gasp* nano. I personally have heavily customized my .vimrc over the last month and even dabbled in zsh. I am not suggesting that Chef will turn you into an emacs user, but it will turn you into a pirate.

Selling it to the Team

Managers love chef. They get the idea in the first five minutes and ask you weren't using Chef five years ago. They love it because 1) big, respected companies are using it 2) it makes system configuration far more transparent, i.e. you can see how systems are actually configured and not how you hope they are 3) it will help them deal with the never-ending requests for new Virtual Machines coming from the developers. It can be harder to sell it to those sysadmins who may know *nix like the back of their hands but are used to doing everything by hand and have never gotten into the habit of automating tasks with their own shell scripts. It can also be hard to sell to those sysadmins who have already invested heavily in their own custom bash/Perl/Python/Tcl infrastructure framework.

Here is my pitch for Chef in a nutshell.

A few years ago we had a ratio of n machines per sysadmin. Now we have 3n machines per admin. Next year we will likely have between 6-9n machines per sysadmin. We are struggling mightily to support 3n now and our current coping mechanisms will break down entirely when we move to 6n. The only way we can cope is by moving from home-brewed infrastructure to a real configuration management system, such as Chef.

If you aren't comfortable with writing your own automation scripts, you are at least comfortable using the scripts of others. I don't expect everybody here to be able write their own cookbooks from scratch but I believe everyone here can learn how to reuse and modify existing cookbooks to meet our needs.

Frankly, I don't yet have a way to sell Chef to the NotInventedHere crowd. Whatever benefits you bring up, Chef still wasn't their idea. One option would be to convince them that Chef just does what they have been doing all along and using Chef will free them from maintenance work for their current tools. Then they can focus on what really needs reinventing. Unfortunately, implementing Chef may require a substantial amount of their help and not leave them so much time to start that new pet project.

Thoughts on the Community

I feared that the Chef community would be populated with developers who were "slumming" as sysadmins. I found quite the opposite. The vast majority of the community are Unix-lovin', bash-spittin' geeks who make their careers as sysadmins and not as a side project. In terms of geography, the community is overwhelmingly concentrated in North America. It can be tough to get a response on IRC during the morning in Europe while you will get tens of responses during the US West Coast's working hours. For all hours outside of the European morning, you can find a large numbers of core developers on IRC and they are extremely helpful.

The vast majority of chef users seem to work on Ubuntu, with Enterprise Linux a distant second. Surprisingly, there is a relatively large presence of OpenSolaris/Ilumos and I am not sure exactly why that is. Perhaps Solaris users are disproportionately drawn to automation tools?

When I can't get a quick answer on IRC, the two best channels are the 1) chef-users mailing list and 2) twitter. For some reason, the devops world has really, really gathered around twitter. I can get a quick answer at any hour by using the hashtag #opschef. However, this only works for questions that entirely fit into 110 characters. It is bad form to link to a mailing list post or extend the question across multiple tweets. For complicated questions, the mailing list is always the best place to go for help.

The community is extremely friendly and so far there are no trolls. However, this is typical of a young project. Early adopters by their nature tend to be more positive and self-sufficient. Most people on #chef are there because they discovered chef on their own and not because their boss mandated they use it.

My Short-Term Chef Wishlist

Here is my wishlist for the next 3-6 months

Clear guidelines on how to collaborate on cookbooks and on how to push changes upstream to opscode/cookbooks (something which jtimberman is already working on)
Lower Memory Usage. Chef can really, really be a memory hog when you have several hundred nodes
Better support for Enterprise Linux, both in core chef and the cookbooks
A few weeks ago I would have said better documentation but they have already made huge strides in this regard
A stronger connection to the FOSS community worldwide

Chef will achieve world domination when the installation instructions for the average software application start not by pointing to a .rpm, .deb, or tarball but to a Chef cookbook. Key to achieving that are 1) making Chef awesome (of course) and 2) better engagement with the communities that manage software packaging in the public sphere--the FOSS geeks. These guys are the guerilla foot soldiers of software packaging. They write the man pages, manage a FOSS project's build servers, maintain wikis, etc. The phenomena of "cloud computing" couldn't exist today if it weren't for all their hard work at every layer of the GNU/Linux/*BSD stacks.

How can Chef win over the FOSS geeks? First, having a presence at conferences like FOSDEM, OSI, FUDCON, FISL, etc. would be relatively inexpensive and give them a lot of bang for their buck. Secondly, offering some version of free hosted chef for up to n nodes for open-source projects and foundations like Software Freedom Law Center, Python Software Foundation, Software Freedom Conservancy, etc.

Wrapping up

So far, Chef has lived up to the hype. It has made me more productive in a relatively short amount of time. I can't imagine my future as a sysadmin without it. Configuration management--whether using chef, puppet, or other--will soon be an essential skill for a professional system administrator. Now how does Chef affect my work in the larger context of Devops? Where does it fit in my devops-toolchain? Well, those are subjects for another blog post entirely.

Comments for this post are on Hacker News

Why Chef?

2011-10-16T02:22:00.000-07:00

Being a system administrator is alternately thrilling and tedious. The thrilling part is setting up an awesome, rock-solid system. The tedious part is doing that 10 or even 100 times more. Further, throughout your career you will configure certain applications again and again. You know the ones, sendmail, apache, samba, etc. Well, there have been many attempts to build a framework, an abstraction, for automating the process of configuring these well-worn tools and others. Today there is one such framework that finally gets it right, Chef. There are many, many reasons to use Chef but the main reason you should use chef is this: You, System Administrator, are too damn smart to spend the rest of your professional life reinventing the wheel.

Let me tell you a little more about how wonderful Chef is, then I will show you briefly some technical highlights, and finally I will clear up some Chef myths.

The Top 5 Reasons to use Chef

Writing reams of documentation sucks. Chef drastically reduces the amount of documentation you have to write.
Bash doesn't scale. Seriously.
Technical Awesomeness
Chef grows with you
You can stop reinventing the wheel

Now let's talk about that awesomeness in more detail.

Write Less Documentation

Less docs? You're kidding me right? Actually, I'm not. If you're like me, you'r sick of annotating the your entire bash history. If you're not like me, you don't even bother. With Chef, your infrastructure literally is code rather than a collection of unique artifacts. Top-level readmes and a few well-placed code comments should be sufficient. Take a look at this recipe for a basic sendmail configuration.

basic sendmail configuration with chef

Loading ....

Bash Doesn't Scale

Bash is a wonderful thing, but like all UNIX tools, it is fundamentally limited by design. Bash doesn't have a code reuse mechanism more powerful than functions. It uses a single global namespace. These and other limitations have made it hard for us sysadmins to reuse and generalize our shell scripts across different distributions, let alone different versions of *nix. Lastly, it is quite difficult to write shell scripts that are idempotent, that is, they have the same effect whether run once or 100 times. This is particularly true when manipulating configuration files. Let's take a look at how you would configure /etc/sudoers with chef.

Loading ....

All the variables in this template are passed in from this recipe. If you think you can replicate this template using sed, please submit it as a comment so we can compare results.

Last but not least, Bash can be just as unreadable and obfuscated as the darkest perl code. With Chef you have a high level, intuitive DSL for describing your configuration. Remember what I said about needing less documentation?

Technical Awesomeness

NOSQL FTW

One of the virtues that many *nix tools share is that they store their configurations in text files rather than binary formats or in a database. Chef stores your system configurations in text and in a database. It accomplishes this by using the document-oriented database, CouchDB. This makes the configuration searchable and higher performance. One of my annoyances with nagios is that I have to restart it every time I change any service or host. There are no annoying restarts with Chef. Each Configuration change you make takes effect instantly. Further you can use your favorite text editor and the command line to make configuration changes. Finally, using a database means that Chef is easily accessed through a GUI. We sysadmins often look down on GUIs as crutches for the Windows crowd. The Chef web UI is excellent for visualizing your own infrastructure, something that is hard to do while staring at plain text.

Knowing is Half the Battle

Chef uses Ohai to collect data about your system. Your recipes can access these attributes and make decisions based on them. For example, you can determine which version of Red Hat you are using simply by looking up the value of node['platform_version']. You don't have to cat | grep | awk to find out which release you are on.

Ohai makes cookbooks more dynamic and able to support different distributions. As we will see later, this is one of the reasons there are so many high-quality cookbooks available.

Search

Search is a feature in Chef Server that allows you to query the configuration information of all other servers and of globally-defined databags. This allows you to do things like configure clusters where a member of cluster needs to know not only about its own configuration but about the configurations of the other members of the cluster.

Example of using search with data bags. For the full recipe go here.

Loading ....

Knife
Knife is one of the truly great command line tools. It is your primary mechanism for interacting with the chef-server. Knife shares many usage patterns with git. If you love git, you'll love knife.

shef

shef works the way you work, in an iterative manner. Most of us system administrators are self-taught and we learn best by doing. Fire up shef and you can on the fly play with attributes and create recipes. Further, you can connect to your server and download the cookbooks.

Chef Grows with You

Chef uses pure Ruby as its configuration language, not a shackled subset of ruby, nor yet another custom configuration language. You only have to learn a small amount of ruby to get started with chef. Once you get beyond the basics of Chef, you can go farther with Ruby. Many of you are grumbling now because ruby sucks to compared to Perl/Python/TCL/<insert-interpeted-language-here>. Well, Ruby may pale in comparison to Python but it is still a powerful, full-featured language.

Just like Perl, there is a lot of dark magic inside Ruby. It can't be used carelessly or it will bite you in the ass. Unlike Python, Ruby does not make it difficult to do the wrong thing.

You can stop reinventing the wheel

Until Chef, we sysadmins did not have a truly modular way to abstract and share our system configurations. Please stop reading and browse http://community.opscode.com/cookbooks. Later on, you should look at the code in https://github.com/opscode/cookbooks. You will discover that some turncoat sysadmins are giving away our trade secrets. Now is the time for you to do the same. In fact, you can rip out a large chunk of your shell scripts and replace them with Chef cookbooks. You will find that many existing recipes meet your requirements and you can easily add new recipes to existing cookbooks for your unmet requirements.

Recently, I replaced a 500 line shell script with 7 cookbooks, 5 reused from community.opscode.com and 2 new ones written from scratch. In the process of replacing the shell script I actually wrote 4 cookbooks that unwittingly duplicated the functionality of existing cookbooks.

The emergence of the cloud-computing, whether on the public cloud such as EC2 or internally with openstack, means that we systems administrators will have to manage at least 10 times more systems. We have to become 10 times more productive. We have to shift from "managing servers" to "managing configurations." Chef is one of key tools to accomplishing this.

One Big Fat Chef Caveat

Chef makes good sysadmins more productive. It does not turn junior sysadmins into experts. In fact, it makes them more dangerous. As I overheard on IRC one day, "Chef is to sysadmins what C++ is to software engineering." This is very true. You can automate system configurations with off-the-shelf cookbooks but you have to understand exactly what they do and test them very rigorously.

Chef Myths

There are a few myths about Chef floating around the intertubes that need to be exploded. The biggest one is that you have to be a professional programmer to use Chef. This is untrue for a couple of reasons, including the assumption that system administrators don't program already. Writing complex shell scripts is programming. Creating Apache rewrite rules is quite close to programming. Further, Chef doesn't require you to be a Rubyist to get started, just as you don't have to be a Bash Hacker to use the command line. Take a look at this tutorial and you will see what I mean.

I hear murmurs now and then that Chef is "insecure" because the recipes are executed on the chef-client rather than on the chef-server. I argue that chef is more secure because of this. If recipes were executed on the chef-server, there would be a risk that a single malignant recipe modified other recipes to turn your entire network into a botnet. This would turn your chef-server into a single point of failure and make it a powerful attack vector.

Finally, some will argue that Chef sucks because every server you configure should be hand-crafted and beautifully unique in its own way. If you believe that, perhaps you should go into the virtual performance art and get out of the business of making the Internet awesome.

Wrapping Up

Chef is a truly powerful tool that every serious system administrator should consider using. It offers immense personal and professional benefits.

Please post comments on Hacker News

Marcus van Dam (m4rcu5) helped edit this post

Note: The title of this post is a reference to Eric S. Raymond's great post "Why Python?" which inspired me to learn Python, a long, long time ago.

Puppet vs. Chef, Fight!

2011-10-09T07:46:00.000-07:00

Update: see Puppet vs. Chef en francais, not a translation but an original article inspired by this one

If you aren't yet using Puppet or Chef for managing your *nix infrastructure, you should seriously consider it. The benefits are enormous. Frankly, I don't think you __can__ manage a modern infrastructure without using Puppet, Chef, or reinventing a configuration management framework with your own custom scripts. However, you can't use both Chef and Puppet together, so which should you choose?

If you're like me, you obsess over tool choices. You agonize for weeks over these decisions. Why? Because you will end up putting a huge personal investment in the chosen tool. There isn't all that much on the relative benefits of Puppet vs. Chef besides this quora thread. Below you will find my completely subjective, non-comprehensive study of the matter. TL;DR, Chef wins but by only a narrow margin.

The Criteria

Community Strength
Leadership
Corporate Adoption
Technical Merits
Hands-on Experience

Community Strength

Community Size

Number of people on IRC on Sunday Oct 9th, 2011 10 AM GMT
#puppet 541 users
#chef 233 users

Mailing Lists
puppet-users 3363 members, 5961 topics
chef: 600 members, # of topics unknown

Regarding IRC, Chef seems much more U.S.-centric. It can be hard to get answers during the European workday. The situation is probably better for Asian users as they can catch the North American community when it is early evening on the West Coast of the United States. That is not true for Puppet, where you can get an answer at any time of day or night. R.I. Pienaar is based in London and seems to be on IRC at all hours. Somehow he manages to kick out awesome code and be continuously available. Further, James Turnbull doesn't seem to sleep either and is incredibly helpful.

Quality of Cookbook Sites

http://community.opscode.com/cookbooks
http://forge.puppetlabs.com/

Github Repositories:
puppet: 1622 repositories
chef: 1299 repositories

While Chef is much younger project than Puppet, it is really impressive how many cookbooks there are and their respective quality, especially when Chef is less than half the age of Puppet. From what I can tell, most people bypass the community sites and go straight to github to find their Chef cookbooks or puppet modules.

Documentation, Official and Informal

Both Puppet and Chef have excellent reference documentation. In my opinion, the Puppet docs help you get up and running much faster. Pro Puppet, by Puppet Labs engineer James Turnbull, is an awesome introduction to Puppet. I found it immensely helpful. There is a 50-page O'Reilly publication on Chef, Test-Driven Infrastructure with Chef, but it cannot match the breadth of Pro Puppet. I have heard that O'Reilly are going to be publishing a feature-length definitive guide, scheduled to be out by the Summer of 2012.

The Chef community definitely tends more towards programmers and rubyists than systems administrators. I think that puppet meets the needs of the majority of its users so well that they never try Chef.

Leadership

We all know that leaders can make or break open-source projects. The BDFL's personality shapes the project more than the actual code. What would linux be without Linus Torvalds' stewardship? I am wary to bet against Puppet founder Luke Kanies. He is both a genius and a visionary. He reminds me a little of another Reed College alumnus. I highly recommend listening to Luke Kanies interview on DevOps Cafe. He shows himself to be very intelligent and very opinionated. My understanding is that his original motivation for creating Puppet was to make life better for the average sysadmin.

The resumes of the Opscode team are just amazing. CTO Chris Brown and co-founder Jesse Robbins were core infrastructure engineers at Amazon, the company that perhaps more than any other understands how to manage infrastructure at scale. Listen to co-founder Jesse Robbin's Interview and you will find him to be very pragmatic about infrastructure and less opinionated than Luke. His motivation was to create an open-source tool that solved the kinds of infrastructure problems he dealt with while working at Amazon.com.

On the funding side, both companies have closed substantial Series B funding rounds. We're talking more that 10 million dollars each. Both of these companies can throw a number of talented software engineers at the devops problem domain for a while to come.

Corporate Adoption

I believe that both Puppet and Opscode corporations will become extremely successful. Jesse Robbins explains this best in Virtual Strategy Magazine.

Our primary competitor is the in-house, home-grown tooling that every systems engineer has had to cobble together to deal with their legacy environments. That’s our principal competitor.

That's right. The biggest competitor to Chef and Puppet isn't each other but that hairball nest of Bash and Perl that holds your data center together.

Still, Puppet has an amazing customer list but Chef is also racking up wins. If Puppet Enterprise 2.0 delivers half of what its press release promises, it will be a huge success. Puppet Enterprise's integration with mcollective is particularly exciting.

I am particularly impressed that a number of companies have built their own internal provisioning tools on top of Chef, notably Rackspace Cloud Builders and Dell's Openstack team.

Technical Merits

Puppet is designed to be approachable by systems administrators. It is not necessarily written to empower rubyists. Writing Puppet manifests, at least initially, feels much like writing nagios or apache configurations. However, it starts feeling much like programming once you get beyond simple resource specification.

Chef is designed to help engineers manage the configuration of their infrastructure at scale. It does not have the same focus on easy-of-use, nor is it as opinionated as Puppet. Now let's look at a few salient features.

Syntax

Here are examples of how to create a user in first Puppet and then chef

from http://docs.puppetlabs.com/learning/ral.html
user { 'dave':
      ensure     => present,
      uid        => '507',
      gid        => 'admin',
      shell      => '/bin/zsh',
      home       => '/home/dave',
      managehome => true,
}

Now in Chef

user "root" do
    username "root"
    password "$1$PdV3HPti$WXmqrQEVj88fVTHevErxq."
    shell "/bin/bash"
    system true
    supports :manage_home => true 
end

They don't look terribly different, but now let's throw some Ruby into the mix. In the following example, I iterate through a list of users stored in a data bag

data_bag('users').each do |user_name|
   u = data_bag_item('users', user_name)
   user user_name do	
        username user_name
        password u['password']
        shell u['shell']
        supports :manage_home => u['manage_home'] 
   end
end

Now you should have a better sense of the difference between Puppet and Chef. Some may have a valid criticism that Chef lets you do too much by allowing you use regular Ruby. I have to admit that undisciplined use of Ruby is as dangerous if not more dangerous as perl or bash abuse. At the same time, it is quite empowering. The Chef community really needs some kind of chef-lint tool to tell you when you're doing something you shouldn't.Update: Foodcritic now excellently fills that need.

Debugging

I miss noop mode in puppet, though this is on the way for Chef. It is a wonderful way to see how your puppet configuration differs from what is actually on disk without actually changing anything. I have heard some refer to noop mode as "Lie to me mode" though I have yet to have it bite me in the ass. I might feel very differently if I had experienced that personally. I also liked how puppet spit out the text of rendered templates to the console. Maybe you can do this with Chef as well, but I haven't figured out how to yet.Update:Chef now has the equivalent of noop mode with its why-run mode.

Chef's Secret Sauce

Chef's Data Bags are incredibly useful. They are a very nice way to feed lists of users to multiple cookbooks. Think of them as global variables for your chef-server. One the benefits of data bags is that they allow you to separate your corporate information from your cookbooks. Thus, making it easier for Chef users to open-source their cookbooks. Puppet has extlookup but that is quite a bit more limited than data bags. It looks like Puppet maybe getting something similar to data bags with Hiera.

I also have to praise the Search function. It is an extemely intuitive way to to tie together dependencies between nodes, such as between a nagios server and its clients. Puppet has a way to "export" resources but I could never get my head around how that worked.

In my experience, Chef has a more powerful read-eval-print-loop (shef) than Puppet does. I use shef quite frequently and I cannot understate its helfulness. Puppet does not have a similar repl as far as I am aware.

Server-side, Chef is much more complex but also seems to be better thought out. I love the fact that Chef uses a document-oriented database. This combines the benefits of using files to store your configuration (easy to understand) with the benefits of using a database (speed).

Hands-On

I don't believe there are many in the Puppet community that have extensively experimented with Chef before choosing Puppet. There are many in the Chef community that have worked previously with Puppet and found that it did not meet their needs.

I started working with Puppet in June of this year and quickly fell in love. I read Pro Puppet and did all the exercises. Next I set to rewriting my 400 line linux provisioning script in puppet manifests. At first it went well but then things got more complicated. I kept tripping over the DSL's syntax and was surprised that there weren't more puppet modules in puppetforge or github. I felt that I had to learn both the Puppet DSL and Ruby to really to be effective with Puppet. I started thinking that the grass might be greener on the other side.

Two things convinced me to try out Chef. The first was the announcements by Dell and Rackspace that they had built there Openstack provisioners on top of Chef. If the people pioneering devops chose to build their platform on top of Chef, then it definitely deserved my attention. The next was Jesse Robbins' interview on DevopsCafe. He made all the action in the Chef community sound so exciting that I had to go check it out for myself.

It took me three days to get Chef Server up and running on RHEL 5.7. Next, I started writing my own cookbooks. Now I have nearly converted that linux provisioning script to a cookbook. Not counting the time it took me to set up the server, it took me roughly half the time it took with Puppet.

It was easy to get started with Puppet, but things became more complicated with time. With Chef I am having the opposite experience. Again, the single biggest drawback to Chef is that it has a steeper learning curve than Puppet. Further, most of the existing tutorials focus on the deeper concepts in Chef rather than giving novices quick gratification. To this end, I have started the "Dirty Chef" series of tutorials.

Summary

Final Scorecard
Community -- Puppet wins
Leadership -- Draw
Corporate Adoption -- Draw
Technical Merits -- Chef
Hands-On -- Chef

According to this non-scientific criteria, Chef is the better tool. In my humble opinion, Chef is technically superior. Don't buy the argument "Chef requires you to be a programmer, Puppet doesn't." While Puppet is geared towards non-programmers, it becomes regular programming once you do anything moderately complicated. That's when Puppet's DSL becomes more of a liability than an asset and where Chef really shines. Now while chef is the winner of my little bake-off, it is not without faults. Puppet has far better support for RHEL/Centos and it does have a larger, more responsive community.

Dear Puppet community, please know that I write this blog post with great ambivalence and I still love you. Jame Turnbull and R.I. Pienaar, thanks for all your help via IRC, Twitter, and the mailing list. Puppet Labs is developing a toolchain that is much larger than just the puppetmaster. This toolchain may turn out to be superior to what Opscode and its respective partners come up with.

On a final note, please know that both of these tools are good enough for configuration management and are only one part of a larger toolchain.

P.S. I will delete any comments that trash either Puppet or Chef. They are both excellent tools, built by
visionaries.

Most of the discussion related this article seems to be happening on Hacker News.

A Stupid, Simple Chef Tutorial for Dirty, Lazy Fry Cooks (Like Me) Part I

2011-10-07T00:55:00.000-07:00

disclaimer: I am not a chef expert. To really learn chef, you need to go to the wiki

I really like the chef documentation, I really do. It is an excellent reference. It is the devops equivalent of the manual they must use at the Le Cordon Bleu cooking school. But I have to admit that sometimes I am just a dirty, lazy fry cook, especially when playing with a new technology. I want instant gratification or at least in under 30 minutes. What I love about the Puppet documentation, is that it gets you started quickly, before you begin optimizing and modularizing and all that other crap that good devops chefs do. To that end, I have created a chef repository that is exactly what you shouldn't do in production but is hopefully useful for learning.

In this tutorial, I am going to use chef-solo for the purpose of the tutorial. Chef-solo simply means running chef with out connecting to central chef server that provides you with cookbooks and some other functionality. Chef-server is awesome but currently a pain in the butt to set up. I am also going to use RHEL 6.0 but you can use CentOS.

Why Chef?

If you aren't yet using configuration management with Chef, Puppet, or similar tool you're just plain doing it wrong. I'm doing it wrong. I cobble together my infrastructure with a combination of kickstart, bash scripts, fabfiles, and random python. But the diversity of applications I support and the sheer number of virtual machines are increasing rapidly. My agglomeration of virtual glue and duct tape is starting to look more like a house of cards and less like modern infrastructure.

In the immortal words of Puppet founder Luke Kanies:

"SSH in a for loop just doesn't cut it anymore."

To build Luke's statement, our imperative bash/perl/python scripts are lousy for customizing server templates to specific node attributes and they suck at maintaining system state. I have tried both Puppet and Chef and I prefer Chef for a number of reasons. The main ones are that it has a more elegant and easier to use DSL and that chef-server has the awesome search functionality.

Installation

Install my chef repository somewhere on your system. I recommend /opt.

git clone git://github.com/bryanwb/dirty-chef.git

Next install chef. Here is the main install guide

This should work on either CENTOS/RHEL 5 or 6

For RHEL 5

$ rpm -Uvh http://rbel.co/rbel5

For RHEL 6

rpm -Uvh http://rbel.co/rbel6

Finally, on either


$ yum install -y rubygem-chef ruby-shadow

Now edit solo.rb, you have to edit these paths to reflect the actual location of dirty-chef otherwise chef-solo will not work

file:solo.rb


file_cache_path  "/opt/dirty-chef/"

cookbook_path "/opt/dirty-chef/cookbooks"

data_bag_path  "/opt/dirty-chef/databags"

If you are using RHEL 5 see the install link above, it has steps for RHEL 5 but they are more complicated Run the following command

$ ohai | less

This shows you all the info that chef knows about your system. You can do reasoning in your recipes based on these values.

Start Cooking

Here is a stupid hello world

$ cd dirty-chef
$ chef-solo -c solo.rb -j node.json
ohai!

What the heck did we just do? The solo.rb tells chef where the cookbooks are, i.e. where the recipes live, and node.json tells chef-solo which recipe to actually apply.

file:node.json


{

  "run_list": [ "recipe[dirty]" ]

}

This loads the recipe in cookbooks/dirty/default.rb

If you get any errors here, you should make sure that you changed the paths in solo.rb to reflect the location of dirty-chef/

Now let us do something more significant. You can use chef to declare "resources" in this case a package. Let's install subversion. In chef recipes, you spend most of your time declaring resources, in this case I declare that I want subversion installed and chef takes care of the rest.

file:cookbooks/dirty/recipes/example1.rb
package "subversion"

$ chef-solo -c solo.rb -j example1.json

file:example1.json

{
  "run_list": [ "recipe[dirty::example1]" ]
}

This loads the recipe in cookbooks/dirty/example1.rb

Here is an example of installing multiple packages, take a look at cookbooks/dirty/recipes/example2.rb

$ chef-solo -c solo.rb -j example2.json

What is that syntax of the configuration language? Why it is pure ruby. If you like perl, you will like ruby as it is the lovechild of perl and smalltalk. Here is a good quick reference

You still are not impressed? True you could do all these things in a kickstart more easily and without having to go to so much trouble. There are several reasons chef is better than kickstart. You can use chef to maintain the state of your server while you can only use kickstart to initially set up your server. Once you install your server from a kickstart, you cannot rerun it w/out reformatting it. Chef on the other hand is "idempotent" Meaning it has the same effect whether run once or 100 times. Say for some reason subversion is removed from your machine. The above chef recipe will reinstall it w/out needing your intervention. You can seamlessly handle different platform versions. You can write one recipe that handles both RHEL 5 and RHEL 6 by doing reasoning on node attributes. Maybe you are like me and love nothing better than a good REPL (Read-Eval Print Loop), i.e. console. Chef comes with an awesome REPL, shef. Let us use it to play node attributes and even load recipes interactively

[root@woof-chef recipes]# shef

loading configuration: none (standalone shef session)

Session type: standalone
Ohai2u root@woof-chef!
chef > if node["platform_version"] =~ /6./
chef ?> puts "hi rhel 6!"
chef ?> end
hi rhel 6!
[ type ctl-D to exit ]

In shef, you have access to all the node attributes and cookbooks on your system. Now let us put the same logic in a recipe.

$ chef-solo -c solo.rb -j example3.json

Now we still haven't actually done anything useful. We will get to that in Part II

What is devopsanywhere?

2011-10-03T01:47:00.000-07:00

Why DevopsAnywhere? Because our profession is undergoing some profound changes. On the positive side, the best of us are in increasing demand. On the other hand, it will take increasing sophistication on our parts to keep up w/ the blistering pace of change in our industry. I believe that to really keep up-to-date, you have be an autodidact, that is someone who is primarily self-taught.

In my experience, autodidacts are randomly distributed around the planet. Linux obsessing, python hacking, ruby scripting geeks are even more randomly distributed. Should companies really want to recruit us, they will have to hire us where we are, be that Kathmandu, Sicily, or Utah. Gone are the days where system administrators lived at the datacenter, as I explain in greater detail later.

As system administrators we were tied to the datacenter. Even if we worked remotely, we had to be within easy driving distance. We were software/hardware hybrids that understood all the various permutations of RAID 0/1/5/6/10. We looked on with envy at the alpha programmers who could do their jobs while lying on the beach in Thailand or while huddled in a cozy cabin in the French Alps.

Our profession is transforming from System Administrators to Devops Engineers. Devops is a bit tricky as it is as vague a term as agile. Still, it points to a new class of engineers that are focused on operations and work seamlessly with software developers. We are self-taught engineers who have the luxury to work remotely. This blog will be entirely focused on the concerns and interests of devops engineers working remotely. Further, I won't talk about expensive proprietary software or hardware. Actually, I will rarely talk about hardware at all.

Some, like John Willis and Damon Edwards of the awesome Devops Cafe, will argue that process is more important tooling. I agree with this 100%. However, as system administrators we often aren't in the position to reorganize our team's management structure. We are in a position to introduce new tools and lobby for better ways of working.

Here is a comparison of the requirements for being a SysAdmin and that for Devops Engineer

A new Set of Skills

System Administration today requires:

deep knowledge of *nix, including linux, solaris, and even some proprietary unices like HP-UX
you really __know__ bash
some basic version control w/ cvs/svn
simple scripting w/ bash/perl/python
understanding of virtualization
You can troubleshoot hardware, firmware issues, even flip a dip switch or change SCSI IDs if necessary

Devops requires:

deep knowledge of Linux -- generally not Solaris, AIX or HP-UX
you __know__ bash
you are very proficient in svn and/or git
You are a practicioner of puppet or chef
system scripting with ruby
System packaging -- ruby gems, rpms, debian packages
You can setup package repositiories on a whim
you understand unit testing and behavior-driven testing, hopefully you practice it w/ Cucumber or something like it
AWS, Rackspace, cloud provisioning -- but of course

Hardware is now (mostly) irrelevant

The only NIC cards a devops engineer cares about are the Intel e1000 or whatever paravirtualized driver your hypervisor of choice offers. We know that working on hardware issues only lowers our hourly billing rate. Now, more than ever, the power is in software and hardware-oriented solutions are to be avoided.

I must confess that I have an ulterior motive for starting this blog. I am a linux system administrator who is following his wife's career around the world. Previously we were in Nepal, now we are in Italy, next up who knows! But I want to work with top-notch teams wherever I end up and not be tied to whatever opportunities are locally available. If you are a devops guy and you feel the same, let me know!

How Ruby is beating Python in the battle for the Soul of System Administration

2011-09-16T11:12:00.000-07:00

Roughly three years ago I purchased the book "Python for Unix and Linux System Administration" and it convinced me that Python would be the default scripting language for linux system administrators for the foreseeable future. I was working on the One Laptop Per Child project at the time, where Python was the lingua franca. It seemed that Red Hat was using Python for almost every new project. Further, Unladen Swallow was making rapid progress. I couldn't have been more wrong.

Through a combination of historical accident and some seemingly minor feature differences, ruby is inexorably becoming the dominant scripting language for linux system administration.

Before I go any farther you die-hard sysadmins are probably screaming "We already have a scripting language, Bash!" While we love bash and commandlinefu, Bash becomes a liability not an asset once your script exceeds 100 lines and a total nightmare if you need to parse or output HTML, CSV, XML, JSON, etc.

Historical Accidents

In 2004, Luke Kanies set about building Puppet, a configuration management system to improve upon the deficiencies he saw in CFEngine. From the excellent On Ruby blog

I tried to implement my idea in perl, but I just couldn’t get the class relationships to work (the attributes and resource types each needed to be classes, according to the design in my head). This was back when Python was the shiznit, so I naturally tried it, but Python just makes my eyes bleed (and no, it wasn’t the whitespace, it was things like the fact that ‘print’ was a statement instead of a function, and ‘len’ was a function instead of a method).

I had a friend who had heard Ruby was cool but hadn’t actually tried it himself. Since I was just messing around at the time, I figured I’d give it a go. Four hours in, never having seen a line of Ruby previously, I had a functional prototype.

A couple years, a number of Puppet developers felt that Puppet didn't meet their needs. They started their own configuration management tool, Chef, heavily inspired by Puppet. The biggest outward difference between Puppet and Chef is that Chef uses pure ruby as its "recipes" whereas Puppet uses its own configuration language based on ruby.

Both Puppet and Chef are seeing rapid adoption by big companies according to BusinessWeek. If you aren't yet using Puppet or Chef, you should be planning on it in the near future. Whether you choose Chef or Puppet, you are effectively scripting your infrastructure with ruby. After spending 25% of your time working with Puppet, you will be much more likely to reach for ruby for your next scripting task.

Popular Projects

Off-hand here is a non-definitive list of sysadmin/devops related projects in ruby

puppet
chef
vagrant
mcollective
cucumber (behavior-driven testing)
capistrano
rake (ruby Make)
aeolus project/openshift
cloud foundry
graylog2
logstash
travis-ci

and in Python:

What's important here isn't the length of the respective lists but the importance of the individual projects. Chef, Puppet, and Vagrant are the new hammers and screwdrivers of system administration. If you are a sysadmin and aren't yet using these tools, don't worry, you will be sooner or later.

Openstack deserves special attention as it is a very exciting but early stage project. It could play a big role in your future as a system administrator.

Please notify me of significant devops projects that are missing from this list.

What a ~~Girl~~ Sysadmin Wants

Here are the features in a scripting language that a sysadmin wants

A DSL for the problem domain
High productivity, i.e. concise and expressive syntax
Easy to interaction with shell commands
Regular Expressions
powerful one-liners

Notice that performance is not on this list of requirements. Ruby is significantly slower than Python and this was particularly true for the 1.8.* series of Matz Ruby Interpreter (MRI). However, performance just isn't critical for 90% of our work as sysadmins. We care about productivity more than we care about performance. Readability is nice but it is a distant second to productivity.

Python doesn't have regular expressions embedded in the language, probably to improve readability. It also limits the number of top-level built-in global variables. From a language design perspective, this is much cleaner. From the perspective of your average street-trained, stress-laden, vi-addicted sysadmin, it is annoying.

These top-level variables also make one-liners more concise. Here are just a few

$_ last line read from STDIN

$? last exit code from a child process

$stdin reference to stdin

$stderr reference to stdout

$stdout reference to stdout

Here is an irb session to show you these values in action

hitman@hiroko:~/pr$ irb

irb(main):001:0> %x[ ls -a ]

=> ".\n..\nfoo1\nfoo2\nfoo3\n"

irb(main):002:0> puts $?

0

=> nil

irb(main):003:0> %x[ ls xys ]

ls: cannot access xys: No such file or directory

=> ""

irb(main):004:0> puts $?

512

=> nil

"IRB?" you scoff "You will have to tear IPython from my cold dead hands."

I love IPython. I have used IPython for 4+ years and IRB doesn't hold a candle to it. That said, ruby's shortcuts are really useful, enough to compensate for IRB's shortcomings. There is something called wirble that I haven't tried yet that may make irb a lot more productive.

Here some python code to detect whether a machine is a VMware VM.

# edit: fixed python code tks to kstrauser

  import os
    if 'vmware' in os.popen('dmidecode').upper():
        print 'this is a vmware vm'
    else:
        print 'this is not a vmware vm'

Here the same code in ruby

`dmidecode`
if $_ ~= /vmware/i
puts 'this is a vmware vm'
else
puts 'this is not a vmware vm'

Frankly, this kind of code makes my eyes bleed. The python example is far more readable and maintainable. That said, a lot of sysadmins would appreciate the terseness of the ruby example.

Let's look at writing one-liners in both languages.

This code prints out the first 10 lines in a file using Python

python -c "import sys; sys.stdout.write(''.join(sys.stdin.readlines()[:10]))" < /path/to/your/file

here is the same in Ruby


ruby -ne 'puts $_ if $. <= 10 ' < /path/to/your/file

Compare for yourself between these collections of oneliners in python and ruby. If ruby reminds of perl, your eyes do not deceive you. In many ways it is the love child of perl and smalltalk.

DSLs FTW

Some time ago, I tried to watch all the Structure and Interpretation of Computing lectures. This endeavor failed miserably but I recall Harold Abelson saying that every large program should have its own internal DSL suited to the problem space. This is debatable in regards to the larger world of programming but I think it is very apt for system administration. We spend our entire careers with n different DSLs. Each different configuration file format is its own DSL.

If you compare rake (Ruby Make) to rails code, they look quite different, almost like different languages. If you compare fabric code to a django class, they look quite similar. This is both a strength and a liability. I am not a language lawyer but it seems much easier to create DSL's (Domain Specific Languages). Ruby certainly spawns DSLs with much greater frequency than python. No single pythonic build tool dominates the problem space like rake does in the ruby community. Most python projects seems to use setup.py for administrative tasks even though that is not its explicit purpose.

Both puppet and chef are DSLs for system administration. Capistrano is a DSL for application deployment. Ruby's operator overriding and blocks lend themselves to the easy creation of DSLs.

In Summary

Ruby's greatest strength is its amazing flexibility. There is a lot of "magic" in ruby and sometimes it is dark magic. Python intentionally has minimal magic. It's greatest strengths are the best practices it enforces across its community. These practices make Python very readable across different projects; they ensure high quality documentation; they make the standard library kick ass. But the fact is that we sysadmins need flexibility more than we need raw power or consistency. Still, these are not the real reasons that ruby is overtaking python.

Ruby is fast becoming the default scripting language for sysadmins because back in 2004, Luke Kanies looked at Python and felt ill (I had the opposite reaction). As a sysadmin you either currently are or soon will be using Puppet or Chef on a daily basis, spending a heck of a lot of time essentially coding in ruby. Personally, I much prefer writing python but am shifting to writing my scripts in ruby because I spend so much time with Chef.