DevOpsCasts

2012 Hiatus

2012-01-09

Hello again, everyone. As you may have noticed it’s been awhile since I’ve done any videos. I have a number of big events going on this year, including having a new son that will be taking up my time for awhile. As such I’m hanging my microphone up until sometime around fall 2012.

In the mean time if you or anyone you know is interested in screencasting devops topics, I’d be happy to help out with production and post them here. Feel free to email me at mat@mashion.net if you’re interested.

Thanks for watching and see you later this year!

Chef Server Usage

2011-09-21

Welcome back! Sorry for delay in getting this episode out. I’m currently on vacation overseas so it’s taken a bit more time to prepare everything.

In todays episode we’ll go over how to upload cookbooks to your Chef Server as well as connect and provision a node. This episode is a continuation of our last episode so you’ll want to follow along with that before trying out this one.

Prep

Chef server VM at “Pre Part 2” snapshot
Move ~/.chef to ~/.chef.orig
Copy ~/.chef.part1 to ~/.chef
open slides

Script

(slide start)

Hi, I’m Mat Schaffer and welcome to DevOpsCasts where we go over topics that help tear down the wall between development and operations.

(change)

Previously on DevOpsCasts we talked about Chef Server. If you followed along you should now have a functioning Chef Server that we’ll build on in this episode.

(change)

Now we’re in Part 2 where we’ll talk about how to upload cookbooks to your Chef Server and attach application nodes so that they can run your cookbooks and provision themselves on demand. So let’s get to it.

(terminal)

$ knife client list

First, run a quick knife client list to make sure you can talk to your chef server. If you have any trouble getting a list back, you may need to debug a bit. One common problem that we saw last time was clock skew, so make sure your laptop and server are in sync with a time server.

(overlap: sudo ntpdate pool.ntp.org)

Remember we used the ntpdate command to fix the the clock.

(terminal)

$ vi ~/.chef/knife.rb
# show chef server URL

If you still can’t get a list, check your knife.rb. If you’re running Chef on a VM, double check that the server URL is correct. Some VMs will rotate IP addresses between boots.

(overlay: mat@mashion.net)

If you’re still having trouble, feel free to email me or comment on the show notes.

(browser: https://github.com/opscode/chef-repo)

(terminal)

$ git clone https://github.com/opscode/chef-repo.git devopscasts
$ cd devopscasts

First we’ll need space to keep our cookbooks. When using chef server we’ll use the Opscode chef-repo as a base. This allows us to use some built-in knife commands to manage our cookbooks and their dependencies.

(browser: opscode mysql cookbook page)

In this episode we’ll be installing a mysql server. We’ll use the off-the-shelf mysql cookbook this time instead of writing our own. This is a much more complex cookbook than the nginx cookbook we wrote in episode 1 so it’ll serve as a good base for upcoming screencasts on the more advanced features of chef.

(terminal)

$ knife cookbook site install mysql -o `pwd`/cookbooks
$ ls cookbooks

Knife comes with a built in tool for downloading these cookbooks. The command to run here is knife cookbook site install mysql and we specify the dash-O option with the full path to the cookbooks folder. This will download the mysql cookbook from opscode as well as the cookbooks it depends on. In this case there’s only one cookbook dependency, openssl. The mysql cookbook uses the openssl cookbook to generate random secure passwords for the mysql server.

(terminal)

$ vi ~/.chef/knife.rb
  cookbook_path [ '/Users/mat/devopscasts/cookbooks' ]

To avoid setting the cookbook path every time we run a command, we can change the cookbook path listed in knife.rb.

We can use these cookbooks as-is, so next we’ll upload them to the chef server using the knife cookbook upload command.

$ knife cookbook upload mysql -d

Here we specify the dash-D option so that the openssl dependency will also be uploaded.

Now that we’ve uploaded the cookbooks we’ll attach a node to this chef server that we can run them on. I have another VM ready here but it hasn’t been prepared yet.

$ ssh ubuntu@192.168.139.132
$ sudo hostname mysql
$ sudo vi /etc/hosts
    192.168.139.132 mysql mysql.devops.mashion.net
$ hostname -f

Like we did with our Chef server, we’ll set the hostname first. Chef uses host names as the unique identifier for each node so it’s important to set this early on. While you can change the host name later it’s somewhat involed, so better to have it set right from the beginning.

The last gotcha before we bootstrap our host is that we’ll also need the validation key from our chef server so we can upload it to our new node. This key is used by the node to ensure it’s talking to the expected chef server.

$ ^D
$ ssh ubuntu@192.168.139.130
$ sudo cat /etc/chef/validation.pem

We’ll do this simply by copying the text and pasting it into a file in our local chef directory.

$ ^D
$ vi ~/.chef/validation.pem
$ vi ~/.chef/knife.rb
  validation_key '/Users/mat/.chef/validation.pem'

Then we fix our knife.rb to reference the file we’ve just created

$ knife bootstrap -x ubuntu -P ubuntu -d ubuntu10.04-gems --sudo -r 'recipe[mysql::server]' 192.168.139.132

Now to prepare the node for running Chef server recipes we’ll use the knife bootstrap command. This command is provided with Chef. It’s a little less automatic than the knife prepare command that comes with the knife solo gem, but it works in a way that fits a bit more closely with Chef Server.

The first arguments are my username and password followed by the dash-D option which tells the bootstrap command what distro I’m using and how I’d like to install chef. In this case it’s ubuntu using rubygems to install chef. The sudo argument tells knife to preface any commands with sudo since the ubuntu user doesn’t have root privileges.

The bootstrap command will install the components necessary to run Chef, configure the node to talk to the same server in our local knife config and set up a node configuration that includes the mysql server recipe.

$ knife node list

Once this is done we can run knife node list and see the node we just set up in the list.

Now let’s try connecting to our new mysql server.

$ ssh ubuntu@192.168.139.132
$ mysql -u root

As you can see, we need a password to connect. That password was generated randomly and stored safely in Chef’s node database.

$ ^D
$ knife node show mysql -m

To access it we use the knife node command with the dash-M option to show normal attribute data. This will show us the node data that was created during a recipe run but skip all the system information that was discovered by ohai.

So here’s our password which I’ll copy so we can connected to our server.

$ ssh ubuntu@192.168.139.132
$ mysql -u root -p

And now we’re in. Normally you won’t need to copy and paste this around since you can access that data from inside the chef recipes. We’ll talk more about how to do that in later episodes.

Let’s say we want to make a change a configuration setting on mysql, for example the max allowed packet size. Since our configuration is controlled by chef now, we’ll look in our mysql cookbook to see how it’s getting set up.

(redcar, cookbooks/mysql)

In the mysql cookbook directory there’s a templates/default folder that contains the my.cnf.erb that gets used to generate the mysql configuration on the server.

(redcar, cookbooks/mysql/templates/default/my.cnf.erb) Show line 141

We can see here that the max allowed packet size gets set from the mysql tunable max allowed packet node attribute.

(redcar, cookbooks/mysql/attributes)

The default values for node attributes are read from the attributes file that matches the recipe we ran on this node. In this case server.rb.

(redcar, cookbooks/mysql/attributes/server.rb) Show line 45

If we scroll down to find the max packet size, we can see the default here is 16MB. Now we could change it here, but that would increase the size for any node we used this cookbook on. I’d rather have this parameter default to 16MB but be 32MB on our new mysql node.

(terminal)

$ knife node edit mysql

To do that we can use the knife node edit command. This will open the node configuration file in our preferred editor. For me this is vim.

(overlay: export EDITOR=vim)

To set your editor place a line like this in your bashrc or local user profile.

(vim)

"tunable": {
  "max_allowed_packet": "32MB"
}

Now in the mysql section, we can add the tunable max allowed packet attribute, save and quit the editor and the node configuration will be fixed the next time chef runs. Now we could use the knife ssh command to explicitly invoke the chef client on our node. But that’s not nearly as much fun as having our node continuously update itself.

To set up our node to continuously update itself we’ll need to install the chef client. Of course, there’s a cookbook for this.

(terminal)

$ knife cookbook site install chef-client
$ knife cookbook upload chef-client -d

We’ll use knife again to download it from opscode and upload it to our own chef server.

$ knife node edit mysql

Then we’ll use the knife node edit command to add the chef-client recipe to the node’s run list.

# Add chef-client to run list
# Add attributes
  "chef_client": {
    "interval": "5"
  }

Now by default the chef client checks in once every half hour. For this demo we’ll set it to check in every 5 seconds. So we can see the effects of our work more quickly.

$ knife ssh -m 192.168.139.132 -x ubuntu -P ubuntu sudo chef-client

And finally we’ll use the knife ssh command to run the chef-client manually. The knife ssh command can also be used search for nodes and execute commands on all node of a given type, but we’d have to take a bit more time to properly set up DNS and SSH to allow for that. So today we’ll just specify the host and authentication information manually.

$ knife ssh -m 192.168.139.132 -x ubuntu -P ubuntu cat /etc/mysql/my.cnf | grep max

We can use knife ssh again to show that our max packet size has indeed been increased to 32MB.

$ knife node edit mysql
$ # Change max to 64MB

Now we can edit the node again and change it to 64 MB. If we wait 5 or so seconds, the chef client will come around again, find the updated configuration and update the max allowed packet size.

$ knife ssh -m 192.168.139.132 -x ubuntu -P ubuntu cat /etc/mysql/my.cnf | grep max

Of course, this gets even more exciting as you add more nodes to your Chef server. But to really make use of it we’ll have to cover a bit about how recipes, roles and attributes work together to coordinate nodes across your environment. So we’ll save those topics for next time.

(keynote)

For our next episode I’m planning to take a break from Chef and talk about some monitoring packages that you can use to help make sure your environment is operating correctly. Of course if you have any feedback, feel free to leave a comment on the show notes.

Thanks for watching!

Chef Server Installation

2011-08-19

Welcome back! This time we’ll start in on Chef Server. It’s a big topic, so we’ll do this in two parts, in this first part we’ll cover the installation then cover how to use it in the next episode.

Prep

Make serverdemo kitchen
Prep VMWare VM, “Chef Server”, upload keys, knife solo prepare
open slides
Move ~/.chef to ~/.chef.orig

Script

(slide start)

Hi, I’m Mat Schaffer and welcome to DevOpsCasts where we go over topics that live in that nebulous space between development and operations.

(change)

Today we’ll start in on Chef Server. As I mentioned last time, Chef Server is a larger, more scalable approach to using Chef.

(change)

Since Chef Server is a somewhat large topic, we’ll be covering it in two parts. Today we’ll do an overview of Chef Server and I’ll show you how to install it.

(change)

Chef Server introduces another server to your architecture.

(change)

This is your Chef Server. And it serves as a central repository for cookbooks, recipes and other configuration information.

(change)

And your application servers will now have the chef-client daemon running on them.

(change)

Like last time, you’ll develop your cookbooks on your local machine. But when we’re finished, we’ll use knife to upload them to the Chef server.

(change)

Since the chef-server and chef-client processes are in constant contract, the application servers will then pick up on the new cookbook and run through them in the same way we saw with chef solo.

(change)

Chef Server is especially useful if you have a large number of servers. With Chef Server you don’t have to wait for each machine to run the recipes before moving on to the next one and tranfering cookbooks to the app servers is done for you.

(change)

It’s also good if you have inter-dependent services. This is because Chef Server stores not only just cookbooks, but also serves as a system of record for configuration information. You can even search your infrastructure for nodes that serve certain roles or configuration. For example, your application recipe could search for any available database node and use that rather than hard-coding the database information in the recipe.

(change)

And finally it’s useful in situations where you have multiple deployment engineers. With Chef Solo you have to be careful to coordinate the provisioning so that two people don’t try to run cookbooks at the same time. Since Chef Server runs cookbooks asynchronously, many people can upload cookbooks as they work on them and the chef client daemon will pick them up on the next run.

(virtualbox: ChefServer)

We’ll be running Chef Server on this VM I have handy. I’ve already installed my ssh keys and prepare the box to run chef by using the “prepare” command from knife solo.

(overlay: knife prepare ubuntu@192.168.139.130)

I did this by running “knife prepare” and passing in the username and host of the VM. One thing to note is that I’ve also given this VM a bit more memory. Chef Server can be somewhat memory intensive as your cluster grows so I recommend giving it at least 1 gigabyte of memory for testing and at least 2 for production work.

There are a number of ways to install Chef Sever including options that use OS-specific packages. Today we’ll use the “bootstrap” method which uses Chef Solo to build the Chef Server.

(terminal: ssh ubuntu@192.168.139.130)

The first thing our server will need is a proper host name. To set it on a running system we can use the hostname command and make a matching entry in /etc/hosts. To persist it across reboots on ubuntu we also need to set the hostname in /etc/hostname.

$ sudo hostname chef.devops.mashion.net
$ sudo vi /etc/hosts
    -127.0.0.1 ubuntu*
    +192.168.139.130 chef.devops.mashion.net chef
$ sudo bash -c 'hostname > /etc/hostname'

If you got it set right, you should see the full hostname when you run hostname -f.

$ hostname -f

Now since we’re using chef solo we’ll need a node config

$ vi chef.json
    {
      "run_list": [ "recipe[chef-server::rubygems-install]" ],
      "chef_server": {
        "server_url": "http://localhost:4000",
        "webui_enabled": true
      }
    }

The run list will include the chef-server rubygems-install recipe. And we’ll set two attributes that tell chef the expected server url and that we want the chef web UI to be installed and available.

$ sudo chef-solo -c /etc/chef/solo.rb -j ~/chef.json -r http://s3.amazonaws.com/chef-solo/bootstrap-latest.tar.gz

And now we’ll run chef solo using that config and pulling our cookbooks from the S3. Note that this will take some time since it installs a lot of packages.

You may have noticed that I’m using VMWare Fusion today. The reason for this is that VirtualBox on Mac OS Lion seems to over-consume resources when running large apt-get installations. One way I’ve found to work around this is the cputhrottle tool for Mac OS. If you run into similar issues check the show notes for some information.

(cputhrottle info)

Download from http://www.willnolan.com/cputhrottle/cputhrottle.html

# Grab the PID of the VM called "Chef Server"
$ VMPID=`ps ux | grep '[C]hef Server' | awk '{print $2}'`
# Limit it to 80 percent of total CPU
$ /path/to/cputhrottle $VMPID 80

(Configuring knife)

Now that Chef is installed we’ll configure the root user’s knife utility. We’ll use the local knife tool to create certificates that we can use on our own laptop.

$ sudo su -
$ knife configure -i --defaults -r .

To verify that it was configured correctly, we’ll run knife client list. The “ubuntu” entry here tells us that it was configured correctly and it can talk to the Chef server:

$ knife client list

To create a knife client entry for myself I’ll use the knife client create command. The options here tell Chef to create and admin account with default settings called ‘mat’. And to place the private auth certificate into /tmp/mat.pem

$ knife client create -a -n -f /tmp/mat.pem mat

(close ssh)

Now back on my computer I’ll scp that key over to my .chef folder and configure my knife client.

$ mkdir .chef
$ scp ubuntu@192.168.139.130:/tmp/mat.pem .chef/

To configure our local client we’ll use knife configure. The options here specify ‘mat’ as our chef user name. This matches the client we created back on the server. The path to the certificate we copied over. The URL to the chef server, which defaults to port 4000. And finally defaults for the rest.

$ knife configure -u mat -k .chef/mat.pem -s http://192.168.139.130:4000 --defaults -r .

To test that we have it configured correctly we run knife client list as we did before.

$ knife client list

When running Chef Server on VMs it’s easy to run into clock synchronization problems. If you encounter an authentication error that mentions synchronizing your clock, go back to the server and run ntpdate to fix it.

$ ssh ubuntu@192.168.139.130
$ sudo ntpdate pool.ntp.org

The happens because the authenication mechanism in Chef only allows for 15 minutes of difference between the clocks on the two systems.

Now that our clocks are in sync we should be able to run knife client list back on our laptop without any problems.

$ knife client list

(browser)

One more thing I’ll mention before we finish is that we’ve also installed the Chef web UI. If we go to our chef server on port 4040, we’ll get a nice Web UI for all of the functions we talked about here. The default password is in the sidebar. Just enter it here, give it a new password and you can then browse around your chef configuration this way too.

(client tab)

If we look at the Clients tab we can see the “mat” and “ubuntu” clients that we’ve created.

(slides)

And that’s it for today. Next episode we’ll go over how to hook up the chef-client daemon to this chef server and upload cookbooks so they get run on our app servers. Thanks for watching and see you next time!

Chef Solo Basics

2011-08-04

Welcome to our first episode! In this episode we’ll cover the basics you’ll need to get started with Chef Solo.

Update: I forgot to add the link overlays so here’s my blog post on setting up an Ubuntu VM and a pre-made image that should work for testing. The username/password is ubuntu/ubuntu.

Prep

Prep virtualbox VM: get IP and upload keys, snapshot, then shutdown
Remove ~/.chef/knife.rb
Switch to empty gemset
start redcar
open chrome to github.com/opscode/cookbooks
open slides

Script

(slide start)

Hi, I’m Mat Schaffer and this is the DevOps Screencast. This is the first episode of the screencast, so while I haven’t nailed down the format yet, I plan to cover a range of topics including infrastructure automation, systems-level concepts and unix utility tutorials.

(slide change)

In this episode we’ll be looking at the basics of using Chef Solo.

Chef is an server configuration framework written in Ruby. Using chef, we can configure various software packages automatically. The recipes we build can then be reused across many machines to ensure uniformity across all your servers.

Chef works in two modes, chef server and chef solo. Today we’ll talk about Chef solo and we’ll cover Chef server in future screencasts.

(slide change)

In chef solo, there’s no separate chef server. You only have your development machine and your application server. You develop your cookbooks and recipes locally on your development machine.

(change)

You then copy these cookbooks and any other configuration information up to your server using SCP or Rsync.

(change)

Then you invoke the chef-solo command on the server. The Chef-solo command will then run through all the recipes you’ve specified in your node configuration file.

As you can see, this is somewhat manual. But there are a handful of tools that you can use to make building and running your recipes a little easier.

(slide change)

Today we’ll use knife-solo. Knife-solo is a plugin for Knife, the main command line utility that drives Chef.

(change)

I just recently released knife-solo, you may encounter some bugs. If you do, please let me know. There are also other tools that help with this job that you might want to try out like spatula, soloist or littlechef. Writing your own is also fairly straight forward.

(terminal)

$ gem install knife-solo

Knife solo is based on chef’s own knife helper tool. It gives you a few extra commands that make working with chef solo a bit easier. If this is the first time you’re using chef, the install make take a bit of time since it will also install the chef gem. You’ll need to generate a configuration file.

$ knife configure -r . --defaults

The options I specified here just set up some defaults. Since we’re only using chef solo we don’t need this configuration, but knife is a bit noisy if the file doesn’t exist. Now I can use the knife kitchen command to make a place to hold my recipes.

$ knife kitchen solodemo $ cd solodemo $ tree

This will create the standard layout for a chef repository. We’ll go over these in more detail in future screen casts. For today, we’ll focus on the cookbooks and nodes directories.

(chrome: github.com/opscode/cookbooks)

Often you can find cookbooks for various packages online. Opscode, the company that created chef maintains a cookbook repository on github. 37signals and other companies are beginning to do the same. While these cookbooks don’t always work for your particular environment they at least provide good examples and starting points.

(terminal)

Today we’ll keep things simple and just install a nginx server. We could use the opcode cookbook, but we’ll make one from scratch to get some practice working with cookbooks.

To create a cookbook, use the knife cookbook command.

$ knife cookbook create nginx -o cookbooks

We specify the name of the cookbook, nginx. And use the -o option to tell knife to store it in the cookbooks directory. Each cookbook contains a default recipe. We’ll use this recipe to install and start the nginx server.

(redcar: cookbooks/nginx/recipes/default.rb)

Chef uses a Ruby DSL to define recipes. But before we start coding it we’ll have a look at what we want to do.

(virtualbox, start it)

I’ve prepared a minimal Ubuntu system running on VirtualBox that we’ll use to work through this installation. If you need pointers on creating your own I have a blog post that goes over the process. Or you can download it from the link you see here.

(terminal)

Now before we can run any recipes on our VM, we have to install chef solo on it.

$ knife prepare ubuntu@ip

To do that we run knife prepare. This is a knife-solo command that will install ruby and chef so that we can run chef solo. It also generates an empty node configuration.

(redcar: nodes/ip.json)

This file is what tells chef which recipes should run on a given host. To include the nginx recipe we just wrote, we add recipe[nginx] to the run list.

(terminal)

$ knife cook ubuntu@ip

Now to install nginx on our VM using chef solo we run the knife cook command. This will copy our cookbooks over to the VM and run chef solo using the node configuration that matches the IP of the box.

Now of course this didn’t do anything because we haven’t written the recipe. We first want to install nginx, so we’ll take a look at the machine and find out what the package name is.

(virtualbox: logged in)

$ apt-cache search nginx

As you can see it’s just “nginx”, so we’ll add a package statement to our recipe for that and cook again.

(redcar: default.rb)

package 'nginx'

(terminal)

$ knife cook ubuntu@ip

(chrome: ip)

Now nginx is installed, but it’s not yet running.

(virtualbox)

$ ls /etc/init.d/

As we can see here, the nginx package installs a standard init script. We can tell chef to use that by defining a service that supports the ‘status’ command. This tells chef to use the init script rather than trying to inspect the process table directly. Then we cook again.

(redcar: default.rb)

service 'nginx' do supports [:status] end

(terminal)

$ knife cook ubuntu@ip

(chrome: ip)

As we can see, it’s still not running. All we’ve done here is defined a service that chef now knows about. To start it, we add the :start action and cook again.

(redcar: default.rb)

service 'nginx' do
  supports [:status]
  action :start
end

(terminal)

$ knife cook ubuntu@ip

And now if we open a browser to the VM’s IP. We’ll get a nice welcome message.

Now to demonstrate that we can reuse this recipe, we’ll roll back the VM and run the whole thing in one sweep.

(virtualbox: rollback)

(terminal)

$ knife prepare ubuntu@ip $ knife cook ubuntu@ip

(chrome: ip)

Now if we open that IP again, we have Nginx just as we did before with the same configuration.

And that’s it! We’ll go over more of the details in upcoming sceencasts, but hopefully this is enough to get you started exploring chef for your own servers. Thanks for watching!