I already wrote about how to get started with the Opscode Chef Platform. In this article I want to show you a very elegant way to deploy a Ruby on Rails stack with Chef. One of the strengths of Chef is the decent set of available cookbooks. @jtimberman does an especially excellent job in writing them. His chef cookbooks really help you to configure your systems neatly. One of his cookbooks is the Application cookbook. It enables data driven application deployment. Currently, it supports Ruby on Rails apps. The preferred stack is currently Matz Ruby with Unicorn, but, in a later post, I’ll show you how to use it cleanly with Ruby Enterprise Edition (REE).
Let’s get started!

Use The Chef Command Line Tool knife to Pull Down Required Cookbooks

First of all we need to get all required cookbooks on our local workstation. I assume your local box is setup to develop chef cookbooks. The preferred way to use existing cookbooks is to “vendor” them:

$ knife cookbook site vendor application -d

The -d parameter tells knife to pull all dependencies. The application cookbook will pull down quite a lot of deprecated dependencies like passenger_enterprise or passenger_apache2. Even if you want to install a unicorn based stack, those cookbooks will be downloaded (but never used). Just ignore them.

You may use the same command to pull a new version of any cookbook (no -d required then).

Configure Your Application Using a Data Bag

One of the great features of opscode chef is the support of data bags. A data bag is a JSON file containing any data structures (like configurations, etc) for your recipes to use. The Application cookbook makes heavy use of a data bag called apps.

First, we need to create a data bag file. Just name the data bag file like your application (in this example I use “my_app”):

$ mkdir data-bags
$ vi data-bags/my_app.json

Copy a sample data bag from application/README and change to your liking.

NOTE: Make sure your deploy key string is in one line with \n "deploy_key": "-----BEGIN DSA PRIVATE KEY-----\nMIIBv...EW1auXCaV\nb24T...TiaHI\n...\n-----END DSA PRIVATE KEY-----\n",

Now, it’s time to store your data bag on the chef server:

$ knife data bag create apps
$ knife data bag from file apps data-bags/my_app.json

Note the “apps” part before the actual file name. This is the name of the data bag and must be “apps” for the application cookbook to work correctly. You can load multiple application configurations into that same data bag (just use different JSON files).

Define Your Roles

You already put some role names into your data bag. Now, it’s time to create those roles to be able to assign them to nodes.

$ cd ./roles
$ vi my_app.rb
$ vi production.rb
$ vi my_app_database_master.rb

You might want to create a “staging” role as well to differentiate between your staging and your production environment by assigning the respective role to your nodes.

Here are samples for each role:

my_app.rb

	name "my_app"
	run_list "recipe[application]"
 
	override_attributes :apps => { 
	  :my_app => { 
	    :production => { 
	      :run_migrations => true,
	      :force => false
	    },
	    :staging => {
	      :run_migrations => true,
	      :force => true
	    } 
	  }
	}

Here we define that we want to run rake db:migrate on deployment and want to force git checkout on staging, but we do not want to force git checkout on production.

production.rb

name "production"
default_attributes :app_environment => "production"

For the initial upload of several cookbooks and roles there’s a handy rake task

$ rake install

Install MySQL as Database

Now, let’s add a database to the mix:

$ knife cookbook site vendor database -d

And, add the database::master recipe to the run_list of the “helpster_database_master” role

my_app_database_master.rb

name "helpster_database_master"
run_list "recipe[mysql::server]", "recipe[database::master]"

Assigning Roles To Nodes

To simplify the example, we install the whole stack on a single node. Just assign the “production”, “my_app_database_master”, and “my_app” roles to your production server node:

$ knife node run_list add www.example.com "role[production]"
$ knife node run_list add www.example.com "role[my_app_database_master]"
$ knife node run_list add www.example.com "role[my_app]"

NOTE: The order in the run_list matters. Make sure you have database_master BEFORE your application role!!!

The application cookbook will install ruby, rails, all packages and gems defined in your data bag and unicorn as a runit service. Sweet!

NOTE: Your application log file is now under:

/etc/sv/my_app/log/main/current

Upload your role and cookbooks (using rake install) and run chef-client on your node!

Related posts:

• Automated Configuration Management With Opscode Chef: The Basic Moving Parts
• Getting Started With The Opscode Chef Platform – Configuration Management In The Cloud
• Aegis: Role-based Permissions for your Ruby on Rails application

A lot of Sysadmins and developers all over the world write, meet and talk about DevOps: How to collaborate better so we can deliver business value faster. The aim of DevOps is to get rid of the traditional way of thinking in silos inherent to development and operations. But how can You find out whether your organization is ready for DevOps? Here are a few hints.

DevOps State of Mind

Ask your developers the following questions:

• Can you describe the runtime environment of your web application?
• Do you know the release process for your app?
• What are the most critical issues operations is dealing with currently?
• When is your work done?

The answers to these questions tell you whether your developers are thinking out of their “code creation” box and consider deployment of their app as part of their work. Only if they show at least some basic interest in the operations realm are they ready for DevOps. If your developers think they can just “throw their work over the fence to QA” (or the build server), or, even worse, that they’re done once they commit their code to version control, you’ve got a long way to go.

Ask your sys admins the following questions:

• Which new features are upcoming?
• Do you look forward to the next release?
• When was the last time you talked to the developers?
• Is your work creating business value?

The answers to these questions show you how the sys admins think about their work and their areas of responsibility. Only if they are involved in development to a certain degree and welcome frequent change (i.e. creating business value) are they ready for DevOps. If they only think about avoiding change and don’t care at all about what’s upcoming or what’s creating business value, you’ve got a ways to go with your sys admins as well.

What is DevOps?

DevOps is a culutre of talking and working together. Starting DevOps means starting a mind shift for your developers and sys admins. Only if people are ready to talk to each other and to care about each other’s work will you be able to create more business value, faster. Only then does it makes sense to look at the processes and tools that support the DevOps culture.

Related posts:

• DevOps: Why Silos Suck And How To Break Them
• Why Automated Testing is a Must for DevOps
• 20 DevOps guys you should follow

When inflexible and wasteful processes are making your organization inefficient, it’s time to introduce agile methodologies. Scrum vs. Kanban then becomes an essential question: Which one is better suited for my own situation?

Scrum – A Fundamental Shift

Scrum is a well defined process framework for structuring your organization. Introducing Scrum is quite a change for a team not used to agile methodologies: They have to start working in iterations, build cross-functional teams, appoint a product owner and a scrum master, as well as introducing regular meetings for iteration planning, daily status updates and sprint reviews. The benefits are well understood: Less superfluous specifications and less hand overs due to cross-functional teams, more flexibility in roadmap planning due to short sprints, etc. Switching your organization to use Scrum is a fundamental shift which will shake-up old habbits and transform them into more effective ones.

Scrum Leverages Commitment As Change Agent

The initial introduction of Scrum is not an end in itself of course. Working with Scrum you want to change habits inside your team: Take more responsibility, raise code quality, increase speed. As your teams commit to sprint goals, they are intrinsically motiviated to get better and faster in order to deliver what they promised. Scrum leverages team commitment as change agent. It’s amazing to see how much teams demand from themselves – often way more you as a manager ever dared ask for.

Kanban – Incremental Improvements

Kanban is way less structured than Scrum. It’s no process framework at all, but a model for introducing change through incremental improvements. You can apply Kanban principles to any process you are already running (even to Scrum . In Kanban, you organize your work on a Kanban Board. There you have states which every work item passes through from left to right. You start left with the Backlog, and push your work items along through the in progress, testing, ready for release, and released columns. And you may have various swim lanes – horizontal “pipelines” for different types of work. The only management criteria introduced by Kanban is the so called “Work In Progress (WIP)”. Nothing else needs to be changed to get started with Kanban.

Kanban Leverages Work In Progress (WIP) Limits as Change Agent

For every column (state) on your Kanban board you should define a “Work In Progress”-Limit (WIP Limit). The WIP limit tells you how much work items are allowed to stay in a certain state. If the state is “full”, no new work can enter that state. The whole team has to help clear the filled up state first. That way your team will find out about bottlenecks in the progress simply by looking at the Kanban Board and is challenged to change the way they work to avoid such bottlenecks in the future. In that way, the WIP limit acts as change agent in Kanban.

Scrum vs. Kanban

Looking at both agile methodologies it should be more clear what to introduce when: If your organization is really stuck and needs a fundamental shift towards a more efficient process, Scrum seems to be more appropiate. If you already have working processes, which you want to improve over time without shaking up the whole system, Kanban should be your tool of choice.

Related posts:

• Scrum What? New Community Edited Q&A Site About Agile, Lean, Kanban and Scurm
• Scrum alone won’t cut it
• Kanban WIP Limits – The Fine Art of Focus

I’d recently ordered a new round of servers for NetDoktor and was positively dreading having to setup Nagios & Munin on them. This is where the fact that I’m a “born & raised” developer really shines through. The configuration of Nagios is simply beyond me. No matter how much documentation I read, I just can’t get all the pieces moving right. Try to bolt Munin on top of this and I simply walk away in frustration. There had to be a better way…

The Competition

I had actually outsourced this initial setup, so of course I toyed around with the idea of having the same guys do it again on the new servers. But then I thought to myself, why should I have to do this at all? Aren’t there “all-in-one” products that can do this for me? I started with Centreon and then Zenoss. But I hadn’t really understood SNMP well enough to have one monitor cover my disparate network of servers.

Arrgggh – weren’t there companies out there that did this sort of thing? Why do I have to reinvent the wheel here?

The Tweet Heard ‘Round the World

LogicMonitor to the Rescue

Over the weekend, I dug around a little on the LogicMonitor site and figured I’d give them a shot. Monday I was contacted by their sales and we scheduled a desktop session for that evening. I was asked to install a lightweight java agent on my test server and configure SNMP properly. They guided me through that process and within a few minutes, all the basic datapoints were being gathered and graphed. All well in fine, but was it worth the cost?

Later, I started to get interesting alert emails – warnings about excess TcpRetrans and query cache prunes:

www2 has 21.42 query cache prunes per second due to low memory. The Query cache has a hit ratio of greater than 50% - so it is likely to benefit more from increasing the cache size to alleviate this memory pressure.
This state started at 2010-07-23 20:36:35 CEST and has been going on for 0h 12m.


Wow – not just a plain alert because a given threshhold was tripped, but a bit of background information and even a suggestion on how to fix it! Suddenly, the light went on LogicMonitor! I now had a virtual system administrator watching my servers and pointing me in the right direction when it noticed something strange. Just yesterday, I created my first custom datasources with graphs to keep track of my website response time for different pages.

It took about 10 minutes to setup 2 of these graphs and it’s the first step to being able to easily gather and plot relevant business data. An example next step would be how many people have logged in within the past 5 minutes? How many people have posted articles or questions? And the “holy-grail”, how does our response times (and load) compare when plotted against these other metrics?

Naturally, I’ve had second thoughts about pushing such a core business value like server metrics out to a SaaS provider. But are the metrics themselves a core value? Or is it the visualization and business decisions that are made after comprehending these metrics that are the real value?
I’m going with the latter and am extremely happy to not have to worry about how to setup my own metrics system.

Understand that LogicMonitor is based in California and this caused some minor headaches during the sales & support process as I’m located here in Munich, Germany. The 9hrs time difference basically meant my problems/questions were answered next business day. Nevertheless, within a week I had full monitoring setup and configured on my test servers. The week after that I moved it into the production systems and haven’t looked back since.

Related posts:

• Getting a Grip on your Operations with Munin
• Monitoring tools essentials: Munin vs. Nagios
• Give Your Servers Some TLC: 10 Seconds A Day With Munin

Every organization has to deal with a mix of ongoing and project oriented work. But, even if you structure your teams into departments to optimize ongoing work, they keep trying to self organize into project focused teams.

Matrix Management vs Project Organization

There are the regular tasks and chores, and there are the bigger undertakings which try to bring the organization as a whole forward. To balance both, you have two ways to structure your organization: Structure by department and have cross-departmental project teams or structure by project. Structuring by department is often called Matrix Management whereas structuring by project is often referred to as “project-oriented organization”.

Departments Focus On Specialization

The advantage of splitting teams into departments is that even though they might work in different projects they still can share their experiences and exchange ideas and learnings in their specific field. That way everyone can improve his specialization e.g. in product management. A disadvantage of that structure is that your project teams are distributed over various departments. This makes it harder for everyone to focus on the project goals as top most priority.

Project-oriented Organizations Foster Cross-Functional-Work

If your main way of structuring your organization is by project you group all specialists around those bigger undertakings. Everyone required for project success sits and works together and everyone can focus on the project goal. The disadvantage of this structure is that all people with the same specialization like product management hardly get to exchange experiences and learnings. There is a need for cross-project exchange, which you need to enforce to make it happen.

Teams Gravitate To Project-oriented Organizations

So, which organizational form is better? Of course, it depends ™. But, more often than not, people given a choice prefer working in cross-functional project teams. Even loosing the advantages of working with collegues of the same trade, they seem to gain more profit out of working together in a project. Co-location and constant informal information flow within the project team gives a team a great boost in productivity and reduces stress and friction between team members.

Related posts:

• Successful Teams Are Small And Dedicated
• Agile Methodologies: Scrum vs. Kanban
• Waterfall, SCRUM and Lean Software Development simulation as teaching platform

Browsers load static images from your website again and again if your web server does not send an expires header with a date far in the future. To avoid that unnecessary traffic on your servers and unnecessary load times for your users, it’s a good idea to let your nginx send those expires headers. But, what if you tell the browser not to come back for your css files until next year and now you change some style within it? The Rails URL helpers automatically attach a timestamp representing the last modification date to each URL (like ?2345346654). That changes the URL every time you deploy making the browser load the modified file. In this post, I want to show you all the not so obvious things to consider when you introduce far future expires headers with rails and nginx.

Far Future Expires Headers In nginx

The first thing you have to do is to tell your nginx to send those expires headers with all your images, css and javascript files. Just add the following section to your server configuration in nginx:

	server {
	    ...
	    location ~* \.(ico|css|js|gif|jp?g|png) {
	        if ($args ~ [0-9]+) {
	           expires max;
	           break;
	        }
	    }
	    ...
	}

First, you match the location with a regular expression catching all your static resources. If you have additional file types you want to equip with a far future expires header, just add them to the regex. The important part, which a lot of people get wrong, is to additionally check for the timestamp parameter. You only want to add the expires header to files if the URL has the timestamp attached. Otherwise you have no chance to push out your changes anymore! In quite a few places around the net you find people attaching \?[0-9]+$ to the regex matching the location. This cannot work as location does not include the $args.

Adding Timestamps To URLs In CSS Files

Using far future expires headers in nginx only for files having a timestamp is great for every URL you generate using the rails helpers. But what about e.g. background images you define in your static css files? As they do not have a timestamp attached they will not get any expires headers and therefore will not be cached by the browsers. Especially if you use larger image maps for css sprites, that hurts. You can use the yslow and live HTTPheaders firefox plugins to validate the headers.

There are a few ways to add timestamps to URLs in your css:

• Make your css files ERB templates and use page caching to avoid re-generating them for every request
• Use an initalizer to parse through your css files and add the timestamps when loading your Ruby on Rails app
• Monkey patch the stylesheet_link_tag helper to parse your css files and add timestamps when collecting all individual css files and building the one big one.

We are currently using the monkey patch, but I would recommend you to do it in an initializer.

Using far future expires headers with rails and nginx is simple and very effective. Your servers and your users will love it!

Related posts:

• Seed Data In Ruby On Rails
• Setting up a test database on a ruby on rails continuous integration server using SQL instead of schema.rb
• How to Deploy Ruby on Rails With The Opscode Chef Application Cookbook

Getting Started With The Opscode Chef Platform – Configuration Management In The Cloud

In “The Moving Parts of Opscode Chef” there was an interesting discussion about the need of a highly available chef server if you want to use opscode chef as your configuration management tool of choice. Especially for small to medium sized enviroments running your own chef server is overkill. If you don’t want to use chef-solo (a local “push” tool instead of the client-server model of chef), you can sign up for an account at the opscode chef platform. The opscode chef platform gives you an highly available chef server in the cloud. After sign up, it’s only a matter of minutes to get your first client (or ‘node’ in chef speak) under configuration management control.

Bootstrap A Chef-Client With Knife

If you have a box you want to have managed by the chef platform all you need is ssh access to it and chef installed on your local workstation. Just run

knife bootstrap www.example.com

(be sure you have the Net::SSH::Multi gem installed on your local workstation: $ sudo gem install net-ssh-multi)
to get chef-client installed and hooked up to the chef platform. After a few minutes you’ll find your brand new node ready to run cookbooks.

Roles, Nodes, And Cookbooks

Now you have the opscode chef platform acting as your chef server and another box acting as chef-client. It’s time to get something installed on that client using chef. Before we write any code, let’s have a look at how everything plays together:

• Node: Every box you want to manage with Opscode Chef is a node. The node definition lives only on the chef server and can be modified by the command line tool knife. To display all the attributes of a node you simply type knife node show www.example.com at your local workstation.
• Role: Roles define what a node should be. Examples of roles could be mysql_master, load_balancer, gateway, etc. Within roles you define so called “run_list”s of recipies (or other roles) telling chef what to install on every box having this role assigned.
• Recipes: A recipe is a set of instructions which ensures that a node has everying setup as you need it. You can define a variety of resources like users, directories, packages, files, etc. You could have e.g. a recipe called nginx::source which will grab the source tarball of nginx form a website, unpack it, configure, make, make install it, and upload configuration files as well as setting up a runit service for it. As you’ve already seen, you tell a role which recipes it shall run on every node having that role.
• Cookbooks: Cookbooks are collections of recipes. There are simple cookbooks including only one recipe (like the logrotate cookbook) and others having multiple recipes (like the nginx cookbook which includes recipes for installing nginx from source or using a package manager). Opscode provides a huge set of ready made cookbooks. If you want to use them you can include them into your chef setup by using knife: knife cookbook site vendor nginx -d.

Assign A Role To A Node

Assuming you have a set of roles and cookbooks ready (that means: edited them on your local workstation and then uploaded them to the chef platform using e.g. knife cookbook upload load_balancer) its time now to get something installed on your new node. Just type knife node run_list add [NODE] [ENTRY] where [NODE] could be www.example.com and [ENTRY] could be role[load_balancer] if you want to assign the role load_balancer to your node www.example.com.

Now just run chef-client on www.example.com and it will grab the required cookbook and run its default recipe.

Related posts:

• Automated Configuration Management With Opscode Chef: The Basic Moving Parts
• How to Deploy Ruby on Rails With The Opscode Chef Application Cookbook
• Configuration Management remixed: Introducing Carpet

Scrum is a great framework for organizing projects. It defines exact roles and procedures to structure your work environment. You gain a lot of visibility and you empower your teams. All that is great. But in software development or operations it’s not sufficient. You need an underlying set of values and practices which drive quality and speed.

Scrum is optimized for projects, not recurring work

In every company there are regular tasks like out of band user requests, really urgent bug fixes, the weekly report generation and distribution, etc. Scrum doesn’t really tell you how to deal with those types of issues. They don’t fit into iterations and can’t be put on a backlog. They make it hard for a team to commit to a sprint goal.

Kanban for recurring work

Kanban is all about optimizing flow. Especially recurring issues should be optimized to flow as fast as possible through your organization to minimize impact on project work. That’s where Kanban shines. One option is to organize big feature packages as projects using Scrum and have one or two teams dedicated to ongoing operations using Kanban.

Clean Code

Another important ingredient for a successful software business is to focus on Clean Code. Keeping your software maintainable is one of the most easily sacrificed but also most important factors for keeping speed and quality up.

By combining various agile practices like Scrum, Kanban, and good old Extreme Programming Ideas you can optimize your software process. Focusing on only one of them implies the risk of missing to address critical areas in your organization.

Related posts:

• Agile Methodologies: Scrum vs. Kanban
• Scrum What? New Community Edited Q&A Site About Agile, Lean, Kanban and Scurm
• Waterfall, SCRUM and Lean Software Development simulation as teaching platform

It’s hard to find the right structure for any organization. A lot of existing management wisdom comes from a time when you had to organize a physical work force. However, with today’s “knowledge workers” those structures don’t work as nicely anymore.

Everyone needs to prioritize

Every developer or devops has to prioritize the current work – nearly constantly. Do I work on a new feature next? Shall I answer those emails first? My box needs an upgrade – is now the right time for it? You hardly find a pointy haired boss standing behind you dictating your every step (hopefully ). Because of this, it’s extremely important to ensure that everyone shares the same goals and has all information needed to decide what’s most important right now. If only an archaic elite circle of managers know the whys and whats of a project and just delegate tasks, bad decisions by the executing team are inevitable.

Empowerment leads to better results, faster

Giving your team the responsibility for the success of their work is the next logical step. By letting the team know all the surrounding conditions and letting them add to the task list produces the best picture of work to be done you can get. I see it way too often that mostly the technical people in a team know exactly what needs to be done but do not feel to have the power to voice their concerns. Of course, those concerns like “we need to rethink this module as it will not scale as it is now” have to be taken into the prioritization of the backlog. If you ignore them for too long, you’ll see two bad effects:

1. You run into serious technical issues
2. You stress out your best and most dedicated people

Both effects can and should be avoided by choosing a fitting organizational structure: Empowerment of your best people.

Related posts:

• Launch Dates – The Good, The Bad, And The Ugly
• A Luxury Problem: How Emerging Iterations Eat Team Commitment For Breakfast
• Open Communication Stops De-Motivating your Team

If you have a lot of big user stories, your velocity will jump up and down wildly. This makes it extremely difficult to tell when a user story will be done. Breaking down your huge user stories into smaller ones will help you smooth the flow and give you a clearer picture.

User Stories Start Big

Often you have an idea for a new feature. There is a vague picture in your mind how it could look and work. You put it on the backlog to be able to estimate and prioritize it. Teams new to agile often make the mistake to keep it that way. They assign it a huge amount of story points and start hacking away.

Big User Stories Are Abstract

The problem with this approach is that you skip an important design step in your process. A rough feature idea needs further break down to find out how exactly it should look like. By doing that you will replace the big, abstract story with a lot of smaller, more concrete ones. If your feature idea was “mobile support” you might start breaking it down into stories like “show my location”, “show nearby friends”, etc. These stories will replace the “mobile support” story. Now, it might be possible to estimate those stories. Usually the sum of story points for the descendent stories is bigger than the very vague estimate for the original user story.

Big User Stories Block Your Flow

Another issue of keeping user stories too big is that they will block your flow. If you use one week long iterations a big story might span multiple iterations. That means you deliver 0 story points for a couple of weeks and then suddenly you earn all 40 or so points for that one huge story. Unfortunately even after quite some weeks you have no idea which velocity you could deliver ongoing. Dividing the 40 story points by number of iterations will not cut it.

Only if you use small enough stories, which you can complete within one iteration, will you get a reasonable value for your velocity iteration after iteration. Only then you’ll be able to tell when stories could be finished in the future.

Related posts:

• How To Estimate User Stories When Using PivotalTracker
• Estimation of User Stories With Story Points as Abstract Size Measure
• Kanban vs. Iterative Development