sysadvent

Day 25 - Learning from Other Industries

2011-12-26T14:35:00.000-08:00

This was written by Jordan Sissel (semicomplete.com).

This post is late because, two days ago (December 24th), I went to the emergency room with appendicitis. Pretty sweet timing, eh?

After surgery, I was bored and took to observing nursing shift rotations - hand-offs of patients, documentation maintenance, etc. The entire process semeed to last about an hour for each shift change. I thought to myself, "Self, you have never been involved in any operations project, task, or fire hand-off that went as smoothly." Pretty sad realization.

Further, throughout my hospital stay (emergency room, surgery, and recovery), not once did I observe any two individuals debating passionately about how best to treat my illness. Data was available at every decision point. There were specializations in surgeons for the surgical tasks; nurses to help stabilize, comfort, and feed me (no small task); and administrative staff to handle the beaurocracy stuff.

I'm not expecting systems administration to follow exactly what the medical industry does, just pointing out that it's nice to see a well-orchestrated system in action.

And the best part? I never once heard anyone say, "hey! We should try this new cool thing I heard about."

Further, I never once heard anyone suggesting that we remove my appendix using node.js and mongodb.

Happy New Year, everyone :)

Day 24 - Implementing Configuration Management in Legacy Environments

2011-12-24T00:10:00.001-08:00

This was written by Dean Wilson (www.unixdaemon.net).

Implementing configuration management is perfect for green field projects where you can have freedom to choose technical solutions free from most existing technical debt. It's a time where there are no old inconsistencies and you have a window to build, deploy and test all the manifests and recipes without risking user visible outages or interruptions.

Unfortunately, most of us don't have the luxury of starting from scratch in a pristine environment. We have to deal with the oddities, the battle scars, and the need to maintain a good quality of service while evolving and maintaining the platform.

In this article, I'll discuss some of the key points to consider as you begin the journey implementing a configuration management tool in an existing environment. I happen to use puppet and mcollective, but any config management tools will do - find one that works for you.

Discovery

You'll need to start with some discovery (manual, automated, whatever) to learn how your current systems are configured. How many differences are there across the sshd_config files? Where do we use that backup users key? What's the log retention setting of the apache servers? Is it identical between those we upgraded from apache 1 and the fresh apache 2?

When I first started to bring puppet in to my systems, I wrote custom scripts, ssh loops, and inventory-reporting cronjobs to find the information I needed. Now, there is an excellent solution to gather information you need in real time: MCollective.

With the appropriate mcollective agents you can compare, peer, and probe in to nearly every aspect of your infrastructure. The FileMD5er agent, for example, will show you groups of hosts that identical config files. Using this, you can partition the environment and the work, section by section. This will help you find smaller amounts of work to build into puppet.

One of my favourite current tricks is to query config file settings using Augeas in the AugeasQuery agent.

# Does your ssh permit root logins?
$ mco rpc augeasquery query query='/files/etc/ssh/sshd_config/PermitRootLogin' --with-agent augeasquery -v
...

test.example.com : OK
    {:matched=>["no"]}

test2.example.com: OK
    {:matched=>["yes"]}

No more crazy regexes to find config settings!

You can learn more about how the Blueprint tool can help you with this configuration discovery process in "Reverse-engineer Servers with Blueprint" (SysAdvent 2011, Day 12).

Short Cycles

One of the key factors to consider when starting large scale infrastructure refactoring projects is that you'll need information and you'll need it quickly.

You should always be the first to know about things you've broken. Your monitoring system, centralised logging, trend graphing, and config management systems reporting server (like Puppet Dashboard and Foreman) will become your watchful allies.

The speed at which you can gather information, write the manifest, deploy, and run your tests is more important than you'd think. Keeping this cycle short will encourage you to work in smaller sections, reducing the time between you actually testing your work and keeping the possible problem scope small. Tools like MCollective, mc-nrpe and nrpe runner will enable you to rapidly verify your changes.

Version Control

The short version is: version control everything.

There's no reason, whether you are in a team or working alone (especially when working alone!), to not keep your work in a version control system (such as subversion or git). "That change didn't work, so I've rolled it back". Really? By hand? Are you sure you didn't forget to remove one of the additions? Why would you shoulder the mental burden of knowing all the things you changed recently when there are so many excellent tools that will do it for you and allow easier, auditable rollbacks to old revisions?

Beyond the benefits of providing an ever-present safety net, using a VCS provides a nice way to tie your exact change into incident, audit, or change control systems - a perfect place to do smaller code reviews and a basic, but permanent, collection of institutional knowledge. Being able to search through the puppet logs / reports when debugging an issue to find exactly which resources changed and when, pull up the related changeset and the actual reason for it (and who you should ask for help) changes your process and reports from best guesses based on peoples memories to quick fixes backed by supporting evidence.

On a positive note, it's amazing how much you can learn from reading through commits from things like DBAs doing performance tuning. Seeing which settings were changed and why can save you and your team from forgetting something or making the same mistake or redoing parts of the work again in the future.

Use Packages

Many older environments have fallen in to the habit of pushing tarred applications around (and sometimes compiling software on the hosts as needed). This isn't the best way to deploy in general, and it will make your manifests ugly, complicated, sprawling messes. In most cases, it's not that hard to produce native packages, and tools like fpm make the process even easier.

By actually packaging software and running local repositories you can use native tools, such as yum and apt, in a more powerful way. Processes become shorter and have simpler moving parts, more meta information becomes available, and it's possible to trace the origin of files. You can then use other tools, such as puppet and MCollective to ease upgrades, additions, and reporting.

You can learn more about the world of software packages from "A Guide to Package Systems" (SysAdvent 2011, Day 4).

As an aside, there are differing opinions on whether packages should handle related tasks, such as deploying users. However, once you've reached the point where you can have this argument, you've surpassed most of your peers and will have an environment that gives you the freedom to do whichever you choose.

You can learn more about overlaps in functionality of packaging and config management systems in "Packaging vs Config Management" (SysAdvent 2010, Day 23).

Baked-in Goodness

As you bring more of your system under puppet control, you'll begin to experience the continuously improving system baseline. Every improvement you make is felt throughout the infrastructure, and regressions become infrequent, easy to spot, and quick to remedy. For example, you should never wonder if you've deployed nginx checks to a new server, it should be impossible to have a mysql server without your graphing applied, and every wiki you deploy should be added to the backup system without you intervening.

By having infrastructure manifests such as a monitoring module and writing some generic "add monitoring check" definitions, you can sprinkle reuse throughout other, more functionality-focused modules. In my test manifests, for example, nearly every class has nagios checks associated with it, and every service declares the ports it listens to. This might not sound like much, but it's impossible for me to deploy a server without monitoring, every time I add a new nagios check, all the hosts with that role receive it. Further, generating security policies, firewall rule sets, and audit documents is done automatically for me and requires no manual data gathering. Being able to supply people with links to live documentation is a wonderful way to remove awkward manual steps from reports, audits and inventories.

Involve Everyone

You don't have to agree with the DevOps movement and ideas to use configuration management, but some of its principles (such as communication, openness, shared ownership and responsibility) are worthwhile and have a low overhead once you start to config manage everything. Look for simple requests for information from other teams. Do they want to know which cronjobs run and when? What modules the apache servers have configured? Which rubygems you've deployed on the database servers? (from packages!)

By sharing configuration modules, you help others develop a balanced understanding of the platform. You enable them to regain time wasted raising support tickets through sharing commit logs, audit and error reports, etc, without feeling like your coworkers or customers are wrestling your time away.

At $WORK, a large percentage of our developers are comfortable reading puppet manifests, running MCollective commands, and using custom Foreman pages and cgi-scripts to investigate issues answering their own queries without the sysadmins being involved. This has dropped our workload, given them quicker answers and allowed them to ask targeted questions after doing their own research.

Such things increase efficiency, communication, and happiness.

It's not all about developers either. With a custom puppet define and a little reporting wrapper, we can expose the list of ports required by every machine to our network teams and auditors along with accurate timestamps showing when information was gathered. Using MCollective and custom database facts, our DBAs have dashboards showing deployed packages and services, current running configuration, and the ability to gather ad-hoc real time information from any of our systems: production, development or QA and even compare the differences between them.

Pick your fights

While you're growing accustomed and skilled with your new tools, you need to be aware of the strengths. As the person pushing this change, you have to prove your way is at least as good as (and hopefully better than) the current practise. Because of this, some tasks are bad places to prove your point and you need to be weary of them. Try to find some low hanging fruit

Large monolithic deployments or complex configuration scripts can be handled if you break them down to manageable components. Unfortunately, tight coupling and intertwined code makes these situations bad places to start. It's never a good opening move to start explaining that it took three hours to pick apart a shell script that seems to work fine. Vaguely-defined processes are the other common tar pit. Bringing attention to processes that people seem to do slightly differently is another great way to unite people against what you are trying to accomplish.

You want to avoid resistance caused by confusion. Find and tackle tasks that make sense to put into configuration management, first.

User accounts, for example, are a complex area to automate. While it starts with a simple "deploy the admin accounts, add them to wheel, and push the SSH keys," it can easily become a sprawl of teams needing different sets of access on a semi-rotating basis.

It's worth noting that your config management efforts will often be considered slower, especially when starting out. Writing a ten line throwaway shell script or hand-hacking something will be quicker in the short term than writing clean manifests that include monitoring, meta data and tests but the comparison is unfair. Remember, you're building flexible, reproducible, systems that support future change - not focusing on a single server that just needs to work now and only now.

Roles, not hosts - No more snowflakes

As you build your collection of configuration management techniques, you'll find special snowflake hosts: Machines performing several small but important tasks; ones hand-constructed under time constraints (and needing that one extra special config option) - boxes you'll "only ever have one of," right? Don't fall in the trap of assuming any machine is special or that you'll only ever have one of them. Building a second machine for disaster recovery, a staging version, or even an instance for upgrade tests means there will be times when you'll want more than one of them. If you find yourself adding special cases to your config management "If the hostname is 'foo.example.com'" then it's time to stop, take a step back, and consider alternatives. Any list, especially of resource names, that you hand maintain is a burden and will eventually be wrong.

A sign that things are starting to click is when you stop thinking in terms of hosts and start thinking in roles. Once you begin assigning roles to hosts you will find the natural level of granularity you should be writing your modules at. It's common to revisit previously written modules and extract aspects of them as you discover more potential reuse points.

You can learn more about configuration in terms of roles instead of machines in "Host vs Service" (SysAdvent 2008, Day 7).

Conclusion

With a little care and consideration it's possible to integrate configuration management and even the largest legacy infrastructures while enjoying the same benefits as the newer, less encumbered projects.

Take small steps, master your tools and good luck.

Day 23 - All The Metrics! Or How You Too Can Graph Everything.

2011-12-23T00:14:00.000-08:00

This was written by Corry Haines.

As your company grows, you may find that your existing metric collection system(s) cannot keep up. Alternately, you may find that the interface used to read those metrics does not work with everything that you want to collect retrics for.

The problems that I have seen with existing solutions that I have used are:

Munin: Fails to scale in all respects. Collection is more complicated than it should be, and graphs are pre rendered for every defined time window. Needless to say, this does not scale well and cannot be used dynamically.
Collectd: While this system is excellent at collecting data, the project does not officially support any frontend. This has led to a proliferation of frontend projects that, if taken together, have all of the features you need, but no one frontend does everything.
XYMon: It's been some time since I used this, and I have not used it on a large set of systems. My guess is that it would suffer from some of Munin's issues.

Enter Graphite

Graphite is a collection of services that can replace or enhance your existing metric collection setup. Yes, it's written in python... but I like python.

The major components are:

Whisper: Replaces RRD with a vastly simpler storage only system. Unlike RRD, whisper cannot graph data directly. Also unlike RRD, you can actually read and understand the entire codebase in less than an hour (only 725 lines of well commented python).
Carbon: Takes data on a variety of interfaces (TCP, UDP, Pickle, AMQP) and stores the data into whisper.
Graphite-webapp: Graphs data from whisper or RRD files.

The best thing about the components being independent is that you can run graphite on your existing RRD data with no hassle. While there are advantages to using whisper, it is not required to get the power of graphite.

The only negatives that I currently hold against graphite are:

The documentation is still a bit lacking, though they are working to improve this. You can invoke the community (mailing lists, etc) as a workaround.
The learning curve can be a bit steep. While there is an interface to see all of the functions, you still need to learn how they are applied. This is offset by the ability to save named graphs for all users to see.
Feedback is a bit lacking. After a graph is requested it is difficult to tell if it is being rendered, or simply failed in the backend.
They use launchpad and thus bazaar, for their project management and source control. In a post-github world, this is starting to get a bit painful.

The Power of Filters and Functions

As wonderful as whisper and carbon are (and they really are worth using), the true power of graphite lies in its web interface. Unlike some other interfaces, graphite treats each metric as an independent data series. So long as you have an understanding of the system, you can apply functions (think sum, avg, stddev, etc.) to the metrics either by themselves, or more often, in aggregate.

In addition, you can use wildcards to select multiple machines quickly. While you could do a sum operation like this: sumSeries(host1.load,host2.load,etc) you could more easily type sumSeries(*.load).

Filter Example

As an example, if I wanted to find overloaded webservers I could construct a query like highestAverage(webservers.*.load.longterm, 3) producing:

higher resolution graph here

Stacking example

Another example, graphing the amount of unused memory on the webservers (time for more memcached if so!) movingAverage(webservers.*.memory.free, 10) producing:

higher resolution graph here

Note that I am also creating a moving average over 10 datapoints here. Also, the series are stacked to produce a sum while still showing the responsible server

Functions Are the Best!

And this is only a small selection of the functions available to you. Moreover, you can write your own! And easily too! Here is an example function in graphite:

# A function to scale up all datapoints by a given factor
def scale(requestContext, seriesList, factor):
  for series in seriesList:
    series.name = "scale(%s,%.1f)" % (series.name,float(factor))
    for i,value in enumerate(series):
      series[i] = safeMul(value,factor)
  return seriesList

Graphite in Production

We are currently collecting >93,000 metrics every 10 seconds. Most of the data is gathered on machines using collectd and then passed to a proxy written by sysadvent's editor. The proxy then ships all of the data, via TCP, to our central Carbon node.

All of the data is consumed by carbon and stored on a single machine with six 10k SAS drives in a RAID 10 array. Although this disk setup is not enough to write the data in real time, it batches up the data and writes sets at once. It only needs to use about 300 MB of RAM for caching.

In reality, this hardware is probably overkill for our current workload. While testing, I was running about 50,000 metrics on four 7.2k SATA drives in a RAID 10 and the machine was doing just fine. It was using several GB of RAM to cache the data, but it was still able to keep up.

In Closing

If you are considering the installation of a metric gathering system, I would absolutely recommend Graphite. If you are using Collectd or Munin already, you can try the graphite web interface without changing how you collect metrics. It only takes a few minutes to setup and might give you better insight into your systems.

Day 22 - Load Balancing Solutions on EC2

2011-12-22T00:28:00.000-08:00

This was written by Grig Gheorghiu.

Before Amazon introduced the Elastic Load Balancing (ELB) service, the only way to do load balancing in EC2 was to use one of the software-based solutions such as HAProxy or Pound.

Having just one EC2 instance running a software-based load balancer would obviously be a single point of failure, so a popular technique was to do DNS Round-Robin and have the domain name corresponding to your Web site point to several IP addresses via separate A records. Each IP address would be an Elastic IP associated to an EC2 instance running the load balancer software. This was still not perfect, because if one of these instances would go down, users pointed to that instance via DNS Round-Robin would still get an error until another instance would be launched.

Another issue that comes up all the time in the context of load balancing is SSL termination. Ideally you would like the load balancer to act as an SSL end-point, in order to offload the SSL computations from your Web servers, and also for easier management of the SSL certificates. HAProxy does not support SSL termination, but Pound does (note: that you can still pass SSL traffic through HAProxy by using its TCP mode, you just cannot terminate SSL traffic there.)

In short, if Elastic Load Balancing weren’t available, you could still cobble together a load balancing solution in EC2. There is no reason to ‘roll your own’ anymore however now that you can use the ELB service. Note that HAProxy is still the king of load balancers when it comes to the different algorithms you can use (and to a myriad of other features), so if you want the best of both worlds, you can have an ELB upfront, pointing to one or more EC2 instances running HAProxy, which in turn delegate traffic to your Web server farm.

Elastic Load Balancing and the DNS Root Domain

One other issue that comes up all the time is that an ELB is only available as a CNAME (this is due to the fact that Amazon needs to scale the ELB service in the background depending on the traffic that hits it, so they cannot simply provide an IP address). A CNAME is fine if you want to load balance traffic to www.yourdomain.com, since that name can be mapped to a CNAME. However, the root or apex of your DNS zone, yourdomain.com, can only be mapped to an A record, so for yourdomain.com you could not use an ELB in theory. In practice, however, there are DNS providers that allow you to specify an alias for your root domain (I know Dynect does this, and Amazon’s own Route 53 DNS service).

Elastic Load Balancing and SSL

The AWS console makes it easy to associate an SSL certificate with an ELB instance, at ELB creation time. You do need to add an SSL line to the HTTP protocol table when you create the ELB. Note that even though you terminate the SSL traffic at the ELB, you have a choice of using either unencrypted HTTP traffic or encrypted SSL traffic between the ELB and the Web servers behind it. If you want to offload the SSL processing from your Web servers, you can choose HTTP between the ELB and the Web server instances.

If however you want to associate an existing ELB instance with a different SSL certificate (say for instance you initially associated it with a self-signed SSL cert, and now you want to use a real SSL cert), you can’t do that with the AWS console anymore. You need to use command-line tools. Here’s how.

Before you install the command-line tools, a caveat: you need Java 1.6. If you use Java 1.5 you will most likely get errors such as java.lang.NoClassDefFoundError when trying to run the tools.

Install and configure the AWS Elastic Load Balancing command-line tools
- download ElasticLoadBalancing.zip
- unzip ElasticLoadBalancing.zip; this will create a directory named ElasticLoadBalancing-version (latest version at the time of this writing is 1.0.15.1)
- set environment variable AWS_ELB_HOME=/path/to/ElasticLoadBalancing-1.0.15.1 (in .bashrc)
- add $AWS_ELB_HOME/bin to your $PATH (in .bashrc)
Install and configure the AWS Identity and Access Management (IAMCli) tools
- download IAMCli.zip
- unzip IAMCli.zip; this will create a directory named IAMCli-version (latest version at the time of this writing is 1.3.0)
- set environment variable AWS_IAM_HOME=/path/to/IAMCli-1.3.0 (in .bashrc)
- add $AWS_IAM_HOME/bin to your $PATH (in .bashrc)
Create AWS credentials file
- create file with following content AWSAccessKeyId=your_aws_access_key AWSSecretKey=your_aws_secret_key
- if you named this file aws_credentials, set environment variable AWS_CREDENTIAL_FILE=/path/to/aws_credentials (in .bashrc)

Get DNS name for ELB instance you want to modify

We will use the ElasticLoadBalancing tool called elb-describe-lbs:

# elb-describe-lbs
LOAD_BALANCER  mysite-prod  mysite-prod-2639879155.us-east-1.elb.amazonaws.com  2011-05-24T22:38:31.690Z
LOAD_BALANCER  mysite-stage   mysite-stage-714225413.us-east-1.elb.amazonaws.com    2011-09-16T18:01:16.180Z

In our case, we will modify the ELB instance named mysite-stage.

Upload SSL certificate to AWS
I assume you have 3 files:
- the SSL private key in a file called stage.mysite.com.key
- the SSL certificate in a file called stage.mysite.com.crt
- an intermediate certificate from the SSL vendor, in a file called stage.mysite.com.intermediate.crt
We will use the IAMCli tool called iam-servercertupload:
```
# iam-servercertupload -b stage.mysite.com.crt -c stage.mysite.com.intermediate.crt -k stage.mysite.com.key -s stage.mysite.com
```

List the SSL certificates you have uploaded to AWS

We will use the IAMCli tool called iam-servercertlistbypath:

# iam-servercertlistbypath
arn:aws:iam::YOUR_IAM_ID:server-certificate/stage.mysite.com
arn:aws:iam::YOUR_IAM_ID:server-certificate/www.mysite.com

Associate the ELB instance with the desired SSL certificate

We will use the ElasticLoadBalancing tool called elb-set-lb-listener-ssl-cert:

# elb-set-lb-listener-ssl-cert mysite-stage --lb-port 443 --cert-id arn:aws:iam::YOUR_IAM_ID:server-certificate/stage.mysite.com
OK-Setting SSL Certificate

That's it! At this point, the SSL certificate for stage.mysite.com will be associated with the ELB instance handling HTTP and SSL traffic for stage.mysite.com. Not rocket science, but not trivial to put together all these bits of information either.

Day 21 - Automating Web Monitoring

2011-12-21T00:01:00.000-08:00

This article is written by Brandon Burton, who can mostly be found posting lolcats and retweeting @solarce, though he occasionally posts interesting links to things sysadmin, devops, and unix.

As systems administrators, we all know that it's not in production until it's monitored, but this isn't always as simple a rule to live by as it may sound. Not all web applications, for example, are easily monitored through traditional monitoring solutions such as Nagios, Zenoss, or various commercial tools. These tools tend to take a "curl | grep" style monitoring, or they may support somewhat more complex POSTing of XML or JSON data and validation of the returned data. But often the most key parts of applications being deployed into production involve complex browser interactions and behaviors - AJAX, or some other session or transaction that traditional monitoring frameworks don't have an easy way to accommodate.

Enter Selenium. Selenium is a mature and robust framework for doing complex interactions with web applications. It originated as a tool at the consulting company ThoughtWorks as a way to do testing against web applications by driving a web browser. Since its release, it has seen the development of numerous tools, including browser plugins to make it easy to develop Selenium tests quickly and easily, language bindings to write tests in pretty much every major language, and tools to run many browsers across many operating systems, in parallel.

Additionally, services, such as BrowserMob and Sauce Labs, have grown around the Selenium ecosystem to help you do testing and monitoring in a scalable and offsite fashion. It is these services that we'll focus on utilizing in this blog post.

So what does all this mean? It means that we have a mature and robust toolset that we can utilize and perform testing and monitoring of the complex web applications that we are deploying into production.

Getting started

So how do we get started? My preferred method is to begin by developing tests locally. You can use the Selenium IDE, but for this example I'll show a Firefox extension called Sauce Builder which makes it a snap to build and run your first test locally.

To get started you'll need Firefox installed, then go to the Sauce Builder download page and walk through getting the extension installed.

Once you've got the Sauce Builder extension installed, it is time to build our first test.

I'm going to walk you through building a test to search for jelly beans on Amazon.

Open Firefox
Click on Tools -> Sauce Builder
Enter ''amazon.com'' in the Start Record prompt and click Go
Enter ''jelly beans'' for the search term
Click Go
Click on the first search result, for me this was '''Kirkland Signature Jelly Belly Jelly Beans 49 Flavors (4 Lbs)'''
Go back to the Sauce Builder window and click Stop recording.
Now that we've recorded a test, we should save it for safe keeping. Click File -> Save or Export -> Choose HTML as the format and name it, then click Save.

As you can see from the test we've recorded. The test is composed of a series of actions and each action will have one or more options associated with it.

Here is a short video of recording your first test

Digging into how to modify and adapt tests is beyond the scope of what I want to cover in this post, but the following links are some good places to go deeper:

Now that we've recorded our first test, it is time to run it.

Click on Run and choose Run test locally.
The test will begin running in the currently selected tab in Firefox.
Obviously this is a pretty simple test and you could do a lot more with it, including go through adding it to a cart, checking out, and buying the order. But for the purposes of getting started, it's a good place to stop.

Here is a video of running your first test

The next thing we want to do, since our focus is on monitoring, is add some verification steps to each page load. This step is crucial in making our test doing the same kind of checking that your traditional curl URL | grep STRING style monitoring did, but now it's integrated into our browser-driven mode of execution.

Go to the Sauce Builder window
Mouse over the second step and choose New step below
Select the new step
Choose edit action
Select the assertion option
Choose page content
Choose assertText
Click Ok
Choose locator and enter ''link=Your Amazon.com'
Click Ok
Choose equal to and enter the string ''Your Amazon.com''
Click Ok
Click on Run and Run test locally

The test should run successfully, if it does not, then you may want to click on locator and choose Find a different Target and use the tool to select the element you're asserting text with.

This is a critical step as the assertions are somewhat brittle and must be maintained as your application changes over time. For more details, see help on choosing good locators.

Here is a video of adding the assertion to your test and running it locally

Using Sauce Labs for Testing

Now that you've gotten your test running locally and you've added some assertions to make the test useful for monitoring, it is a good idea to run the test externally. As previously mentioned, the Sauce Labs folks run a service to run your tests in the Cloud, and they are nice enough to offer a free plan that gives you 200 "execution" minutes per month and the ability to run your tests under multiple browsers and operating systems with ease. Plus you'll get your jobs stored, logs, screenshots, and a video recorded of the whole test for later review and analysis. So now that you're thinking "where do I sign up?!"

To sign up for the free plan, do the following.

Go to https://saucelabs.com/signup
Enter a username
Enter your email address
Enter a password
Click Sign Me Up

Now configure your Sauce Builder installation to use your free account

Login to https://saucelabs.com/ and click on View My API key
Copy your API key
In your test, choose Run -> Run on Sauce OnDemand
Leave the default Linux - Firefox 3.0
Click Run
When prompted if you have a Sauce Labs account, choose Yes
Enter your username and API key
Choose Save
Your test will start running. Grab a snickers.
You'll end up with a Job URL that looks something like https://saucelabs.com/jobs/6f4629f04dad85cd7803d8049ec00888 (which I've made public, since there is nothing private in it.)
Review the details of the test, as you can see, you get the following for each test
Platform
Start and End Times
Duration
Status
Break down of each Selenium command that's executed
Screenshot of the final page of the test
Video recording of the whole test run.

At this point you've successfully executed a test on Sauce Labs. I recommend you review the following to get a full idea of Sauce Labs features, which includes being able to use it programmatically from various languages, which is beyond the scope of what I'm covering this post.

Using BrowserMob for monitoring

So you've succeeded in getting your test run locally, you've run it externally in the "Cloud", and now you're thinking "wasn't I promised I could use this for monitoring?". Yes, you were, and that's where BrowserMob comes in.

While BrowserMob's primary product is focused on load testing, they've also built a great monitoring product and that's what we'll using to get our monitoring up and running.

BrowserMob is kind enough to offer a free plan, so let's start with getting signed up.

To sign up for the free plan, do the following.

Go to https://browsermob.com/website-monitoring-load-testing-signup
Enter all the required info.
Click Sign Up
Complete the email verification.
You're done.

Now upload and verify your first test.

Go to https://browsermob.com/account/overview
Click on Scripts
Click on Upload Selenium Browser Script
Give it a Name
Click Browser, locate your test file you saved from Sauce Builder
Click on Upload
It should automatically validate.
If it passes validation, you should then see Revalidate, View Log, and Screenshot links
Check out the log and screenshot to get an idea of what will be recorded for each monitoring test run.

Here is a short video showing uploading and verifying your first test

Let's configure an email address for notifications

Click on Monitoring
Clic on Notifications
Click on create one
Enter a name
Confirm the contact name and email, it will default to what you registered with
Click Create

Now let's set up a monitoring job.

Click on Monitoring
Click on Schedule
Give the job a name
Select a Frequency * With the free account, you can run a simple test every 12 hours, for higher frequency or more complicated tests, you'll need to purchase a paid account.
If you want to do an alert, click create
Select a location
Select your notification preference
Click Activate Now
The job will be scheduled and will run at the next internal after the minute the job was created.
Since you just signed up for a trial, you can get the test to run a bit sooner, but only a couple times, so we'll do that now, so we can see what it looks like.
Click Edit next to the test
Change Frequency to 10 minutes
Click Save and Activate
Set a timer for 12 minutes and wait, once it is done, we'll review what things look like. * Once you're done with this, you may want to revert to every 12 hours so that when you're trial expires you won't be over your credits, or just pause/delete the monitoring job.

Here is a video of creating the monitoring job

So now that that test has run, let's take a look at what it looks like.

Click on Dashboard
Mouse over the name of the job and click on the URL, it should look like something like this: https://browsermob.com/monitoring/view/{some_id_here}
You should see a chart that defaults to 1 day and shows you each test, with a bar showing each data point, based on the overall time it took to run the test. * This gives you some quick insight to how performance (as measured by execution time) is doing over time.
You can drill into each data point, and you'll get a waterfall style break down of each test run: how long each element of the page took to load, etc.

Below is a screenshot of a test that has run for a few days.

So a couple tips on how you can use custom stuff from BrowserMob's API to make your tests that much more effective.

Setting variables.

Since the BrowserMob scripts are written in JavaScript, doing variables is as simple as doing var zipcode = '90210'

Getting back data from a webpage.

I've only ever used this to get back the whole response from a page and use it as is, so you'd need to break out a bit of your own JS-fu if you want to use part of a response, but here's how I did it. The code below also shows using a previously declared variable in your request.

var response = c.get("http://api.example.com:8080/id?"+zipcode)
var testid = response.getBody()

At this point the testid variable contains the string returned in the response from the request to http://api.example.com:8080/id?90210

Extra Logging

BrowserMob's JS API has a nice function called browserMob.log() which lets you log arbitrary data and it will show up in the raw logs that BrowserMob keeps for each test run. An example of this is

browserMob.beginStep("Step 2");
selenium.waitForPageToLoad(60000);
selenium.type("id=twotabsearchtextbox", "jelly beans");
browserMob.log('searched for jelly beans')
browserMob.endStep();

For more info on these and more functions, check out the BrowserMob API Documentation

What Next?

At this point you've successfully built a test, run it locally, run it in the "cloud", and deployed it to monitor every 12 hours and are getting alerted by email, you're wondering what's next.

Well, amongst the things you could would would be

Load Testing through BrowserMob
Get called or pager by sending your email alerts into PagerDuty
Interact with your own web services by using the ''getting data back'' example from above

I've made a github repository with my Amazon.com example.

As a challenge and a way to motivate people to contribute and give feedback, the 5 most interesting tests that people submit as pull requests on Github, I will send them a package of stickers, including SysAdvent, Github, Riak, and more!

I hope you've found this post to be informative and would love feedback via email or Twitter on how you do end up using any or all of the services in this post.

Day 20 - Thoughts On Load Testing

2011-12-20T00:00:00.000-08:00

This was written by Adam Fletcher (@adamfblahblah

One task that often falls on the lonely sysadmin is load testing. In this article I'm going to talk about some philosophies and processes I used when doing load testing in my past roles.

I'm going to focus on testing the server side. There's a lot of articles on how to optimize the client side experience, and it is very important that you are aware of both the client side and server side tuning changes so that you can give the customer the best experience.

Use Science

Science

(credit to XKCD. Buy the shirt!)

What I mean by that is that load testing is really experimentation. You're testing a hypothesis: Is setup A better than setup B? Develop your hypothesis, experiment, and measurements, and make conclusions based on data, not feelings. Don't forget that you need to control as many variables as possible. Don't test on your VMs or your staging server that the customer is also using.

Have Defined Targets

You can't determine if your architecture scales/is fast enough/can handle traffic during a disaster/won't crash when you launch/etc. without have something measurable that determines success. For example, instead of saying "make each page load in 400 milliseconds", it is better to say something like "Every page load must have also resources loaded within an average 400 milliseconds with a standard deviation of 25 milliseconds with 1000 clients performing actions every X seconds on Y series of pages." You will then know when you are done load testing because all the measurements you are taking show you have achieved success.

Scale First

If the Y axis is latency and the X axis is number of workers, then scaling is keeping Y constant while you increase X to infinity. This is much harder than keeping the number of workers constant and lowering latency. The first thing to look at during load testing is the shape of your latency curve as load increases. Keep that curve flat and you're most of the way there.

Understand Your Traffic

If you're already suffering a load problem in production, great! Track the pages being requested and the path of the requests through your system. Use a tool like Google Analytics to get a picture of the diversity of the pages hit and the flow your users take through the product. You'll want to be able to model those flows in your load generation software.

For example, an online store may have a few different paths users commonly take through the system: arrive at the home page organically, search for a product, add to a cart, and check out; arrive at a special landing page for a sale, add to cart, checkout; arrive at the home page organically and use the customer services features; etc. If you viewed this store's traffic at single point in time you could divide your simultaneous traffic up by percentage of pages hit: at a point in time, 30% of the requests are to /search, 20% are to /checkout, 40% are to /, and finally 10% are to /customer-service. This is the model you should use for load generation.

Your traffic also follows patterns that depend on the time of day. With the advent of (somewhat) elastic capacity allocation, you can model these usage patterns and adjust your capacity to fit the pattens of usage.

Furthermore, you need to be aware of client side changes such as allowing your users to use HTTP pipelining or inlining Javascript versus loading the Javascript in another HTTP request. Making the server scale requires you understanding the client.

Know Your Resource Limits

You don't have infinite computing power. You don't have infinite money. Most importantly, you don't have infinite time. Be smart about how you use that time.

It's expensive to have people do all the load testing science we're talking about in this article. With a little thought, you can probably guess where your bottleneck is - I'm going to guess it is something related to your data storage. Use your systems knowledge to make your first hypothesis "If I remove the obvious bottleneck, the system will be faster".

Also, to paraphrase Artur Bergman, don't be a backwards idiot - buy some SSDs. They are expensive per GB but they are dead cheap per IOP/S. They're also cheaper than the time you are spending doing the load testing. You'll want to use these SSDs in the machines that have the highest IO load (and you know which machines those are, because you're measuring IO load, right?).

Graphs Lie

There was an excellent talk at Velocity this year about the dangers of trusting your graphs given by John Rauser entitled Look at Your Data. He pointed out that you have to be careful of the trap of representing many points of data at the same X as the mean of those points of data as this representation hides the distribution of that data. This most commonly occurs when measuring latency and during load testing, when you have many requests at time X that, when averaged, come to Z milliseconds. Plot Z for many Xs and you miss the distribution of the latencies at X.

John's video explains it better, but if you look at this graph:

Request Latency Over Time

You'd think from this graph that everything is great - your latency went down!

But if we look at the distribution of our data at each sampling point:

Request Latency Over Time With Sample Distribution

We see that some of users are having a really bad experience on our site.

A good example of a tool that doesn't have this problem is Smokeping. Here's an example of Smokeping telling me that my home internet connection has some jitter in latency:

Comcast ICMP Ping Latency

I've also put a gist up with the R code used to generate the graphs above here.

Measure Time and Resources Spent in Each Component

If you aren't instrumenting each piece of software in your stack you should start doing so. Instrument the entry point to your software and the exit point and graph this data over time. Combine this with even simple data from sar, iostat, other *stat tools, etc, and you can learn a lot about your code without ever firing up a profiler.

Learn And Use The Right Tools

Good tools will allow you to export the raw data in such a way that you can then do analysis on it. Tools that expose your system resource consumption metrics are critical, and it probably doesn't matter what you use as long as you are storing and graphing roughly what iostat, sar, vmstat, netstat and top give you. Learn what each metric really means - do you know why your software is context switching 4000 a second? Do you know if that is bad (hint: probably)? How would that manifest itself in top?

Learn to use the profiler that comes with your product's implementation language. Profilers are amazing things. If you can't use a language-specific profiler try a system-wide profiler such as oprofile or similar.

When you have all this data, use a real data analysis tool to look at it. Learn some R or NumPy/SciPy. Instead of using Excel or a clone for data analysis, consider learning a numerical computing language such as R. For example, in R or NumPy you can write a script that takes all of your raw resource consumption data (CPU, RAM, IOPS, etc) and runs correlation tests against the latency data. Try to do that in Excel! Oh, you can then use that script in your monitoring.

People often call load testing an art, but all that really means is that they're not doing science. Load testing can be challenging, but hopefully this article has given you some things to think about to make your load testing easier and more effective.

Day 19 - Why Use Configuration Management?

2011-12-19T19:29:00.000-08:00

This was written by Aleksey Tsalolikhin (http://verticalsysadmin.com/blog/). Illustrations by Joseph Kern

If you ask Wikipedia, "Configuration management (CM) is a field of management that focuses on establishing and maintaining consistency of a system."

Configuration management tools increase sysadmin efficiency and make sysadmin life better. As our systems grow larger and more complex, we need better tools to help us increase control and reliability of ever growing quanities and complexities in computing. Examples of such tools include Bcfg2, Cfengine, Chef, and Puppet - all of which are open source!

Configuring systems manually in interactive sessions is error-prone and extremely labor-intensive. Even with mostly-automated scripts, such as the typical "ssh and a for-loop" solution, pushing ad-hoc changes are still error prone. For example, if a system is down for maintenance while a change is being pushed out over ssh, it will miss that change, and "state drift" will occur between it and other systems in the same class.

You want a tool that helps keep actual and desired state the same.

System imaging is a common strategy for dealing with complexities of config management - make a copy of a system image, label it "gold master", and clone it to make new systems. While this approach helps to crank out identically configured systems, it has the weakness that updating the master image can be a pain and it does nothing to maintain the systems configured after the initial deploy. It is also not very auditable (what changed between golden image v1 and v2?).

Many sysadmins still configure systems with more traditional manual, ad-hoc, and hard-to-audit methods. In some cases, sysadmin teams build home-grown tools to solve these problems. An example of this is Ticketmaster, who released their own config management, "ssh and for loop" tool, and provisioning systems.

Why do we care to do this? Well, why do we administer systems? Correct configuration helps keep computer systems in use by human civilization.

CM tools free sysadmin's time for more challenging and creative system engineering and architecture work and for taking naps which power such work.

Minimize Manual Effort

Minimize manual effort by automatically configuring new systems. This works well because repeatable work is best left to computers; they don't get bored, and they don't forget steps.

"Go away or I will replace you with a very small shell script" - you've probably seen this shirt before, right? How about hearing someone recommend "automating yourself out of a job"? Building systems and fighting fires without any tools is a slow task that is difficult to repeat accurately, and with many sysadmin skills being software-related, it is in your interest to automate system turn up, maintenance, and repair. Automation helps reduce time spent in corrective actions, reduces mental energy consumed, reduces stress, and increases business value and agility. Winning!

In using a config management system, you are implicitly documenting the system's "desired state" - Why is the system configured this way? What are its dependencies? Who cares about the system? This documenting capability helps protect against knowledge loss by moving configuration knowledge out your brains and into a version control system. This helps defend against data lost through forgetfulness or staff changes, and it also facilitates alignment of efforts on a multi-sysadmin team.

In general, configuration management is in the realm of "Infrastructure as Code". Once your infrastructure is represented in code, you can think about apply release engineering and other tools - tag a new policy as "unstable", test it, then move the new policy into the "stable" branch where servers will apply it.

A Visualization

Sys Admin configures a server manually, ad hoc, and hands-on.

Sys Admin writes a configuration management tool program to configure a server. Then the CM tool (like a little sysadmin robot) configures the server.

Sys Admin takes a nap, while the CM tool configures more servers, and keeps checking and re-configuring the servers (as needed) to keep them in compliance with the program.

Getting Started

To encourage sysadmins to start using Configuration Management, the following is a rough manual of how to do some small tasks in a few different, open source configuration management tools demonstratiing what policies look like in common open-source server. Bourne shell examples are provided to help aid in understanding.

Using these examples

Bourne shell: Can be run on the command line or via cron
CFengine: Follow the quick start guide In a nutshell, put into a promise bundle inside a policy file (example.cf) and run from the command line with "cf-agent -f example.cf -b $bundlename"; or integrate into the default policy set in promises.cf in the CFEngine work directory, often found in /var/cfengine/inputs.
Chef: Follow the Chef Fast Start guide
Puppet: Follow the Getting Started guide. For quick testing of these examples, you can write them to a file 'foo.pp' and execute them with puppet apply foo.pp. Puppet also supports a client-server model that is more common for production deployments.

Set Permissions on a File

Bourne shell
```
chmod 600 /tmp/testfile
```

CFengine

files:
    "/tmp/testfile"
         perms   => m("600");

Chef

file "/tmp/testfile" do
  mode "0600"          
end

Puppet

file { "/tmp/testfile":
   mode => 0600;
}

Create with some content

Bourne shell

echo 'Server will be down for maintenance 2 AM - 4 AM' > /etc/nologin

CFengine

files:
   "/etc/nologin"
        create     => "true",
        edit_line  =>  insert_lines("Server will be down for maintenance 2 AM - 4 AM");

Chef

file "/etc/nologin" do
  content 'Server will be down for maintenance 2 AM - 4 AM' 
end

Puppet

file { "/etc/nologin":
  ensure => present,
  content => "Server will be down for maintenance 2 AM - 4 AM";
}

Install a package

Bourne shell
```
yum -y install httpd
```

CFengine

packages:  
    "httpd"
        package_policy => "add",
        package_method => yum;

Chef
```
package "httpd" 
```

Puppet

package { "httpd":
  ensure => present;
}

Make sure a service daemon is running

Bourne shell

ps -ef | grep httpd >/dev/null 

if [ $? -ne 0 ]  
  then /etc/init.d/httpd start 
fi

CFengine

processes:
   "httpd"
        restart_class => "restart_httpd";

commands:
 restart_httpd::
   "/etc/init.d/httpd start";

Chef

service "http" do 
  action :start   
end

Puppet

service { "httpd":
  ensure => running;
}

Final Thoughts

There's going to be a learning curve to any config management system, but I have found that the benefits in being able to audit, repeat, test, and share "desired state" in code far outweigh any time spent learning the config management tools.

Day 18 - Why Businesses Do Things

2011-12-18T01:02:00.000-08:00

This was written by Joseph Kern (www.semafour.net).

Imagine your whole professional career as a sysadmin and you never understood the OSI model. Those seven simple layers that allow you to build an effective internal framework of network communications. Without this model how would you even begin to understand larger and more complex systems or the complex interactions between multiple systems?

You might get by for a time; hard work and dedication can take you a long way. But you would never be able to progress beyond a certain point. The problem space becomes to complex to brute force.

Now imagine that successfully managing and running a business is at least as complex as managing a network. Managing a 1000 computers is much easier than managing a business of 1000 people. I'd like to take you into the shallow end of business management and show you how the services that we sysadmins maintain are viewed from a business perspective.

Fortunately this framework is simple and avoids any hand waving. We just need this three word phrase, "differentiation and neutralization."

Differentiation builds services that create competitive advantage. Neutralization builds services that seek to maintain competitive equilibrium. This interplay is the heart of what drives business and directs the supporting activities of enterprise level IT. Working DNS is needed for almost all aspects of modern enterprise IT infrastructure, and it will serve as a technical example for this discussion.

Building services that neutralize competitive advantage usually involves buying solutions; these are often disguised as "industry best practices", and have many accompanying white papers offered as proof. More often than not this niche is filled by Microsoft or other large software vendors. You can buy your way out of a problem.

When building a DNS service it is most often thought of as a way to neutralize the advantage of other organizations. Seldom is thought given to how an organization might run DNS 30% "better" (not necessarily faster or any particular quality) than its competitors. In most cases, "better" would not matter at all.

"Better" does not create an advantage with a neutralizing service. Instead, it in fact creates a disadvantage. Time, attention, and resources are being funneled into a project that creates no value from a business standpoint. The business does not (and should not) care about innovating in services that they consider neutralizing. (See footnote #1)

Building services that create competitive differentiation is much different than neutralization as most of these services are built rather than bought. These tend to be very custom to the environment. The prime consideration for these services is adaptability. You must be able to extend the software providing the service as this allows you to out maneuver your competition. You are able to think your way out of a problem.

Turning Neutralizing into Differentiating

OpenDNS took a service that was neutralizing and rebuilt it from the ground up and adding many other services such as anti-phishing, content filtering (based on domain), and reporting. These services created a differentiation in their business model and offered something new to the market. OpenDNS created a reason to build "better" DNS services, as this is their core business model and their competitive advantage.

As it turns out, setting security and content filtering at the DNS level works equally well across all devices all the time and requires no client installation. Now other businesses must appear to neutralize the differential advantage by creating their own services to match. Norton, for example, has followed suit with their Norton Everywhere product offering DNS services that largely mirror OpenDNS. (See footnote #2)

OpenDNS must now continue to differentiate their services from their competitors. OpenDNS recently started offering DNSCrypt, which creates an encrypted channel for DNS queries between the client and the DNS server. Consider it to be SSL for DNS. No doubt, there will be other service providers that follow suit, creating their own DNSCrypt implementations. (See footnote #3)

Why do businesses seemingly chase the tail of their competiors? This is because if organization declines the opportunity to neutralize the advantage of their competition, they will be excluded from further innovation in this field and may be locked out of the market entirely. A technical term for this is a "feature". As the differentiation of services increases, the cost to enter the market (the table stakes) increases accordingly.

Why Should You Care?

Senior sysadmins and engineers need to not only understand how to build a service, but we must also understand why we are building it and what the business requires from this deployment. Understanding the complete picture, we will understand what technology is required, how it needs to be implemented, and how much effort we should put into a project.

Both the engineer and the business get something valuable from this understanding - keeping time and attention focused on important projects. The next time you are asked to deploy a new service ask yourself (and your management) one simple question:

"Is this a service that neutralizes or differentiates?"

Knowing this helps you set your own expectations. If you find yourself wanting to spend energy improving a service, knowing whether it is neutralizing or differentiating will help you make the case to your team and managers that you should be working on it. Knowing it is a neutralizing service might help you set expectations such that you don't spend time and energy thinking hard about how to improve a service that doesn't benefit the business if improved, and having that knowledge and expectations can help keep you from burning out optimizing things that effectively are unimportant.

Footnotes

Why do you think sharepoint is so popular? It's not because it does everything well ...
In the light of Windows 8 coming preloaded with Anti-Virus software, Norton is facing an almost complete lockout of their traditional market.
The great thing about standards, there are so many to choose from.

Day 17 - Speaking the Same Language

2011-12-17T00:51:00.000-08:00

This was written by Jordan Sissel (semicomplete.com).

Language is important.

You've probably had disagreements or confusions that, during or after, you realized were caused by miscommunication or misinterpretation. Something got lost in translation, right?

Language is important, so do you resolve these issues? Lawyers do it by dedicating great lengths of text to defining terms to eliminate confusion. What was the last legal document you read? Perhaps the constantly-changing iTunes EULA? Did you read it? Did you skip reading it because it was 70 pages long? Was it readable? Was it plain english?

What was the last legal document you saw that seemed approachable? Can you even read this document at the default font size?

Legal documents are long for many reasons, but the main reason I believe is to reduce loopholes or confusion by defining and using a common language and vocabulary. Vocabulary is important. Most of these documents have embedded definitions of just about every major term used. Look at your current employment contract - how much of it is simply defining words?

Centrally defining all words allows two parties who speak different languages to speak on a common ground. Look at the GPL3 and Apache 2 licenses compared to the MIT license. Both the GPL3 and Apache 2 licenses specify more requirements or allowances than the MIT one, but my point is that much of the raw text in both GPL3 and Apache 2 are definitions. Compare this to the MIT license which has practically no definitions embedded.

Verbal Communication

In your average day, it is unlikely you have the time or energy to spend defining terms in the middle of every conversation. Most of the time you might assume the other person (or people) know what you mean when you say it.

I propose that the likelihood of one person understanding your words is in inversely related to their distance from your context, job role, and other factors you could have (or have not) in common.

Explaining something job-related to a fellow sysadmin will require a different set of terminology, a different language, than doing the same to your manager, someone outside your engineering group, someone in marketing, etc. The knee-jerk reaction is often to assume the other person is stupid. They aren't understanding you! It sucks. It increases tension and distrust.

Like I said, you probably don't have time to work with each person (or group) to design a common terminology. You have stuff to do. What other options do you have?

My best recommendation here is to study everyone. Watch what they say, how they say it, and what you think they mean when they say something. Study their reactions when you say things. If they appear confused, ask and clarify. Don't treat them like they're an idiot. Speaking loudly and slowly doesn't help anyone; ask for what is confusing, and ask what needs clarification and definition.

Know your audience.

Avoid analogies. Analogies are you translating your words into other words you hope the audience will understand. Bad analogies are very easy to make, and you can accidentally increase confusion and distrust like the famous "the internet is not a truck" failure. If you have bidirectional communication with a person, instead of making an analogy, why not ask for clarification and definition?

Visual Communication

Pictures can help, too. Bought anything from Ikea recently? If not, check them out, or just look at this table assembly manual. All the Ikea furniture assembly manuals I've seen are pretty effective with only pictures. No words.

Study your audience.

If you study and interact with a given set of folks frequently enough, you should learn to more easily speak a language they understand. Marketing and PR folks probably don't care about disks being full, but they care about the external impact of that problem. Try to understand what a person knows and what they care about and frame your words accordingly.

Software Similarity and Language Problems

Language problems affect even groups with small distances between them. Look at logging tools: You'll see terms like facility, severity, log level, debug level, source, and more all referring to pretty much the same things. What about monitoring? Nagios calls it a 'service'; Zabbix calls it an 'item' and sometimes a 'check'; Xymon calls it a 'service'; Sensu calls it a 'plugin'.

All of monitoring term examples above mean pretty much the same thing, and before you disagree, consider that I say they mean the same thing because they look like the same thing from a distance. Knowing Nagios and learning a new monitoring tool requires learning new term definitions as well as new software, and overloaded terms (like 'service') can have different meanings in different projects. It trips up the brain!

Look at features provided by similar tools, and each feature is likely to have a different name for the same thing. Intentional or otherwise, this is a language-equivalent of vendor lock-in, and it sucks.

Puppet calls it a module and manifest; Chef calls it a cookbook and recipe. Ruby and Python use the word "yield" to mean different things, and this causes much confusion for people trying to learn both languages. You can reduce learning curves if you use a common language in your systems and projects.

When to Define Terms

It is worth defining terms if you need to have a long-lived common ground. You want defined common terms and features of a project that stay defined through out the project's life cycle.

As an example, there's a small group of folks who say, "Monitoring sucks," myself included. We got together to discuss ways to solve crappy monitoring problems, and one of the tasks was to define a common set of terms - we went the route lawyers go because having a common terminology ground would strengthen the #monitoringsucks movement. Agree or disagree with the definitions we came up with, the point was to lay a path for discussion that could avoid religious wars over confused terminology. The common terms were also chosen to help steer any new projects to use the same terms and reduce the learning curve as a result.

Parting Thoughts

Much of technical writing education focuses on knowing the audience. While 'technical writing' leans towards one-way communication (writer communicating with readers), the ideas are important in general communication.

Who are you talking to? What are their interests? What are the boundaries of their knowledge?

Day 16 - Shipping Some Logs

2011-12-16T16:55:00.000-08:00

This was written by Jordan Sissel (semicomplete.com).

Logging is messy. Ever have logs fill up your disk and crash services as a result? Me too, and that sucks.

You can solve the disk-filling problem by rotation logs and expiring old ones, but there's a better solution that solves more problems: ship your logs somewhere else. Shipping logs somewhere centralized helps you more quickly access those logs later when you need to debug or do analytics.

There are plenty of tools in this area to help you solve log transport problems. Common syslog servers like rsyslog and syslog-ng are useful if syslog is your preferred transport. Other tools like Apache Flume, Facebook's Scribe, and logstash provide an infrastructure for reliable and robust log transport. Many tools that help solve log transportation problems also solve other problems, for example, rsyslog can do more than simply moving a log event to another server.

Starting with Log Files

For all of these systems, one pleasant feature is that in most cases, you don't need to make any application-level changes to start shipping your logs elsewhere: If you already log to files, these tools can read those files and ship them out to your central log repository.

Files are a great common ground. Can you 'tail -F' to read your logs? Perfect.

Even rsyslog and syslog-ng, while generally a syslog server, can both follow files and stream out logs as they are written to disk. In rsyslog, you use the imfile module. In syslog-ng, you use the file() driver. In Flume, you use the tail() source. In logstash, you use the file input plugin.

Filtering Logs

Most of the tools mentioned here support some kind of filtering, whether it's dropping certain logs or modifying them in-flight.

Logstash, for example supports dropping events matched by a certain pattern, parsing events into a structured piece of data like JSON, normalizing timestamps, and figuring out what events are single-line and what events are multi-line (like java stack traces). Flume lets you do similar filter behaviors in decorators

In rsyslog, you can use filter conditions and templates to selectively drop and modify events before they are output. Similarly, in syslog-ng, filters let you drop events and templates let you reshape the output event.

Final Destination

Where are you putting logs?

You could put them on a large disk server for backups and archival, but logs have valuable data in them and are worth mining.

Recall Sysadvent Day 10 which covered how to analyze logs stored in S3 using Pig on Amazon EC2. "Logs stored in S3" - how do you get your logs into S3? Flume supports S3 out of the box allowing you to ship your logs up to Amazon for later processing. Check out this blog post for an example of doing exactly this.

If you're looking for awesome log analytics and debugging, there are a few tools out there to help you do that without strong learning curves. Some open source tools include Graylog2 and logstash are both popular and have active communities. Hadoop's Hive and Pig can help, but may have slightly steeper learning curves. If you're looking for a hosted log searching service, there's papertrail. Hosted options also vary in features and scope; for example, Airbrake (previously called 'hoptoad') focuses on helping you analyze logged errors.

And then?

Companies like Splunk have figured out that there is money to be made from your logs, and web advertising companies log everything because logs are money, so don't just treat your logs like they're a painful artifact that can only be managed with aggressive log rotation policies.

Centralize your logs somewhere and build some tools around them. You'll get faster at debugging problems and be able to better answer business and operations questions.

Day 15 - Automating WordPress with CFEngine

2011-12-15T01:35:00.000-08:00

System administration is a relatively new profession. Without a standard curriculum, practitioners have different philosophies and practices. It is challenging for new sysadmins because every organization implements differently: the how and why of system setup, how and why of maintenance, and the how and why of disaster recovery and growth.

A software tool can respond faster than a human sysadmin to a deviation from configuration policy (something being broken). The corrective action can be automated, so chaos is kept to a minimum while not requiring human action.

Why WordPress? Installing WordPress involves coordinating multiple system components into a harmonious whole. It is a great demonstration of the power of automated configuration management. It involves copying and editing files, installing packages, and starting and restarting services.

Manually installing WordPress often takes tens of minutes. An automated install under CFEngine greatly shortens the time required and most importantly provides a repeatable and auditable experience.

Lastly, an introduction to CFEngine is out of scope for this post, but you can learn more here, here, and here.

Automating WordPress Installation

The two main parts of infrastructure involved in making WordPress work are a web server and a database. In this example, we'll use Apache httpd and MySQL as well as assume a Red Hat (or derivative) system.

The most up to date version of the cfengine implementation of this post can be found here: https://github.com/cfengine/contrib/raw/master/wordpress_installer.cf

You can run this policy with:

cf-agent -f /var/cfengine/inputs/wordpress_installer.cf

The rest of this post covers the manual steps you might do to install WordPress and also the equivalent implementation in CFengine.

Ordering Things

Below shows the control promise which controls the behavior of cf-agent including which files it should import (the standard Cfengine library) and in what sequences to examine and keep bundles (collections) of promises.

body common control 
{

        bundlesequence => {
                                "packages_installed",
                                "services_up",
                                "wordpress_tarball_is_present",
                                "wordpress_tarball_is_unrolled",
                                "configuration_of_mysql_db_for_wordpress",
                                "wpconfig_exists",
                                "wpconfig_is_properly_configured",
                                "allow_http_inbound",
                          };

        inputs =>        { "/var/cfengine/inputs/cfengine_stdlib.cf" };
}

Get the Right Packages

With that order given above, let's start by ensuring we have all the necessary packages. We will use the "yum" package_method since we are using a Red Hat derivative.

The packages_installed bundle depicted in below promises to restart the httpd if any packages are added to cover the case where httpd is up and running, but "php" and "php-mysql" are missing, and Cfengine installs them.

bundle agent packages_installed
{

vars: "desired_package" slist => {
          "httpd",
          "php",
          "php-mysql",
          "mysql-server",
         };

packages: "$(desired_package)"
    package_policy => "add",
    package_method => yum,
    classes => if_repaired("packages_added");

commands:
  packages_added::
  "/sbin/service httpd graceful"
    comment => "Restarting httpd so it can pick up new modules.";
}

Apache and MySQL

Now let's make sure httpd and mysqld are running with the services_up bundle shown below:

bundle agent services_up {
processes:
  "^mysqld" restart_class => "start_mysqld";
  "^httpd"  restart_class => "start_httpd";

commands:
  start_mysqld::
    "/sbin/service mysqld start";
  start_httpd::
    "/sbin/service httpd start";
}

The "restart_class" is used to scan the "ps" output for the named string, and if not found, the right hand side class will be set. We can then use that to launch a command to start the server.

Downloading WordPress

The next section shows the wordpress_tarball_is_present bundle where we make sure we have a copy of WordPress in an arbitrary location - let's say in /root. We'll need it later to install WordPress under the httpd document root.

We test using Cfengine built-in test function "fileexists()". If the file exists the "wordpress_tarball_is_present" class gets defined. (A class is Cfengine implicit if/then test. If it is defined, the test passes. If it is not defined it does not. In other words, defined = true, not defined = false.)

If the file does not exist, the "wordpress_tarball_is_present" class will not be defined and the commands promise will download it. If the file does exist, no action will be taken.

bundle agent wordpress_tarball_is_present
{
classes:
  "wordpress_tarball_is_present" expression =>
    fileexists("/root/wordpress-latest.tar.gz");

reports:
  wordpress_tarball_is_present::
    "WordPress tarball is on disk.";

commands:
  !wordpress_tarball_is_present::
    "/usr/bin/wget -q -O /root/wordpress-latest.tar.gz
http://wordpress.org/latest.tar.gz"
    comment => "Downloading WordPress.";
}

Unpacking WordPress

Next, we test if the WordPress directory exists under the document root (assumed to be "/var/www/html").

If it doesn't, we'll extract our WordPress tarball to the docroot using "tar".

Note that the "tar" extract promise depends on the earlier promise that the tar ball is on disk. Because Cfengine does three passes through the promises when it runs: on the first pass, the tar ball will be downloaded if necessary; on the second pass, Cfengine will extract it. This is an example of convergence to desired state, part of the basic philosophy of Cfengine.

Because Cfengine is convergent in its operation, the above cf-agent command can be run multiple times, and the system will always stay at or approach the desired state, never get further away from it. It can fight entropy and system state drift.

bundle agent wordpress_tarball_is_unrolled
{
classes:
  "wordpress_directory_is_present" expression =>
    fileexists("/var/www/html/wordpress/");
reports:
  wordpress_directory_is_present::
    "WordPress directory is present.";
commands:
  !wordpress_directory_is_present::
    "/bin/tar -C /var/www/html -xvzf /root/wordpress-latest.tar.gz"
      comment => "Unrolling wordpress tarball to /var/www/html/.";
}

Configuring MySQL

Next, we use the "mysql" command to create the database for the application data store as well as credentials to access it:

bundle agent configuration_of_mysql_db_for_wordpress
{
commands:
  "/usr/bin/mysql -u root -e \"
    CREATE DATABASE IF NOT EXISTS wordpress;
    GRANT ALL PRIVILEGES ON wordpress.*
    TO 'wordpress'@localhost
    IDENTIFIED BY 'lopsa10linux';
    FLUSH PRIVILEGES;\"
  ";
}

Please note the above command (like all these promise bundles) is convergent to desired state - it will either get us to the desired state if we are not there, or keep us there if we are there already.

The desired state is a "wordpress" database that can be accessed via a "wordpress" user with the password "lopsa10linux".

Adding the WordPress Config

Let's copy the sample config file WordPress ships with to wp-config.php if it doesn't exist.

First, we check if wp-config.php exists using the built-in "fileexists()" function. If wp-config.php exists, this will set a "wordpress_config_file_exists" class.

This class will be used to control what happens next: if the class is set, no changes will be made to the system; we'll just report wp-config.php is there. If the class is not defined, we'll report wp-config.php is not there, and then put it there by copying it from wp-config-sample.php

bundle agent wpconfig_exists
{
classes:
  "wordpress_config_file_exists"
  expression => fileexists("/var/www/html/wordpress/wp-config.php");
reports:
  wordpress_config_file_exists::
    "WordPress config file /var/www/html/wordpress/wp-config.php is present";
commands:
  !wordpress_config_file_exists::
  "/bin/cp -p /var/www/html/wordpress/wp-config-sample.php \
    /var/www/html/wordpress/wp-config.php"
    comment => "Creating wp-config.php from wp-config-sample.php";
}

Here is the wp-config-sample.php sample config:

// ** MySQL settings - You can get this info from your web host ** //
/** The name of the database for WordPress */
define('DB_NAME', 'database_name_here');

/** MySQL database username */
define('DB_USER', 'username_here');

/** MySQL database password */
define('DB_PASSWORD', 'password_here');

Taking the sample config above, we can use the "replace_patterns" in cfengine_stdlib.cf to replace database_name_here with our database name, and so on. Just like using a template, we replace placeholders with actual values.

bundle agent wpconfig_is_properly_configured
{
files:
  "/var/www/html/wordpress/wp-config.php"
    edit_line => replace_default_wordpress_config_with_ours;
}

bundle edit_line replace_default_wordpress_config_with_ours
{
replace_patterns:
  "database_name_here" replace_with => value("wordpress");

replace_patterns:
  "username_here" replace_with => value("wordpress");

replace_patterns:
  "password_here" replace_with => value("lopsa10linux");
}

Configure IPTables

As a finishing touch, let's make sure our host firewall allows inbound connections on port 80 TCP (Figure 9).

The is our most complicated promise bundle. There are three levels of abstraction: a "files" type promise that edits a file using "edit_line" type promise bundle uses "insert_lines" (from cfengine_stdlib.cf) which has an attribute "location" which is defined (in a separate promise attribute body) as before the iptables rule for accepting established TCP connections.

Incidentally, this promise bundle will also restart iptables if it edits the iptables config file.

Abstracting the details allows the sysadmin to see at a high level what's going on without being blinded by too many details at once, yet the details are accessible to examination if needed.

bundle agent allow_http_inbound
{
files:
  redhat::  # tested on RHEL only, file location may vary on other OSs
  "/etc/sysconfig/iptables"
    edit_line => insert_HTTP_allow_rule_before_the_accept_established_tcp_conns_rule,
    comment => "insert HTTP allow rule into /etc/sysconfig/iptables",
    classes => if_repaired("iptables_edited");
commands:
  iptables_edited::
  "/sbin/service iptables restart"
    comment => "Restarting iptables to load new config";
}

bundle edit_line insert_HTTP_allow_rule_before_the_accept_established_tcp_conns_rule
{
vars:
  "http_rule" string => "-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT";
insert_lines: "$(http_rule)",
  location => before_the_accept_established_tcp_conns_rule;
}

body location before_the_accept_established_tcp_conns_rule
{
before_after => "before";
first_last => "first";
select_line_matching => "^-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT.*";
}

To summarize, here is what the policy can do:

Install Web server and required httpd modules (php, php-mysql);
Install Web app in httpd docroot
Install and configure the database for the Web app
Configure the Web app to use the database
Configure the host firewall

There is a more sophisticated version of this automated WordPress installer in Aleksey's "CFEngine 3 Examples Collection" ( http://www.verticalsysadmin.com/cfengine/cfengine_examples.tar ), see 2030_More_Examples._EC2._Example102_wordpress_installation.cf

Day 14 - Write Your Resume in Markdown Already!

2011-12-14T00:06:00.000-08:00

This was written by Phil Hollenback (www.hollenback.net)

Like most of you, I hate dealing with my resume. That means I don't keep it as current as I should. Over the years, I've kept my resume in all sorts of formats: straight text, nroff, TeX, etc. I think I might have even handcrafted a postscript resume once.

However, I got so tired of recruiters always asking for a Word version of my resume that somewhere along the way I gave up and wrote one in Word (probably OpenOffice now that I think about it). I saved that as a pdf and an html version and forgot about it. That seemed to work ok since I've been gainfully employed for the last decade or so.

However, I recently got a promotion at work so I decided it was time to revisit my resume. Frankly it was a little embarrassing as much of my resume was no longer particularly current or relevant. Does anyone really care that I once worked as a Tcl programmer? Also there's probably no real need for me to list jobs from 10 years ago that I barely remember. Thus, I decided to rewrite my resume.

Speaking of promotions, I often see folks only maintaining their resumes when considering job changes, but you should keep it updated! A record of accomplishments at your current job is useful in times of performance reviews or promotion consideration. For company internal things, you can maintain an internal company resume.

Selecting a Markup Language

You sysadmins can see where this is going: I used my resume rewrite as an excuse to try new tools. Specifically, I wanted to answer the question of how a system administrator should be managing their resume in 2011. I decided on a few criteria:

human-readable source format
- I should be able to look at the original document and understand it. That presumably means straight ASCII text.
- (yes I know that's US English only).
- This also eliminates the 'oops, my deleted text is still in the file' problem with Word.
PDF output
- PDFs seem the most portable way to exchange documents and ensure that formatting remains intact.
HTML output
- I want a version of my resume I can post on a website.
Open-source tools
- No lockdown in proprietary formats.
the cool factor
- I wanted something fresh and interesting that I could (for example) blog about.

With these constraints, it was pretty clear I'd have to choose something like Markdown or TeX for the my source document and then figure out a way to convert those to various output formats.

I have some experience writing TeX / LaTeX documents, and I knew that would probably be sufficient. For example, I knew I could convert TeX into HTML and PDF fairly easily.

However, while TeX files are somewhat human-readable, they aren't really presentable in that format. The big advantage of formats such as Markdown is that the source document can stand on it's own. That way if I needed an ASCII version of my markdown document I could just use the source file. That wouldn't work for TeX.

As I mentioned, I wanted to try something interesting and fairly new for this exercise, and Markdown certainly is the lightweight markup language of choice these days. Lots of blogging tools support Markdown as a native format, so if I pasted chunks of my resume onto websites they would magically show up as formatted text.

Finally, Markdown has momentum. There are lots of other perfectly good alternatives like textile and reSTructured Text. However, none of them are the default markdown language on github.

Tools

I'm a hardcore emacs user so I knew I wanted to write my markdown in Emacs. Naturally there's a Markdown mode for Emacs. I run Emacs 23.3.1 on my Mac, by the way, and I recommend it. Getting markdown-mode running was as simple as copying the .el file and adding a few bits to my .emacs:

(autoload 'markdown-mode "markdown-mode.el"
  "Major mode for editing Markdown files" t)
(setq auto-mode-alist
   (cons '("\\.md" . markdown-mode) auto-mode-alist))

Once that was installed I could open a .md file, make my edits, and preview it as html in my browser with a simple C-c C-c p.

Next I tackled the problem of converting markdown to pdf. Luckily Brandon Burton happened to mention the free markup converter Pandoc on a mailing list I subscribe to. Pandoc comes with a markdown2pdf script. Based on the name of that script I was pretty sure pandoc could do what I needed.

I installed pandoc from macports, and I should mention that I also had TeX installed from the same source. This is important because pandoc uses TeX in the markdown->pdf pipeline.

Constructing A Resume

Now that I had some tools for creating and processing Markdown documents, I needed to figure out how to structure my resume in Markdown. For this I just stared googling "markdown resume" until I found something I liked. Nathaniel Welch's resume is an excellent place to start. I ended up using a similar although somewhat simplified design. I basically just used standard headers and paragraphs all the way down: my name in the largest header size, major headings one size smaller, etc. Something like this:

# Philip J. Hollenback

philiph@pobox.com

## Overview

I do stuff.

## Experience

### SysAdvent, On The Interwebs

#### Blogger, 2010 to present

Wrote some awesome and some mediocre SysAdvent articles.

et cetera, et cetera. Note that you should consider using reference links for readability as this puts the links at the end of your Markdown document for a more natural text flow.

Generating Output

TeX Time

Once you have your resume written in Markdown, it's time to produce some output. I put my resume.md file in a separate resume directory and wrote a makefile to handle generating the html and pdf versions. I'll go over that later in this article.

To generate a pdf from my markdown source, I first tried this:

markdown2pdf resume.md

which produced a usable pdf file, but there were some problems:

there was a page number at the bottom of every page
the margins were much too large
the font looked like garbage when I printed the pdf file

After reading the pandoc documentation and researching on the web I figured out how to fix the first two problems. You just need to include a TeX header file to control how the output gets generated. Here's a good starting point for that header:

\usepackage[top=1in, bottom=1in, left=1.25in, right=1.25in]{geometry}
\pagestyle{empty}

That gave me a resume with much more narrow margins, and turned off the page numbers. All I had to do was include the header file when running markdown2pdf:

markdown2pdf -o resume.pdf -H header.tex resume.md

that fixed all my problems except the ugly font. Fixing that turned out to be a little more complicated: I had to turn on XeTeX and create an entirely new output template. That's the only way I could find to easily select new fonts for TeX on my mac. I'm hoping people who read this have better solutions.

I ran fc-list to find all my available system fonts. I ended up using that delightful old standard, Times New Roman.

Next, I dumped out the LaTeX template that pandoc uses by default to use as a starting point:

pandoc -D latex > resume-template.tex

and modified that file to incorporate my header changes and add font selection to the xetex section. I also forced the font size to 11 points:

@@ -1,9 +1,16 @@
-\documentclass$if(fontsize)$[$fontsize$]$endif${article}
+\documentclass[fontsize=11pt]{article}
 \usepackage{amssymb,amsmath}
+
+\usepackage[top=1in, bottom=1in, left=1.25in, right=1.25in]{geometry}
+\pagestyle{empty}
+
$if(xetex)$
 \usepackage{ifxetex}
 \ifxetex
 \usepackage{fontspec,xltxtra,xunicode}
+  \setmainfont{Times New Roman}
+  \setsansfont{Verdana}
+  \setmonofont{Courier New}
 \defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
 \else
 \usepackage[mathletters]{ucs}

(you can download the full template from my website).

The final step was to use the --xetex option to markdown2pdf to use xetex and enable my custom font. My markdown2pdf invocation thus became:

markdown2pdf --template=resume-template.tex --xetex resume.md

the end result of this is a nice clean pdf file. Compare this to the original Markdown.

HTML

The final piece of the output puzzle is HTML. Convert the markdown to html by invoking pandoc directly:

pandoc -o resume.html resume.md

That results in a serviceable but not very attractive html file. Fortunately pandoc allows you to also use a css file to change the html output. I used Michael White's css file as starting point and tweaked the colors slightly. Then, I invoke pandoc as follows:

pandoc -t html -o resume.html -c resume.css resume.md

You can see the final result on my website. I'm quite happy with the fit and finish.

Automation

No SysAdvent article is complete without a section on automation. To wrap this whole thing together, I wrote a makefile (as I mentioned earlier). I also wrote a perl script to automatically update my resume files on my website when they changed. First, the makefile:

all:    resume.html resume.pdf

%.html: %.md
    pandoc -t html -o $@ $< -c resume.css
    ./resume-uploader $@ $< resume.css

%.pdf:  %.md
    markdown2pdf --template=resume-template.tex --xetex $<
    ./resume-uploader $@ $< resume-template.tex

clean:
    rm -f *~ *.html *.log *.pdf

The perl script is a bit too long to post here, but you can find it on my website as well. This script takes a filename as input, and checks if the file exists on the server. If it exists and the version on the server is older than the one just generated, the script uploads the file. Note that authentication is handled via the .netrc file.

Now I have a one-stop mechanism for updating my resume. I just load up my resume.md file in emacs and use markdown-mode to edit it. I can hit C-c C-c p at any time to view it in html in my browser. When I'm happy with the contents, I run my makefile with C-x C-k to build fresh html and pdf versions. Finally, the uploader script uploads the new versions of my resume and support files to my website as appropriate.

Another enhancement to consider is putting this whole thing in git and hosting it on github pages (did I mention that github supports Markdown as a native format)?

Conclusion

So there you have my account of how I converted my resume to Markdown. In general, I'm very pleased with this approach: I get the simplicity of a text source file combined with decent HTML and PDF output. With a little bit of sysadmin-compatible automation, I've turned this whole thing in to a repeatable workflow. The best thing about this approach is the flexibility. If I want to host my resume on github, it's ready to go. If I want to change my fonts or html output, I just have to make a few simple changes to the input files. Most importantly, I'm not locked in to a proprietary format like Microsoft Word.

If you are interested in using my resume as a starting point for your own, I suggest you consider some hardcore github forking action.

Day 13 - Test Driven Infrastructure with Vagrant, Puppet and Guard

2011-12-13T00:36:00.000-08:00

This was written by Patrick Debois (www.jedi.be)

What's the next step once you implement some config management tools? Testing, right? Wouldn't you like to automate that with some nice clean and sandboxed environment for testing your config management changes?

You can use Vagrant to help ease the VM management. It is a great tool often used as a sandbox environment to develop Chef recipes or Puppet manifests, and this workflow usually looks like this:

you create a vagrant vm
share some puppet/chef files via a shared directory
edit some files locally
run a vagrant provision to see if this works
and if you are happy with it, commit it to your favorite version control repository

Specifically for puppet, thanks to the great work by Nikolay Sturm and Tim Sharpe, we can now also complement this with tests written in rspec-puppet and cucumber-puppet. You can find more info at Puppet unit testing like a pro.

So we got code, and we got tests, what else are we missing? Automation of this process: it's funny, if you think of it, that we automate the hell out of server installations, but we haven't automated the previous described process.

The need to manually run vagrant provision or rake rspec actually breaks my development flow: I have to leave my editor to run a shell command and then come back to it depending on the output.

Would it not be great if we could automate this whole cycle? And have it run tests and provision whenever files change?

How

The first tool I came across is autotest: it allows one to automatically re-execute tests depending on filesystem changes. Downside is that it could either run cucumber tests or rspec tests.

So enter Guard; it describes itself as a command line tool to easily handle events on file system modifications (FSEvent / Inotify / Polling support). Just what we wanted!

Installing Guard is pretty easy, you require the following gems in your Gemfile

gem 'guard'
gem 'rb-inotify', :require => false
gem 'rb-fsevent', :require => false
gem 'rb-fchange', :require => false
gem 'growl', :require => false
gem 'libnotify', :require => false

As you can tell by the names, it uses different strategies to detect changes in your directories. It uses growl (if correctly setup) on Mac OS X and libnotify on Linux to notify you if your tests pass or fail. Once installed you get a command guard.

Guard uses a configuration file Guardfile, which can be created by guard init. In this file you define different guards based on different helpers: for example there is guard-rspec, guard-cucumber and many more. There is even a guard-puppet(which we will not use because it works only for local provisioning)

To install one of these helpers you just include it in your Gemfile. We are using only two here:

gem 'guard-rspec'
gem 'guard-cucumber'

Each of these helpers has a similar way of configuring themselves inside a Guardfile. A vanilla guard for a ruby gem with rspec testing would look like this:

guard 'rspec' do
  watch(%r{^spec/.+_spec\.rb$})
  watch(%r{^lib/(.+)\.rb$})     { |m| "spec/lib/#{m[1]}_spec.rb" }
  watch('spec/spec_helper.rb')  { "spec" }
end

Whenever a file that matches a watch expression changes, it would run an rspec test. By default if no block is supplied, the file itself is run. You can alter the path in a block as in the example.

Once you have a Guardfile you simply run guard (or bundle exec guard) to have it watch changes. Simple hu?

Vagrant setup

Enter our sample puppet/vagrant project. You can find the full source at http://github.com/jedi4ever/vagrant-guard-demo

It's a typical vagrant project with the following tree structure:(only 3 levels shown)

├── Gemfile
├── Gemfile.lock
├── Guardfile
├── README.markdown
├── Vagrantfile
├── definitions # Veewee definitions
│   └── lucid64
│       ├── definition.rb
│       ├── postinstall.sh
│       └── preseed.cfg
├── iso # Veewee iso
│   └── ubuntu-10.04.3-server-amd64.iso
└── vendor
    └── ruby
        └── 1.8

Puppet setup

The project follows Jordan Sissel's idea of puppet nodeless configuration. To specify the classes to apply to a host, we use a fact called: server_role. We read this from a file data/etc/server_tags via a custom fact (inspired by self-classifying puppet node).

This allows us to only require one file, site.pp. And we don't have to fiddle with our hostname to get the correct role. Also if we want to test multiple roles on this one test machine, just add another role to the data/etc/server_tags file.

├── data
│   └── etc
│       └── server_tags

$ cat data/etc/server_tags
role:webserver=true

The puppet modules and manifests can be found in puppet-repo. It has class role::webserver which includes class apache.

puppet-repo
├── features # This is where the cucucumber-puppet catalog policy feature lives
│   ├── catalog_policy.feature
│   ├── steps
│   │   ├── catalog_policy.rb
│   └── support
│       ├── hooks.rb
│       └── world.rb
├── manifests
│   └── site.pp #No nodes required
└── modules
    ├── apache
    |    <module content>
    ├── role
    │   ├── manifests
    │   │   └── webserver.pp # Corresponds with the role specified
    │   └── rspec
    │       ├── classes
    │       └── spec_helper.rb
    └── truth # Logic of puppet nodeless configuration
        ├── lib
        │   ├── facter
        │   └── puppet
        └── manifests
            └── enforcer.pp

Puppet - Vagrant setup

These are the settings we use in our Vagrant file to make puppet work:

config.vm.share_folder "v-data", "/data", File.join(File.dirname(__FILE__), "data")
# Enable provisioning with Puppet stand alone.  Puppet manifests
# are contained in a directory path relative to this Vagrantfile.
config.vm.provision :puppet, :options => "--verbose"  do |puppet|
  puppet.module_path = ["puppet-repo/modules"]
  puppet.manifests_path = "puppet-repo/manifests"
  puppet.manifest_file  = "site.pp"
end

Puppet tests setup

The cucumber-puppet tests will check if the catalog compiles for role role::webserver

Feature: Catalog policy
  In order to ensure basic correctness
  I want all catalogs to obey my policy

  Scenario Outline: Generic policy for all server roles
    Given a node with role "<server_role>"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve

    Examples:
      | server_role |
      | role::webserver |

The rspec-puppet tests will check if the package http gets installed

require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"
describe 'role::webserver', :type => :class do
  let(:facts) {{:server_tags => 'role:webserver=true',
      :operatingsystem => 'Ubuntu'}}
  it { should include_class('apache') }
  it { should contain_package('httpd').with_ensure('present') }
end

Guard setup

To make Guard work with a setup like our puppet-repo directory we need to change some things.

This has mostly to do with conventions used in development projects where Guard is normally used.

Fixing Guard-Cucumber to read from puppetrepo/features

The first problem is that the Guard-Cucumber gem standard reads it's features from features directory. This is actually hardcoded in the gem. But nothing a little monkey patching can't solve:

require 'guard/cucumber'

# Inline extending the ::Guard::Cucumber
# Because by default it only looks in the ['features'] directory
# We have it in ['puppet-repo/features']
module ::Guard
  class ExtendedCucumber < ::Guard::Cucumber
    def run_all
      passed = Runner.run(['puppet-repo/features'], options.merge(options[:run_all] || { }).merge(:message => 'Running all features'))

      if passed
        @failed_paths = []
      else
        @failed_paths = read_failed_features if @options[:keep_failed]
      end

      @last_failed = !passed

      throw :task_has_failed unless passed
    end
  end
end

# Monkey patching the Inspector class
# By default it checks if it starts with /feature/
# We tell it that whatever we pass is valid
module ::Guard
  class Cucumber
    module Inspector
      class << self
        def cucumber_folder?(path)
          return true
        end
      end
    end
  end
end

Orchestration of guard runs

The second problem was to have Guard only execute the Vagrant provision when BOTH the cucumber and rspec tests would be OK.

Inspired by the comments of Netzpirat, I got it working so that the block vagrant provision would only execute on both tests being complete.

# This block simply calls vagrant provision via a shell
# And shows the output
def vagrant_provision
  IO.popen("vagrant provision") do |output|
    while line = output.gets do
      puts line
    end
  end
end

# So determine if all tests (both rspec and cucumber have been passed)
# This is used to only invoke the vagrant_provision if all tests show green
def all_tests_pass
  cucumber_guard = ::Guard.guards({ :name => 'extendedcucumber', :group => 'tests'}).first
  cucumber_passed = cucumber_guard.instance_variable_get("@failed_paths").empty?
  rspec_guard = ::Guard.guards({ :name => 'rspec', :group => 'tests'}).first
  rspec_passed = rspec_guard.instance_variable_get("@failed_paths").empty?
  return rspec_passed && cucumber_passed
end

Guard matchers

With all the correct guards and logic setup, it's time to specify the correct options to our Guards.

group :tests do

  # Run rspec-puppet tests
  # --format documentation : for better output
  # :spec_paths to pass the correct path to look for features
  guard :rspec, :version => 2, :cli => "--color --format documentation", :spec_paths => ["puppet-repo"]  do
    # Match any .pp file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/.*/[^.]*\.pp$}) { "puppet-repo" }
    # Match any .rb file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/.*/[^.]*\.rb$}) { "puppet-repo" }
    # Match any _rspec.rb file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/.*/[^.]*_rspec.rb})
  end

  # Run cucumber puppet tests
  # This uses our extended cucumber guard, as by default it only looks in the features directory
  # --strict        : because otherwise cucumber would exit with 0 when there are pending steps
  # --format pretty : to get readable output, default is null output
  guard :extendedcucumber, :cli => "--require puppet-repo/features --strict --format pretty" do

    # Match any .pp file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/[^.]*\.pp$}) { "puppet-repo/features" }

    # Match any .rb file (but be carefull not to include any dot-temporary files)
    watch(%r{^puppet-repo/[^.]*\.rb$}) { "puppet-repo/features" }

    # Feature files are monitored as well
    watch(%r{^puppet-repo/features/[^.]*.feature})

    # This is only invoked on changes, not at initial startup
    callback(:start_end) do
      vagrant_provision if all_tests_pass
    end
    callback(:run_on_change_end) do
      vagrant_provision if all_tests_pass
    end
  end

end

The full Guardfile is on github

Run it

From within the top directory of the project type

$ guard

Now open a second terminal and change some of the files and watch the magic happen.

Final remarks

The setup described is an idea I only recently started exploring. I'll probably enhance this in the future or may experience other problems.

For the demo project, I only call vagrant provision, but this can of course be extended easily. Some ideas:

Inspired by Oliver Hookins - How we use Vagrant as a throwaway testing environment:
use sahara to create a snapshot just before the provisioning
have it start from a clean machine when all tests pass
Turn this into a guard-vagrant gem, to monitor files and tests

Day 12 - Reverse-engineer servers with Blueprint

2011-12-12T00:00:00.000-08:00

This was written by Richard Crowley (rcrowley.org)

The benefits of using a configuration management tool can be felt by even the smallest of teams, yet most still rely on wiki pages and handwritten shell scripts to keep things straight because the barriers to entry in traditional configuration management tools are too high, both mentally and technically.

You could be comfortable with how to configure a server by hand, but you don't know how to use the right tools to automate that configuration. We set out to build a configuration management tool that addresses common problems in an approachable way.

Blueprint is a simple configuration management tool that reverse-engineers servers. It figures out what you've done manually, stores it locally in a Git repository, generates code that's able to recreate your efforts, and helps you deploy those changes to production.

Once you get it installed, creating a blueprint takes but a single command:

blueprint create example

where 'example' is the name of your blueprint. Blueprint will list packages managed by APT, Yum, RubyGems, Python's easy_install and pip, PHP's PEAR and PECL, and Node.js' NPM. It will also determine which configuration files in /etc have been added or modified from their packaged versions and also collect files in /usr/local that are part of any software packages installed from source. Finally, it will build a list of conditions under which System V init or Upstart services should be restarted, including package upgrades and configuration changes.

Blueprints are stored in a local Git repository as JSON and tarballs and from there you can turn them into shell scripts, Puppet modules, or Chef cookbooks.

Generate a shell script rendering of this blueprint as example/bootstrap.sh:

blueprint show -S example

Generate a puppet module named 'example'; the main manifest will be example/manifests/init.pp:

blueprint show -P example

Generate a bit of chef with the recipe as example/recipes/default.rb:

blueprint show -C example

The generated shell scripts can render mustache.sh templates to tailor configuration to the hardware: {{CORES}} for the number of CPU cores in the system, {{MEM}} for total memory, {{FQDN}} for the fully-qualified domain name, and more. See blueprint-template(7) for the complete list and blueprint-template(1) to learn how to add your own template data.

There will undoubtedly come a time when you want to use a bit more finesse in configuration management than Blueprint's automatic reverse-engineering can provide, even with hints in /etc/blueprintignore. For that time, there are rules files that enumerate only the resources you want included in the blueprint. Take, for example, example.blueprint-rules describing a tiny Sinatra application:

:source:/usr/local

/etc/init/example.conf
/etc/nginx/sites-*/example
/etc/unicorn.conf.rb

:package:apt/libmysqlclient-dev
:package:apt/mysql-client-5.1
:package:apt/nginx-common
:package:apt/nginx-light
:package:apt/ruby-dev
:package:apt/rubygems
:package:rubygems/*

:service:sysvinit/nginx
:service:upstart/example

Regardless of what else is installed on the system, example will only contain these resources:

blueprint rules example.blueprint-rules

Blueprint still extracts file content and metadata, package versions, and service dependencies from the system but instead of including everything and the kitchen sink, stays focused only on the resources you declare.

There's much more detail in the newly-revamped documentation and copious man pages. Give Blueprint a spin and send questions to .

Day 11 - Simple Disk-Based Server Backups with rsnapshot

2011-12-11T00:00:00.000-08:00

This was written by Phil Hollenback (www.hollenback.net)

I've helped a friend administer the network at his engineering firm for about 10 years. For most of that time, they've used a standard Amanda configuration with a separate backup server and a DAT drive. That setup worked well enough as long as someone remembered to swap the tape every day and occasionally run the cleaning tape.

However, this backup system suffered from a few serious limitations:

Annoyingly difficult to retrieve individual backup files for users
20GB tape size required small filesystems and other 'creative' tricks

There was also a desire to simplify the configuration and eliminate the standalone backup server. With those goals in mind I set out to find a suitable replacement system.

Physical Media

The big change compared to ten years ago is the easy availability of very large and inexpensive external hard drives. This site is a small company so price is a major factor, and larger tape systems were just too expensive compared to large drives. At the time this system was being considered (2008), you could buy 500GB SATA drives for around US$120. Compare this to many thousands of dollars for a larger capacity tape system and media.

Based on this, we purchased two 500GB SATA drives and two external SATA/USB enclosures for the backup system. The plan was to rotate the drives every few days and assign someone to take the spare drive home in case the office burned down.

I should also note that this site was and is connected to the internet with consumer DSL, so the very slow upload speeds prevented use of network backups.

Backup Software

We knew we wanted a backup mechanism that would copy important user data to the external drive at regular intervals. Bare metal restore capability was judged unimportant because the main server that was being backed up was a stock CentOS box. Backing up user data and system config files was enough of a safeguard, especially since the filesystems were already running on RAID1.

Of course the other criteria was that the software needed to be free as in beer and free as in Richard Stallman. Google quickly told us we should look at rsnapshot.

Hot Swap

This small site did not have a lot of technical expertise other than my friend who served as a part-time sysadmin. We needed to find a solution that was as simple and bulletproof as possible, particularly for swapping out the external drives. This meant hot swap and a system that could handle being unplugged any time backups weren't actually running. I decided to use the autofs automounter to keep the drive unmounted when it was inactive.

We had hoped to use eSATA as the hot swap mechanism to maximize throughput, but that proved unworkable. The CentOS 4 server would autodetect the drive and load the appropriate storage drivers. However, it refused to disconnect the drive and would spew console errors if you then manually unplugged it. While I suspected this might be fixed in a newer Linux distro, we didn't want to go through the effort of upgrading the server to something like CentOS 5. Instead, we tried a USB2.0 connection since the external drive enclosures we had purchased supported that as well as eSATA. USB hotplug worked just fine - as long as the filesystem wasn't mounted, you could plug and unplug all day long. Since this was just a backup system on a small office server, the speed of USB2.0 has proven to be just fine.

Configuring File Systems and the Automounter

For simplicity and robustness, I configured the external backup drives with one large, journaled, ext3 filesystem. Again, I didn't try to optimize for performance, the goal was just to create a large backup filesystem that wasn't terribly slow. Note, I did label the filesystem on each disk as backup1 and backup2 respectively. This is important because it provides an easy programmatic way to determine which drive is connected. The format command was thus:

mkfs -t ext3 -j -L backup1

By default CentOS detects the usb device and mounts it as /media/<label>. This is not desirable for backups as I wanted to filesystem paths to always be consistent. Remember, I also wanted to use the automounter so that the disk could be physically removed any time it was not being actively used.

Automounter setup was as follows:

Enable autofs with /sbin/chkconfig autofs on
In /etc/auto.master configure mounts under /misc with a very short timeout of 5 seconds:
/misc /etc/auto.misc --timeout=5
In /etc/auto.misc specify that /dev/sdd1 will be mounted under /misc/backup:
backup -fstype=ext3 :/dev/sdd1

After that I was able to verify the configuration by running /etc/autofs start, plugging in one of the backup drives, and verifying I could access it as /misc/backup.

Time for rsnapshot

rsnapshot uses rsync and hard links to make snapshots of a filesystem. Since it uses hard links, only files actually changed in each snapshot take up additional room. While rsnapshot does not come with CentOS 4, I was able to install one from DAG with no difficulty. It was then necessary to make a few edits to /etc/rsnapshot as follows:

# where to make backups
snapshot_root   /misc/backup
# if the usb drive isn't mounted, don't create the root dir!
no_create_root  1
# number of each level of snapshot to keep
# tune so you don't fill up the backup drive too fast
retain  hourly  3
retain  daily   7
retain  weekly  4
retain  monthly 3
# extra verbosity, for analysis scripts
verbose 4
loglevel 4
logfile /var/log/rsnapshot
# collect stats for later analysis
rsync_long_args --delete --numeric-ids --relative --delete-excluded --stats
# don't cross filesystem boundaries
one_fs          1
# filesystem trees to back up
backup  /home/          localhost/
backup  /etc/           localhost/
backup  /usr/           localhost/
backup  /var/           localhost/

That was about the extent of the customizations I needed. After this, I was able to enable hourly rsnapshot backups with an entry in /etc/crontabs:

0 */8 * * * root /usr/bin/rsnapshot hourly 2>&1 | tee \
/tmp/rsnapreport-hourly.log | /usr/bin/rsnapreport.pl \
>>/root/cron/rsnapshot-hourly.log && /bin/echo >> \
/root/cron/rsnapshot-hourly.log

I realize that is kind of an ugly cronjob but I'll come back to why I set up that complicated job. Also note that we are only doing backups of the main server currently, but we did do backups of an additional machine via rsync over ssh which rsnapshot supports fully.

Finally I needed a few other crontab entries to perform the daily, weekly, and monthly backups. Note that all other backups past the hourly are really just renaming the latest hourly snapshot.

50 23 * * * root /usr/bin/rsnapshot daily > /tmp/rsnapreport-daily.log
40 23 * * 6 root /usr/bin/rsnapshot weekly > /tmp/rsnapshot-weekly.log
30 23 1 * * root /usr/bin/rsnapshot monthly > /tmp/rsnapshot-monthly.log

Then finally I wanted to run a daily disk usage report script that I wrote:

1 6 * * * root /usr/local/sbin/rsnapshot-du

That script is the reason for the ugly hourly crontab entry. On each hourly run I do two things:

Run the hourly log through the supplied /usr/bin/rsnapreport.pl script to produce a report for the individual backup run showing how long the run took and how much was backed up from each filesystem.
Collect all hourly reports each day so they can be combined with additional housekeeping information into one simple mailed report. This report includes information such as how much space remains on the attached backup drive, and the filesystem label for the drive.

This nightly email report is critical because it allows us to monitor the amount of free space on the backup drive and verify that the drives get switched out on a regular basis.

Here's what the daily report looks like.

In Practice

This backup system has now been running on the main server for about two years, with great success. In particular the ability for non-admins to recover files from the snapshots has been very popular.

Initially we started with two 500GB external drives in the rotation. After one year, one of these drives failed, and space was getting a little tight, so we upgraded to 2TB drives. The modular nature of the automounted disk backups makes this change (and switching backup drives in general) very simple. All you need to tell an untrained operator is to wait until the drive light isn't flashing, unplug it, and plug in the new drive. When we added the new larger drives to the rotation we just had to format the filesystems ad label them backup3 and backup4. Then we just rotated the new drives in and rotated the one remaining non-failed old drive out.

While this sort of hotswapping of backup drives may sound a little risky, in a small office environment the backup drives end up sitting idle most of the time. We have had no problems with rotating drives every few days, and the external cables and drive enclosures have held up just fine.

Conclusion

The key things I want to emphasize about this setup are that it is cheap and reliable. While this design would probably not scale well in a more demanding environment, for a small office it works just fine. The daily management load is substantially reduced from using a tape drive, and reliability has been excellent. I found it particularly satisfying that we were able to assemble this backup tool using existing software, off the shelf external drives, and a bit of custom scripting. This did not take a huge amount of time, and the hardware investment was only a few hundred dollars. This modularity illustrates the greatest strength of open source software.

Day 10 - Analyzing Logs with Pig and Elastic MapReduce

2011-12-10T12:01:00.000-08:00

This was written by Grig Gheorghiu

Parsing and analyzing individual log files can be done fairly easily with standard Unix tools such as find, grep, sed, awk, wc etc. The difficult part is doing this at scale when you are dealing with large quantities of logs. Hadoop is a suite of technology that has proven itself to be capable of scaling well as you are throwing more and more data at it. What makes it even better is the large ecosystem that has grown around it with tools such as Hive and Pig which offer high-level programming constructs that make your life easier as a "data analyst." I chose Pig for this article because I find it a bit more friendly for programmers, whereas Hive is more appropriate for SQL die-hards.

Pig is an Apache project which, as the official documentation says, “is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets.”

Pig runs on top of Hadoop. You could build out your own Hadoop cluster, but for quick experimentation you would be advised to choose a platform like Amazon’s Elastic MapReduce (EMR) which abstracts away the operational details of a Hadoop cluster and lets you focus on writing your data analysis scripts. In the rest of the article, I’ll show you how to launch an EMR cluster running Pig, how to use Pig to analyze sendmail log files, and how to terminate the cluster once your data analysis is finished.

A major help in my experiments with Pig was an AWS article called Parsing Logs with Apache Pig and Elastic MapReduce. Although the article deals with Apache logs and not mail logs, the techniques it presents are the same.

Launching an EMR cluster with Pig interactively

In the examples that follow, I’ll use the elastic-mapreduce Ruby command-line tool. Here are the steps you need to install and configure this tool:

Installing the EMR Ruby CLI

Download the zip file from here, then unzip it somewhere on an EC2 instance where you can store your AWS credentials (a management-type instance usually). I installed in /opt/emr on one of our EC2 instances. At this point, it's also a good idea to become familiar with the EMR Developer Guide, which has examples of various elastic-mapreduce use cases. I also found a good README on GitHub. , Next, create a credentials.json file containing some information about your AWS credentials and the keypair that will be used when launching the EMR cluster. The format of this JSON file is:

{
  "access-id": "YOUR_AWS_ACCESS_ID",
  "private-key": "YOUR_AWS_SECRET_KEY",
  "key-pair": "YOUR_EC2_KEYPAIR_NAME",
  "key-pair-file": "PATH_TO_PRIVATE_SSH_KEY_CORRESPONDING_TO_KEYPAIR",
  "region": "us-east-1",
  "log-uri": "s3://somebucket.yourcompany.com/logs"
}

Running Pig on an EMR Cluster

Here’s a script that will launch an EMR cluster with 1 master instance and 2 slave instances, all m1.large, and will install Hadoop 0.20 and Pig. The cluster can be accessed interactively via ssh because --pig-interactive is specified as an option to elastic-mapreduce:

#!/bin/bash
# File name: run_emr_pig.sh

TIMESTAMP=`date "+%Y%m%d%H%M"`
EMR_DIR=/opt/emr
LOG_FILE=$EMR_DIR/run_emr_pig.log.$TIMESTAMP

START=`date "+%Y-%m-%d %H:%M"`

echo $START > $LOG_FILE

SSH_KEY=/root/.ssh/emrdw.pem
NAME=piggish
CREDENTIALS=/opt/emr/credentials.json
NUM_INSTANCES=3
MASTER_INSTANCE_TYPE=m1.large
SLAVE_INSTANCE_TYPE=m1.large

CMD="$EMR_DIR/elastic-mapreduce -c $CREDENTIALS --create --name "$NAME" --alive --num-instances $NUM_INSTANCES --master-instance-type $MASTER_INSTANCE_TYPE --slave-instance-type $SLAVE_INSTANCE_TYPE --hadoop-version 0.20 --pig-interactive"

echo Launching EMR cluster with command $CMD >> $LOG_FILE

JOBID=`$CMD| egrep 'j-.*' -o`
echo JOBID: $JOBID >> $LOG_FILE
while true;  do
   $EMR_DIR/elastic-mapreduce --list | grep $JOBID | grep WAITING
   if [ $? = 0 ]; then
       break
   fi
   sleep 10
done

$EMR_DIR/elastic-mapreduce --list | grep $JOBID >> $LOG_FILE

MASTER=`$EMR_DIR/elastic-mapreduce --jobflow $JOBID --describe | grep MasterPublicDnsName | egrep 'ec2.*com' -o`
echo Master node: $MASTER
echo Master node: $MASTER >> $LOG_FILE

Using Pig Interactively

To ssh into the EMR master node, you can do:

$ ssh -i $SSH_KEY hadoop@$MASTER

One note: the version of Pig available in EMR currently is 0.6.

Once on the master node, you have the option to run pig in either local mode (which will not go against Hadoop/HDFS) or in Hadoop mode. The local mode is recommended for trying out your scripts against small data sets. In my case, I wanted to analyze gzipped log files stored in S3, and there’s a bug in Pig which prevents copying files from S3 to the local file system. The only way I could test my scripts was to run Pig in regular Hadoop mode, and point it to files in S3.

$ pig

grunt> RAW_LOGS = LOAD 's3://pig.mycompany.com/mail/test.maillog.gz' as (line:chararray);
grunt> DUMP RAW_LOGS;

The above lines load a gzipped mail log file stored in S3 into what is called a Pig relation (a collection of tuples) which I named RAW_LOGS. Then the DUMP statement prints the relation to standard out. Each element of the relation is a tuple with 1 element called line:

(Nov  8 18:39:58 mail1 sendmail[18842]: pA8Ndwvm018842: from=<sender@example.com>, size=9549, class=0, nrcpts=1, msgid=<201111082339.pA8Ndwvm018842@mail1.example.com>, proto=ESMTP, daemon=MTA, relay=relay1 [10.10.10.152])
(Nov  8 18:39:58 mail2 sendmail[18784]: pA8Ndtvm018781: to=<recipient@somedomain.net>, delay=00:00:03, xdelay=00:00:03, mailer=esmtp, pri=107011, relay=relay.somedomain.net. [A.B.C.D], dsn=2.0.0, stat=Sent (pA8NduK8032663 Message accepted for delivery))

Note that the file test.maillog.gz contains just a small subset of a regular mail log file. This is the recommended way of experimenting with your data before analysing it at scale: start small, understand the structure of your data, play with it.

There is not much analysis we can do at this point unless we refine the parsing of the log file. Pig supports regular expressions, so we’ll use that. Before I go into more detailed examples, let me say that I advise you to become familiar with the Pig tutorial and the Pig Latin reference manuals, (manual 1 and manual 2).

We first register the "piggybank", which is a collection of useful Pig functions exposed via a jar file which gets installed automatically with Pig in EMR. We import the EXTRACT function which allows us to use regular expressions for parsing the Pig relation. We then go through each tuple of the relation (with the FOREACH statement) and split it into fields by means of a threatening-looking regular expression which will match all lines that contain a destination email address (a "to" field).

Note that you need to escape the backslash everywhere. We save the result of this processing into another relation called LOGS_BASE.

grunt> REGISTER file:/home/hadoop/lib/pig/piggybank.jar;
grunt> DEFINE EXTRACT org.apache.pig.piggybank.evaluation.string.EXTRACT();
grunt> LOGS_BASE = FOREACH RAW_LOGS GENERATE
>> FLATTEN(
>>  EXTRACT(line, '(\\S+)\\s+(\\d+)\\s+(\\S+)\\s+(\\S+)\\s+sendmail\\[(\\d+)\\]:\\s+(\\w+):\\s+to=<([^@]+)\\@([^>]+)>,\\s+delay=([^,]+),\\s+xdelay=([^,]+),.*relay=(\\S+)\\s+\\[\\S+\\],\\s+dsn=\\S+,\\s+stat=(.*)')
>> )
>> AS (
>>  month: chararray,
>>  day: chararray,
>>  time: chararray,
>>  mailserver: chararray,
>>  pid: chararray,
>>  sendmailid: chararray,
>>  dest_user: chararray,
>>  dest_domain: chararray,
>>  delay: chararray,
>>  xdelay: chararray,
>>  relay: chararray,
>>  stat: chararray
>> );

If we dump the LOGS_BASE relation to stdout (via the DUMP statement), we see something like this:

(Nov,8,18:39:58,mail2,18817,pA8Ndvvm018807,user1,yahoo.com,00:00:00,00:00:00,mta6.am0.yahoodns.net.,Sent (ok dirdel))
()
()
()
()
(Nov,8,18:39:58,mail1,13466,pA8NdvaN013451,user2,gmail.com,00:00:00,00:00:00,gmail-smtp-in.l.google.com.,Sent (OK 1320795598 v8si2644603yhm.107))
(Nov,8,18:39:58,mail1,13389,pA8NdtaN013380,user3,me.com,00:00:02,00:00:02,mx.me.com.akadns.net.,Sent (Ok, envelope id   0LUD00A9Q8EK4BQ0@smtpin135.mac.com))

Each tuple of the relation is either empty (if the regular expresssion doesn’t match the line) or contains the elements we specified in the EXTRACT statement (month, day, time, mailserver etc).

Now that we have the lines split into individual fields, we can start thinking about analysing the data. One thing that you’ll find when doing data analysis is that often one of the hardest things to do is to ask meaningful questions. There may be wonderful stories waiting to be told by the data, but you need to be able to extract those stories.

In the example that follows, I want to see the most common scenarios where mail did not get sent correctly. I will be looking for a status that does not start with "Sent," and I will want to see the top mail domains that were involved in non-successful mail delivery.

First we select only the mail domain and the status from the LOGS_BASE relation:

grunt> DOMAIN_STAT = FOREACH LOGS_BASE GENERATE dest_domain, stat;

We filter out the empty tuples:

grunt> NOT_NULL = FILTER DOMAIN_STAT BY NOT $0 IS NULL;

We also filter out tuples with the ‘stat’ field not starting with "Sent":

grunt> NOT_SENT = FILTER NOT_NULL BY NOT stat MATCHES 'Sent.*';

We now group the remaining tuples by domain and status, which are indicated by their position in the NOT_SENT tuple (0 and 1 respectively):

grunt> GROUPED = GROUP NOT_SENT by ($0, $1);

The GROUPED relation contains tuples of this form:

((alumni.myuniversity.edu,Deferred: 451 Requested mail action not taken: mailbox unavailable),{(alumni.myuniversity.edu,Deferred: 451 Requested mail action not taken: mailbox unavailable),(alumni.myuniversity.edu,Deferred: 451 Requested mail action not taken: mailbox unavailable)})

The first element of a tuple in the GROUPED relation is the tuple we grouped by containing the original fields $0 and $1. The second element is what is called a "bag of tuples" which is a set of tuples denoted by {} bracketing. This bag contains as many instances of the group as were found in the GROUPED relation. This allows us to count those instances, sort by count in decreasing order and mail domain in increasing order (as specified by ‘ORDER COUNT BY num DESC, $0’), then limit to the top 50 results:

grunt> COUNT = FOREACH GROUPED GENERATE FLATTEN(group), COUNT($1) as num;
grunt> SORTED = LIMIT(ORDER COUNT BY num DESC, $0) 50;

The result looks something like this:

(austin.rr.com,User unknown,2L)
(comcast.net,User unknown,2L)
(aol.com,User unknown,1L)
(earthinlink.net,Deferred: Connection timed out with earthinlink.net.,1L)

Making a Pig Script

After experimenting with the grunt command line tool, it’s time to put together a script. You can find the statements you ran from the command line in the ~/.pig_history file on the master node. You can simply copy and paste them into a script. I called mine mail_domain_stat.pig. I also parameterized the input and output of the script by including the variables $INPUT and $OUTPUT which will be passed to the script when it will be called. Here’s the script:

REGISTER file:/home/hadoop/lib/pig/piggybank.jar;
DEFINE EXTRACT org.apache.pig.piggybank.evaluation.string.EXTRACT();
RAW_LOGS = LOAD '$INPUT' as (line:chararray);
LOGS_BASE = FOREACH RAW_LOGS GENERATE
FLATTEN(
    EXTRACT(line, '(\\S+)\\s+(\\d+)\\s+(\\S+)\\s+(\\S+)\\s+sendmail\\[(\\d+)\\]:\\s+(\\w+):\\s+to=<([^@]+)\\@([^>]+)>,\\s+delay=([^,]+),\\s+xdelay=([^,]+),.*relay=(\\S+)\\s+\\[\\S+\\],\\s+dsn=\\S+,\\s+stat=(.*)')
)
AS (
    month: chararray,
    day: chararray,
    time: chararray,
    mailserver: chararray,
    pid: chararray,
    sendmailid: chararray,
    dest_user: chararray,
    dest_domain: chararray,
    delay: chararray,
    xdelay: chararray,
    relay: chararray,
    stat: chararray
);
DOMAIN_STAT = FOREACH LOGS_BASE GENERATE dest_domain, stat;
NOT_NULL = FILTER DOMAIN_STAT BY NOT $0 IS NULL;
NOT_SENT = FILTER NOT_NULL BY NOT stat MATCHES 'Sent.*';
GROUPED = GROUP NOT_SENT by ($0, $1);
COUNT = FOREACH GROUPED GENERATE FLATTEN(group), COUNT($1) as num;
SORTED = LIMIT(ORDER COUNT BY num DESC, $0) 50;
STORE SORTED INTO '$OUTPUT';

Now I can call the script from the command line like this:

$ TIMESTAMP=`date "+%Y%m%d%H%M"`
$ pig -p INPUT="s3://pig.mycompany.com/mail/test.maillog.gz" -p OUTPUT="s3://pig.mycompany.com/mail/output/run_$TIMESTAMP" mail_domain_stat.pig

The output will consist of a series of files called part-NN (00, 01, etc.) stored in the S3 bucket specified as the OUTPUT parameter. To get back the final result, download the partial files and concatenate them.

Note that INPUT can be an expression such as INPUT="s3://pig.mycompany.com/mail/2011*.maillog.gz" which would automatically read in all files in S3 matching the expression 2011*.maillog.gz. As a matter of curiosity, when I ran the above Pig script against all our mail logs for 2011, the most common errors where of the form "someisp.com User unknown" followed by "Some mail server Connection timed out".

Asking more questions of the data

Here are some more questions you can ask from a mail log, together with short Pig examples:

Who are the top mail recipients?

RAW_LOGS = LOAD '$INPUT' as (line:chararray);
LOGS_BASE = FOREACH RAW_LOGS GENERATE
FLATTEN(
    EXTRACT(line, '(\\S+)\\s+(\\d+)\\s+(\\S+)\\s+(\\S+)\\s+sendmail\\[(\\d+)\\]:\\s+(\\w+):\\s+to=<([^>]+)>,\\s+delay=([^,]+),\\s+xdelay=([^,]+),.*relay=(\\S+)\\s+\\[\\S+\\],\\s+dsn=\\S+,\\s+stat=(.*)')
)
AS (
    month: chararray,
    day: chararray,
    time: chararray,
    mailserver: chararray,
    pid: chararray,
    sendmailid: chararray,
    dest: chararray,
    delay: chararray,
    xdelay: chararray,
    relay: chararray,
    stat: chararray
);
DEST = FOREACH LOGS_BASE GENERATE dest;
DEST_FILTERED = FILTER DEST BY NOT $0 IS NULL;
DEST_COUNT = FOREACH (GROUP DEST_FILTERED BY $0) GENERATE $0, COUNT($1) as num;
DEST_COUNT_SORTED = LIMIT(ORDER DEST_COUNT BY num DESC) 50;
STORE DEST_COUNT_SORTED INTO ‘$OUTPUT’;

At what hours of the day do we send the most mail?

RAW_LOGS = LOAD '$INPUT' as (line:chararray);
LOGS_BASE = FOREACH RAW_LOGS GENERATE
FLATTEN(
    EXTRACT(line, '(\\S+)\\s+(\\d+)\\s+(\\d+):(\\d+):(\\d+)\\s+\\S+\\s+sendmail\\[\\d+\\]:\\s+\\w+:\\s+\\S+=<[^>]+>')
)
AS (
    month: chararray,
    day: chararray,
    hour: chararray,
    minute: chararray,
    second: chararray
);
HOUR = FOREACH LOGS_BASE GENERATE hour;
HOUR_FILTERED = FILTER HOUR BY NOT $0 IS NULL;
HOUR_COUNT = FOREACH (GROUP HOUR_FILTERED BY $0) GENERATE $0, COUNT($1) as num;
HOUR_COUNT_SORTED = ORDER HOUR_COUNT BY num DESC;
STORE HOUR_COUNT_SORTED INTO '$OUTPUT';

Which mail servers are sending the most email (they should be almost equal if you are using a round-robin mechanism):

RAW_LOGS = LOAD '$INPUT' as (line:chararray);
LOGS_BASE = FOREACH RAW_LOGS GENERATE
FLATTEN(
    EXTRACT(line, '(\\S+)\\s+(\\d+)\\s+(\\S+)\\s+(\\S+)\\s+sendmail\\[(\\d+)\\]:\\s+(\\w+):\\s+from=<([^>]+)>,\\s+size=(\\d+),\\s+class=(\\d+),\\s+nrcpts=(\\d+),\\s+msgid=<([^>]+)>.*relay=(\\S+)')
)
AS (
    month: chararray,
    day: chararray,
    time: chararray,
    mailserver: chararray,
    pid: chararray,
    sendmailid: chararray,
    src: chararray,
    size: chararray,
    classnumber: chararray,
    nrcpts: chararray,
    msgid: chararray,
    relay: chararray
);
RELAY = FOREACH LOGS_BASE GENERATE relay;
RELAY_FILTERED = FILTER RELAY BY NOT $0 IS NULL;
RELAY_COUNT = FOREACH (GROUP RELAY_FILTERED BY $0) GENERATE $0, COUNT($1) as num;
RELAY_COUNT_SORTED = LIMIT(ORDER RELAY_COUNT BY num DESC) 50;
STORE RELAY_COUNT_SORTED INTO '$OUTPUT';

One other interesting question I am working on is finding out how fast we deliver a given piece of mail. This will involve defining two relations, one for lines containing mail sources and one for lines containing mail destinations, then joining them together based on the sendmail ID.

The Elasticity in EMR

One of the nice things about using EMR is that you can launch an EMR cluster at night, process your data, then terminate the cluster, thus paying only for the period of time the cluster was in use. Here’s a script that does that.

It launches a cluster, it copies your Pig scripts to the master node, then it runs a Pig script. When that finishes, it terminates the cluster. The results of the Pig script will have been stored in S3 at that point.

#!/bin/bash

TIMESTAMP=`date "+%Y%m%d%H%M"`
EMR_DIR=/opt/emr
LOG_FILE=$EMR_DIR/run_emr_pig.log.$TIMESTAMP

START=`date "+%Y-%m-%d %H:%M"`

echo $START > $LOG_FILE

SSH_KEY=/root/.ssh/emrdw.pem
NAME=piggish
CREDENTIALS=/opt/emr/credentials.json
NUM_INSTANCES=5
MASTER_INSTANCE_TYPE=m1.large
SLAVE_INSTANCE_TYPE=m1.xlarge

CMD="$EMR_DIR/elastic-mapreduce -c $CREDENTIALS --create --name "$NAME" --alive --num-instances $NUM_INSTANCES --master-instance-type $MASTER_INSTANCE_TYPE --slave-instance-type $SLAVE_INSTANCE_TYPE --hadoop-version 0.20 --pig-interactive"

echo Launching EMR cluster with command $CMD >> $LOG_FILE

JOBID=`$CMD| egrep 'j-.*' -o`
echo JOBID: $JOBID >> $LOG_FILE
while true;  do
   $EMR_DIR/elastic-mapreduce --list | grep $JOBID | grep WAITING
   if [ $? = 0 ]; then
       break
   fi
   sleep 10
done

$EMR_DIR/elastic-mapreduce --list | grep $JOBID >> $LOG_FILE

MASTER=`$EMR_DIR/elastic-mapreduce --jobflow $JOBID --describe | grep MasterPublicDnsName | egrep 'ec2.*com' -o`
echo Master node: $MASTER
echo Master node: $MASTER >> $LOG_FILE

scp -i $SSH_KEY -r $EMR_DIR/pigscripts hadoop@$MASTER:
ssh -i $SSH_KEY hadoop@$MASTER "cd pigscripts; ./mail_domain_stat.sh" >& /tmp/emr_pig.log

cat /tmp/emr_pig.log  >> $LOG_FILE
$EMR_DIR/elastic-mapreduce --jobflow $JOBID --terminate
STOP=`date "+%Y-%m-%d %H:%M"`
echo $STOP >> $LOG_FILE

Using this script (which launches 1 m1.large master node and 4 m1.xlarge slave nodes) I was able to process 100 GB worth of compressed mail logs in a little under 4 hours.

Conclusion

The combination Apache Pig + Elastic MapReduce is a pretty powerful one when it comes to doing large-scale data analysis. The learning curve for doing simple, but useful, data analysis with Pig Latin is not very steep. Elastic MapReduce has the advantage of abstracting away the operational details of a Hadoop cluster, and it also makes sense financially if you only use it a few hours per day.

Resources

Apache Pig Wiki
Elastic MapReduce Developer Guide
Programming Pig - O’Reilly book by Alan Gates

Day 9 - Data in the Shell

2011-12-09T00:55:00.001-08:00

This was written by Jordan Sissel (semicomplete.com).

The shell is where I live most days, and the shell has pipes. Pipes simply transmit plain text, and many pieces of data I deal with are structured data where staples like grep and sed are not the best.

Fortunately, there are lots of tools available to help you deal with these structured formats.

Delimited Data

I'll start with data you're probably already familiar with, simple text delimited by some characters like spaces or commas. The general tools used here are awk and cut. I often want to get some simple stats using awk, so I keep some handy shell functions to help me do sums and counts by field.

function _awk_col() {
  echo "$1" | egrep -v '^[0-9]+$' || echo "\$$1"
}

function sum() {
  [ "${1#-F}" != "$1" ] && SP=${1} && shift
  [ "$#" -eq 0 ] && set -- 0
  key="$(_awk_col "$1")"
  awk $SP "{ x+=$key } END { printf(\"%d\n\", x) }"
}

function sumby() {
  [ "${1#-F}" != "$1" ] && SP=${1} && shift
  [ "$#" -lt 0 ] && set -- 0 1
  key="$(_awk_col "$1")"
  val="$(_awk_col "$2")"
  awk $SP "{ a[$key] += $val } END { for (i in a) { printf(\"%d %s\\n\", a[i], i) } }"
}

function countby() {
  [ "${1#-F}" != "$1" ] && SP=${1} && shift
  [ "$#" -eq 0 ] && set -- 0
  key="$(_awk_col "$1")"
  awk $SP "{ a[$key]++ } END { for (i in a) { printf(\"%d %s\\n\", a[i], i) } }"
}

It's not exactly pretty, but it lets me do things like this:

# Simply sum the first field
% seq 10 | sum 1
55

# Sum one field grouping by another
% printf "hello:1\nworld:1\nworld:3\n" |sumby -F: 1 2
4 world
1 hello

# Count instances by field
% printf "hello:1\nworld:1\nworld:3\n" | countby -F: 1
2 world
1 hello

fex

Over the years, I've found awk to be useful for some cases, but the majority of my uses of cut (with cut -d) and awk were essentially setting the field separator and picking out a few fields. I wanted something with better syntax to solve some more complex field selection problems, so I wrote a tool called fex.

The best example I can give of fex's power is by chopping up an apache log entry:

% echo '208.36.144.8 - - [22/Aug/2007:23:39:05 -0400] "GET /hello/world HTTP/1.0" 200 3595' \
  | fex 1 '"2 2'

208.36.144.8 /hello/world

The above asks for two values, the client ip and the request path. The first field (default delimiter is space) and the 2nd field (delimited by doublequote), then inside that the 2nd field by space. I believe this tight syntax is still readable while being expressive enough to get most field splitting and selections done. Doing the same thing in awk (or worse, cut) would require much more effort to capture.

Structured Data

Awk, cut, and fex are useful, but there are better tools to use when dealing with structured data formats.

JSON

Lots of web APIs speak JSON these days, so there's an increasing likelihood that you will have to mangle it on the command line at some point

For this format, I use jgrep to search json data.

Let's use it to look for github public gists with comments:

% curl -s https://api.github.com/gists | jgrep 'comments>0' -s url     
https://api.github.com/gists/1450770
https://api.github.com/gists/1450726

Boom, only seeing gist urls with comments.

XML

Lots of programs use XML for configuration and data transport, so it is helpful to have a tool you can use to quickly poke at this data format.

Many of the tools in this area use a language called XPath, which I originally learned from here. The XPath language is in most cases fairly simple to learn. The other language you will likely see in these XML processing tools is XSLT, but that is a bit outside the scope of this article.

Tool: xmlstarlet

Data: Hadoop [core-default.xml](https://github.com/jordansissel/sysadvent/blob/master/2011/09/code/core-default.xml

# Get all the properties set in this file:
% xmlstarlet sel -t -v '/configuration/property/name' core-default.xml

# The above can also be written:
% xmlstarlet sel -t -v '//name' core-default.xml
hadoop.common.configuration.version
hadoop.tmp.dir
io.native.lib.available
...

# Read the 'hadoop.tmp.dir' property:
% xmlstarlet sel -t -v '//property[name/text() = "hadoop.tmp.dir"]/value' *.xml  
/tmp/hadoop-${user.name}

This kind of thing is very useful when you need to feed other tools from data in xml config files (like monitoring settings, etc)

CSV

In most cases, awk, cut, or fex will be sufficient in dealing with CSV or other delimited format files. There are edge cases in CSV (like quoting) that make using normal delimiter tools insufficient.

In this case, there's nicely a whole suite of tools for mangling CSV data, csvkit:

Tool: csvkit

csvkit comes with tools to cut, grep, sort, and more to csv data sets.

For a fun example, let's look outside the sysadmin world at some UK Government data on mergers and acquisitions.

After downloading the csv data, let's get some stats on quarterly domestic acquisitions (listed as code AIHA in the csv data):

First, since there's both yearly and quarterly data, we'll need to grep for only quarterly data, second we'll want to get stats on column 2 (AIHA):

% python csvgrep -c 1 -r ".*Q[1-4]" AM_CSDB_DS.csdb.csv | python csvstat -c 2            
  2. AIHA
        <type 'int'>
        Nulls: No
        Min: 56
        Max: 464
        Sum: 27967
        Mean: 164
        Median: 140.5
        Unique values: 116
        5 most frequent values:
                120:    5
                139:    4
                148:    4
                124:    3
                88:     3

Row count: 170

# What quarter had 464 acquisitions?
% python csvgrep -c 2 -r '^464$' AM_CSDB_DS.csdb.csv  | python csvcut -c 1
1988 Q3

# What quarter had only 56?
% python csvgrep -c 2 -r '^56$' AM_CSDB_DS.csdb.csv  | python csvcut -c 1
1975 Q1

This tool is pretty slick. As a tip, if csvkit has features you like (column selection/grep, stats, etc), remember that if your data set is not already in csv, it might be easy to convert to csv for use with csvkit.

Parting Thoughts

Sysadmins are often required to deal with multiple systems that speak different languages - one might log in plain text, another expose data over a HTTP API and send JSON, etc.

You're not alone, and having the right data mangling tools in your toolbox will help you answer the right questions faster without spending energy fighting with data formats. This lets you be more effective, and have more time at the pub with friends, at home with the family, or wherever you happen to find happiness.

Day 8 - Running services with systemd

2011-12-08T01:42:00.001-08:00

This was written by Jordan Sissel (semicomplete.com).

In the past three years of sysadvent, I've covered various process monitor tools like monit, supervisord, and upstart. A little while back I put Fedora 15 on my laptop and found a new one, systemd.

Since every tool seems to invent different terminology for the same things, for the purposes of this article when I say 'process' or 'service' I mean the same thing - a systemd service.

So let's dig in a bit.

A First Look

The first thing you'll need to know is how to interact with systemd: starting and stopping things - the usual business. The main tool for this is systemctl. Run it with no arguments, and it gives you a list of all services. You can also ask for status:

% systemctl status rsyslog.service
rsyslog.service - System Logging Service
          Loaded: loaded (/lib/systemd/system/rsyslog.service)
          Active: active (running) since Tue, 06 Dec 2011 22:32:58 -0800; 1 day and 1h ago
        Main PID: 803 (rsyslogd)
          CGroup: name=systemd:/system/rsyslog.service
                  └ 803 /sbin/rsyslogd -n -c 5

From the above, you can see a few notable and strange things:

it's called "rsyslog.service" not simply "rsyslog"
the config file is in /lib/systemd/system/rsyslog.service
it's running, and for how long
the pid, and that it said 'main pid' indicating it might support child processes somehow
it uses cgroups.
and finally, the process command and arguments

A Sample Service

What's the config file look like?

[Unit]
Description=System Logging Service

[Service]
EnvironmentFile=-/etc/sysconfig/rsyslog
ExecStartPre=/bin/systemctl stop systemd-kmsg-syslogd.service
ExecStart=/sbin/rsyslogd -n $SYSLOGD_OPTIONS
Sockets=syslog.socket
StandardOutput=null

[Install]
WantedBy=multi-user.target

Most of the above should be fairly straight forward, though the 'unit' and 'install' sections have strange names. I'll explain it from top to bottom.

The 'Unit' section is documented in the systemd.unit(5) manpage. This section seems to cover things like ordering and dependencies. There's a separate section for defining the rsyslog service itself because systemd supports many more things than simply services, according to systemd.unit(5) -

A unit configuration file encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a start-up target, a file system path or a timer controlled and supervised by systemd(1).

That's a lot of stuff, but I'm mainly interested in how to run things in systemd.

Controlling a Service

Let's stop it.

% sudo systemctl stop rsyslog.service

There's no output. If you run 'stop' again, it will again have no output and will still exit with success - a nice touch making scripted management easier. Check the status:

% sudo systemctl status rsyslog.service
rsyslog.service - System Logging Service
          Loaded: loaded (/lib/systemd/system/rsyslog.service)
          Active: inactive (dead) since Thu, 08 Dec 2011 00:36:46 -0800; 3s ago
        Main PID: 803 (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/rsyslog.service

Remember that stop and start only affect the current run-time and don't impact other events that might cause rsyslog to start (like the system booting). For that, you'll want enable and disable. You can disable things fairly intuitively:

% sudo systemctl disable rsyslog.service
rm '/etc/systemd/system/multi-user.target.wants/rsyslog.service'

And enable it again:

% sudo systemctl enable rsyslog.service 
ln -s '/lib/systemd/system/rsyslog.service' '/etc/systemd/system/multi-user.target.wants/rsyslog.service'

It's an odd thing. The 'stop' and 'start' commands output nothing, but 'enable' and 'disable' output shell commands? Additionally 'disable' and 'enable' do those actions for you, so I don't know what I am supposed to do with the output. Is systemd trying to encourage me to use those commands myself instead of using systemctl? By the way, if you enable an already-enabled service, you get no output and success. Same for disabling. Confusing!

Add Your Own Service

Here's the config file I used, and I put it in /lib/systemd/system/fizzle.service:

[Unit]
Description=Hello World

[Service]
ExecStart=/bin/sh -c 'echo Hello World; sleep 5'
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

Nothing else is required to make systemctl aware of our new service, so we can start it normally like shown before:

% sudo systemctl start fizzle.service

And since I told it to take stdout and ship it over syslog, /var/log/messages has the output:

% grep 'Hello' /var/log/messages
Dec  7 17:45:26 nightfall sh[22616]: Hello World

That's a nice feature and is similar to supervisord and daemontools (with multilog).

Checking the status, I see it has died:

% systemctl status fizzle.service
fizzle.service - Hello World
          Loaded: loaded (/lib/systemd/system/fizzle.service)
          Active: inactive (dead)
          CGroup: name=systemd:/system/fizzle.service

I don't want it to be dead. As described in past sysadvents covering process monitoring, "if it dies, restart it." What can be done? The systemd.service(5) manpage says to add 'Restart=always' to the 'Service' section.

Once that is added, starting 'fizzle.service' again will get it rolling. After 5 seconds it will die and be started by systemd:

% sudo systemctl status fizzle.service | grep 'Active'
          Active: activating (auto-restart) since Thu, 08 Dec 2011 00:52:50 -0800; 5s ago
% sudo systemctl status fizzle.service
          Active: active (running) since Thu, 08 Dec 2011 00:52:55 -0800; 771ms ago

Supporting Odd Services

I originally wanted to use nagios as the example, not rsyslog as above, but when installing nagios on Fedora 15, I received the usual /etc/init.d/nagios startup script. However, when I ran it, I saw this output:

% sudo /etc/init.d/nagios start
Starting nagios (via systemctl):                           [  OK  ]

Huh? I thought that was strange, and when I dug into the script, I saw no mention of systemctl. It loads /etc/init.d/functions which, by default, seems to pass itself into systemctl. Asking systemctl what's up, it says:

% systemctl status nagios.service
nagios.service - SYSV: Starts and stops the Nagios monitor
          Loaded: loaded (/etc/rc.d/init.d/nagios)
          Active: active (running) since Thu, 08 Dec 2011 00:59:22 -0800; 31s ago
         Process: 23567 ExecStop=/etc/rc.d/init.d/nagios stop (code=exited, status=0/SUCCESS)
         Process: 23590 ExecStart=/etc/rc.d/init.d/nagios start (code=exited, status=0/SUCCESS)
        Main PID: 23601 (nagios)
          CGroup: name=systemd:/system/nagios.service
                  └ 23601 /usr/sbin/nagios -d /etc/nagios/nagios.cfg

Funky, though it shows two features of systemd. First, it supports old sysv init scripts. Second, having explicit 'ExecStart' and 'ExecStop' settings can be useful in putting together systemd and software that insists on being run with its own management tools to stop and start.

Concerns

This section is more sourced from my feelings on systemd than from facts, so take this with a grain of salt.

My first problem with systemd is the huge feature list. It looks to be trying to replace /sbin/init, SYSV init scripts and runlevels, inetd, udevd, automount, and supports cgroups, inotify, and more. That's a pretty big feature space, and it reflects in the size of the code base. At this time of writing, the lines of code in systemd around 82000 lines of code. Of those, only about 1000 are tests. Only 1.2% of the code appears to be tests? Yikes.

Further, systemd is a major consolidation of several components of the system. I don't really want software replacing major components (/sbin/init, cron, etc) with practically no tests and a fixation on problems I don't have. A bug in cron doesn't crash init, but now it just might.

Lastly, systemd relies on DBus. It's pretty rare for sysadmins to have work experience with DBus. It's another layer to debug when things break. Are there debugging tools? I hope I'm just failing to google for this, but I always come up empty when looking for a decent tracing tool for DBus messages - all the ones I run across are graphical.

The above problems are not terrible things on desktops which tend to have much looser expectations on software reliability and more flexibility on outages. However, put these on a server, and what do you have? DBus usage, major software consolidation into a single binary, 82000 lines of code and basically no tests - all this adds up to great worry and concern. How long until systemd ships with RHEL or your preferred production Linux distribution?

Conclusion

As stated, please take my opionated concerns detailed above for what they are - they are not facts. I'm not writing off systemd as a failure by any stretch.

Systemd itself has some fairly nice features for running services. Pretty much anything you'd want to configure for a service is available: cgroups, user, oom tuning, output logging, cpu and I/O tuning, etc. The command line tools and documentation are also pretty good. It's the default on all Fedora 15 and newer releases, and you can get it on many other Linux distributions, so go on and play with it!

Day 7 - Change and Proximity of Communication

2011-12-07T00:55:00.001-08:00

This was written by Aaron Nichols (www.opsbs.com)

Whether we like it or not, we sysadmins are all about "me". When someone else sends an email about something, our interest in that thing is inversely proportional to the length of the email. When stuff breaks at 3AM and we know there was an email sent out about that, we go looking and can't find it - so we call the person who sent it.

There is a better way.

This post is about communication and documentation for day-to-day stuff. This is not the documentation you dig through when you have all day to sort out a problem. This is the stuff you want, now, without getting intimate with a search engine.

Pop the hood on any car, and you'll see an example of this documentation. Right there - all the stuff you are most likely to care about when you are looking under the hood. It doesn't matter if you've never driven this car or not: the documentation is placed as close to the problem as it can get.

Here are some examples.

Communication in your Config Management System

Today, most of your configuration-related changes should be distributed through some sort of CM system. Since this is where most folks will go looking for the status of things when they break, this is a great place to document the status of things as they are changing. Here are some examples using puppet as the CM.

Identify files that are managed by the CM system

The top of every file managed by your config management system should have a line that looks something like this:

# This file is managed by puppet - any changes made to this file
# directly will be deleted
# Source: puppet/modules/wibble/templates/wibble_a.erb

This makes it perfectly clear where to find this file if you want to edit it.

If you are moving things from one CM to another, tag the old ones too so people can tell which system is the source for a particular file.

Tell users about things they should know when they are running the CM manually or debugging it

If there's important information you want people to know about when they run your CM in debug mode (presumably looking for problems) you can usually add notifications. Comments in the code are great if someone is looking in the right place, but messaging like below directs them to that place:

In puppet you can use 'notify' for this:

class java {
  notify { "WARNING: This module (java) is experimental and may break things!":; }

And you should see something like this when you run the manifests by hand:

notice: WARNING: This module (java) is experimental and may break things!

Send short email messages when things are changing, with links to more details

If you are making major changes to your CM and you need people to be aware, send a short email out with a link to details. Make sure you include enough keywords to make it searchable later on, but short enough to ensure people read it:

Everyone,
I am renaming the 2 puppet modules 'wibble' and 'wobble'. I will be working on this over the next 2 days. This work is being done on the 'wibblewobble' branch. An explanation of these changes can be found here: http://wiki/why-wibblewobble-must-change

And add a notify to those modules warning users. Add comments to those modules where users will see them. Put links to your wiki in both places.

Leverage your MOTD

It's very common to be working on a system and need to let people know you do not want them making changes to the system or disturbing the state of things. It's fine to send email about this, but don't get stabby when someone forgets about your email at 4AM and "fixes" the system.

My preference is always to put a message in the MOTD telling them about this. Let's say I have stopped httpd on a system for a few days & I've acked the alert in Nagios - I will also add this to the MOTD:

***************************************************
NOTE: This system has httpd stopped for maintenance
We are resolving some network problems on this host
contact anichols@example.com if you have questions. 
Time of this message: 2011-12-06 13:55:01Z

If a script shouldn't be run - break it

In the above example we had httpd turned off because we were resolving some issues. If you are in an environment where enabling httpd would cause a service impacting event - make that harder to do.

If I want a script to stop running, and this includes init scripts or others, I will typically disable the script with an exit along with a message to let the user know:

start() {
        # SCRIPT DISABLED - anichols 2011/11/11
        # See http://wiki/why-I-disabled-httpd-on-this-host for more info
        echo "This script is disabled, see http://wiki/why-I-disabled-httpd-on-this-host for more detail"
        exit

So when someone tries to use it they see this:

[~]$ sudo /etc/init.d/httpd start
This script is disabled, see http://wiki/why-I-disabled-httpd-on-this-host for more detail

You can do fancy stuff like conditionals so certain users can run the script even, but the point is: Communicate what's going on and do it where the user will come in contact with the change.

If you have migrated to using a new tool, put a header on the old one

As an example, you are moving some graphing from cacti to Graphite, and you want users to know about it. You want to keep the old cacti instance up because it has valuable historical data you haven't moved yet, but you want folks to stop asking you why the cacti graphs aren't accurate.

At the top of cacti, in the header, put something like this:

<font color=red>
<strong>This cacti data is no longer updated</strong><br>
This data is here for archival purposes, for current data visit
http://awesome-graphite-url
</font>

It's simple and people will see it. If they don't - you can safely ignore their pleas of ignorance.

You get the point, the point is...

When you are working on systems and making changes, think about the other people who will come in contact with your change and how you can make it painfully obvious to them what is going on. Think about your own thoughtless patterns when you are bleary eyed at 3am: what would make it easy for you to know what is going on?

Day 6 - Always Be Hacking

2011-12-06T00:01:00.001-08:00

This was written by John Vincent (blog.lusis.org).

I got my start in computers a long time ago. Our school library had, if I recall correctly, a few Apple IIe systems in a room off to the side. I remember pretty clearly the first time I wrote something in BASIC. It was so awesome watching this machine do exactly what I told it to do. I was in second or third grade at the time.

The only computer I ever had at home as a kid was a Mattel Aquarius; thankfully it had the cassette add-on.

I dabbled in some form of 'programming' over the years from typing code using edlin out of Byte magazine and writing little gui apps in Tk to do most of the system administration related stuff I write today.

Evidently, I've always been hacking, and you should be, too.

Sysadmins of the Future

Adam Fletcher did an open space at DevOps Days Boston this year about what system administrators would look like ten years from now. The truth is that sysadmins in the future will look a lot like they did twenty or thirty years ago - programmers.

It wasn't too long ago that the same guy who ran the system also wrote the code that ran on the system.

That's what sysadmins need to get back to, today, if they have any hope of being employable in the future. I'm not necessarily talking about writing the application code that runs your business, but I'm not ruling it out either. Of course, you might be thinking, "I'm not a programmer! I can't do that stuff". In the words of Stephen Nelson-Smith, "With all due respect, I call bullshit".

You Already Program

When someone tells me that they can't program, I ask them to show me the last shell script they wrote. Look back over the scripts you've taken from company to company (you know you've done it). Look at the hyperconvoluted pipelines of awk, sed, grep, sort, cut and uniq. Look at the command-line argument handling. Look at the bash functions. Look at the loop constructs. Look at the subprocess you spawn.

I want you to listen to something. It's a talk that Uncle Bob Martin gave at RailsConf 2010. It's called "Twenty Five Zeros". Once you do, come back and reread this. If you weren't convinced before, you will be after listening. Bob Martin talks mainly about traditional programming languages and history in, but also talks about interpreted languages and the ever-common ground of sequence (assignment), selection (conditionals), and iteration (loops) - you think you don't do all of those in a shell script?

The fact is that, as a system administrator, you probably know more about the inner workings of the systems you manage than anyone at your company. Not only do you understand the details of what happens at the OS level but more often than not, you're the one person with the most holistic view of how all the components fit together, probably because you or your team were the ones to glue all those components together.

But Why?

There are several reasons WHY you should learn to program, but here are the three main ones in my mind.

So That You're Self-Sufficient

Remember the first time you found a newsgroup post or web site that told you the exact solution to the problem you were having and how to solve it?

No? Me either.

That's not to say it hasn't happened, but what is MORE common is that you found a solution that ALMOST fit, and you munged it until it worked.

Then you left it there hoping that it never stopped working. Just to be sure you could always find it again you bookmarked the page, printed a copy and also ran a wget -m (because there might be pictures) and saved that somewhere. In fact, I probably still have most of Ben Rockwood's Solaris articles from Cuddletech saved off somewhere on a dead hard drive.

But really, wouldn't it be more awesome if you could be the person who WROTE the solution that someone else found? Adding Epic to Awesome, imagine if you could just write it yourself from SCRATCH?

So you'll have a job in the next five years

This one is probably the most painful for folks. The fact of the matter is that sysadmins who CAN'T program and contribute more than just keeping the lights on are going to find themselves less and less employable. The times, they are a changing. Fish or cut bait. Insert other random phrase here. You aren't likely to have a choice in the matter. This isn't a DevOps thing. It's just the reality of the situation. The 'new' crop of operations folk are already ahead of you.

You might disagree with the above, and the future might prove it wrong, but what if you find you like writing code? What if having programming skills improves your business value and thus reflects positively upon your salary and benefits?

Career Flexiblity

I wish I had someone tell me the stuff I'm telling you years ago. Interests change. People change. Economies change. You may discover that you enjoy programming more and want to make a career shift. Better to learn that now than when you're unable to switch financially or you find yourself out of work in your originally chosen field.

Learning To Program

We've already established that you can program and why you should learn more, so how do you go about it? I'm going to make some assumptions, here, based on my experience and how I learn.

Tight Feedback Loop

I was never formally educated in computer science, and I find myself missing quite a bit. One thing that made learning to program easier was using a language with very quick feedback - how quickly can you try something? How many steps does it take? I don't care what the language is that you start with, but my personal preference is either Python or Ruby.

Both of these languages can work in the style you're familiar with in shell scripts, but also ease you into both functional and object-oriented programming concepts. They're very easy to get started with on any modern distro and have amazingly rich REPLs (Read-Eval-Print Loop, an interactive shell for Ruby, Python, etc). I'm not ashamed to say that I learned what I know about both languages more via IPython and irb respectively than ever cracking open vim. REPLs also provide instant gratification when you do something right. This provides encouragement to keep going.

Scratch an Itch

Regardless of which language you pick, don't start theoretical. Find a problem you have or recently had and solve it using that language. I like to credit the phrase 'Hate Driven Development' to Jordan Sissel, but really sysadmins are more likely to practice NDD: Neccessity driven development. We don't typically sit down and write an application just to write one (not that you shouldn't do that).

Once you've solved the problem, rewrite it to be a bit more flexible. Throw in some parameterization and configurability. Try and make it a little more generic.

Don't just write one-offs.

I know for a fact that a few scripts that I thought were one-offs are still being used, today, to keep some core business functionality running.

Don't screw yourself with bad solutions. Don't screw the person who comes after you.

Practice

Rewriting and refactoring also helps you practice the language. The next time you have a problem, don't immediately reach for bash. Take a few minutes and think about possibly writing it in your new language. If you have to solve the problem immediately, go back and rewrite the solution to be less fragile.

Create public repo on Github or Bitbucket and start putting what you write there. Share it with the world.

Don't let ego get in the way.

You never know who you'll be helping down the road, and you never know who will be helping you in the future.

Next Steps

Remember, you should always be hacking. Here are a few things that I think we as sysadmins need to know beyond the basics

Perl

Every sysadmin should know some Perl. It's on every system. You might not be able to get the latest and greatest dynamic language on the system, but Perl will always be there.

At least one functional language

Don't know what 'functional' means? This might help

Functional programming concepts provide a new way to look at problems and often can be a better way to solve them. If you're an emacs user and you've ever modified your config, there's no reason you shouldn't go learn Clojure right now. Otherwise, it probably doesn't matter which language you use.

Learn to read Java

You don't have to learn to WRITE Java, but you should be able to navigate your way through the basics of a Java program. Additionally, the JVM is becoming a popular platform outside of Java with projects like Scala, JRuby, Clojure and others. Those languages will often be coupled with leveraging the existing Java ecosystem. Quite honestly, you should be able to at least read whatever language your company writes its applications in.

The dirty secret is this: once you learn one language, the others become easier.

Yes, you should learn C at some point. It's not going anywhere.

These things won't happen overnight, and I don't know all of these languages, either, but if we can manage to keep track of the differences between 5 different Linux distros and 3 flavors of Unix, we can handle learning a few different languages.

Remember that inside your head is the most amazing computer ever. Use it to the fullest and remember:

Always Be Hacking.

Futher Reading

Learn Python the Hard Way - Free online; teaches you how to program in Python
Learn Ruby the Hard Way - Same as above, but with Ruby.
Learn C the Hard Way - Same as above, but with C.

Day 5 - Tracking Requests with Request Tracker

2011-12-05T00:01:00.000-08:00

This was written by Christopher Webber (cwebber.ucr.edu)

One of the first things that Tom Limoncelli talks about in the "Practice of System and Network Administration" is getting a request tracking system in place. While you are not always tracking requests per-say, I have never seen an environment that didn't have tasks, work orders, requests, tickets or some other variant on that theme. Because my roles have always been rather customer facing, whether internal or external, the idea of a ticket system always appealed to me, so here are my reasons you should dive head first into a ticket system, why I chose Request Tracker by Best Practical, and a quick HOWTO on getting RT up and running on Ubuntu/Debian.

Why a Ticket System?

A ticket system is important because email, conversations, sticky notes, etc get lost and forgotten. That is it, plain and simple, but that just scratches the surface of the real power of a ticket system. For starters, tickets make great collaboration spaces for dealing with issues. Everyone involved gets updated when new info is available, and it provides a great blow by blow when you are dealing with the postmortem. The other thing that comes from all of this effort is metrics: Metrics about which users or customers make the most requests, what kinds of requests happen with what frequency, and how long it takes to get certain tasks accomplished. These metrics provide an easy way to justify the need for improvements in ticket throughput like more staff or more staff in specific areas.

Why RT?

Request Tracker, or RT, is fairly simple to get up and running and is extremely easy to bend to meet the needs of most environments. I personally chose RT because of the general ease of setup and the feature fit: Debian has packages for it, it is email based, it has a REST API, and there is a mobile interface and iPhone app.

Sure, there are a bunch of other way cool things you can do with it, but the above are my personal reasons. Being email-based is a huge advantage because of the limited amount of training required to get people started. Instead of sending mail to you, they are going to send to a special email address you create. The REST API is rather new, and I am looking forward to being able to automate ticket creation from scripts and events from Nagios and Munin.

Getting Dirty

The easiest way to get this up and running is using Vagrant. If you aren't familiar with Vagrant, I highly suggest taking a look at the Vagrant site to get it running, as it is much easier to use Vagrant to start playing with RT. Otherwise, I will provide instructions below on getting it installed on Ubuntu and Debian.

With Vagrant

Once you have Vagrant installed, perform the following steps:

git clone git://github.com/cwebberOps/rt-vagrant.git
cd rt-vagrant
vagrant up

From there, wait for the instance to come up all the way and browse to http://10.0.0.10/rt

Vagrant is neat, isn't it?

Without Vagrant (The Manual Way)

If you are using Debian Squeeze, you will need the backports repo installed and setup. To do this, please see the backports setup instructions for more details.

Install the packages listed below. Add the password as you see fit. On Debian, you will need the '-t squeeze-backport' option to be passed to apt-get:
```
apt-get install rt4-db-mysql rt4-clients rt4-apache2 mysql-server postfix request-tracker4
```

Running the following commands to create all the necessary symlinks.

ln -s /etc/request-tracker4/apache2-modperl2.conf /etc/apache2/sites-enabled/001-rt4
ln -s /etc/apache2/mods-available/actions.conf /etc/apache2/mods-enabled/actions.conf
ln -s /etc/apache2/mods-available/actions.load /etc/apache2/mods-enabled/actions.load

Restart Apache
```
/etc/init.d/apache2 restart
```

Add the aliases for the 'general' ticket queue to /etc/aliases and run newaliases

general: "|/usr/bin/rt-mailgate-4 general --action correspond --url http://localhost/rt"
general-comment: "|/usr/bin/rt-mailgate-4 general --action comment --url http://localhost/rt"

Now you should be able to browse to http://[systemname]/rt to get started.

First Steps

If you used Vagrant to bootstrap a test instance, the default password is root:password. Once you are in there are a few things that are worth taking a look at:

Start by going to Tools > Configuration > Global > Group Rights and granting 'Everyone' the right to create tickets. This will allow for people to send mail and create tickets. You may want to revisit this setting once you have a better handle on how you are going to break up your ticket queues.
Add a group for administrative users. The interface for this can be found at: Tools > Configuration > Groups > Create
Go back to Tools > Configuration > Global > Group Rights. Type the name of the new group you just created in the box below 'Add Groups'. Then grant all of the permissions to that group as you see fit.
Add a user for yourself. The interface for this is located Tools > Configuration > Users > Create. Once the user is created, you will likely want to visit the Memberships tab and add the user to the new admin group.
Goto Tools > Configuration > Queues > Select. From there, choose the 'General' queue and then the 'Watchers' tab. Find the new user you created and add them as an 'AdminCC'. This will cause that user to receive email messages when tickets are created or updated.

And welcome to RT!

Notes About Going Production

Make sure that the $rtname setting (what shows up in the ticket subject when dealing with mail) is set correctly. It breaks things if you change it after you have a number of tickets in the system.
The postfix configuration that makes this all work will likely need a bit of tweaking to fit into any given environment.
The actual configuration file for RT is located in the /etc/request-tracker4/RT_SiteConfig.d directory. The update-rt-siteconfig-4 command builds the `/etc/request-tracker4/RT_SiteConfig.pmq file.
Because of the credentials and other potentially sensitive information, you should get an Apache vhost setup with SSL for this site.

Day 4 - A Guide to Packaging Systems

2011-12-04T01:25:00.001-08:00

This was written by Jordan Sissel (semicomplete.com).

Software packaging systems are a strange phenomenon. They all seem to aim at solving the general problem of shipping some named, versioned collection of files to the world. Yet, this common ground seems insufficient given the overwhelming number of incompatible packaging systems in the wild.

The "overwhelming" part of above is what we need to talk about. Specifically, despite having similar goals, one package system is rather unlikely to have anything in common with another. Different terminologies, different tools, different technologies, different distribution mechanisms, different policies. The side effect with this phenomenon of "similar goal, nothing meaningful in common" is that you get punished.

Know how to use Red Hat's package system? Knowledge of rpm and its world doesn't help you much with the Solaris packaging system. Each solves roughly the same problem but with fundamentally different tools, techniques, policies, and terminology. Even package tools with historical ties like Gentoo Portage and FreeBSD Ports have strong divergence in technology, policies, and distribution.

I've had enough frustration over the years in dealing with learning new packaging systems, so today's post aims to give you a general guide to this strange world of packaging. As a caveat, the information detailed here is to the best of my knowledge and assisted with some googling. For the sake of brevity, I'm going to leave out (for now) language or non-OS platform packaging systems like Ruby's rubygems, Python's eggs, etc.

Terminology

Terminology first, because knowing the right terms will help you find answers in search engines faster.

Tool terms:
- Red Hat: rpm, rpmbuild, spec file, yum
- Debian: dpkg, debuild, control file, dh_make, apt-get
- FreeBSD: ports, make, pkg_add
- Solaris: pkg, pkgadd, pkgmk, pkgtrans
Versioning terms:
- Red Hat: epoch, version, release
- Debian: version
- FreeBSD: epoch, version, revision
- Solaris: version
Relationship terms:
- Red Hat: requires, obsoletes, provides
- Debian: depends, conflicts, provides, replaces
- FreeBSD: depends, requires
- Solaris: nothing as far as I can tell
Scripted action terms:
- Red Hat: %pre, %post, %preun, %postun
- Debian: preinst, postinst, prerm, postrm
- FreeBSD: pkg-install, pkg-deinstall, pkg-req
- Solaris: preinstall, postinstall, checkinstall
Support or repo tools
- Red Hat: mrepo, createrepo
- Debian: apt-ftparchive, reprepro, apt-file
- FreeBSD: portmaster, portupgrade, portsnap,
- Solaris: pkgbuild

As you can see, some things have very similar terms, some do not. FreeBSD calls 'pkg-req' what Solaris calls 'checkinstall' but Red Hat and Debian lack this specific feature (though it can be implemented in pre-installation scripts). Red Hat and FreeBSD agree on 'epoch' in meaning but use 'release' and 'revision' to refer to the same feature.

In Solaris, to do dependency checking, you are supposed to do this in a script named 'checkinstall' and bail out if the dependencies are not installed.

Distributing Packages

Many package systems support network installation and dependency resolution. It is these two features that makes 'yum install foo' fetch and install all dependencies of 'foo,' possibly aborting due to dependency conflicts, version mismatches, or other relationship problems.

A common practice here is to mirror packages locally in your infrastructure. This enables you to survive upstream failures (like package removal, outages, etc). Tools like mrepo (for rpm) and rsync will help you here as most package repositories support rsync.

Rolling your own package repository can be done with mrepo and creatrepo for rpm/yum. For Debian (dpkg/apt-get), apt-ftparchive or reprepro are useful.

Building Packages

Have you ever needed a piece of software (or specific version) not available in your upstream package repository? You don't want to be at the mercy and whims of upstream. Your own software policies and needs are likely to be different (and even in conflict), for example, with the packaging policies and cultures of Debian, Red Hat, FreeBSD, Ubuntu, or whoever is your package provider.

Given this likely mismatch in your needs and upstream's available software, it is worth your time learning how to build packages and how to host them internally in your own package repository.

Learning to build packages can be a frustrating experience and, at least for me, has resulted in many shaved yaks and lost hours, but there are resources available to help you out:

Converting Packages

Using multiple package systems on the same platform can make for a confusing array of moving parts. For example, using both rpms and rubygems. The confusion stems largely from the huge differences already described - it's yet another learning curve you must ascend. What if you converted those rubygems packages you needed to rpms? Your team and your tools could then manage ruby libraries the same way as the rest of your software, and that sounds promising.

This idea of converting package types is not new. The common ground between package systems is common enough that there's often a tool available to help translating one package type to another.

What package translating tools are there?

FreeBSD has 'BSDPAN' for automatically registering perl modules installed with CPAN in the FreeBSD package system. This allows you to uninstall them later with standard FreeBSD tools.
Python setuptools (the 'setup.py' stuff) usually responds nicely to 'python setup.py bdist_rpm'
Ruby has gem2rpm will help you build rpms from gems
Perl has cpan2rpm

Build Packages Easier

It's pretty lame needing one tool for every conversion method you want to use. While one tool doing one task well is deeply rooted in Unix philosophy, there's still lots of overhead in requiring many tools to do similar jobs.

Further, the conversion tools don't often help you build the packages; for example, gem2rpm emits an 'rpm spec' which requires you know how to turn that into an rpm. All of this sums up to a burden of knowledge I think is silly.

Luckily, there are projects addressing this problem. Two tools, in particular, aim to solve exactly the 'how to I build a package of this thing?' question.

First, there is CheckInstall (which bears no relation to the Solaris packaging term). This project tries to make package creation a side effect of the normal "make install" task, certainly a nice touch if you are looking to keep you software building workflow the same. Using the same basic workflow that ends with 'make install,' CheckInstall will produce an rpm or deb for you.

Second, there is fpm. The goal of fpm (caveat: I am the author) is to make a common tool for building packages from any source to any target. For example, you can turn a directory into an rpm, or a rubygem into a deb - both tasks can be done with very little knowledge of how each source (rubygem, directory) or target (deb, rpm) work.

At this time, fpm can create Solaris, rpm, and deb packages. To show you the simplicity, the following command lines will create a package of the 'boto' AWS library for Python:

% fpm -s python -t rpm boto
% fpm -s python -t solaris boto

# Example with 'deb' target:
% fpm -s python -t deb boto
Trying to download boto (using: easy_install)
...
Downloading http://pypi.python.org/packages/source/b/boto/boto-2.1.1.tar.gz
...
Created /tmp/z/python-boto_2.1.1_all.deb

Now you have a native deb, rpm, and solaris package of the python boto library and you never had to learn how to build packages on each of those platforms.

Conclusion

For any given platform I am operating, I've always had to learn the native packaging system, build my own custom packages, and host my own package repo. That's why I wrote fpm, after all, to codify my knowledge and skills into a tool that I and others can reuse.

Hopefully this guide has shed some light on a confusing and frustrating area of systems administration and given you the tools and terms to wield whatever packaging systems you are using.

Day 3 - Share skills and permissions with code

2011-12-03T00:40:00.001-08:00

This was written by Jordan Sissel (semicomplete.com).

Last year, I said, "Don't be a human keyboard," and recommended avoiding this situation by building a dashboard for others to consume instead of consuming your time and energy. Going deeper, why not stem 'human keyboard' requests by providing a very simple terminal- or web-based tool?

Before we dive into solutions, we need to go over some of the problems. What are some common reasons someone might invoke you as their computer interface? In general, you probably have something that this person does not, such as skills or permissions.

Both skills and permissions problems can be solved in the same way. You can codify these things into a tool or application that someone other than yourself can use: turn your knowledge and access into a tool usable by others!

The simplest of these tools could be a script in the terminal or a button in the web browser. Buttons are an important interface element in this situation because of exactly the skill mismatch that I mentioned above - not everyone has the skill to use the terminal (ssh, ssh keys, the shell, etc). Going with the 'button' idea, there are nice options for those in the terminal with tools like dialog:

#!/bin/sh

dialog --yesno "Do you want to do this?" 0 0
result=$?
echo

if [ $result -eq 0 ] ; then
  # Do your thing here...
  echo "Doing it."
else
  echo "Cancelled"
  exit 1
fi

Running this script will give you a nice simple terminal tool. This kind of terminal interface is good for some situations, but not all. The dialog tool itself has many more features I won't discuss here, but you will benefit from at least playing with this tool. Anyway, despite this kind of visual interface in the terminal, the permissions problem isn't solved since this script is targeted at users other than you, and you might need to share your permissions.

Permissions are more easily shared over remote interfaces because many access control models lack the detail to express what you want to allow. In a human keyboard situation, the 'server' is you and the 'client' is the person asking for action, which maps pretty well to something like HTTP, so how about exposing permissions and skills through the web browser?

One caveat with doing this with the web is the number of technologies required to make it happen. In the terminal, a button was as easy as a simple dialog invocation, but you'll need a web server, html, and some code to make it happen. Luckily there's lots of open source tools available to make building this quicker.

I'll start with a simple example that has a button restart Apache. I chose Ruby, Sinatra and Bootstrap for this, but you can use any tool that helps you get the job done.

require "rubygems"
require "sinatra"
require "logger"

logger = Logger.new(STDOUT)

# The main page
get "/" do
  haml :index
end

# Handle form submission
post "/" do
  logger.info("Bouncing apache by request from #{request.ip}")

  # Should use Ruby's Open3 here, but let's keep it simple.
  @output = system("sudo apachectl graceful 2>&1")
  exitcode = $?.exitstatus
  @status = (exitcode == 0) ? "success" : "error"

  # Render it.
  haml :result
end

There are support files (the templates and such) for this available here.

The above is pretty short in terms of code. Serve up the button (get "/") and handle the form submission (post "/"). The deployment scenario for this is that since your permissions are what are needed, it should run as your user (or as something with appropriate permissions if you are shipping it to production).

The results look something like this:

Main page

A success after clicking the button

A failure after clicking the button

Keep in mind the point of all of this is to code your way out of the human keyboard situation. Don't know ruby? There's hope! Ruby, Python, Perl, and other communities have lots of ways to serve the web. Beyond that, you probably have a friend who can help you build the parts you aren't able. Lastly, there's lots of resource online for learning how to build simple web applications like this.

Of course, a simple button is just the start. With slightly more effort, you could build a simple interface that allows users to, for example, pick a database backup to restore to their own dev database, or perhaps exposing a one-click 'silence all active nagios alerts for 30 minutes' feature for your on-call folks?

A complex, real-world example of a skills and permissions sharing tool is Etsy's Deployinator. Deployment in many organizations is a complex mixing of different teams, skills, permissions, and goals. A tool like Deployinator allows the operations team to expose deployment functionality outside the operations team while hiding the necessary complexities of the infrastructure. This allows ops to share its permissions and skills by putting those in code and exposing it in the web browser. Because of this, deployments no longer require the full attention of the operations team, development doesn't block on operations, and no one is used as a human keyboard.

Definitely a brilliant win-win situation for everyone.

Proxying permissions and embedding your skills into code allows other people to act with your permissions and skills in a controlled and repeatable way without the burden of knowledge or complex access controls. Reducing the knowledge required to perform a task will greatly reduce the amount of documentation you need to write, too, since you can reduce most things to "If this, then click this button" instead of documenting the complicated orchestra of steps and knowledge.

Day 2 - Strategies for Java Deployment

2011-12-02T01:28:00.001-08:00

This article was written by Kris Buytaert (www.krisbuytaert.be/blog).

After years of working in Java-based environments, there are a number of things that I like to implement together with the teams I`m working with - the application doesn't matter much, whether it's plain java, Tomcat, JBoss, etc, these deployment strategies will help your ops and dev teams build more managable services.

Packaging

The first step is to have the native operating system packages as build artifacts rolling out of your continuous integration server - No .ear, .war or .jar files: I want to have rpms or debs. With things like fpm or the maven rpm plugin this should not be an extra hassle, and the advantages you get from doing this are priceless.

What advantages? Most native package systems support dependency resolution, file verification, and upgrades (or downgrades). These are things you would have to implement yourself or cobble together from multiple tools. As a bonus, your fellow sysadmins are likely already comfortable with the native package tool used on your systems, so why not do it?

Proxied, not running as root

Shaken, not stirred

Just like any other daemon, for security reasons, I prefer to run run Tomcat or JBoss as its own user, rather than as root. In most cases, however, only root can bind to ports below 1024, so you need to put a proxy in front. This is a convenient requirement because proxying (with something like Apache) can be used to terminate SSL connections, give improved logging (access logs, etc), and provides the ability to run multiple java application server instances on the same infrastructure.

Service Management

Lots of Java application servers have a semi functional shell script that allows you to start the service. Often, these services don't daemonize in a clean way, so that's why I prefer to use the Java Service wrapper from Tanuki to manage most Java based services. With a small config file, you get a clean way to stop and start java as a service and even the possibility to add some more monitoring to it.

However, there are some problems the Java Service wrapper leaves unsolved. For example, after launching the service, the wrapper can return back with a successful exit code while your service is not ready yet. The application server might be ready, but your applications themselves are still starting up. If you are monitoring these applications (e.g for High Availability), you really only want to treat them as 'active' when the application is ready, so you don't want your wrapper script to return, "OK," before the application has been deployed and ready. Otherwise, you end up with false positives or nodes that failover before the application has ever started. It's pretty easy to create a ping-pong service flapping scenario on a cluster this way.

One application per host

I prefer to deploy one application per host even though you can easily deploy multiple applications within a single Java VM. With one-per-host, management becomes much easier. Given the availability and popularity of good virtualization, the overhead of launching multiple Linux VM's for different applications is so low that there are more benefits than disadvantages.

Configuration

What about configuration of the application? Where should remote API urls, database settings, and other tunables go? A good approach is to create a standard location for all your applications, like /etc/$vendor/app/, where you place the appropriate configuration files. Volatile application configuration must be outside the artifact that comes out the build (.ear , .jar, .war, .rpm). The content of these files should be managed by a configuration management tool such as puppet, chef, or cfengine. The developers should be given a basic training so they can provide the systems team with the appropriate configuration templates.

Logs

Logs are pretty important too, and very easy to neglect. There are plenty of alternative tools around to log from a Java application: Log4j, Logback, etc .. Use them and make sure that they are configured to log to syslog, then they can be collected centrally and parsed by tools much easier than if they were spread all over the filesystem.

Monitoring

You also want your application to have some ways to monitor it besides just checking if it is running - it is usually insufficient to simply check if a tcp server is listening. A nice solution is to have a simple plain text page with a list of critical services and whether they are OK or not (true/false), for example:

someService: true
otherService: false

This benefits humans as well as machines. Tools like mon, heartbeat or loadbalancers can just grep for "false" in the file. If the file contains false, it reports a failure and fails over. This page should live on a standard location for all your applications, maybe a pattern like this http://host / servicename/health.html and an example "http://10.0.129.10:8080/mrs-controller/health.html". The page should be accessible as soon as the app is deployed.

This true/false health report should not be a static HTML file; it should be a dynamically generated page. Text means that you can also use curl, wget, or any command-line tool or browser to check the status of your service.

The 'health.html' page should report honestly about health, executing any code necessary to compute 'health' before yielding a result. For example, if your app is a simple calculator, it should verify health by doing tests internally like adding up some numbers before sharing 'myCalculator:true' in the health report.

The 'health.html' page should report honestly about health, executing any code necessary to compute 'health' before yielding a result. For example, if your app is a simple calculator, then before reporting health it should put two and two together and get four.

This kind of approach could also be used to provide you with metrics you can't learn from the JVM, such as number of concurrent users or other valid application metadata for measurement and trending purposes.

Conclusion

If you can't convince your developers, then maybe more data can help: Check out Martin Jackson's (presentation on java deployments)[http://www.slideshare.net/actionjackx/automated-java-deployments-with-rpm]

With good strategies in packaging, deployment, logging, and monitoring, you are in a good position to have an easily manageable, reproducible, and scalable environment. You'll give your developers the opportunity to focus on writing the application, they can use the same setup on their local development boxes (e.g. by using vagrant) as you are using on production.

By the way, I will be giving a talk titled DevOps: The past and future are here. It's just not evenly distributed (yet). at this year's LISA in Boston!

Day 1 - Don't bash your process outputs

2011-12-01T00:14:00.000-08:00

This article was written by Phil Hollenback (www.hollenback.net)

The Problem

Like most veteran bash scripters, I have a tendency to abuse shell scripting. In particular, I often fall into the trap of collecting and manipulating process output. This works fine for the simple case of one or two lines of stdout, but falls down horribly for complex cases. Pretty soon a simple

output=`some_command`

turns in to a huge conditional block where you have to check exit codes, redirect stderr in to temporary files, and generally hate yourself.

Welcome to Perl

I should warn you from the start: I've got a secret agenda. My goal is to convince all sysadmins to ditch the bash scripts and use perl instead. With perl you get both the ubiquity of shell scripting and access to a rich data manipulation environment.

Unfortunately, the default perl backtick mechanism for capturing process output isn't a whole lot better than what you get with bash. The good news is that thanks to the CPAN, there are many ways to deal with gathering process input. I'll cover two here: IO::CaptureOutput and IPC::System::Simple. There's some overlap between these two modules but I find I often end up using them both for different tasks. In particular, IO::CaptureOutput is best if you need to deal with both STDERR and STDOUT, and IPC::System::Simple is best if you just need something friendlier than the default perl system call or backticks operator.

IO::CaptureOutput

IO::CaptureOutput is the more comprehensive solution to the problem of capturing output and determining whether the external process ran correctly. Here's how I typically use it:

#!/usr/bin/perl -w
use IO::CaptureOutput qw/capture_exec/;

my ($JobStdout, $JobStderr, $JobSuccess, $JobExitCode) =
         capture_exec("my_command");
if ( ! $JobSuccess )
{
    print "my_command failed with exit code " . ($JobExitCode >> 8);
    print " and printed on stderr:\n";
    print $JobStderr;
    die;
}

my @JobResults = split(/^/,$JobStdout);

my $number = 1;
foreach my $line (@JobResults) {
  print "$number: $line"
}

This provides a comprehensive record of what happened when you ran my_command and is appropriate if any failure is grounds for terminating your script. Note that it also preserves and outputs stderr in case of failure, which is a friendly touch.

After running capture_exec this way, every line of your external command output is sitting there in the array @JobResults, ready for further processing. For example, you could do this next to remove beginning line numbers from your output:

# perl arrays are always references so this is in-place
# editing!
map { s/^\s*\d+\s// } @JobResults;

Or, say you wanted to print all the non-comment lines from the output:

print grep(!/^#/, @JobResults);

Another problem that capture_exec solves nicely is the issue of external commands that print failures to stderr but then return a successful exit code. To trap this, check the contents of $JobStderr:

#!/usr/bin/perl -w
use IO::CaptureOutput qw/capture_exec/;
use List::Util qw/first/;

my $command = "echo hello world and some error text";
my ($JobStdout, $JobStderr, $JobSuccess, $JobExitCode) =
        capture_exec($command);
my @JobResults = split(/^/,$JobStdout);

if ( first { /some error text/ } @JobResults )
{
    # found an occurence of the error text
    print "'$command' command failed";
    print " and printed on stderr:\n";
    print $JobStderr;
    die;
}

Let's put that all together in a trivial data munging script that calls lsof(1) and finds all the open files from processes owned by my user:

use IO::CaptureOutput qw/capture_exec/;
my $username = "philiph";
my $command = "lsof";

my ($JobStdout, $JobStderr, $JobSuccess, $JobExitCode) =
    capture_exec($command);
if ( ! $JobSuccess )
{
    print "$command failed with exit code " . ($JobExitCode >> 8);
    print " and printed on stderr:\n";
    print $JobStderr;
    die;
}
# split lines in to an array
my @JobResults = split(/^/,$JobStdout);

print $#JobResults + 1;
print " total open files on the system, here's what is owned by $username:\n";
print grep(/^\S+\s+\S+\s+$username/, @JobResults);

That will print something like this:

1687 processes owned by philiph:
loginwind    47 philiph  cwd      DIR       14,2       1530        2 /
loginwind    47 philiph  txt      REG       14,2    1754160  3902953 blah
loginwind    47 philiph  txt      REG       14,2     113744    35203 foo
loginwind    47 philiph  txt      REG       14,2     425504 14968261 quux
..etc..

The power of IO::CaptureOutput is that it provides a consistent way to capture stderr, stdout, and the return code when calling an external process. In turn that makes it trivially easy to transform the process output into an array and perform operations on that array.

IPC::System::Simple

If you don't actually care about separately capturing the stdout, stderr and exit value from a process then IO::CaptureOutput is a bit of a heavyweight. Instead, for the ultimate in simplicity you should look at IPC::System::Simple. This CPAN module provides a number of extremely convenient replacements for the perl system and backticks operators. For example, if you just want to run a process and die on failure, you can do this:

use IPC::System::Simple qw(system);
system("my_program");

That's it. If anything goes wrong with calling my_program, this replacement for the perl system builtin will just call die to exit your script with a descriptive error message. Of course, you can always catch the exception instead of just die()ing. The big advantage of this version of system is it prints much more descriptive messages on exit, including the real return code.

This module also includes a capture function which is a replacement for the default backticks operator in perl. It works similarly to IO::CaptureOutput capture_exec, although note you can only capture STDOUT, not STDERR. If you don't care at all about STDERR you could make my earlier example much simpler:

#!/usr/bin/perl -w
use IPC::System::Simple qw(capture);

my $username = "philiph";
my $command = "lsof";

my @JobResults = capture($command);

print $#JobResults + 1;
print " total files on this system, the following are opened by by $username:\n";
print grep(/^\S+\s+\S+\s+$username/, @JobResults);

If anything goes wrong, IPC::System::Simple will just bomb out with a descriptive message, and set an appropriate exit value automatically. This saves you from having to bit shift the system result codes to figure out what the hell actually happened. An illustration:

# the standard way
my $command = "crontab -l -u root";
system("$command") == 0 or die "$command failed: $?";

if you aren't running as root, this produces the following baffling error:

crontab: must be privileged to use -u
crontab -l -u root failed: 256 at /tmp/test.pl line 6.

that '256' actually has to be bitshifted ($?>>8) to get the real exit value, which is 1. Also, you have to remember to manually call die on a non-zero return from the default perl system function.

If you replace the default system fucntion with the IPC::System::Simple one, that is all taken care of automatically:

# Replace system with IPC::System::Simple
use IPC::System::Simple qw(system);
my $command = "crontab -l -u root";
system("$command");

this time you get the real failure:

crontab: must be privileged to use -u
"crontab -l -u root" unexpectedly returned exit value 1 at /tmp/test.pl line 6

And you don't even have to remember to call die! Of course, you can trap the failure in an eval block if you don't want your script to die.

As a further refinement, say you know that there is a list of exit codes that are acceptable when calling an external process. IPC::System::Simple has a shorthand for that:

# Replace system with IPC::System::Simple
use IPC::System::Simple qw(system);
my $command = "crontab -l -u root";
# return codes 0 or 1 are acceptable
system([0,1], "$command");

Now, the script will continue on past the system call as long as it returns an exit code of 0 or 1. Otherwise it will die with a descriptive error message the same way as before.

Conclusion

Dealing with process output in bash scripts is hard! Doing it correctly and safely can quickly end up with you shaving yaks.

Stock perl with the system function and backtick operator is an improvement. You gain access to perl's rich data manipulation features, but you still have to do unpleasant things like bitshifts to determine the real process exit code.

As usual, CPAN comes to the rescue. If you just want to replace system or backticks with more convenient functions, IPC::System::Simple is an excellent choice. If you want to capture all of STDOUT, STDERR, and the process return code, then IO::CaptureOutput is the way to go. Also, note that these modules are portable across unix, mac, and windows.

sysadvent

Day 25 - Learning from Other Industries

Further Reading

Day 24 - Implementing Configuration Management in Legacy Environments

Discovery

Short Cycles

Version Control

Use Packages

Baked-in Goodness

Involve Everyone

Pick your fights

Roles, not hosts - No more snowflakes

Conclusion

Further Reading

Day 23 - All The Metrics! Or How You Too Can Graph Everything.

Enter Graphite

The Power of Filters and Functions

Filter Example

Stacking example

Functions Are the Best!

Graphite in Production

In Closing

Further Reading

Day 22 - Load Balancing Solutions on EC2

Elastic Load Balancing and the DNS Root Domain

Elastic Load Balancing and SSL

Further Reading

Day 21 - Automating Web Monitoring

Getting started

Using Sauce Labs for Testing

Using BrowserMob for monitoring

Setting variables.

Getting back data from a webpage.

Extra Logging

What Next?

Day 20 - Thoughts On Load Testing

Use Science

Have Defined Targets

Scale First

Understand Your Traffic

Know Your Resource Limits

Graphs Lie

Measure Time and Resources Spent in Each Component

Learn And Use The Right Tools

Further Reading

Day 19 - Why Use Configuration Management?

Minimize Manual Effort

A Visualization

Getting Started

Using these examples

Set Permissions on a File

Create with some content

Install a package

Make sure a service daemon is running

Final Thoughts

Further Reading

Day 18 - Why Businesses Do Things

Turning Neutralizing into Differentiating

Why Should You Care?

Footnotes

Further Reading

Day 17 - Speaking the Same Language

Verbal Communication

Visual Communication

Software Similarity and Language Problems

When to Define Terms

Parting Thoughts

Further Reading

Day 16 - Shipping Some Logs

Starting with Log Files

Filtering Logs

Final Destination

And then?

Further Reading

Day 15 - Automating WordPress with CFEngine

Automating WordPress Installation

Ordering Things

Get the Right Packages

Apache and MySQL

Downloading WordPress