asemanfar

Monitoring Unicorn with Bluepill

Fri, 06 Nov 2009 09:32:22 +0000

At Serious Business, we recently switched to using unicorn and bluepill. One of the features we implemented in bluepill was the ability to monitor child processes. This was built with Unicorn's workers in mind. Here's the config file we're using to monitor our Unicorn workers:

Bluepill.application("app_name-production") do |app| app.process("unicorn") do |process|



process.pid_file = File.join(RAILS_ROOT, 'tmp', 'pids', 'unicorn.pid')
process.working_dir = RAILS_ROOT

process.start_command = "/usr/bin/env #{env_vars} unicorn -Dc config/unicorn.rb -E production"
process.stop_command = "kill -QUIT {{PID}}"
process.restart_command = "kill -USR2 {{PID}}"

process.uid = process.gid = 'deploy'

process.start_grace_time = 8.seconds
process.stop_grace_time = 5.seconds
process.restart_grace_time = 13.seconds


process.monitor_children do |child_process|
  child_process.stop_command = "kill -QUIT {{PID}}"

  child_process.checks :mem_usage, :every => 10.seconds, :below => 150.megabytes, :times => [3,4]
  child_process.checks :cpu_usage, :every => 10.seconds, :below => 20, :times => [3,4]
end

end end

One new feature we recently implemented was the support for working_dir, similar to god's config. There is one important difference though: we also set the PWD environmental variable. This is used to reload the unicorn app with a USR2 signal. In order for Unicorn to properly reload the app with the new current symlink, the env variable needs to be set.

Typically, when you chdir into a symlink'd folder, you are stuck in whatever the symlimk referenced, but if we pass the symlink as an environmental variable, unicorn chdir's into the symlink. The alternative would be to put the full path to the config file in the start command.

The montioring of child processes is still fairly new, as is all of bluepill, so please report any bugs you encounter to the github repos' issue page.

Bluepill: a new process monitoring tool

Wed, 04 Nov 2009 06:29:02 +0000

Over the past several weeks, two co-workers and I wrote a simple process monitoring tool: bluepill, because it keeps things up. Bluepill replaces the existing process monitoring tool we were using which we had some issues with.

Bluepill is now running on over a dozen of our machines with no memory leaks. It's monitoring several of our services, including long-running rake tasks, background workers, unicorn processes, and pandemic workers. It's been idling at 18MB (see related post for graph).

Here is a sample of the DSL: Bluepill.application("app_name") do |app| app.process("process_name") do |process|



process.start_command = "/usr/bin/some_start_command"
process.pid_file = "/tmp/some_pid_file.pid"

process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3        
process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]
process.checks :flapping, :times => 2, :within => 30.seconds, :retry_in => 7.seconds

end end

Check the readme for detailed installation and usage.

read, fork, and report bugs.

Why We Wrote Bluepill

Sat, 31 Oct 2009 19:49:33 +0000

At Serious Business, we use god to monitor our long-running processes (mongrel, background workers, and more recently unicorn). We had a basic god config setup that only checks for memory usage, cpu usage, and request queue length (for mongrels only). God was working fine for us except for one problem: the notorious memory leak. If you use god you probably know it leaks memory in correlation with the number of watches on the system. This became a problem for us when god hadn't been restarted for several days; its memory usage would climb and reach several gigs, causing the machine to swap and eventually lock-up. To prevent lock-ups we needed to manually monitor god (what does that make us?) and restart god daily via cron, gross.

Our frustration with this issue eventually reached a point where we decided to write our own process monitoring tool. Rohith Ravi, Gary Tsang, and I got together one weekend and built a first version of what we've come to call bluepill. We spent the next couple weeks massaging the DSL, expanding feature set, and fixing some bugs we found while using it for our apps. The current feature set is small but is sufficient for the most users:

DSL for specifying processes and their respective watches
Built-in support for monitoring memory usage and CPU usage
Support for custom conditions to watch
Daemoniziation of non-daemonized processes
Monitoring child processes (especially useful for monitoring unicorn workers)
Logging
Support for triggers (flapping)

While both bluepill internal and the external interface is heavily influenced by god, we decided do some things differently in bluepill:

Written with long-running daemon in mind (read: low resource consumption)
Simplicity over flexibility:
one process per application; forces separation between multiple apps on the same box
simple state machine; does only what it needs to to keep the process up

This past week, we ran a test to see how well Bluepill will do in the wild compared to god, so we set up a basic bluepill config file and the equivalent god config on two identical machines and recorded their memory usage every 30 minutes for just over 4 days.

In addition to the memory leak issue, we sought to improve god in a few other ways: sequential CLI command processing, improved daemonization, and monitoring child processes:

CLI Command Processing
In god, CLI issued commands are sent to the long-running god daemon which starts a separate thread and returns to the CLI; this led to some race cases when you issue two commands sequentially and expected them to execute in that order (i.e god stop <process_name>; god start <process_name>). This is fixed in bluepill by handling CLI issued commands in a single thread fed by a queue.

Improved Daemonization
One of the things that is great about god is that it can daemonize processes for you. Meaning, for example, you can write a normal rake task that does some heavy offline work and ask god to daemonize it and viola! you have instant background workers. However, there was a problem: god did not do a good job of monitoring the processes it daemonized. God, through it's use of the system Ruby method, executed command inside a sh -c "..." shell and then proceeded to monitor the shell instead of the actual process that the shell spawned. Bluepill fixes this by daemonizing commands and then properly detecting the pid of the spawned process. The only catch is that you need to tell bluepill whether or not it should daemonize the process.

Monitoring Child Processes
We recently switched in Unicorn which starts its own long-running child processes to handle requests. So in order to monitor the unicorn workers, we needed to add support for monitoring child processes. Child process monitoring differs from regular process monitoring because bluepill is not responsible for starting them back up and the PID comes from the parent process and not a PID file.

We're going to continue working on it to improve its feature set and iron outs any bugs that we find.

Read the readme for usage information. Read the design file for technical details.
Fork and contribute: http://github.com/arya/bluepill
Report bugs: http://github.com/arya/bluepill/issues

search.facebook

Thu, 29 Oct 2009 05:43:57 +0000

Today, Facebook announced their new Open Graph API. The Open Graph API allows any website to have all the features of a Facebook Page. So you'll be able to "Become a Fan" of any site, make comments, have discussions, and receive updates published by fanned sites. Think Amazon's Universal Wishlist but not just for things you can buy.

This leads to some interesting business opportunities for Facebook. Facebook will be able to capture socially relevant information from anywhere on the Internet. Facebook will collect information about "what's on your mind" from everywhere on the Web, not just on facebook.com.

Now let's take that and see how Facebook can profit from that: they can do exactly what Google does but enhanced by your social graph. They will be able to give you Web Search Results but unlike Google, they'll be able to tell you that "Joe Smith likes this" and what was "on his mind" when he visited that site. Not only that, but they'll maybe even cross some privacy boundaries that Google couldn't and use that information to target ads.

The challenges Facebook will face in the coming months/year will be dealing with privacy issues and adoption of the Open Graph API. I suspect something similar to a bookmarklet, browser toolbar, or full-blown "socially enhanced" browser will be used in order to gain traction and bootstrap their social indexing efforts. I'm also interested to see how well they'll execute on search results with social graph information and if they can complete with Google's myriad of well-tested and proven search ranking algorithms.

Arya likes this.