Scout ~ The Blog

Monitoring Amazon EC2 Instances

Thu, 3 Jul 2008 12:53:00 CDT

Amazon’s Elastic Compute Cloud (Amazon EC2) Service is changing how we approach computing infrastructure. Scout is an ideal monitoring solution for Ruby on Rails application servers on EC2 instances.

Why use Scout for monitoring your EC2 instances?

Just a single line of code to add the client
No complicated scripts to configure – choose what you want to monitor through the Scout web interface
Easy to re-use or view statistical data once an instance has terminated

Overview

The basic process for EC2 monitoring works like this: You create a new client that you want to use as a template . You can then clone this client (and all of its plugins) using the scout clone command, which can be added to the boot/startup script on your AMI.

Read on for the Step-by-Step.

Step-by-Step

Create a new client using the Scout Web Interface:

Install plugins on this client:

Grab the client key of your newly created “template” from the Client Settings tab:

Now, you can easily create a brand new client from your template using the scout clone command:


scout clone [client-key] `hostname`

The output of the line above is a line for your crontab file, so you could simply add this to the bottom of the crontab:


scout clone [client-key] `hostname` >> /etc/crontab

You’re just one step away from automating the entire process by adding the scout gem dependency to your AMI, and even adding the scout clone line above to your startup scripts.

Here’s my first version of our Scout install script to be run on boot of our Rails AMI:


#!/bin/bash
# Scout Install / Setup Script

/usr/bin/gem install scout

CLONE_CLIENT_KEY = 6ecad312-0a17-3db1-9x2c-a12c45493fg # replace with your template client key
INTERVAL = 10
NEW_CLIENT_KEY = /usr/bin/scout clone $CLONE_CLIENT_KEY

# add to crontab:
echo "*/$INTERVAL * * * *   root   /usr/bin/scout $NEW_CLIENT_KEY" >> /etc/crontab

When your AMIs boot up, they’ll be on the cloud, and under the watch of Scout.

Everything is easier with Scout Client 2.0

Mon, 9 Jun 2008 09:26:00 CDT

We’ve released the 2nd version of our Scout client. What I love about this release is that it doesn’t make anything more complex or break compatibility with our first release. It just makes writing plugins easier.

For example – lets say you want to generate an alert. How’s this for simplicity?


class INeedASandwich < Scout::Plugin
  def build_report
     alert(:subject => "Get me a sandwich")
  end
end

We’ve also made the process for managing memory (data stored on the client between runs of the plugin) easier to work with. Memory is a great tool for storing state.

For example:


class INeedASandwich < Scout::Plugin
  def build_report
    if memory(:am_i_hungry)
      alert(:subject => "Get me a sandwich")
      remember(:am_i_hungry => false)
    else
      remember(:am_i_hungry => true)
    end
  end
end

As always, updating to the latest release is as simple as:

~~gem install scout~~ [1]
gem install scout --source http://gems.scoutapp.com/

[1] Rubyforge is slow to update, in the meantime, use our gem server mirror

Monitor MySQL queries with Scout

Fri, 6 Jun 2008 17:36:00 CDT

Robin Ward of Forumwarz has released a MySQL slow queries plugin for Scout that does exactly what you’d expect: sends an alert when a slow query is run.

Derek Laptop"/>

The alert contains the full query along with the query time:

Alert"/>

Note that you’ll need to enable slow query logging in your MySQL config file, but that’s just a 2 line change. See the MySQL Slow Queries directory entry for more information.

Meet our team at RailsConf

Thu, 29 May 2008 11:19:00 CDT

Charles, myself (Derek), and Andre are boarding a plan soon for RailsConf in Portland.

If you want to do a little coding, chat about monitoring, etc. look for us there!

Giving your web host access to your Scout account

Thu, 29 May 2008 09:14:00 CDT

A couple of times already, we’ve seen users give their web host access to their Scout account.

It’s a big help when debugging problems – especially things that a basic monitoring tool might not pick up (like slow web requests).

Click the ‘People’ link in the header:

Select the ‘Invite a New Person’ tab:

Enter their email address and select the clients they need access to:

Your web host’s support team will now have read-only access to the clients you specified.

How we handle background jobs

Thu, 22 May 2008 10:24:00 CDT

We run background jobs on our Scout servers. Lots of them.

As we’ve grown, they’ve used dramatically more resources. We needed a way to simmer them down. Most of these jobs load the entire Rails environment – that’s a hefty overhead.

We’ve modified how we run background jobs and we’re seeing great results. If you’re interested in what we did, checkout Charles’ post on our Highgroove blog.

And finally, here’s the obligatory semi-promotional graph, generated by Scout, of how the load dramatically decreased after implementation:

Scout Client 2.0 is on the Horizon

Fri, 9 May 2008 15:20:00 CDT

We’ve been working on two major features that we’re sure will be a hit, and that we can’t wait to talk about:

1) Easier Plugin Creation—there’s a reason we love Ruby, and not just because it’s such a powerful language for talking to external system processes and manipulating strings (as pointed out by JEG2 in our RubyInside debut). Ruby makes writing powerful code down-right simple through a very natural language. And with our new plugin code helpers and syntax, you can write a plugin like this:


my_output = `run some wild script here`
remember(:last_time_output => my_output)
report(:data => my_output)
if my_output ~= /Error/
  alert("An Error Occurred", "Details: #{my_output}")
end

And that’s not all. We’ve built a clearer syntax for how you invoke Scout, and how you interact with options and how you can store your own settings and report your own data.

2) Automatic Client Provisioning—if you have any servers in the cloud, or just more than one server, you already know how easy it is to install Scout (one liner to install and another to setup), but it just got easier than that. You can now automatically provision clients using the scout client itself. I know, crazy-talk, but watch. On bootup, of our Amazon EC2 instance, we’ll just run:


scout clone [client-key] `hostname` >> /etc/crontab

And we’ve just setup a new client on scout, with all our pre-configured plugins, cloned from another client, and the name of our hostname, setup to run on our interval.

If you’re interested in helping us beta test the new 2.0 version of our client, you can install it from our beta-gems server:


sudo gem install scout --source http://beta-gems.scoutapp.com/

Let us know what you think. Be sure to check the README to find out more details on what’s new.

How To Customize a Scout Plugin

Thu, 8 May 2008 17:20:00 CDT

Since Scout Plugins are open source, it’s easy to modify them to fit your needs.

Here’s a quick way to create a custom version of a Scout plugin.

1. Grab the current plugin and options

The code:


curl http://plugins.scoutapp.com/mongrel_cluster_monitor/trunk/mongrel_cluster_monitor.rb \ 
> mongrel_cluster_monitor.rb

The options:


curl http://plugins.scoutapp.com/mongrel_cluster_monitor/trunk/mongrel_cluster_monitor.yml \ 
> mongrel_cluster_monitor.yml

2. Edit the plugin code and test

 
scout -p /path/to/customized/mongrel_cluster_monitor.rb \
-o '{"mongrel_cluster_configuration_dir"=>"/etc/mongrel_cluster"}' \
-v -l debug

That says, use the -p option (or --plugin) and the -o (or --option) to test locally, passing these options, be verbose (--verbose) and set the log level to debug (--log-level debug).

Note that you could also add your own “option” there, as the path to one of the configuration files…. however you want to do it.

3. Host the Plugin

Then, all you have to do is make that plugin and its options file (of the same name) URL accessible (put it in your rails public dir, perhaps, or on another server) for the scout server to be able to grab it when you install. When you install, choose, Manual—and type in the URL to your new customized version (http://highgroove.com/new_customized_single_mongrel.rb for example).

Plugin Code is now cached

Tue, 29 Apr 2008 00:00:00 CDT

One of the things people like most about Scout: the plugin code your clients execute is downloaded via a plain-vanilla HTTP or HTTPS connection like this one. This means you can simply update the plugin code once, and any clients that run the plugin grabs the new code. You don’t have to manually install scripts on each monitored server.

But what happens if the web server hosting the plugin code isn’t available? What if you are running a plugin someone else developed and they change the plugin code?

We’ve got you covered. Scout now caches the plugin code when you add a plugin to a Scout Client. By default, Scout locks to the version of the code available when the plugin was installed. However, you can override this and have Scout automatically grab the latest version. You can also perform a one-time update.

We check for code updates every 5 minutes, so if you are debugging a new plugin you developed, click the “Update Code” button when you push out a change to instantly refresh the code.

Read on for screenshots.

The new setting is on the Edit Plugin page:

Bristol"/>

You can manually update the code on the Plugin Code page:

Bristol"/>

Recording Historical Data in Scout

Mon, 28 Apr 2008 08:35:00 CDT

By default, when you create a report in Scout, we assume that the data corresponds to the time the report was submitted.

Sometimes this isn’t the case – for example, you can retrieve the number of subscribers to your RSS feeds in Feedburner, but not from the current day.

You can override the time Scout records the data in a Plugin with the special :scout_time report key.

Here’s an example:


# typical report data 
{:circulation => 1000}

# correspond the data w/ February 24, 2008
{:circulation => 1000, :scout_time => Time.parse("2/24/08")}

When displaying the data on a graph, the data will be shown on 2/24 and not the current date:

Learn more about creating Scout Plugins