Scout ~ The Blog

Scout Agent Updated - and do your Net::HTTP calls ever hang?

Tue, 30 Jun 2009 11:47:00 CDT

UPDATED 6/30 – The fix for the old scout client (run via cron) is now available in version 2.0.7 (sudo gem install scout).

In rare (and difficult to reproduce) cases we’ve seen the Scout Agent not observe a Timeout during a checkin error with the Scout server. Scout uses Ruby’s RestClient gem to connect to the Scout Server and it uses the standard Net::HTTP library to manage the connection. Some versions of the Net::HTTP library can run into a bug in IO.select() on some platforms. This causes the request to hang forever in some rare cases.

Our fix? We added a redundant Timeout for the request, in addition to Net::HTTP’s own Timeout. You have to be careful how you nest those calls though, since they will throw the same Exception by default. We followed Eric Hodel’s advice to get our implementation right.

If you’re using Net::HTTP and notice the same issue, try adding a redundant Timeout with a custom halting Exception (our committed fix for this is on github).

This fix is included in version 3.2.6 of the Scout Agent. We’re planning on backporting the fix to the old client ~~late next week~~ (available now). Follow our Twitter feed to stay updated with the latest releases.

On Sinatra

Mon, 29 Jun 2009 11:42:00 CDT

Sinatra, a Ruby DSL for quickly creating web applications with minimal effort, forms a key part of the Scout infrastructure.

James Gray talks about how we use Sinatra at Scout via RubyLearning.org – 20+ Rubyists are using Sinatra – Do you?

For more on how Scout works:

Rails Machine selects Scout for Performance Monitoring

Thu, 25 Jun 2009 14:14:00 CDT

Rails Machine, one of the first specialized Rails hosting providers, has selected Scout for performance monitoring.

Why Scout? “Scout’s versatility was a key benefit: “the open-source plugins are the killer app of Monitoring,” says Rails Machine CEO Bradly Taylor. “We have so much flexibilty—and we’ll be contributing some great plugins back to the community.”

“With Scout, we have the full picture – Rails, MySQL, CPU, Memory, IO, and Disk Usage – all in one dashboard. You can’t beat that,” said Jesse Newland, Senior Engineer at Rails Machine.

Rails Machine, which recently launched a Managed Hosting offering, finds Scout particularly useful for proactively managing customer accounts. “Scout’s trend detection have more than once caught an application’s jump in memory usage due to RMagick-based image uploads/resizes,” says Newland. “We’ve been able to jump in, restart Apache, and then proposing alternatives to customers. It’s much better than a server running out of a memory minutes later.”

Scout is included with Rails Machine’s new Managed Hosting service which does everything but write the code – from MySQL tuning to backups to performance monitoring. Self-managed customers can request a coupon for a free Basic Scout subscription or $14/off a larger plan. See the announcement on the Rails Machine blog for more details.

From the original five-minute Rails deployment gem, the Rails Machine gem, to Moonshine, an open source configuration management and deployment system, Rails Machine understands automating system administration. Combining Scout’s proactive approach to monitoring with Rails Machine’s encyclopedia-like knowledge-base for scaling Rails apps is a perfect combination for managed hosting.

A tour of Scout's Ruby on Rails Monitoring Plugin

Mon, 22 Jun 2009 16:37:00 CDT

Curious how data moves from your Rails app to Scout? Wondering how Scout handles high-traffic Rails apps? Where does Scout add MySQL suggestions? How are alerts generated? I’ll guide you through the process below.

Logging each web request

At a basic level, Scout’s Rails benchmarking isn’t much different than what gets written to your Rails log. Instead of writing to a log file, the Scout Rails Instrumentation plugin sends a Hash to the Scout Agent like the one below:


{:num_requests=>2, :avg_request_time=>241.0, :actions=>
{"users/index"=>{:db_runtime_max=>156.152, 
:num_requests=>2, :render_runtime_avg=>8.59355926513672, :render_runtime_max=>8.73303413391113, 
:runtime_avg=>241.0, :other_runtime_avg=>77.4774407348633, 
:queries=>[[[156.152, 0]], [[153.706, 0]]], :runtime_max=>243.0, 
:other_runtime_max=>78.1149658660889, :db_runtime_avg=>154.929}}, 
:queries=>["SELECT * FROM \"users\""], 
:scout_time=>"Mon Jun 15 21:04:00 UTC 2009"}

These messages are sent to the Scout Agent every 30 seconds by the Scout#Reporter.report! method:


response = ScoutAgent::API.queue_for_mission(Scout.config[:plugin_id], report)

The instrumentation runs in a background thread that is only active when reporting to the agent, and since the queuing process is asynchronous once the data has been prepared, the impact on performance is negligible. We’ve benchmarked the impact between 1ms-3ms with a small memory requirement (1 MB).

Running the Instrumentation Plugin

Every three minutes, the agent collects the messages sent to the agent, analyzes & aggregates the data, and sends the results to Scout.

Aggregation

A large Rails application can process a lot of requests – for example, a Rails app with 50 web processes may result in up to 300 messages queued to Scout. If we needed to process every request on our server, our hardware cost would dramatically increase for high-traffic Rails apps.

Enter the Scout Agent.

Instead of sending reports directly from your Rails app to our server, the agent steps in between to decrease the load. The agent creates a single report for each action that contains the minimum, maximum, and average for each benchmark as well as the number of requests. No matter how large or how small the Rails application is, Scout generates just 1 report for each unique action – dramatically cutting down the hardware costs for serving Scout.

Analyze

The agent looks through the messages sent to the agent, looking for slow MySQL queries, missing indexes, and more. When an optimization opportunity is found, a Hint is created.

Triggers & Alerts

Triggers are Scout’s way of tracking when things go wrong. In other words, leave the number crunching to Scout. When you install the Rails Monitoring Plugin, 2 default triggers are created:

When a trigger fires, an alert is created. A sample alert from Scout:

Alert Throttling

When things are really going wrong with your application, the last thing you want is a every-minute email blast. Scout sends one alert when something goes wrong then another when the conditions are back to normal. Scout will continue to update the alert on scoutapp.com so you’ll see the most recent data.

Elsewhere & Digging In

You can find everything but the Scout server on GitHub:

It's very easy to create a Scout plugin

Wed, 10 Jun 2009 08:33:00 CDT

How easy? Really easy. There are just two things you need to know to get started:

1. Starting the Agent in test mode

The agent sports a nifty autotest-like mode to help you quickly iterate new plugin code. To start:

scout_agent test my_new_plugin.rb

With that, the agent is watching my_new_plugin.rb, just waiting for you to hit save so it can come to life. Whenever you modify the target file, the Agent will immediately log to console the results of your plugin-in-progress.

2. The basic format of a plugin

Keeping this very simple—a plugin looks like this:

class ExampleReport < Scout::Plugin
  def build_report
    report(:minute=>Time.now.min)
  end
end

Yeah, that’s a simple case all right. It just reports the minute every time it runs, so it will generate a single data series with a value between 0 and 59.

If you haven’t created a plugin before, I encourage you to try this very simple example. It will take you approximately two minutes, and you’ll see how easy the Agent’s test mode makes it to iterate.

Go a little deeper

There’s more: generating alerts, reading from option files, etc. When you’re ready, check out the developer documentation at http://scoutapp.com/info/plugin_anatomy

Learn about the Scout Agent, API, and more at OK.rb

Mon, 8 Jun 2009 12:51:00 CDT

James will be speaking at Thursday’s (June 11th) Ok.rb meetup in Edmond, Oklahoma, giving a tour of the Scout Agent and the API.

See the Ok.rb website for more details.

Now with deep Rails Instrumentation, triggers, a more robust agent, and more

Tue, 26 May 2009 16:47:00 CDT

Scout is nearing 1/2 billon recorded metrics, but bragging about metrics is like bragging about the number of lines of code you’ve written. Metrics, like code, are a means to an end. It’s the quality, not the volume, that matters.

We took a big step in going beyond numbers today with the release of the Rails instrumentation plugin for Scout. It’s the first plugin to leverage Scout’s new features, and at just $7/server, we think it’s at a price that makes sense whether you have a small VPS or a cloud of servers.

A quick tour of Scout’s new features:

Deep Rails Instrumentation Plugin

Using the new Rails instrumentation plugin, you’ll get alerted of trends. Scout wants to help you correct and issue before it becomes a problem:

When things go wrong, Scout tries to lend a helping hand, giving optimization suggestions:

..and highlights the variables that changed the most:

Sometimes you just want an overview of your Rails application as well:

The Rails Instrumentation plugin is open source, like the rest of our plugins. View the source of the Rails plugin and its companion Scout plugin on GitHub.

Triggers

Leverage the same alerting functionality used by the Deep Rails instrumentation plugin in your own plugins with triggers. For example, let Scout notify you if user signups dramatically increase compared to the previous 7 days. Just pick a metric and set the thresholds!

The new Scout Agent

The old cron-based Scout client has been replaced by the Scout Agent, a daemon. The Scout Agent is an easier install and enables the more powerful and flexible metric-collecting goodness used by the Rails instrumentation plugin.

View the Scout Agent source at GitHub.

An easier API

It’s even easier to get your data into Scout with the new Ruby API:

ScoutAgent::API.queue_report({:metric_name => value, :plugin_id => PLUGIN_ID})

Or just use JSON from the command line:

scout_agent q report <<< ‘{“new_users”: 12}’

View our Developer Resources for more information on integrating with Scout.

Just $7/server

You can get started today with a free 30 day trial.

Scout at RailConf

Tue, 12 May 2009 18:07:00 CDT

It’s been a big year for Scout at conferences. We sponsored GoGaRuCo here in San Francisco last month, and we had a booth on the floor at RailsConf last week.

A few highlights from the conference:

We came to the conference armed with all kinds of “selling points” for Scout 2.0. When all is said and done, the number one point people love about Scout is the open-source plugin model. Thanks for everyone’s feedback and support!

It was really cool talking to people in all stages of involvement with Ruby and Rails. Including those who were building a Rails app for the first time. It’s good to get that perspective after working with Rails for the last 3+ years.

Charles Quinn and I had the opportunity to treat Hongli Lai & Ninh Bui to Sushi and say thanks for one of our favorite pieces of software.

One cool interaction—Josh Goebel of Pastie came by and asked us about the new Scout API. Then, he tracked us down the next day to show us the Scout plugin he had created for vmstat! Great to see folks using Scout’s extensibility like that.

I didn’t hear one mention of “what happens in Vegas”—Rails community +1; tired cliches -1.

Interview on akitaonrails.com

Mon, 11 May 2009 23:38:00 CDT

Fabio Akita tracked down our own Matt Todd to talk about Scout.

Listen to the interview:

Download (13:43)

I'm not a ShamWow salesman and other lessons from our first conference

Mon, 20 Apr 2009 08:09:00 CDT

I despise sales pitches, but you wouldn’t know it if you were the first person to visit our Scout exhibit booth. After introducing myself, I delved into a 5 minute whirl-wind tour of Scout 2.0, trying to show every possible feature that might help him monitor his Rails stack.

He watched, mostly quiet.

As he walked away, I felt a little dirty, a little QVC-like. I hoped my career wasn’t heading down the path of another famous pitchman.

We exhibited Scout at our first conference, Golden Gate Ruby Conference, run with Scandinavian precision and held, not ironically, at the Swedish American Hall. I started a little awkward telling the Scout story, but things really smoothed out quickly. 8 lessons from our first conference:

Don’t dive into the product

It’s important to have a specific use case that someone can relate to. Everyone uses Scout in their own unique way. If I do a general demo, I’ll be doing it from my perspective, not their own.

Ask attendees questions for 1-2 minutes before demoing

I asked people a lot of questions about their business. The better I understand their business, the better I can show how Scout can help.

Demo a single feature

Rather than drowning people in features, I tried to isolate a major problem they have and showed how Scout can help. From there, people were a lot more engaged and led the demo themselves.

Stand in front, not behind the exhibit table

Yes, I’m a developer, but I don’t usually bite and a I brought breath mints. Sitting behind a table isn’t engaging.

Don’t need a Mac Cinema Display

I demo’d less and talked more. You don’t need a Mac Cinema display to have a conversation.

Have a curious visual display

Scribd, the exhibit next to us, had a display that provided a real-time view of search queries directed from Google to the Scribed website. Did it have anything to do with the actual product? No – but it really demonstrated the volume of traffic that Scribd handles. It was a great way to start a conversation.

Safari is cleaner, faster

At least on my Mac, Safari is a lot faster than Firefox. It’s cleaner-looking as well.

Be Social

We’re not that well known and my wife’s family isn’t rich, so we need to rely on word-of-mouth instead of marketing dollars to spread Scout’s goodness. Don’t be afraid to spend time away from the booth talking to other attendees.

Golden Gate Ruby Conference was a blast and we met a lot of people that are going to benefit from Scout. Technical conferences are really about people, not technology. That applies to the exhibit booth as well.