Scout ~ The Blog

RVM, Bundler and Cron in Production: Round 2

Fri, 7 Jun 2013 14:02:00 EDT

Back in 2010, we suggested using /bin/bash -l -c to run scout via Cron when using RVM. However, this was a brute approach: /bin/bash -l -c tells bash to behave as a login, interactive process. However, as Daniel Szmulewicz elequently stated in the comments for the original blog post, "Cron jobs are by nature non-login, non-interactive processes".

Fast-forward to today: RVM usage is continuing in production, and to make things more complicated, Cron jobs often need to account for both RVM and Bundler. So, what's our preferred approach when running Ruby executables via Cron in an RVM, RVM+Bundler, or Bundler environment? A shell script.

Cron Shell Script: RVM + Bundler

Lets say we want to run a Ruby executable (scout [KEY]) via Cron with (1) Ruby 1.9.2 and (2) my Rails App's Gem bundle:

Make the shell script executable: chmod +x FILE.sh.

Add the Cron job:

* * * * * shell_script.sh

But that's a lot of typing...

It's tempting to use /bin/bash -l -c when you are busy/lazy. To get around this, the scout install [KEY] command will detect if you are using (1) RVM and/or (2) Bundler. If so, we generate the shell script for you and make it executable.

scout install BNrIneEBMwE8h6VlhO4Bw4WmOVSLmnygSFZEPCfi
=== Scout Installation Wizard ===

It looks like you've installed Scout under RVM and/or Bundler. 
We've generated a shell script for you.

Run `crontab -e`, pasting the line below into your Crontab file:

* * * * * /Users/dlite/.scout/scout_cron.sh

How do we detect RVM and Bundler? We've encapsulated it into an Environment class:

The winding path to server roles

Wed, 29 May 2013 09:39:00 EDT

We're overjoyed with the reaction to server roles, our new feature that makes monitoring many servers as easy as monitoring a few. The end result hits our favorite sweet spot: it makes something that used to be painful into something fun.

Server Roles was the biggest release since the launch of Scout and the path to the release was anything but a smooth, rolling path. It's a story of fast-changing deployment environments, tangents, a failed experiment, listening, first-hand experience, and finally, something we were happy with.

Here's the story of Scout's evolution to roles.

Oct 2007: Before AWS

Scout started as an internal tool at Highgroove Studios (now Big Nerd Ranch) in 2007, or, roughly one year before AWS exited Beta status. For you young chicks out there, this was a time when you couldn't click a button to provision a server.

Since it wasn't as easy to provision servers, there was less churn in the size of environments. When you wanted to monitor a new server in Scout, you'd create it in our UI and then use the provided locally in your Crontab entry. The manual step of copying the key to to the server didn't feel tedious (and was way easier than configuring Nagios, Munin, etc) since our customers weren't provisioning servers frequently.

Oct 2008: AWS Exits Beta

Using a unique key per-monitored server didn't align well with EC2. As our customers encountered this, they initially requested that we provide an API method to provision a new server and its associated key. This would be run as part of the instance provisioning process.

However, we weren't big fans of this: it's vital that monitoring works. If the script to initiate a Scout server failed during the startup sequence, the server wouldn't be monitored. Everything from a temporary network hiccup to a missing gem dependency could cause monitoring to fail.

Jan 2009: The emergence of Chef and Configuration Management Tools

The release of Chef in 2009 rapidly accelerated the configuration management world for our customers. While the overall adoption between Puppet and Chef appears to be fairly equal, we received far more inquiries about configuring Scout with Chef.

Sep 2009: Cloud Images

EC2 made it much easier to deploy more servers and create redundant setups. This meant our customers began to have groups of similar servers ( web servers, database servers, etc) that have the same monitoring configuration.

What if you could re-use an existing server's key in Scout on new servers? Then, those new servers would use all of the same settings as the original server. This was a much better approach than creating servers via the Scout API as well: it's a Crontab entry that runs every minute, so if for some reason the first Scout run fails, no big deal: it'll keep trying.

We launched cloud monitoring in September 2009.

Consistency

A problem soon emerged: when a new server reported with a cloud key, we copied the plugins from the original server, but we didn't keep things consistent (ex: adding a new plugin to the original server didn't add it to the other servers).

Why?

Customers would often use the cloud key as a base template - it monitored key metrics, then they would add additional monitoring plugins as they saw fit.
Inevitably, some servers end up needing slight modifications from their base image (ex: testing out Memcached on a single app server).

In short, taking the step of making everything consistent would be a big jump. We needed more time to see how cloud images were used.

Jan 2010: A Tangent - Copy and Paste

We punted on the consistentcy issue, but we introduced a tool to make it easy to copy one server's settings to another: Copy and Paste launched in January 2010. It allowed you to copy a plugin (and all of its settings) to any number of servers. It made it easier to keep things consistent, but required a manual step.

Sep 2011: Failed Attempt - Synced Groups

Our first attempt at making things consistent? Synced groups: define a group of servers (ex: web servers), pick one server to act as the template server. All other servers in the group will use that server's configuration.

We started working on this in the fall of 2011. However, issues soon emerged:

Too ambitious - We envisioned representing these synced groups differently in the UI. In addition to the standard server views, there would be additional views to represent the aggregate synced group. We've found UI issues to be the longest to rollout - it takes a while to get the experience right.
Overlapping terms - We already had groups - so we thought about calling these "synced groups" or "clusters". We were hesitate to add more Scout-specific terminology. We like clean, easy-to-use tools and adding more terms heads in the opposite direction.
Servers that should belong to multiple groups - For example, a single app server may also perform some utility functions.
One-off changes - For example, testing Memcached on a single app server.

Apr 2013: Roles - A common language

We started using configuration management tools vs. hand-rolled scripts to deploy servers at Scout in the summer of 2011. As our first-hand experience with configuration management tools increased, our ability to make editorial decisions on how syncing should be done was strengthened.

Our first commit to roles was in late September 2012. We put our first customer on roles in November 2012 and opened it up everyone in the begining of April 2013.

Apr 2013: Chef Recipe + Install Instructions

We wanted to make it super-easy to use Chef with Scout, so inspired by Boundary, we added Chef-specifc instructions for Scout when adding a new server.

The original Scout Chef recipe was written by Drew Blas. We forked it on Github and updated it w/role support.

Tl;dr

We realized keeping monitoring settings in sync across servers would be a big win a couple of years ago.
We didn't want to make the customer experience complicated when syncing was enabled.
It took (1) our own experience with configuration management tools (2) regular customer feedback to make something easy-to-use yet powerful.

Debugging request bottlenecks with realtime charts

Tue, 14 May 2013 15:19:00 EDT

Last week, one of our application servers died. We have four app servers, so in theory, the death of one app server shouldn't bring the entire platoon down. However, real-life had other plans: 95% of requests were handled fine, but around 5% were being dropped. Here's the story of how we diagnosed and fixed the issue with our realtime charts.

The Problem

Our web stack looks like this: A web server running HAProxy (using Apache to process HTTPS requests), which then uses round-robin load balancing to send requests to four app servers running Apache + Passenger.

One of our customers reported seeing sporadic "503 Service Temporarily Unavailable" errors in their web browser. From experience, we know this typically means HAProxy is unable to handoff a request to an app server.

1. How frequent are 503 errors?

We scanned our load balancer access logs and checked for the number of 503 errors. There were a lot of 503s:

grep 503 /var/log/apache2/access.log | wc -l
BIG NUMBER

Not good. Next, a quick sanity check. Did we get this many 503s before the app server went out of rotation?

grep 503 /var/log/apache2/access.log.1 | wc -l
SMALL NUMBER

Thankfully, no. This was a pretty good sign the 503s were tied to a bottleneck with the loss of the app server.

2. Can we identify the bottleneck by watching request activity in realtime?

I fired up a realtime chart session, fetching the following metrics from each app server:

Apache Available Workers
Passenger Global Queue Depth

Passenger was showing a rise in the Global Queue Length and Apache Available Workers were dropping to zero. Because the global queue depth was so high, we knew the bottleneck was at the application server (Passenger) and not the web server (Apache).

3. When is the queue spiking?

The chart showed that the vast majority of requests were bunched between 20-30 seconds. Why?

Originally, every Scout agent would checkin on the minute. This resulted in an inefficent request profile: virtually all requests to scoutapp.com occurred between 1-7 seconds. We then added a modifiable variable sleep - later agents reported to scoutapp.com between 15-30 seconds. This spread out the request volume (older agents report close to on the minute, new versions a bit later).

Today, almost all agents are checking in w/an agent version that supports the sleep interval. This means the bunching just moved to later time period. The fix: re-balance the checkins. We adjusted the sleep interval on the agents to be between 0-30 seconds, not just 15-30 seconds.

4. Did it work?

We simulated a dead app server by removing one from our HAProxy configuration and fired up a realtime chart session. With the request distribution spread fairly evenly from 0-30 seconds, there were zero 503s:

Nice. Finally, we added a trigger to alert us in the future when the rate of 503 errors dramatically increases.

RailsConf 2013!

Sun, 28 Apr 2013 18:20:00 EDT

I’m sitting in the Denver Airport – in a couple of minutes, I’ll board the plane to RailsConf in Portland, Oregon. I’m already getting amped for Voodoo Donuts, Stumpdown Coffee, well-trimmed beards, and of-course, lots of Rails-related chats.

I’m bringing a fresh load of Scout T-Shirts. These aren’t your normal heavy-weight, poor-fitting shirts. They are tastefully designed, American Apparel – Tri-Blend (otherwise known as the most comfortable shirt you’ll own). If you’re attending RailsConf, shoot us an email so we can meetup and improve your wordrobe at the same time. Or, just look for us (Andre and Derek). We don’t always rest our arms on each other, but when we do, we look like this:

Three weeks with roles: by the numbers

Tue, 23 Apr 2013 16:42:00 EDT

It's been three weeks since the launch of the largest feature enhancement in Scout's existence: roles. Haven't heard of roles? Nutshell: roles let you monitor many serves with fewer clicks and more joy. Roles were driven by your feedback and it's showing in the fast adoption numbers below.

Time to give an awkward nerd high-five of thanks:

Customers on our Roles BETA program - your feedback and willingness to try new things helped us iron out the edges for the public rollout.
Contributors to our Chef recipe - we've already had six authors commit to our Chef recipe for deploying Scout. It's great to see a hardened Chef recipe based on real-world usage.
Feedback since the launch - we built roles because of your feedback, and we've enjoyed reading your suggestions post-launch.

Haven't tried roles yet? To get started, see the "Roles" dropdown on your account, and read the FAQ on roles.

Server Roles: Fewer Clicks, More Joy

Tue, 2 Apr 2013 09:50:00 EDT

Roles are a new feature available immediately for all new and existing accounts.

You have a carefully thought out architecture. You frequently add new servers as your business grows. In fact, scaling up is part of business as usual. Monitoring should scale easily with you -- that's why we're introducing roles.

Roles make it easy to setup plugins and triggers across many servers. Instead of individually configuring servers, configure roles. Then, apply roles to your servers through our UI, the command line, or your configuration management tool.

Some examples of how roles will make your life easier:

Updating a trigger on 50 app servers
Adding a Memory Usage plugin on 100 memcached servers
Updating a plugin to a new version across all 10 MongoDB servers

Getting started

Roles are available now on your account. Look for the new "Roles" item on the top navigation. If you previously had servers organized by groups, your groups have been upgraded to roles. See documentation here on creating roles and organizing your existing plugins into roles.

Your account now has a single, account-wide key -- use it for any new servers you add. Your existing keys will continue to work, so you don't have to touch any servers you're currently monitoring.

Why Roles?

Most setups have a limited number of server configurations (app, db, utility, for example), and several servers of each configuration. When you add another app server, it probably needs the same monitoring template as your existing app servers. Adding more servers using existing templates is the scenario we wanted to make dead simple in Scout.

There's no need to stick to one template at a time: servers can have any number of roles in Scout, so feel free to mix and combine roles as needed to reflect the functionality of your servers. Is one of your HAProxy boxes also running memcached? No need to create a brand new roles, just apply two of the roles you already have.

Once defined, roles are "active": if you update a role (say by adding a plugin or a trigger), all the servers in that role are automatically updated to reflect the changes. It's a much easier way to to manage your monitoring configuration.

Best friends with Chef

We provide an official Chef Recipe designed to work with roles.

To simplify deployment, Scout now provides a single, account-wide key you can use on all your servers.

Even if you're not using Chef, you can (optionally) specify roles directly through the Scout executable: scout -rdb,app to assign the db and app roles, for example. This makes role assignments highly script-able, whether you're using Chef, Puppet, or Moonshine.

Fine-tuned for large environments

With the recent notification group changes and now roles, we're making monitoring easier for large environments. Our previous tools -- cloud keys and plugin copy-paste -- were useful, but it was easy for things to get out of sync. Roles our our answer for keeping monitoring in sync in large environments.

Roles in Summary

With Roles, we want to make deploying and scaling scout on large environments as easy as possible:

Roles are "active": updating role's triggers or plugins propagates to all the role's servers.
Better than templates: Servers can belong to more than one role.
Account-wide keys: no need to provision keys for new servers - reuse the same 40-character account key in the crontab across all your servers
Specify roles via the crontab: optionally, you can pass a command-line argument to the scout agent to specify the roles it should belong to.
Friendly with Chef: we also provide an official chef recipe for roles-enabled server configuration.

To get started, see the "Roles" dropdown on your account, and read the FAQ on roles here.

Notification Schedules and PagerDuty

Wed, 27 Feb 2013 09:37:00 EST

Whenever we’re asked how to make on-call notification schedules for Scout alerts, we recommend PagerDuty. PagerDuty has invested a ton of time in building a dedicated notification scheduling service, and it’s a great complement to Scout.

With our recent release of notification groups, Scout’s integration with PagerDuty got even more powerful:

Multiple PagerDuty services: add as many PagerDuty services to Scout as necessary.
Trigger-specific escalation policies: assign any PagerDuty escalation policy to any threshold in Scout. If you need to create multiple thresholds on a given metric with different escalation policies, it’s simple to do – just add another trigger.
Automatic incident resolution from Scout: since all integrations are routed through PagerDuty’s API, Scout now auto-resolves any PagerDuty incidents when Scout’s trigger stops firing.

Multiple services in PagerDuty:

... and those same services integrated into Scout:

Adding PagerDuty services within Scout

You need to start in Scout to create a PagerDuty integration:

Click on Notifications (in the top navigation bar),
Click on “Add PagerDuty Integration.”
You’ll be given the option to create a new PagerDuty service, or connect to an existing service within your account.

To assign a PagerDuty service to a trigger, ensure the PagerDuty integration is part of a notification group (the notification group can contain other items too, if needed), then assign that notification group to a trigger.

Sidekiq and Puppet Monitoring Plugins = 70+ plugins strong!

Thu, 14 Feb 2013 22:29:00 EST

With the Sidekiq Monitor Plugin (by Scott Klein of StatusPage.io) and the Puppet Last Run Plugin (by Didip Kerabat of Kongregrate) Scout’s plugin directory count has now passed 70 plugins!

The Sidekiq Monitor Plugin monitors key metrics for Sidekiq, a Ruby message processing library. Didip’s Puppet Last Run Plugin tracks key metrics for the most recent Puppet run on a monitored server.

Have a useful plugin sitting around? Share it! Send a pull request to our scout-plugins repository on Github.

The reward of redundancy: time

Thu, 31 Jan 2013 16:39:00 EST

It’s been a month since I started attaching torture devices disguised as boots to my feet, long wooden sticks to each torture device, and tumbling down mountains. Skiing has changed my outlook on winter. It’s a season to enjoy, not a time where I gaze wistfully out the window, hoping the short, cold days pass by as quickly as possible.

However, there’s a problem when skiing becomes a favorite hobby: not everyday is a great day on the mountain. If it hasn’t snowed in a while, the surface is hard. The temperature might be in the single digits and the wind may be gusting 50 MPH+. It might dump snow in the backcountry, but the avalanche conditions may make it unsafe.

There’s something special about being able to sneak away when the conditions are the best, even if it’s during the work week. It feels a bit like being a kid again (correction: a kid with a receding hairline). It’s a fun reminder that it’s not always bad to feel redundant.

Our DIY Homepage Process

Mon, 21 Jan 2013 11:44:00 EST

We recently decided it was time for a major update to the public side of Scout. We’d start with a more polished homepage. Since we’re both developers, the obvious next step seemed like hiring a designer. However, working with an outside designer isn’t a hire-and-forget experience:

Good designers are difficult to find. Design doesn’t scale like a product business.
Good designers are busy. It could take 30-60 days to start work, then another 30 days for it to come together. This means we could be looking at a 90 day timeline. We wanted to launch it faster.

Instead of starting work with a designer on a blank slate, we decided to start firming up what we wanted the homepage to look like. We’d end up with one of the following outcomes:

We’re terrible at design, but we’ve at least thought it through. Hire a designer.
We can get 80% of the way there, but we’ll need a designer for touchups.
If we iterate enough, we can launch something we’ll be happy with.

Structure

We had a rough idea of what we wanted for the structure of the site:

Tagline and overview – In as few of words as possible, give an overview of Scout. Highlight Scout’s sweet spot.
Social Proof – Our lightweight approach to monitoring can scare off larger teams that think Scout won’t meet their needs. Show this isn’t the case: large teams love using Scout.
Feature highlights
7 days on Scout – Highlight how Scout fits into a developer’s day-to-day life, similar to the Living with Nest section on nest.com.

Inspiration

In the past, finding design inspiration has been a laborious task. Blog posts that highlight good design often don’t cover products. This time, I used Dribbble, creating a Public Site Inspiration Bucket. I searched Dribble with terms like “homepage”, “minimal”, and “landingpage”.

Showing up for work

I was leading the design on this and warned Andre not to expect any works of art in the beginning. Instead, my plan was to simply show up for work: work on design a bit every day, valuing rapid experiments and lots of mockups. The first week was rough – it started out a bit like this:

As we focused in on a style we liked, I started applying more polish and things came together. In all, I created 24 iterations of the homepage. Iterations often had several alternates as well.

About half-way through, it felt like we could probably do this ourselves. It took 30 days (including a break over the holidays) to launch.

Copy

I admire product sites with solid copy – it’s especially hard to find with technical products. Early-on in our process, I replaced placeholder text with actual copy. This gave us lots of time to firm up both the design and the text on the page.

Tagline – “Neckbeard Optional”.

Products like Scout that emphasis simplicity often strike a negative or snobbish-tone with their tagline:

“FOO done right”
“FOO that doesn’t suck”

I wanted to avoid this. I assembled 30 possible taglines for Scout and the term “neckbeard” consistently floated to the top, because, obviously, neckbeards are the awesome:

The general public may not know what we’re referencing, but a developer visiting our homepage probably does. My high school English teacher always emphasized “show, don’t tell”. Referencing a neckbeard shows how every part of Scout is really written by (and for) developers, including the marketing copy.

Decreasing the scope

About halfway through the process, we decided to cut the “7 days on Scout” section. We love the idea, but doing it right would require a significant amount of time. There’s no reason we can’t add it to our homepage later.

Browser Testing with Browserstack

Since Scout doesn’t support Windows, an incredibly small percentage of our visits come from Internet Explorer. However, our homepage attracts a more general audience, so we wanted to make sure it looked solid in IE as well. Instead of dealing with something slow and complicated on my Mac, I just used Browserstack. Browserstack is great: tons of browsers (way more than I’d ever install locally) and support for SSH tunneling straight to my laptop.

Summary

We were able to put together a design we liked by:

Using Dribble to quickly assemble a bucket of designs we liked.
Working on the homepage design every day. Be OK with ugly things in the beginning.

Remember, the worst case DIY design route: you form a much clearer picture of what you’re looking for.