Scout ~ The Blog

Running a freemium web app? Here's a big reason we're growing.

Wed, 4 Nov 2009 13:18:00 EST

Before Scout, my experience developing software was primarily consulting. Success was measured by delivering software on time and on budget.

With Scout, a subscription-based service, my focus isn’t on scheduling. We are self-funded and we didn’t have the luxury of a venture-backed startup. We’re focused on figuring out which pieces of development work can increase revenue the most. What follows is how we’re approaching it.

The 1% paid conversion rule

If there’s a golden rule of a subscription-type service like Scout, it’s this: 1% of visitors will signup for a paying subscription. This 1% rule exposes itself across industries, price point, and size.

The 1% rule is the Pi of freemium web apps.

It’s very easy to think you can improve this because 1% is such a small number, but your business is the rare exception if your paid conversion rate dramatically exceeds 1%. If your paid conversion rate is around 1%, you need to dig somewhere else for treasure.

As our signup rate is around 1%, what have we focused on?

Retention & Lifetime Value

Increasing the rate that customers renew their subscriptions (retention rate) can have a dramatic impact on revenue. For example, increasing retention by 1/2 results in the following revenue per/month:

If you sum the periods, you’ll see that revenue actually doubles with a 50% increase in the retention rate:

To determine the revenue, I’m using an equation for Lifetime Value, which is the revenue expected from a customer over their lifetime. Every period (in our case, 1 month), decreases the chances that they’ll renew. R represents the retention rate and rev is the revenue per/period in the equation below:

LTV = rev + rev*R + rev*R^2 + rev*R3 + …

This can be simplified to:

LTV = 1/(1-R) * rev

The retention rate can be calculated using the equation below:

R = 1-(# of canceled accounts in a given period / # of accounts in a given period)

So, is it as difficult to increase retention as it is increase the signup rate?

Good news – there is no rule for retention rate

While there is a rule for paid signup conversions, you won’t find one for retention rate. Retention rates vary widely and are a great target for increasing revenue.

Focusing on churn also makes sense in another way: increasing retention by 50% is the same as doubling your signup rate. If your paid conversion rate is already around 1% and you think you can double it, you might be a bit crazy.

I’ve had good success increasing retention by:

Closely monitoring application performance. As you’d expect, we use Scout for this.
Getting feedback from users when they cancel.
Asking users for feedback after they’ve setup their account.
Aggregating and prioritizing user feedback. We use UserVoice.
Tracking user behavior. We use Google Analytics.

Focusing on retention rate isn’t glamourous work, but for many subscription-based web apps, it may hold the most untapped potential to increase revenue.

A higher retention rate can have a dramatic impact on signups too

You can double your revenue if you decrease your churn by half. That is a substantial number – it means you could also charge customers the same amount, but double your spending on acquiring customers and have the same income.

Increasing your retention rate gives you room to breath.

Also See

Andrew Chen’s excellent blog post, How to create a profitable Freemium startup. This was my original inspiration.
Paul Graham’s post on What Startups Are Really Like
Our previous blog post, We Just Undid Three Months of Dev work. Here’s What We Learned.

We’ll continue to talk about our experiences growing Scout. Subscribe to our RSS feed or follow us on Twitter.

Bulk Plugin Management — Your Input Wanted

Tue, 3 Nov 2009 19:01:00 EST

Some of our customers are monitoring a lot of servers with Scout. We’re psyched about this, and we’re looking at ways to make Scout even easier to use.

Bulk plugin management is a topic that comes up pretty often.

So we’re looking at the ability to make changes en masse to your servers:

add / delete a plugin on all servers in one shot
add / delete a trigger to a plugin across all servers
clone all plugins and triggers from one server to another

We’d love your feedback. How do you want to be able to bulk update your plugins? Feel free to leave feedback in comments.

Rails Monitoring Improvement: Ignore Specific Actions

Mon, 2 Nov 2009 11:31:00 EST

Sometimes you have actions in your Rails app that you know take a long time. Getting alerts on these actions is just noise.

With the updated Rails Monitoring Plugin, you can filter out any requests on which you don't want to be notified. You supply a regular expression, so you make as simple or complicated as you need to.

Update your Rails plugin

If you already have the Rails plugin installed, you need to update it. Go to plugin->code and click Update:

Then go to plugin->edit and click Update Options:

The new goodness is under "advanced options":

Ignoring Actions

You provide a regular expression for actions you want don't want to be alerted on when they're slow. In the simplest case, don't even worry about it being a regular expression -- just provide a string you want to match. For example, if you don't want to be alerted to any slow actions with admin in the URI, just put admin in the Ignored Actions field.

More Complicated Matches

To ignore all actions under admin and also accounts/new, the regex is(admin|accounts\/new). If you wanted to make sure admin only matches paths starting with admin, just match the beginning slash: (\/admin|accounts\/new)

If you're building a complicated regex, try it out separately to ensure it matches/doesn't match what you expect. I dig on Rubular for quick regex sanity checks. Of course, the plugin will tell you if your regex has a problem, but you'll get faster feedback by running it through Rubular.

Notes

Note that the match is case insensitive -- no need to worry about case.

Finally, note that excluded actions will still be analyzed in the daily Rails Analysis reports, so you'll still get metrics on them -- you just won't get email notifications for actions you already know are slow!

Rails + Google Analytics = easy goal tracking

Tue, 27 Oct 2009 11:34:00 EDT

Google Analytics is an indispensable tool as you optimize the business side of your operation. If you haven't already set up goals in Analytics for viewing your pricing information, accessing the sign-up form, and signing up for an account -- go do it! It's vital information.

However, Google Analytics' goals have to be attached to a specific URL. What if there is no URL for an important goal? For example, the New Account goal for Scout is just the account/show page -- there's no specific URL to represent a newly created account.

flash[:analytics] to the rescue

For this situation, we found a nice clean solution with Rails' flash helper + a touch of Javascript.

Rails' flash helper is exactly the same one you use to display errors and notices. But you can pass anything in a flash. In our Accounts_controller#create action, we call:

flash[:analytics] = @account.free? ? "/goals/free" : "/goals/paid"

Then, in our layout, application.html.erb, within the same <script> tag in which we include Google Analytics:

<%if flash[:analytics]%>
pageTracker._trackPageview("<%=flash[:analytics]%>");
<%end%>

It's as simple as that! We can register arbitrary goals in any action by setting flash[:analytics]. Within Google Analytics, just set the goals with the same names you've given the flash:

Key takeaways

Google Analytics is essential
you can register arbitrary URL's via Google Analytics' _trackPageview() call
Rails' flash helpers make it easy and clean to register analytics for any action in your app

Major chart updates - variability, overlays, scaling & stacking

Wed, 21 Oct 2009 15:07:00 EDT

When debugging performance problems, visualizing server metrics in a variety of ways is a critical part of isolating the cause:

Visualizing variance
Overlaying metrics to identify correlations
Scaling to compare several metrics with different units
Stacking graphs to visualize distributed setups

We’ve just released a major update to Scout’s charting functionality that makes it easier to analyze your metrics.

Variability

The average value of a metric only tells part of the story – where does the average fall in relation to the min & max? How volatile is the metric? Scout now shows the highs & lows for each time period when graphing via a shaded vertical column.

Overlays

What’s the correlation between database load and web requests? Or how about average request and and memory usage? It’s easy to compare different metrics with overlays.

Scaling

Have a complicated setup and need to compare several metrics with different units? No problem – Scout can scale each metric so you can compare them on a single graph.

Stacked Graphs

Stacked graphs help you get a complete picture of your distributed setup. Receiving requests across multiple web servers? Have several database servers? See how it all adds up with stacked graphs.

Accessing Charts

Just click on a metric when viewing a server in Scout and you’ll be directed to the charts page using the selected metric. From there, mix and match your metrics.

30-Day Free Trial

Want to try out charts? We offer a 30-day free trial on all of our plans. Sign up for Scout.

Part II: We Just Undid Three Months of Dev work. Here's What We Learned.

Tue, 20 Oct 2009 09:11:00 EDT

Two weeks ago I covered some of the business lessons learned from a large (~3 months) investment in new features, and the hard decision to roll them back. I discussed how you will underestimate the ongoing cost of complexity in your product, and how cool new capabilities don’t sell themselves.

Continuing this week—more insights gained from undoing development work.

Lesson #3: The best features are ones that support better business decisions

In cost-benefit terms, there are some clear winners in our recent feature development: the winners are the features that support better business decisions and enable more business development. For us, this meant:

front-and-center reports of signups, plan upgrade/downgrades, and cancellations. This data was always there, but making it accessible in a hassle-free way was a huge win for us. You will naturally optimize around what is easily accessible and visible.
an affiliate program and corresponding administrative functionality, integrated with our signup system. This enabled us to pursue a whole new class of business relationships and revenue opportunities.
feedback collection from users who cancel. A surprising percentage of canceling accounts will give you honest feedback on why they are quitting. The subjective feedback goes a long way in coloring your signup metrics.

Lesson #4: There is no substitute for talking to your users.

We launched a major development initiative based on our internal ideas of what users wanted. In retrospect, we should have spent much more time speaking with customers and vetting the ideas. The closer you get to your product, the easier it is to fall into this trap. Two examples of thinking that lead us to assume too much:

A. Extrapolating feature needs from a competitor’s offering … “Competitor X does feature Y. We should do feature Y, since there’s obviously a demand for it.”

Well, maybe. Company X may already “own” that feature in customers’ minds. Or they may be losing money on the feature. The existence of a competitor in the space isn’t necessarily a good argument for entering that space yourself.

The lesson: do some more research before committing serious development resources.

B. Assuming users will invest in more complicated features.

We assumed that if we offered the capability to report structured data into Scout, plugin authors would create more sophisticated plugins. Signup numbers didn’t bear out this hypothesis. Why? Too much complexity. Building and debugging plugins to report complex, structured data wasn’t something you could easily pick up and see immediate results. It was too involved for all but the most dedicated plugin authors to utilize.

The lesson: complexity can change the game. If you’re asking a lot more of your users, do a gut check and have some more conversations.

Lesson #5: Be wary of “comfort work”

We all tend to see challenges in terms of our own skillset. If you’re a good developer, you will think of product solutions to growing your business. If you’re good at business development, you will see business development solutions. That’s not a bad thing, but part of the challenge of building a business is to expand the domain of solutions you are comfortable trying.

It’s easy to be biased toward development work because that’s what you know best.

So, are you getting a warm, comfortable feeling for the execution of the solution? Then do a quick sanity check. You may be guided by your comfort with the execution rather than how appropriate it is for your problem.

Liked this article? Subscribe to our RSS feed or follow us on Twitter

The MySQL MyISAM and InnoDB engines and a grocery checkout

Fri, 9 Oct 2009 16:15:00 EDT

There’s no shortage of resources comparing the MyISAM and InnoDB storage engines. You’ll quickly see it isn’t a black-and-white decision after reading through various discussions debating MyISAM and InnoDB.

Why is the decision so hard?

Setting up your database is one of the first steps when building a web application. You probably don’t have a good idea on the database activity at this point, so you may have little data to work with.
The ordering and number of statements can have a big impact on database performance. It’s difficult to simulate until you have real users.

However, there is a one case where choosing the wrong table type can be crippling.

The Grocery Checkout

Many web applications primarily issue READ queries – SELECT statements. Both the MyISAM and InnoDB table types can execute these concurrently. There isn’t a single queue – you’re checking out at the cash register with 10 possible lines. One long query won’t stop another from running:

UPDATE statements are another story

For the MyISAM engine, an UPDATE can only execute if the other checkout lines are empty. All of the other cashiers must stop working. This is table-level locking.

Normally, this isn’t a problem. However, what if there is long 10 second SELECT query, then an update, and then several other SELECT statements?

The additional SELECT statements must wait unit the UPDATE query finishes:

This means that the 3 queries after the UPDATE statement will take a minimum of 10 seconds to run. The execution time of these queries may be very small – perhaps 0.001 seconds, but it doesn’t matter – they have to wait for the UPDATE to finish, which is waiting for the long SELECT statement to finish.

InnoDB doesn’t have this restriction as it uses row-level locking – only the updated rows are locked. SELECT queries can continue to execute.

This seems like a fatal flaw. Why would MySQL make MyISAM the default storage engine?

In a well-optimized web application, the table type may not matter:

Most web applications primarily issue reads, not writes, so table locks don’t occur frequently.
There should be few (if any queries) that take a 1/2 second or more.

However, new web applications are anything but optimized. You’re more likely to have missing indexes that result in slow queries. You don’t know if you’ll be doing reporting, which often involves slow queries.

There are a lot of unknowns, so it’s often worthwhile to go with the safe choice. This can be a difficult problem to diagnose, and during the post-launch phase, you’ll have plenty of other things to worry about.

More on MySQL

I’ll be following up this post with a walk-thru of several tools for monitoring MySQL performance.

Subscribe to our RSS feed or follow us on Twitter for part 2.

First Impressions Count

Wed, 7 Oct 2009 15:09:00 EDT

Scout is making a better first impression than ever starting today. When you start monitoring a new server, you'll immediately get a high-level summary of the vital stats:

Scout reports this for you automatically. From there, you choose the deeper metrics you need, like Ruby on Rails monitoring, MySQL Slow Queries, Process memory usage, etc.

We Just Undid Three Months of Dev work. Here's What We Learned.

Tue, 6 Oct 2009 13:38:00 EDT

We’ve been deleting a lot of code from Scout. We’re ripping out major infrastructure, and in doing so, pulling the plug on functionality which, just six months ago, we believed would be crucial to our business. Most importantly, we’re simplifying the most complex, error-prone, and poorly-performing parts of the application. At the same time, our revenue and sales pipeline is growing at a faster rate.

How did this happen? How did we get to a place where we can remove code and functionality and see our business will grow because of it?

As they say, “mistakes were made.” You don’t get the satisfaction of throwing out a bunch of cruft and performance-degrading features without having gone through the pain of:

Building those features in the first place.
Fighting the performance problems for a few months before you realize its all untenable and come up with alternatives.

So yes, mistakes were made. But also, lessons were learned.

Lesson #1: You will underestimate the ongoing cost of complexity in your product

When you have a small team, complexity is a killer. But no-one sets out to make their product complex, right? That’s the irony: we grappled with heinous complexity in the name of simplicity—simplicity for the end user. It turns out that simplicity and elegance for the end user can mean some awfully complicated stuff behind the scenes to make it work.

In our case, the move from flat data to nested data was the killer.

Here’s the culprit:

We envisioned a very versatile data store and alerting system in which a broad variety of data could be stored, analyzed and reported on. This wasn’t a pipe dream or an abstract feature wish—it had immediate application to the kind of Ruby on Rails application reporting we were building. We came up with a sweet way of storing the nested data and abstracting away most of complexities of dealing all kinds of data.

However, the load on our database was far more than we envisioned. The performance problems that resulted from this dominated our efforts for the coming months. We upped our hardware budget. We fought the fires, and yes, we made it work.

But, we made it work with very little headroom. We were already spending more than we wanted to on hardware, and we didn’t have room to grow. The database operations on the nested data were just taking too much processing power. From a business perspective, these problems went from annoying to crippling. When the service is slow, you have to address it ASAP. That takes time and focus away from sales and business development, so it’s a double hit. We ended up spending nearly all our effort during the period after launch keeping that elegant and sophisticated system running.

The takeaway

The ongoing cost of complexity is higher than you think. And it’s easy for complexity to sneak in through the back door in the guise of its nemesis, simplicity. Complexity is sneaky that way. Don’t be fooled.

Lesson #2: Cool new capabilities don’t sell themselves

And more to the point, you can’t sell them if you have to spend all your time hand-holding an overloaded database. After a few months of our Cool New Features being live, we had certainly signed up some customers because of them. Not as many as we wanted to however. There are two reasons for this:

The features never reached their full potential. All that time on upkeep? That’s time taken away from incorporating user feedback, fixing annoying little bugs, and putting in that next 20% to make it all really shine. The vision we started with got tested in only superficial ways. What was clear though, is that we couldn’t get there from here without either a major increase in team size (which brings its own set of problems), or much, much more time to work out the kinks. We didn’t want to do either.
We didn’t have time to properly publicize and sell the features. This is key. We’re a lean team, and we all wear multiple hats. If we’re spending more time analyzing the performance of a particular MySQL index, we’re spending less time writing blog posts or forming partnership with other companies. The biggest hit we took during those months of building and stabilization was the corresponding decrease in sales and marketing.

Why you need sales and marketing

If you’re running a web application and the majority of your work is spent delivering the service you advertised to your customers, you’re probably in bad shape. A good rule of thumb for paid web apps: 1 out of every 100 visitors will become a paying customer. With almost 1/2 of all web applications attracting less than 1,000 visitors per/month, it’s reasonable to expect just 10 paid signups per-month. You need to do more than just deliver the web app to make a living.

The takeway

It’s a huge net-negative to build something so resource-intensive that you don’t have time to sell it. You’re much better off spending more time selling a less expansive product.

That’s it … two lessons?

This is part 1 of 2. Subscribe to our RSS feed or follow us on Twitter for part 2.

Simplify. Get an order of magnitude speedup.

Thu, 1 Oct 2009 11:39:00 EDT

Have you noticed Scout feels snappier lately? We made some major simplifications that sped things up a lot. Here’s the CPU load on one of our DB servers:

(and yes, we use Scout to monitor itself!)

Even better, the response time for users improved dramatically.

Scout’s longest actions before and after the speedup:

The simplification

What yielded such a dramatic speedup? Earlier this year, we implemented a very generic datastore and reporting system. It could handle all sorts of data, relationships within the data, etc.

Unfortunately, we never got to demonstrate all the benefits of this cool system. It wasn’t viable from either a maintenance or a performance standpoint.

So we rolled it back. And we got back a ton of performance, as you can see.

The lessons …

I will be writing up some business lessons we learned from this experience—stay tuned!