Eric Ries, Startup Lessons Learned, popularized the actionable metrics meme for making product and design decisions iterative and more fact-based versus faith-based. Instead of endlessly debating a better checkout flow on the whiteboard, go set up an experiment and test it.

Eric further describes the 3 AAAs of metrics for properly setting up experiments for success:

Actionable
An Actionable metric ties specific, repeatable actions to observed results. The key here is picking the right “macro” metric to track. So in the checkout experiment above, rather than tracking button clicks, you should be tracking revenue.

Accessible
In order to keep the process objective, the results should be available for everyone on the team to see and in a format that is simple to understand e.g. a conversion funnel.

Auditable
People usually have high emotional stakes in the outcome of an experiment and some might even outright refute the data using the ever so popular statistical significance argument. In order to minimize on the drama, it helps to tie the metrics back to actual users so they can be audited if needed.

This last point of “Metrics are People too” is the one I want to build upon here. I don’t believe it goes far enough. While I am a big proponent of building a metrics driven culture, there is a lot more to building a great product than numbers – you have to go to the people behind the numbers.

Metrics can’t explain themselves

When you first launch a product or new feature, lots of things can and do go wrong. Metrics help identify where things are going wrong, but they can’t tell you why.

My last product was a downloadable desktop application. My conversion funnel showed a huge drop off between signup and first login. I knew I was loosing people after the download but didn’t quite know why. This led to a series of hypotheses that took us weeks to build and test: We tried different install methods, we reduced the download size, we added lots of sub-funnel events to get better visibility into exactly where users were dropping off.

Then we tried something different. We re-ordered the sign-up flow and started collecting an email address before the download. This allowed us to reach out to our users with an offer for help when things went wrong. Surprisingly, we started getting some responses back that focussed our efforts enough to nix our user activation problems.

We learned a couple things here:

User Motivation Matters

While users were motivated enough to try a new product, they weren’t invested enough (yet) to put up with it. Despite having our contact information (including a 1-800 number) plastered on every page, users weren’t reporting these issues on their own.

User Motivation Decays Fast

There was a critical time window for reaching out that yielded the highest response rates. After that window, the user had moved on and we never heard back from them.

Not all metrics are equal

Do you really know who your users are? When you just look at numbers, you get an averaging effect that can be greatly skewed if you don’t yet have a lot of traffic (or the right traffic). Behind these numbers are actual people with very different personas and motivations which need to be taken into account. A busy mom interacts with a service very differently from a teenager. Andrew Chen frequently references the IDEO human-centric design process on his blog and Alan Cooper talks a lot about the importance of personas in design here.

Services like FlowTown and RapLeaf let you build rich user personas by mapping email addresses to public social networks. Segmenting metrics into buckets this way made it possible for us to better identify and service the right “types” of users.

Validate in 2 phases – First Qualitative, Then Quantitative

The biggest issue you run into when relying solely on quantitative metrics is that you need a lot of data to call a result and there is sometimes a danger of calling it too soon: How we improved signups by 30% by doing nothing.

Validate Qualitatively, Verify Quantitatively

When you first launch a product you don’t have enough traffic to afford waiting for statistically significant results. The good news is that you typically don’t need a large sample to qualitatively test a hypotheses using usability testing or customer interviews.

Your initial goal is getting a strong signal – positive or negative. A strong negative signal indicates that your hypothesis most likely won’t work and lets you quickly abandon or refine it. On the other hand, a strong positive signal also doesn’t necessarily mean it will scale up to statistical significance either. But it gives you permission to move forward on the hypothesis until it can be verified later through quantitative data.

This is best illustrated with an example:

Variant 1:

Variant 2:

We setup an A/B split test against these 2 variations in Google Website optimizer and ran customer interviews with our target group (moms) in parallel . We interviewed 7 moms over 2 days and all of them had a strong resonance with the second variation for very similar reasons. By the end of the first week, the A/B test was at an 87% probability in favor of Variant 2 but still didn’t have enough data to call a winner conclusively. In this case, the qualitative testing signal was so strong that we didn’t need to wait to pick our own winner.

You learn more in one 15 minute customer interview than you do ploughing through a sea of numbers.

Metrics are People First

While I am a big proponent of metrics and use them quite heavily, there is a lot more opportunity for learning if you also make it a point to go to the people behind the numbers.

I love metrics, but I love users more.

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.

The ultimate goal of any new product launch is building something people want. But you can’t do that unless you really understand your users.

I’ve been a long time fan of Kathy Sierra’s work on Creating Passionate Users.
This is one of my favorite diagrams:

The “Kick Ass Curve” shows how your users progress through your product’s adoption curve. Your job is getting users as quickly as possible from the “suck threshold” where they find your product frustrating to the “passion threshold” where they feel like experts.

When you first launch a product lots of things can and do go wrong. Many of these things aren’t even about what we traditionally think of as “the product”. For instance, you might be loosing people due to poor positioning, design, or pricing.

Well guess what – design, positioning, and pricing are all parts of the product.

But tackling the whole product all at once can be overwhelming. This is where an understanding of the User Lifecycle is helpful in prioritizing what you focus on first and build your product in an iterative manner.

What is the User Lifecycle?

Every product has a user lifecycle. For a web-based product or service, the user lifecycle describes the path a user takes from first landing on your website, to signing-up and using your product, to eventually becoming a passionate user.

Joshua Porter (Bokardo) describes this flow as the “Usage Lifecycle” on his blog.

The Usage Lifecycle describes how far a person has progressed in using your web application, helping to identify the hurdles someone needs to overcome to become regular, passionate users.

He uses the “Usage Lifecycle” model to highlight practical design approaches to overcoming usability obstacles which he outlines in his book: “Designing for the Social Web”. I particularly like Joshua’s model because it describes the conversion funnel from a “user’s perspective” and captures the psychological hurdles between each stage.

The one shortcoming of the “Usage Lifecycle” is that it doesn’t tell you how to measure each stage. For instance, how do you measure passionate use?

To answer that, it’s helpful to consider another model that is based on actionable metrics – Dave McClure’s AARRR Model:

Both models are quite complementary and the User Lifecycle model we’ll be covering in this blog is a synthesis of both. While Joshua focusses on user motivation, Dave focusses on metrics. The stages up until “Regular Use” / “Retention” match up very closely.

So what do passionate users do? In Dave’s model, passionate users tell others about your product (Referral) and get you paid (Revenue).

The final step to realizing the User Lifecycle for your application is mapping specific user actions (or events) to each stage.

Here’s an example:

An understanding of your User Lifecycle is not only useful for prioritizing how you tackle design and positioning (which we’ll cover in future posts), but it’s also invaluable in segmenting your users based on the stage they are in and targeting them differently which is the basis for Lifecycle Marketing.

What is Lifecycle Marketing?

Lifecycle Marketing is targeting your users with differentiated messaging based on where they are in the lifecycle.

In this post, Josh Kopelman from FirstRound Capital describes several examples of how he applied lifecycle marketing at his company Half.com.

Here’s an excerpt from one of his examples:

“The average fiction book is read within two weeks of purchase. So if you purchased a John Grisham book for $8.75 on Half.com, chances are that you will finish it within 14 days. We decided to implement an auto-email that was sent 17 days after purchase that said “Want your $8.75 back, click here to list your Grisham book for sale”. We found that the open (and conversion) rate of that email was amazing — and it greatly added to our ability to “turn” the same book multiple times.”

Lifecycle Marketing typically kicks in once you can identify a user in your system (e.g. with an email address) which is usually after the sign-up process. Before that other forms of marketing such as outbound and inbound marketing are more effective at driving traffic and conversions.

Right Message, Right Audience, Right Time

The core premise of Lifecycle marketing is delivering the right message to the right set of users at the right time. When you first launch a product, lots of things can and do go wrong and your initial objective is reducing user abandonment i.e. getting them over the suck threshold. Over time, that focus shifts towards increasing retention and engagement which gets your users into the passion threshold which is the key to building a successful product.

Lifecycle Marketing isn’t a new concept and has been practiced as a marketing discipline in several industries (both online and offline) for some time now. But for web-based products, beyond some simple applications, it’s been too hard and expensive to take full advantage of these concepts. This has mostly been because it’s hard to track and segment users over time (into cohorts) especially since the product itself is also changing continuously. Only recently, have the technology, tools, and knowledge evolved to a point where advanced forms of Lifecycle Marketing are now practical.

In this blog, we’ll cover techniques for applying Lifecycle Marketing to your products using social media, email marketing, and metrics.

Thanks for reading…

If you enjoyed this, please consider subscribing for updates:
Get Email Updates or Subscribe to RSS

No related posts.

Related posts brought to you by Yet Another Related Posts Plugin.