The multiplicative nature of retention
Retention metrics, like viral marketing, have a powerful multiplicative approach that makes them important to optimize. This is an often-overlooked metric that is easily sacrificed to the Great God of Viral Traffic I wanted to dig into this issue a bit more, and give my thoughts on how your retention numbers can be an incredibly powerful method of driving up total traffic numbers to your site, not just the user acquisition piece.

Matt Humphrey from Bumba Labs on user retention curves
A frequent collaborator of mine, Matt Humphrey, has often referenced some of these ideas in a public posting below, which I’ll excerpt. I’ve worked with Matt closely for over a year and a half now, and know a lot about his skillsets, background, and interests. His background is influenced by his time as a Carnegie Mellon alum, YCombinator entrepreneur, and working on quantitative marketing projects with me.

In my conversations with Matt, we often discussed the metric of “user retention rate” which tells you what % of users are revisiting after some period of time. So you might define 50% user retention as, 50% of the users on your site come back after a month, or what % come back after an initial signup, or whatever suits your particular business.

What’s amazing is the multiplicative aspect of this number:

[...] Having month-to-month user retention of 92%, 96%, and 97.3% will get you on average 1, 2, and 3 user-years respectively per user that ever signs up on the site.

Okay, in English? If each month you lose 8% of your existing users (92% retention) from the previous month, the average use will stay for 12 months. If you can hold just 4% more of your users (96% retention), then they will stick around for 2 years. If you can hold only 1.3% more than that (97.3% retention), they will be in for 3 years.

It’s easy to think of retention percentages in the 90’s as good. It just feels good. But over the course of time, products in the low-to-mid 90’s will fade super-fast, and ones only slightly more sticky will do much, much better. Single percentage points here are mission critical, that’s why attention to detail and rigorous analytics become so important on the web. [...]

To restate this, as you approach high levels of user retention, you begin to see powerful multiplicative effects.

Let me state, for reality-checking purposes, that retention rates over 90% are unrealistic, but are useful for discussion purposes because they bring out the extreme cases. More realistically, the numbers I’ve seen are generally much closer to 30-60% revisit rates after the initial registration. Similarly, the typically retention rates are not linear – you see the most churn initially, but then the cohort usually settles and becomes much more loyal.

But the core issues still hold, and the reason, of course, is simple – it’s simple arithmetic, which we’ll examine below.

An example of two subscriber cohorts
Let’s say you’re running a subscription site, and you compare two sets of subscribers, both starting in the same month, both numbering 1000. Let’s say that one cohort has 80% monthly retention, and the other is 90%.

In the short run, the numbers are close to the same:

• 80% monthly retention, after 1 month = 800
• 90% monthly retention, after 1 month = 900

Although obviously you’d want to have the 90% retention, the differences are not huge – you end up with 12.5% more users, after one period.

But let’s look at these numbers once you get to 12 months:

• 80% monthly retention, after 12 months = 1000*(0.8)^12 = 68
• 90% monthly retention, after 12 months = 1000*(0.9)^12 = 282

Of course, this ends up being a staggering 3X difference. Wow! And when you compare aggregate lifetime value, the numbers are even bigger.

Retention-focused features are very powerful
The point of all of the above is that retention-focused features are very powerful because they let you create dramatic improvements in all the important metrics, across the board – be it pageviews, total time usage, revenue, etc.

This means that you can, just as people do with addressbook importers and the like, put a tremendous amount of time into a whole host of retention-driven features like:

• A great product and value proposition
• Targeted notifications
• Fresh news and content on every return
• Desktop app-integration (which has a much lower rate of uninstall)
• The number of friends on the site (the more that are there, the more notifications can be generated)

All of the above contribute meaningfully to this user retention number.

As weird as it is to imagine that something as pedestrian as how you deliver your notifications can cause success or failure to your web business, in fact it can. The reason is that how you deliver notifications can have a huge impact on whether or not your users come to your site, and every percentage point of improvement may lead to many times the revenue, pageviews, and content. This is certainly an area worth focusing on, as much as Bay Area companies have focused on just the viral metrics.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

UPDATE: Thanks for Bhanu Sharma for correcting some dumb typos on the post – I changed the numbers halfway through and forgot to update all of it.

Take the poll below, and please leave comments if you have extended thoughts:

Thanks for the feedback!

The perils of ignoring brand value
The nature of internet marketing makes it easy to have a highly accountable, metrics-driven view – but companies that are highly metrics driven easily overlook hard-to-measure issues like brand and user experience. The reason is that when all product decision-making is run through metrics-driven reports, soft things like “Brand” show up as costs, but never as benefits.

This leads to systematic erosion in many “soft” but important factors, like customer experience, brand value, and “love.” And ultimately you need all of these things to create a massive, enduring consumer brand – it’s not enough to optimize funnels.

Let’s discuss why:

Two worlds: Direct marketing and brand marketing
In the advertising industry, there’s been a long, historic distinction between brands and direct response – and this distinction echoes its way into the online startup building world as well.

In the brand world, you have companies like Coca Cola, Apple, and others who pour millions of dollars into high-reach vehicles like TV which lack any real accountability. Thus the saying:

Half the money I spend on advertising is wasted; the trouble is I don’t know which half
– John Wanamaker, US department store merchant (1838 – 1922)

To many people, the brand advertising world is irrational and fashion-driven, because of the complex interactions between agencies, their partners, and the publishers that rely on them. Just watch Mad Men.

On the other hand, you have direct marketers who thrive on accountability. They buy into marketing channels like direct mail, coupons, infomercials, and most recently online remnant ads, because they can purchase cheaply and use sophisticated statistical techniques to optimize their media buys.

Startup engineers tend towards metrics-driven
So what side do startups tend to side on? It obviously depends, but because of the highly accountable and measurable nature of online, it’s much easier to become metrics focused. Similarly, startups are mostly poor Thus, expensive brand efforts are mostly out of reach. (Probably for the better!)

Also, with the possible exception of GoDaddy, I don’t know a single startup that made it or not based on their brand advertising strategy. The typical path is focused on products and technology, and large organic growth which builds large consumer audiences.

And obviously, readers of this blog will tend to be much more metrics driven compared to the average entrepreneur!

You optimize what you measure
The first issue that causes metrics-driven startups to ignore brand value has to do with the fact that it’s very hard to measure brand, and you tend to optimize what you can measure. As soon as you throw some numbers on a big report, there’s an inherent human desire to make the numbers go up!

This is why one of the fundamental tenants of metrics-driven startups is to build lots of highly accessible reports that everyone in the organization can look at. Even if it’s easy enough to pull something out via a SQL query, it’s another thing for everyone to be able to hit a URL and load it instantly, no matter who they are on the team.

Measuring brand value is hard!
But measuring brand value, or user experience, or community “feel” or other soft things like that is very hard. I think they’re hard because while it’s clearly important, at the same time:

• The quantitative effects accumulate over large periods of time
• These might be “source” variables that drive lots of behavior, but it’s hard to measure past surveys and explicit information collection
• Some of the most important datapoints may be qualitative, not quantitative
• Changing these soft things may require big efforts above and beyond small A/B-testable changes

The companies out in the marketplace that try to measure brand value mostly just use surveys to detect changes. Or, many companies simply resort to a pretty ineffectual number like “reach,” which refers to the number of people who saw the campaign. This can sort of work, but self-reporting also sucks, and the quantitative data you get out may not be as useful as the qualitative data.

In my previous online ad career, I was shocked to hear that the standard way to measure a brand advertising campaign online was to fork $50k over to Dynamic Logic, whose job was to run a dinky little survey and tell you if your campaign worked. $50k to run a survey!

Reports show the cost of branding, but not the benefits
As a result of brand advertising being hard to measure, you get two systematic, interrelated issues:

1. Product changes that result in brand value are overlooked, whereas the costs of delivering that value is not
2. Features that negatively impact brand value but show short-term quantitative value are accepted

Here are two examples – let’s say that you think your site’s interface looks like crap, and you want to improve it to make it higher class and more trustworthy. But your metrics czar says, let’s make a really small improvement and see if it affects anything before we revamp the whole site. That sounds reasonable, but then you find out that in fact, making a visually compelling site just doesn’t drive better metrics, and in fact, it’s expensive and maybe lowers certain metrics. What do you do? (This is case #1)

Another example is that you make it really hard to unsubscribe from your mailing list. Maybe you don’t have a link, or you have to login first, or whatever. Making this change clearly affects your ability to retain users, but you get a small percentage of complaints, but the overall quantitative metrics look good. Should you keep this hard-to-unsubscribe mailing list issue? (This is case #2)

Ultimately, it should be clear that both cases are not clear cut issues at all. I could find reasons to go either way, but when you’re trading off a qualitative metric versus a quantitative thing, the numbers-driven approach tends to win. But this may not be the right thing. Similarly, sometimes the numbers may justify the decision, and the brand costs are actually quite low.

How do you make these decisions then? I’ll just wave my hands and say, “Entrepreneurial judgement”

Who’s the brand advocate?
One of the big, important roles that you need on every team as a result is someone who can advocate for the soft things. Who’s your brand advocate? Or customer experience advocate? Having someone on your team who can make logical arguments to balance out the quantitative stuff is hugely key, otherwise you’ll inevitably go down a path of brand-eroding quantitatively driven decisions.

Similarly, if you find that you’re never making decisions that go against the numbers, then frankly, you’re probably doing something wrong. If the data drives all the decision-making, then a lot of “soft” data is getting ignored.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

Don’t worry, I’m not a hippie
From the title of this blog post, you might think that I’m going to make a touchy-feely argument about why you should respect the right of your users to do all the terrible things that every entrepreneur fears:

• delete their accounts
• unsubscribe from email lists
• cancel their subscriptions
• uninstall their apps

… but you’d be wrong.

In fact, I’m going to argue that for every early product out in the market, making it really easy to quit is completely aligned with self-interested thinking. I’ll make the assumption that all the entrepreneurs reading this post are greedy, self-interested individuals, and target the appeal straight into your dark hearts

My central argument is that if you believe that every startup is an iterative learning process that converges towards product/market fit, then you need extremely high-fidelity signals to tell you if you’re going in the right direction. That means that along with trying to charge people money from early on, which is the highest form of “I love this!” you should give people valves to tell you “I hate this!” so that you can learn more faster.

Let’s drive into this further…

Product/market fit
There’s a notion of product/market fit that Marc Andreessen references in his blog, and he calls it the “only thing that matters” and says that every startup should do everything they can to get to this point. Let’s see what he writes:

The only thing that matters is getting to product/market fit.
Product/market fit means being in a good market with a product that can satisfy that market.

… and Marc continues:

Lots of startups fail before product/market fit ever happens.

My contention, in fact, is that they fail because they never get to product/market fit.

Carried a step further, I believe that the life of any startup can be divided into two parts: before product/market fit (call this “BPMF”) and after product/market fit(”APMF”).

When you are BPMF, focus obsessively on getting to product/market fit.

Do whatever is required to get to product/market fit. Including changing out people, rewriting your product, moving into a different market, telling customers no when you don’t want to, telling customers yes when you don’t want to, raising that fourth round of highly dilutive venture capital — whatever is required.

When you get right down to it, you can ignore almost everything else.

If you believe what he says, that gives you a pretty firm set of marching orders. And for early products on the market, getting to to this point in which your product is good enough and the market is compelling enough is a tough slog. So the question is, how do you navigate your way to product/market fit?

At the heart of every startup is a learning loop
For the idea that every startup is inherently a learning machine, we can turn to two of my favorite startup people, Steve Blank and Eric Ries. Eric has blogged in a lot of detail about how he believes that inside of every startup is an OODA loop that involves trying stuff out, learning, and trying more stuff again. And of course a lot of these ideas are built off of Steve Blank’s Customer Development framework that I’d encourage my readers to look into as well.

In this light, to combine the two ideas: Every startup is a series of iterative experiments that gets you from zero to product/market fit, and if you can do it before running out of money, then you might get rich

And the decision-making process in this approach is totally different. In most product strategy conversations I’ve been involved in, the most heated debates center around whether a particular product will work, and all the pros and cons of the situation. Contrast this to a learning-centric approach, which emphasizes whether or not experimenting with an idea will yield insights, and how much it’ll cost to learn these insights.

In other words, you’re much more likely to try things that will fail, if those failures teach you something important about the market.

Of course, all of the decisions that power these iterations rely data – and the better the data, the better your decisions will be, naturally. So where do you get the data to tell you if customers are happy or not about your product?

Explicit signals beat implicit signals almost every time
One of the key lessons I took away from my time from the behavioral targeting ad industry is that explicit data is much, much better than implicit data, when it comes to predicting user behavior.

That is, you’d prefer explicit “intent” data like:

• made a purchase
• used a student loan calculator
• searched for “palo alto bmw dealership”
• filled out a form

versus the less valuable implicit “interest” data like:

• have similar demographics to other people who buy
• visit the same publications as similar customers
• having a pattern of reading finance articles

So if you are looking to collect data to drive decisions, then the best kind comes from the explicit data of having users specifically take action, whether it’s positive or negative. Purchase intent data, as illustrated above, is positive – and quitting intent gives you the negative half. In fact, if you only look at the positive feedback, you might be ignoring 50% of your data.

As a result, you want lots of explicit data points in the axis of “I love it!” to “I hate it!” which includes people giving you money (maybe donations being the ultimate form of love) to allowing them to easily quit. Make it easy  for your users to quit, unsubscribe, or otherwise cancel – it gives you the strong signal when you’re doing wrong! And make sure to track it and include it in all of your quantitative experiments as well.

Better data = better learnings = Better product
So to summarize my key arguments here:

• Give users lots of explicit ways to show appreciation and hatred
• These datapoints will help you iterate your product
• Better product iterations will let you reach product/market fit faster
• Reaching product/market fit will lead to more money faster

You can only learn so much from reacting to positive data, and trapping your users in unwanted subscriptions won’t get you to product/market fit any faster.

And finally, don’t do it because it’s annoying
‘nuf said.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

Measuring value created rather than optimizing for yourself
In my last blog post, I talked about the idea that value creation generates revenue, traffic, and other metrics, not the other way around. This is a particularly interesting idea to implement because it goes against much of the standard analytics reports that are out there.

The reason is that ultimately, most metrics tend to focus inwards, on self-interested gain, rather than outwards on the value you’re creating for your customers. Let’s take a couple examples of inward-focused metrics that people often cite:

• account registrations
• pageviews
• unique visitors per month
• revenues

I’m sure you measure many of the above, as I do as well. It’s OK to measure this stuff, but if you start to optimize for it, you are starting to focus on the business of value extraction, not value creation.

Measuring # of life preservers sold versus the # of lives saved
Thus we come back to the title of the post. Most people are using standard analytics packages that are commoditized to focus inwards on metrics like pageviews and revenue. To use an analogy, that’s akin to the idea of measuring the # of life preservers that you sell, and trying to optimize for that, rather than optimizing for the benefit, which is the # of lives saved.

If you focus primarily on selling life preservers, then you’ll tend to do all sorts of stuff like:

• making them cheaper to build
• adding doo-dads to them that are flashy to customers
• using aggressive sales tactics
• etc.

These things might generate revenue in the short to medium run, but if you prioritize this at the expense of actually delivering on the product benefit, then that’s a bad optimization in the long term.

Now contrast that to the idea of trying to save as many lives as possible. You might still want to make them as cheap as possible, so that every ship in the world can have as many of them as needed. You might still want to add upgrades to them, but only if they help save lives. etc. While these changes may be similar in execution, they are different in spirit than the changes you’d make when optimizing for sales.

Introducing “Benefit-Driven Metrics”
So ultimately to start this exercise, you should throw out all the standard metrics (conversion rates, pageviews, etc.) and just focus on one thing:

What are your customers measuring?

By looking at how they define value, then you get yourself aligned to them as closely as possible. Answering this question sets your company up for value creation, which then unlocks the ability to gain something from that value, then you have to start here.

I’ll deem these quantitative measurements as “Benefit-driven metrics.”

How do you measure it?
Here’s the interesting part – everyone’s benefit-driven metrics will be completely different, because most people’s customers and value proposition and product are ultimately very different. Unfortunately, you don’t have the crutch of standardized numbers like pageviews or uniques to lean on.

But let me give you some examples for reference:

For dating sites:
Why do customers join dating sites? To find their soulmates. Thus, measure the quantity of successful matches you make, not the lifetime value of the customer. Focusing on LTV can easily lead you to do things like creating fake accounts to make people come back, or optimizing it so that they find their best matches several months down the line, or trying to get everyone to pre-pay for the service rather than making the product experience awesome.

For marketplaces:
Why do customers sell on a marketplace? To make money and get rid of their stuff. Why do customers buy on a marketplace? So that they can get things cheaply and quickly, and are happy with their purchase. Thus, measure the quantity of how much your sellers take home, and how many buyers are happy with their experiences. Contrast this with overfocusing on listing fee revenues, which might get you into a spiral of raising prices rather than creating the best commerce experience.

For social networks:
Why do customers use social networks? To “connect” with their friends – let’s boil that down to communicating (though it’s obviously much richer than that). Then ideally, you might want to focus on the number of messages/comments/posts that end up getting replies from their friends. If you overfocus on something like user registrations, then you might get a ton of users, but maybe they won’t be getting to experience the benefits of the product.

For online publishers who sell to advertisers:
Why do advertisers buy ads on websites? To generate traffic to their own sites, which in turn leads to revenue. In this case, you should quantify the amount of revenue you generate for your advertiser customers, or at least the number of conversions they receive. Contrast this with the approach of measuring and optimizing your own CPMs as a publisher, which results in potentially delivering a lot of crappy traffic to advertisers who will drop their payments in the long term.

Special note for ad-driven startups
Now ad-supported startups have a particularly interesting issue in this, because I keep using the words “benefits” and “customers” and perhaps it’d be easy to think this refers to the users of the product. But maybe not, as I’ve outlined before in Your ad-supported Web 2.0 site is actually a B2B enterprise in disguise. The reason is that your customer may actually be the advertiser on the site, not your user!

And in fact, if you overfocus on pleasing your users to the detriment of your advertiser customers, which is very easy – then that leads to very bad things.

Start this benefits-driven approach now, not later, so you can learn the right things
Finally, I want to emphasize that I believe it’s important to start thinking about these benefit-driven metrics from the beginning of your business, not later. The reason is that every learning that a startup makes is often hugely applicable to a specific context, but not at all applicable to other variations.

If you’re going to start a website that churns users like crazy but hits massive user goals, you will build an entire organization to optimize for those metrics. And once you’ve gone far on this, it’s not clear that you’ll have the DNA, the technology, the ideas, or the willpower to execute in a different direction.

• Over-optimizing leads to local maxima, particularly in product design
• Focusing too much on pageviews/uniques ignores actual product/market fit
• Relying on quantitative models leads to anti-innovative behavior
• etc.

For all the data geeks reading my blog – my opinion on all of this is, these are absolutely all true, and are all very important and relevant conversations that every data-driven startup needs to be having. Are you having them?

All of these have gotten me focused on one of the core questions of any business: What value are you actually creating?

Distribution-led approaches can lead to local maxima on value creation
Many new companies in this age of quantitative virality easily fall into hitting local maxima on value creation, all for very good reasons. By focusing on viral invites, addressbook scraping, A/B testing, and other techniques, you end up getting a big inflow of traffic and your question becomes, “What is the best product I can make to keep all these users around?”

There become three major temptations:

1. First, there’s a huge desire to build as efficiently as possible. That is, build in just enough to satisfy the user, but don’t overpolish
2. Similarly, there’s a big temptation to build for the lowest common denominator, because you’re trying to appeal to a huge audience. As a result, a lot of designs err towards persistent low-brow internet “recipes” – like quizzes, polls, forums, and other mechanics
3. In addition, your product veers towards a portfolio of experiments, rather than one cohesive experience. After all, you’re still trying stuff out, and it’s a lot easier to add a new feature or use a crazy headline to get people to your site, rather than really going through the difficult synthesis process that’s at the heart of every design discussion

As a result of all of this, it’s very easy to build a shitty product that generates small to medium value, but doesn’t do something amazing.

I won’t go too much into the solution of how to solve this, but I think the key thing to think about is that the quantitative lean philosophy doesn’t allow you to skip the difficult process of coming up with a hugely value-creating product. You still have to do it, but you have a framework in which to think about the process.

Maximizing the source rather than your share
Another issue in all of this is that the focus for quantitatively driven companies ends up being on outputs rather than inputs. For example, it’s easy to start to optimize traffic as an entity in itself, rather than thinking about the fact that traffic comes out of product/market fit. Or similarly, you can optimize revenue, but I think it’s misguided to do it without considering the fact that you have to be creating value for whoever is paying you.

Thus, one can argue that “value creation” is the ultimate source for all of these secondary variables like revenue, traffic, etc. And you can make the decision to focus on extracting as much as possible from the secondary variables, but you become fundamentally limited by the primary value creation process within your product.

Another way to think of this is that ultimately, every product creates a bunch of “value” (however you want to define it) and then you end up taking some % of that value back as revenue. Abstractly, this is true regardless of whether your product is ad-based, freemium, or otherwise. If you think about things this way, the following two approaches are fundamentally different strategies:

1. Create a massive amount of value, and capture a small amount
2. Create a moderate amount of value, and totally dominate the economics

(and obviously this is a spectrum as well)

I would put companies like Wikipedia, Craigslist, Open Source, and others as extreme examples of #1. And unfortunately, I think a lot of short-lived apps on Facebook are really more or less examples of #2.

I think this is why, for people who question the value of internet companies like Facebook and Twitter, the natural thing to ask is, are these companies generating real value? If they are, I think the process of turning that into cash is much easier than the process of creating the huge value in the first place!

The biggest value drivers are qualitative
So the question then becomes, how do you systematically create value? I think that this is a very hard question, and one that it may be difficult to use quantitative tools to define, because the biggest value drivers are often qualitative. They are things like:

• What’s does your product do?
• Who’s your customer?
• Why do people give you money?
• etc.

Now, a lot of these you can turn from qualitative to quantitative. After all, after you build your product, you can generate hypotheses around how people ought to use it and make it better in the most common flows, by optimizing the page flows. Similarly, you can figure out how much money you should charge for something.

Yet simultaneously, the process of figuring out the core product requires the entrepreneur to have an opinion, perhaps one that is difficult to test or takes many years to test. And whether you do this quantitatively is its own thing – after all, companies like IDEO have a very evidence-driven design process, but is one that uses qualitative evidence gathering to generate the product prototypes.

Looking at landing page optimization as a value creator
I think that this entire perspective about maximizing value creation rather than optimizing outputs leads to a lot of interesting, subtle changes in how you approach things. Let’s take landing page optimization as an example of this.

Typically, the entire discussion around landing page optimization is just one about conversion rates, and all the different possible candidates to get to a conversion. Instead of this perspective, you might ask: What value does an optimized landing page generate in the first place? Ultimately, I think this optimization makes it so that people can grok what they’re signing up for better. It helps them scan the page better for relevant pieces of information. And it could make them less confused about the page they’re on.

Compare that line of thought with, “hey, let’s make a lot of random headlines and see what people react to” as two different ways to approach the same problem, with one prioritizing value creation and the other prioritizing the output (conversion rates in this case).

Looking at viral loops as a value creator
Same for viral loops, and the process of getting people to invite their friends to a site. If you are sincerely value-oriented, then the entire question is:

• why do people WANT to invite their friends to the site?
• how does having your friends on the site make the product a better experience?
• what conveniences can you build in to make people expose their friends to the process?
• etc.

Contrast this to a perspective that an outcome like the viral factor is all you care about optimizing, and however you can get that number >1.0, then the better off you are. I think that this numbers-centric model absolutely can lead to viral websites and apps, but also sucks at actually creating a huge base of value that you can recoup later.

Conclusion
My point with all of the above is simple: No matter what your product is, the only way to make money long-term is to make a lot of people happy, and then getting some % of the value you created back, in return. The right strategy to build a long-term sustainable business is to build long-term sustainable value. No amount of viral tricks or optimization will allow you to escape that truth!