Not all of these programs are Marketing, some are Service, and some could be considered “Operations”.  But they all touch the customer, so we were thinking changes in customer value might be a common way to measure and compare the success of these programs.

A: Absolutely!  I just answered a question very much like this the other day, it’s great that people are becoming interested in customer value as the cross-enterprise common denominator for understanding success in any customer program!

If I am the CEO, I control dollars I can invest.  How do I decide where budget is best invested if every silo uses different metrics to prove success?  And even worse, different metrics for success within the same silo?

By establishing changes in customer value as the platform for all customer-related programs to be measured against, everyone is on an equal footing and can “fight” fairly for their share of the budget (or testing?) pie.  By using controlled testing, customers can be exposed to different treatments and lift in value can be compared on an apples to apples basis – even if you are comparing the effect of a Marketing Campaign to changes in the Service Center.

But are you sure you want to use LifeTime Value for this application?

Q: From what you stated on your website, I will not be able to develop a LifeTime Value model unless I understand the customer Lifecycle.  The customer lifecycle is something that I could get a good understanding from using doing a RFM analysis.

My question is, once I complete the RFM analysis, what would be my next steps in developing a customer LifeTime Value model?   At this point in time, the hardest thing that I am trying to wrap my head around are the variables to include in the model.  I visited Arthur Middleton Hughes’ website:

http://www.dbmarketing.com

and he suggests the following variables (download spreadsheet, if interested):

http://www.dbmarketing.com/special_ltv.htm

Jim, could I simply use those variables going forward to calculate the LifeTime Value of a customer at my company?  I would appreciate any assistance you may be able to provide to me on this matter.  Thanks.

A: Well, that’s a big tangle of related issues!    Let’s unpack first, then answer the question.  First, the relationships between these ideas:

Lifetime Value versus Lifecycle – LTV is a number, LifeCycle is a trend over time that contains trigger events.  You don’t need the LifeCycle to develop (calculate) LTV, you need the LifeCycle to most efficiently and profitably act on and manage LTV issues.

RFM versus Lifecycle – RFM is a tactical model that is a “snapshot” of customer state at a point in time, the customer’s likelihood to respond.  Frequently used names for these customer states include active, lapsing, lapsed, defected.   Lifecycle is the “movie” one might put together from these snapshots of RFM states; the migration from one customer state to the next are the Lifecycle trigger points.

Now, let’s make sure we understand each one of the ideas:

LifeTime Value

Strictly speaking, LTV is not a very flexible concept and is best used for determining how much you can spend to acquire a customer and still make a profit.  This is the equation that Mr. Hughes has provided, a man by the way that I have a lot of respect for.  His model is quite detailed and useful for the purpose of finding break-even cost to acquire a customer.

To use Arthur’s LTV model, you have to find historical values and plug them in.  You could assume nothing will change and the LTV of certain segments of past customers will be the same; this is great for “benchmarking”, for example.  However, this approach is not measuring LTV, it’s predicting LTV based on historical data.  This is fine, and a valid method for certain types of analysis.

But, the premise of your question is you will be testing, and testing implies something new will occur.  So while you could use LTV to estimate results, you’d have to wait quite a while to prove the results one way or another.  LTV is really “forensic” in this way – you won’t know the final answer until the customers defect.

You could certainly go back 2 – 5 years after the tests, and prove one group had higher LTV than another, but that’s not typically a very useful approach when doing testing.

RFM (Recency, Frequency, Monetary)

RFM is a predictive model that takes a “snapshot” of the customer base and gives you a score for each customer, a prediction of likelihood to respond relative to all customers.

By itself, RFM doesn’t tell you if you are making money or not.  It is used to classify the “state” of customers at a point in time, usually for targeting purposes – are they active, lapsing, lapsed, defected?  In other words, it’s a customer segmentation tool.

For example, RFM could be used to choose your test and control groups for a campaign using Lift measurement – you would want test and control to have the same range and balance of scores.  In fact, one of the tragic campaign measurement mistakes people often make is not taking into account the likelihood to respond when selecting test and control groups, resulting in biased test results.

Customer LifeCycles

One of the great features of RFM is the idea of “ranking” customers relative to each other; this gives allocation of budget and success measurement a standard to follow.  A single  customer can have many different scores over the course of their LifeTime, with the likelihood to respond the score at a specific time.  In fact, if you looked at RFM scores over time for a single customer, you would have a clear understanding of the LifeCycle of a customer – the most powerful segmentation available in terms of message and offer targeting.

The problem with looking at RFM scores over time is complexity; the beauty of individual customer scores at a single point in time becomes unbearable when you are talking 125 different scores on 50,000 customers over 6 months.  That’s the internal or analytical problem.  Externally, this kind of information is extremely gnarly to present and explain to senior managers, it’s presentation hell.

The way I solve this problem is with a tool I call LifeCycle Grids.  The Grids takes the same fundamental drivers used in the RFM model and instead of ranking, uses thresholds or “hurdles” to classify customer states.  This creates a standardized customer LifeCycle “dashboard” so comparisons of customer value between different segments can be made more easily.  It works for both short and long term observations and is easy to represent either numerically or graphically.  And because it uses finite thresholds for activity rather than ranking, the same calculations that create the dashboard can be used to actually drive or trigger actions.

So the dashboard is actually the controller as well.  This is extremely beneficial in terms of linking presentations, plans, and results. People can literally point to a segment on the LifeCycle framework and say, “Let’s deliver message X to each person from segment Y who enters this cell” and see the results right where they pointed when the dashboard is updated.

Once you test some ideas and find out which approach generates incremental profits for a cell in the Grid, you can automate delivery of the program as customers enter that cell of the Grid.  This is the classic “sense & respond” approach to marketing communication – right message, right person, right time.

The LifeCycle Grids are demonstrated in a lot of detail for different applications in the series here, but probably of most interest to you as it relates to customer analysis, see here.

And now, to answer your question:

Which approach above, if any of these, would be best for standardizing measurement of ROI in widely diverse customer programs?

LTV would be appropriate if what you want to know is breakeven cost to acquire.  Since we are talking about customer programs, I doubt that’s what you want to use.  Plus, if you want a hard number rather than a prediction, you could be waiting a long time for the answer.

RFM is a “snapshot” model and so not really suited to long-term studies of customer value.

Customer Lifecycle models are more likely to be involved in the execution of a program, not the success measurement.  LifeCycle tracking could be (and often is) used to predict the financial success of campaigns before they have run their course, but you’re only predicting success, not delivering numbers into an ROI model the CFO would accept as “fact”.

Answer: None of the above.

What you need is an approach designed for the task, which in this case, is:

Lift Measurement or Near-Term Value

Lift is a measure of the performance of a test group of customers compared with a control group of similar customers who are not exposed to the test.  You can read more about control groups here.  In the analysis of value contributed by each group, many of the same values from Arthur’s LTV model are used – product margin, costs of program, fulfillment costs, payment parameters, etc.  However, if you are talking about a program to existing customers, cost to acquire is probably not relevant, though you might use source (campaign) to segment your test approach.

Lift is typically measured at intervals, say every 30 or 60 days, to see how test versus control populations are tracking, and can continue after the test is over to pick up residual value created in the customer.  However, this is not a Lifetime Value measurement, Lift models measure incremental contribution to LTV created by the Marketing, Service, or Operations program execution.

This means if you get lift from program test versus control, when you go back 2 – 5 years later and measure true rather than predicted LTV – after the customer has defected – you should in fact see the LTV in the test group higher than in the control group, barring any radical downstream difference in customer experience between test and control.  In this way, Lift models are actually predictive of changes in LTV.  That’s why the output of Lift models is sometimes referred to as the measurement of “Near-Term Value” and used much more often than the forensic approach of waiting for customers to defect.

Summary

All the above are core concepts in customer value measurement and management.

LTV is a measurement of net financial value contributed by a customer, and Lift measures  are like a “time slice” of the overall LTV curve.

LifeCycles are a management framework for programs designed to affect LTV, and models using Recency, Frequency, and Monetary are used to look at a “time slice” of the LifeCycle.

LTV can generally be increased in two ways: by creating more value during the existing LifeCycle, or by extending the LifeCycle.  Marketing (including Product) is typically used when doing the first, Service and Operations – customer experience and satisfaction – are largely what affects the second.

So it is completely appropriate to establish a unified approach to the measurement of customer programs intended to increase the value of a customer across all these disciplines, in order to ensure the allocation of  scarce resources to highest and best use.

A great question, and for a great cause!

Jim

Update:

Listrak asked me to do a podcast with them on these and related topics, check it out (MP3 link) here, or see list of all their Email Marketing Today podcasts here (I’m on Episode 42).

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

LTV, RFM, LifeCycles – the Framework

So here, I will attempt to answer other kinds of questions I think people might have based on my discussions with WAA members.

Update: The WAA has answered many Certification questions here.

However, I’m going to approach this topic a bit differently than most of the published documentation – from a Product / Marketing perspective, rather than an Educational / WAA POV.  I can do this because (if you don’t know) I have worn all the hats on this project – developer, marketer, WAA project owner – and I think it might be helpful to tell the business story of the WAA Certification, from the bottom up.

And if you have other questions, feel free to leave them in Comments and I will do my best to answer them!

Where did the idea for Certification come from?

The WAA is a member-driven organization; we listen to the membership and try to accomplish what they would like us to accomplish.  We heard from hiring folks and managers that ”web analysts today know a lot of the buzz words and can follow instructions as far as reporting goes, but what we’d be willing to pay a premium for is web analysts who discover things on their own, who add value in areas we don’t already know about”.

So that’s where WAA Certification came from.  It addresses a specific need identified by members, what came to be known internally as the “Book Smart versus Sherlock Holmes” problem.  Sure, you can read a ton of books or blogs and be a  good web analyst by following best practices.  But so can a lot of other people.  What you need to pass the Certification Test is different; you have to be able to turn data into insight and recommend a best action given the scenario presented.

This is Why the WAA’s Educational efforts lack “tool focus”?

Sure.  And of course, the tool vendors already own that focus, and by definition they have the resources to be much better at tool education / certification than the WAA, so why would the WAA want to compete with the tool vendors in the same space?

Better to add value on the business side, where there is demand we can fill and a lack of trusted resources.  And if you think about it, this approach simply expands the overall WA opportunity.  People who want to become experts on the tool side have a path (through the vendors), and people who want to become experts on the analysis / business side also have a path through the WAA.  And if you want to be a Universal Web Analytics Soldier, I guess you could do both!

Does that mean I can pass the Test with No Tool Knowledge?

Not at all.  The threshold we set is you need to be able to communicate effectively with tool experts to pass the test.  That means you will need to know the basics of how the web works, how the tools accomplish their mission, and know what all the web analytics terms mean.  Example: To pass the Test, you don’t need to know how to write a tag, but you do need to know when a  custom tag  is required and how to communicate your need effectively.

So Marketing people can Pass the Certification Test?  eCommerce Managers?  Usability people?  Media Buyers?  Etc.?

Absolutely, if they are good at transforming the data generated by web analytics tools into business insight AND have broad knowledge across the entire scope of web analytics.

Who Created the Certification Test and How?

About 50 WAA members from all over the world volunteered to take on the task.  We created questions, tested them across different audiences, gathered feedback, rewrote the questions based on the feedback, tested the questions again.  You know, the continuous improvement thing?

If you want to participate in the ongoing process of creating the Certification Exam, there is more info here.  Please note you have to be a member of the WAA to be on any WAA Committee.

Where did the Requirements to take the Test Come From?

From the 4 Test the Test sessions we held at various eMetrics events, where we asked people to volunteer to take the Test.  We looked at the backgrounds of  people with high scores versus people with low scores and established the  benchmarks.  People with higher than average scores had these characteristics:

Years of Web Analytics Experience:  5.4
Interprets reports / suggests actions to be taken: 100% of population
Training / Courses in web analytics:  100% of population
Education post High School: 4.8 Years

People with lower than average scores had these characteristics:

Years of Web Analytics Experience:  2.3
Interprets reports / suggests actions to be taken: 50% of population
Training / Courses in web analytics:  63% of population
Education post High School: 3.6 Years

But inside these averages (segmentation!), it gets much more interesting.  Turns out the less experience you have, the more formal education / training helps you get a higher score.  Education could be college / advanced degrees, vendor training, or classes in web analytics / e-commerce.  Logical, and expected.

Not so intuitive was this on the mix of education and experience: when you have a lot of one and little of the other, you tended to get a lower score.  For example, both Ph.D’s with low years experience and people with 10 years experience but lacking education / training tended to get lower scores.  Likewise, people who indicated they “read blogs and books” as the only source of education did not tend to have high scores unless they had a lot of direct web analytics experience.   So somewhere in the middle there is a “magic mix” of experience and education that results in higher scores.

Interestingly, the single most reliable predictor of a higher score on the test was whether or not in the current job the person regularly suggests actions to be taken based on the analysis.  This data point is more subjective than years of education or experience so we did not include it as a requirement to take the Test, but it’s worth mentioning since it aligns closely with the purpose of the test.

In the end, it’s tough to predict tangible business analysis skills based on just education or experience alone, and this is why the Certification Test should be an important tool for people hiring web analysts.

I’ve heard the Test is Difficult to Pass; can you Explain Why?

In short, because we are a young industry and people tend to have narrow experience relative to the scope of the topic.

You can be an expert in e-mail and Display analytics and still not pass the test because you don’t know enough yet about PPC analysis or Optimizing Web Sites.  You don’t have to be an expert at everything to pass the Test, but you do need to have some knowledge across the entire scope of web analytics to get a high score.  See the Knowledge Required for Certification document for an overview of topics.

That said, I’m sure many of you have been faced before with challenges you did not understand or have any experience with – and then you figured out how to produce insight.  That brainset is precisely what the WAA is testing for.  So if you can take what you know from e-mail analysis and use it to figure out a question about PPC analysis, you could answer the PPC question correctly.  Do that enough times across the different knowledge areas and you could pass the Test, because you essentially demonstrated the ability to think analytically – the objective of the Test.

In opposition to that scenario, blindly following best practices in any knowledge area without recognition of the changes in approach a particular business situation or model might require means you probably will not pass the Test; you will need the capacity to modify your thinking based on the business goals presented.  Example: the correct answer for the publishing model may not be the correct answer for the commerce model.

How Do I Decide if I Should Take the Test?

Honestly, I personally think the Certification has much more value to people who are in the earlier stages of their web analytics  career.   Let’s say you have the same training and read the same books as a lot of other folks.  And you are trying to establish yourself as a person who can create business value but don’t have the resume to back that position up quite yet.  Passing the Certification Test could give you the edge you need to make things happen faster for you.

Conversely, if you have an awesome resume of accomplishments and references for those deeds, then why would you need the additional “proof” the Certification Test provides?  Plus, experienced people often specialize to distinguish themselves from the crowd, and a Test across the universe of Web Analytics would not be particularly relevant.

So I’d expect the majority of people taking the Certification Test to be say 3 – 4 years into their WA careers, or perhaps  earlier if they have been focused on WA and exposed to the right training or experience environments when doing the actual work.

The above is from the perspective of an individual.  However, an agency, consultancy, or service provider might decide having their analysts Certified (including senior people) creates a competitive advantage in their particular space.  Companies looking outside for analytics help may feel more comfortable hiring a resource with WAA Certified talent on staff.

Are there any Questions?

Feel free to ask about anything,  and please see the WAA FAQ for questions on execution details.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Inside WAA Certification: Any Questions?

You may recall I wrote last year about the explicit or implicit pressure put on Analysts to “torture the data” into analysis with a favorable outcome.  In a piece called Analyze, Not Justify, I described how by my count, about 50% or so of the analysts in a large conference room admitted to receiving this kind of pressure at one time or another.

Since then, I have been on somewhat of a personal mission to try to unearth more about this situation.  And it seems like the problem is getting worse, not better.

I have a theory about why this situation might be worsening.

Companies that were early to adopt web analytics were likely to already have a proper analytical culture.  You can’t put pressure on an analyst to torture data  in a company with this kind of culture – the analyst simply will not sit still for it.  The incident will be reported to senior management, and the source of “pressure” fired.  That’s all there is to it.

However, what we could be seeing now is this: as #measure adoption expands, we find the tools in more companies lacking a proper analytical culture, so the incidents of pressure to torture begin to expand.  And not just pressure to torture, but pressure to conceal, as I heard from several web analysts recently.

One bright young analyst went “beyond the call of duty” on his analytical project.  The analyst gathered relevant data not just from the WA tool, but from Finance, Customer Service – all around the company.  The report painted a detailed picture of cost to acquire customers through various methods and campaigns, and was presented to the head of Marketing – also the analyst’s boss.

The analyst was told under no circumstances was this report to ever be produced again.  Further, the analyst was told to destroy any “evidence” this project / report ever existed.  And finally, the analyst would now be required to send all analysis through the boss first before anybody else sees it.

That’s shameful behavior for an exec.  And apparently, this kind of thing is happening more and more often.  I’ve heard plenty of “if we want your opinion, we’ll ask for it” stories, but this is the first time I’ve heard so many stories about concealing results.

Here’s a scary thought: what if the stories about web analytics not driving business value are primarily concealment stories?   What if the tool / analysts actually did provide value, which was then hidden from Senior Management?

My concern about this issue is wider than screwed up company culture and management.  What I’m more concerned about is screwed up people, analysts who may come to think this kind of behavior is normal and just part of being an analyst.

This matters because as this new generation of analysts moves to other companies and throughout the ecosystem, these pressure to torture situations could become “accepted” and even spread as “part of the game”.

It is never, ever OK to manipulate or hide the results of an analysis.  It’s not part of the job.  The role of an analyst is to analyze, not justify or conceal bad news.

Now, I realize some folks are thinking, “Yea, that’s great Jim, I’ll just get myself fired by being an analytical hero”.

I’m not saying you should respond to data torture pressure by falling on your analytical sword.  What I am saying is you – and management – need to know this kind of pressure from a superior is shameful, not a “normal” part of being an analyst.  And as soon as you can, you should get a job somewhere people respect your professional opinions.  Don’t have to agree; but must respect.

Like the company you work for?  Ask a buddy in Finance if they could use a web analyst.  Pretty sure Finance would be interested in fully-loaded cost to acquire new customers by source!

What really troubles me about this situation is it’s rarely ever talked about, so could be worse than people might think.  At the very least, Senior Management should know about the potential for this to happen and lay down some rules.  Perhaps even seek some cultural guidance on this topic (here’s a start – Fear of Analytics).

So, I want to put this message out there, perhaps create a resource for people who are looking for information on this topic.  It would be great to have examples so managers can understand and be on the lookout for these situations.  Plus, I’m sure there are some terrific stories out there about either giving in to the torture pressure or resisting it!

What about you?  Were you ever pressured to torture the data?  What happened?  Did you comply?  How did things come out?  Tell us with a Comment.  Feel free to post anonymously, leave out company names.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Tortured Data – and Analysts

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: Thank you for your recent article about Control Groups.  Our organization launched an online distance learning program this past August, and I’ve just completed some student behavior analysis for this past semester.

Using weekly RF-Scores based on Recently and Frequently they’ve logged in to courses within the previous three weeks, I’m able to assess their “Risk Level”– how likely they are to stop using the program.  We had a percentage who discontinued the program, but in retrospect, their login behavior and changes in their login behavior gave strong indication they were having trouble before they completely stopped using it.

A: Fantastic!  I have spoken with numerous online educators about this application of Recency – Frequency modeling, as well online research subscriptions, a similar behavioral model.  All reported great results predicting student / subscriber defection rates.

Q: I’m preparing to propose a program for the upcoming semester where we contact students by email and / or phone when their login behavior gives indication that they’re having trouble.  My hope is that by proactively contacting these students, we can resolve issues or provide assistance before things escalate to the point they defect completely.

A: Absolutely, the yield (% students / revenue retained) on a project like this should be excellent.  Plus, you will end up learning a lot about “why”, which will lead to better executions of the “potential dropout” program the more you test it.

Q: However, in light of your newsletter, I realized that we should probably have a control group with whom we do NOTHING (just as we did this past semester) in order to prove the effectiveness (or not) of the program.

A: Correct.  Otherwise, you won’t be able to make a valid claim to the “saved students”. People can always argue a variety of other factors were in play – seasonality, topic, course sequence, etc.

Q: Since the actual number of students is confidential, can you please tell me what percentage you would use for a control group if we had 400, 800, 1200, 1600, 2000, 3500, or 5000 students?  You mentioned 10% in your newsletter, but the population you were referring to exceeded millions.

A: Well, there are online calculators you can use confidentially, example right here.

If you don’t understand the variables they are asking for, explanations at bottom of page, though this is very simple – what is confidence level and interval plus population size.

Q: Our population is MUCH smaller, and each customer is therefore even more critical.  I don’t want to recommend an unnecessarily large control group that would prevent us from retaining future students when we could see they were having trouble.

I suspect that our defection rates will be lower 2nd semester than 1st since students should be beyond the “learning curve,” so I don’t think we can justly say that the program alone is the reason for lower defection rates if we don’t use a control group.

A: Yes, well, this desire to “get as much test as we can” was the main point discussed in the newsletter.  And that’s the challenge with very small populations – to hit statistical confidence levels at say population = 500, you need over 300 or so in control.

Not so great.

So we go back to the question of company culture and how intuitively confident people will be with the results.  Do they in fact need true statistical significance for a program like this?

There is a way around the significance issue – repetition. The stats part of this is all about the “likelihood you get the same results again” – real important for drug testing, not so much for 500 folks in a marketing program.

The question you need to ask: do you really need “prediction”?  Or does prediction just make the whole test more complex and expensive than it’s worth?  What if you repeated the test a couple of times and got roughly the same results, is that “proof”?

Here is what I might do.  I would ask whoever needs to believe in the results of this test a question like this:

“Let’s say we took a random 20% sample of the students and excluded them from the marketing.  We apply the marketing to the other 80% and their retention rate is 15% higher than the 20% who had no marketing. We do this test 2 more times and the retention rate of students in the test is 13% and 17% higher than the students in the 20% who do not receive the marketing.  Would you at that point believe that without question, the marketing drives at least a 13% improvement in retention among students?”

Do you see where I’m headed with this?  The more times you repeat the test, the more confident you will be in the results – regardless of sample sizes and statistical mumbo jumbo. At some point, the reality of the differences between test and control performance has to be accepted.  It may help to define up front how many repetitions the “boss” needs.

There are two clues to help you evaluate the validity of your results / how many times you need to repeat the test to be “confident”.

One clue is the variability of the results – the more inconsistent the results are, the more likely the data is “noisy” and the more times you need to repeat the test to be confident.

If the spreads between test and control for the first 3 tests are 20%, 5%, and 10%, then you’ll need more repetitions of the test to get a good feeling for the actual impact.  If the results tend to cluster as in the example above (15%, 13%, 17%) then you can be more confident earlier in the test series the actual impact is somewhere around 15%.

The other clue is in the “spread” between test and control.  If the spread is consistently  ”wide”, say +10% (or more), this provides additional confidence a positive impact is being made.  The result over a series of tests may not actually be +10% (confirm by repeating the test), but it’s more likely to be positive.  If you consistently get a spread more like 1% or 2%, it’s more likely the actual result could be zero or negative and you need to keep repeating the test to gain confidence you have a positive result.

In the end, you may not want or be able to repeat the test enough times to know with statistical confidence what the result is.  But if the spread between test and control is wide and consistent, and the cost relative to the benefit is small, then does it really matter if there is statistical confidence?

For example, if you can make the statement you’re confident the program generates at least $10 in profit for each $1 invested, does it really matter if the statistically confident  number is $11 or $12 profit for $1 in cost?  We’re doing Marketing here, not drug testing.  There is an opportunity cost (profit left on the table) to not rolling out a program based on a test with results like this; rather than repeat the test to death just to be more confident I’d roll it out and continue to monitor the results.

One more tip, on this idea of sequencing / semesters / experience with the program.

There is no doubt in my mind that 2nd semester students would have what is called a “survivor bias” and be less likely to drop out; you will get the best performance in a program like this with 1st semester students.  So if at all possible, run the test / control on only 1st semester students , or segment by semester.

But, just because you run it on only 1st semester students does not mean you don’t have an effect in 2nd semester.  Continue to follow test and control into 2nd, 3rd, 4th semesters and you may see the dropout rate of the original 1st semester group continue to widen versus control.

This is not only great for the profitability of the initial 1st semester program but also provides you the baseline you have to beat (control) for those 2nd, 3rd, 4th semesters.  When you decide to see if you can have an additional effect by intervening in those periods, you’ll have 2 groups: those affected by Marketing in the 1st semester, and those new to any Marketing intervention.

My guess: a 1st semester intervention will have tremendous impact, both then and throughout the 4th.  The impact of intervention at each subsequent semester will diminish compared with acting in 1st semester, as will the “tail” value created over the student life, since the number of months left in the student life is shrinking each semester.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Control Groups in Small Populations

Which brings me to the latest Marketing Science article I have reviewed for the Web Analytics Association, Dynamic Customer Management and the Value of One-to-One Marketing, where the researchers find “customized promotions yield large increases in revenue and profits relative to uniform promotion policies”.  And what variable is most effective when customizing promotions?

The researchers took 56 weeks of purchase behavior from an online store, and used the first 50 weeks to construct a predictive model of purchase behavior.   Inputs to the model included Price, presence of Banner Ads, 3 types of promotions, order sizes, number of orders, merchandise category, demographics, and weeks since last purchase (Recency).

The last 6 weeks of data were used to test the predictive power of the model, and the answer to which variable is most predictive of purchase is displayed in the chart below, click to enlarge:

Weeks since last purchase dominated the predictive power of the model, controlling not only the Natural purchase rate (labeled Baseline in chart above, people who received no promotions) but the response to all three different types of promotion.

The  Natural buying rate (here, as much as 50% of campaign response) has tremendous implications for the measurement of Campaign profitability, and can also be used to measure the success of customer-centricity / experience / social programs.  These are the issues I cover in my review of the article.  If you’re interested in that take, you can read it here.

But for this post, what I’d like to do is  explore the Recency measurement idea itself, because I suspect a lot of people may not understand what it really means.  And since many Marketing folks are not used to taking action on this kind of data, also talk about what you can do with this information.

Most people think of time in a linear way.  A graph that includes time typically starts at some point in the past and churns through time in a sequential fashion.  Not so with the graph above, which is looking at Purchase Cycles.

In this style of cycle measurement, customers are moved back to the time = zero segment (left side of chart) as soon as a purchase is made, and time starts all over again for these customers.  If they don’t make a purchase, they continue to slide out down the curves to the right.  Can you picture this activity in your mind?

You can have customers who stay at the top end of the graph, rapidly cycling round back to zero weeks each time they purchase.  You can have customers with longer cycles that loop back to zero weeks in slower purchase cycles from the middle.  You can have customers who purchase only once and every week just slide out further way from zero until they fall off at the right.

That means the same person might be in different  places on a curve in different weeks.  The same person can buy 2 weeks after last purchase or 4 weeks after last purchase, or a person can buy every week, or every month.  All of these purchase cycles summarized produce the series of likelihoods you see on the chart.

The point of the chart is, no matter which promotion customers are exposed to, no matter when their previous purchase was made (2 weeks ago or 20 weeks ago), their likelihood to purchase again can be very simply and accurately predicted by knowing one simple data point: weeks since last purchase.

Said another way, because this is a core concept to customization using behavior:

Customers with all kinds of different purchase patterns, demographics, categories of purchase, campaign exposure, and so forth tend to behave in the same way, that is, their likelihood to purchase at any given point in time from this  online store is primarily a function of how long it’s been since they last purchased from the store.

There are some pretty significant online marketing implications from a statement like that.  But how do you act on this information?

You’ve probably heard of the “sales pipeline” idea from B2B.  Sales management gathers data to inform them on which deals are likely to close and when, and build a flow chart of expected revenues.  This helps management take action on any deals that seem to be “floundering” -  special exec attention, discounts, bundling, etc.

You can do this in B2B because the value of the customers is usually quite high, and you have sales people or account managers who are close to the customer and can provide this data.

In B2C, you can’t afford to have account people for each customer, but using Recency you can predict which groups of customers are most likely to purchase again, and then build the same kind of sales pipeline.  And then, customize your Marketing action based on whether the customer seems likely to buy or is ”floundering” and drive increased profitability.

Building a sales pipeline model can also be used to predict how well the business will be doing in the future, and what kinds of products or tactics are really driving future profits.  Like other kinds of optimization, moving focus or resources towards products and tactics that are driving value, and away from those destroying it, results in a more profitable business.  But using Recency, instead of optimizing the Present, you are really optimizing the Future.

Look at the chart above.  There is a discount promotion and a free shipping promotion.  The coupon promotion outperforms the free shipping promotion as long as the customer has purchased in the past 6 weeks.  After this point, free shipping outperforms coupons.  That is something, as a Marketer, I think I’d like to know.  It means to optimize this system, I should deliver campaigns not based on my calendar, but based on the customer’s calendar as evidenced by their purchase cycle behavior.

Similarly, around week 8 since last purchase, coupon performance drops below the baseline performance of people in the loyalty program.  And finally, at 20 weeks, coupon performance is basically equal to the Natural buying rate, meaning virtually everyone using a coupon would have purchased anyway without the coupon.

Please understand, I’m not saying these Recency curves will be the same for your commerce site – they will depend on the type of products you sell, how good your service is, and so forth.  You have to do your own analysis.  What I am saying is the Recency effect is universal and can be the most important variable you could ever use for segmentation if you are concerned about campaign profitability.

For a practical perspective however, data in the format above is difficult to use and explain to other folks.  I much prefer what I call the LifeCycle Grid format below, click to enlarge:

People are more used to seeing data in a format where “up and to the right = better” so I have flipped the zero Recency boundary to the right side.  The customers with the lowest future value are in the lower left (Pink) and highest future value are in the upper right (Green).  I have also cross-tabbed Recency with Frequency so we have an idea of the value of a customer; the value of the customer helps decide how to approach the customer.   For Recency, we have chosen “hard breaks” rather than a smooth curve.  This creates specific populations so we can target certain groups and measure results.

Example:  If I send a 10% off promotion to all customers, you will see dramatic differences in response and profitability across these different cells.  Working the grid this way with various offers, you will find that allocating the same Marketing budget and promotions evenly across all the cells is truly a suboptimal approach.

Additionally, the general location of the cell gives clues to customizing campaign content or angle of attack as well as customizing the offers.  In general, for the four colored segments:

Green: Best customers who are Engaged – this is a segment where aspirational messages and services are extremely effective.  Think ”Special VIP treatment” in campaign copy and offers.

Orange: Best customers with declining likelihood to purchase again – if you are truly customer-centric, it’s time to analyze (or survey) these customers for broken products, processes, and service.  Why is a best customer dis-engaging?  Can we help you?  Did we do something wrong?  Would you recommend us?

Yellow: Potential Best Customers – new customers and those who are “floundering”.  What can you do to turn them on?  This is a group that benefits from category or affinity analysis to inform campaign content; help them try new product ideas.

Pink: Defected Low Value Customers - high value, broad discounting (30% off anything) is probably the only thing that’s going to drive response from this group – is it really worth it / do you actually generate profits here?

From a management perspective, feeding specific populations through the Grids can inform strategic decisions.  If you believe the Grids essentially represent a sales pipeline, then how do the pipelines for different customer segmentations compare?

An obvious place to start is Campaigns – what do the sales pipelines look like for different Campaigns, which Campaigns generate the highest percentage Green segment 1 month after Campaign drop?  What about at the end of month 3?

Run Product or Category analysis through the Grids.  For example, new customers whose first purchase is in a certain category – does this category create customers with high pipeline value?  What about customers who continue to buy in the category?  Softgoods versus hard goods?  Software versus hardware?  Shouldn’t we feature products that drive high pipeline value in campaigns and on the home page, as opposed to products that generate 1x buyers?

How about channel analysis, which sources generate new customers with the highest likelihood to continue purchasing?  Are most of our PPC customers in the Green segment, and most of our Affiliate customers in the Pink segment?  Where do the Social customers end up?  At 1 month after first purchase?  At the end of month 3?

The beauty of this approach is it can be used over and over, on any platform, in just about any situation, to answer the same question: which activities generate customers with the highest future value?  The Grids provides a consistent way to compare investments in all types of activities – products, campaigns, service initiatives, usability, centricity.  Just take the population exposed to the test, run them through the Grid, and compare to average (or better yet, control).

Most Marketers grew up with a linear view of execution – just keep Pushing, the more impressions the better.  Taking this approach in an Interactive environment completely ignores the fact that many customers will come back and Purchase again without any Push at all - and especially so if you are nailing all the centricity angles.

The trick is to optimizing Interactive commerce for Profit is:

1.  Understand which tactics create customers with high pipeline value – those likely to re-purchase on their own - then,

2.  Take Marketing action based not on a linear calendar, but a cyclical one – the calendar defined by the customer’s own behavior, customizing the message by location of the customer in the purchase likelihood Grid.

Execution Tips: List selection for this customization program is easily automated, right?  Just use the Grid cell boundaries as selection variables.  Many people decide to keep a regular generic “Brand” email communication to all customers while running the hyper-targeted communications based on cycle behavior underneath.  In this case, consider backing off discounting in the Brand communication and stick to new products, new hires, content marketing, etc. and let the cycle-driven email handle the behavioral discount program.  Test for the optimal balance / frequency between the 2 different emails by tagging e-mails with Grid cell.

Questions on this?  Also, with this background you might now want to read my review of the study.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Acting on Buyer Engagement

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

 Q:  I am a big fan of your web site and read your Drilling Down book. Great work!

A:  Thanks for the kind words!

Q:  I was wondering if you could help me picking the right control group size for a project of ours?  The population is 25 million telco customers that for which we want to do a long term impact analysis (month by month) in regards to revenue increase versus control group.  The marketing initiatives are mix of retention, lifecycle and tactical/seasonal activities.  We want to measure revenue increase through any of the marketing activities compared to control group.

A:   Great project, this is the kind of idea that can really improve margins if you can find out which specific tactics drop the most profit to the bottom line.

Q:   I have searched the web for some help and found calculators that say: On 25 million and smallest expected uplift of 0.1% and highest likely rate of > 5% the calculator gives 250k (1%).  Is that sufficient to calculate the net impact on the remaining base?  Would be very grateful if you could give me your thoughts.

A:  Well, it could be and might not be…

If you were manufacturing widgets, where the outcomes are clear (unit is defective or not defective), you might use this approach to the question.  But in Marketing we’re talking about human behavior, and there is quite a lot more variability in outcomes and more room for interpretation.  You can encounter a number of problems down the road by running a control so “tight” to the statistically correct size.

From a practical perspective, when you do a test of this magnitude (and I assume strategic importance), you don’t want test to just “beat control”, you want to beat control beyond a shadow of any executive’s possible doubt.

From personal experience, I can tell you that executives tend to be non-believers with a 1% control versus a 5% control or a 10% control. So some of this control size choice is culture-based – if the exec team is a bunch of engineers that understand / believe in statistical sampling methods, then 1% is probably OK in terms of believing the results are predictive of future events.

But if you need to convince a CFO or somebody who will be working from gut or risk management rather than “science” then 1% may not be enough, there is too much perceived “room for error” with a 1% sample (even with the science).

This is in effect a “perceived confidence interval” argument – the difference between 95% confidence and 99.999% confidence. Engineers may be OK with 95% because they intimately understand the derivation of it; CFO’s not so much.  CFO’s may not even understand the math behind confidence but intuitively, they perceive that 10% control is “more likely to be accurate” than 1%.

Said another way, do you want people to argue about the math and stats and waver on their belief in the outcome, or do you want them to just look at a simple chart of test versus control numbers and say, “Congratulations, that’s a tremendous success!”.  A 10% control gets you complete agreement on the results without any quibbling.  At 1%, you may get “what about the chance we are wrong” arguments.

Now, there are financial implications to using very large controls – some positive (reduced expense), and some negative (potential revenue foregone).  So choosing control group size can be impacted by these other issues.  In small population tests these financial impacts are usually quite small, so negligible and I always go for large controls.

But in a population of 25 million, maybe not so.

Which brings us to the second consideration -  segmentation or “drill down” after the test.

Nothing is quite so painful as gearing up for a test of this magnitude, producing a stunning positive result on a “macro” basis across all initiatives, and then having the execs ask, “What is the driving force behind this increased profitability in the test group?  Is it retention, lifecycle or tactical / seasonal?”  Or as often happens in telco (usually from an ops GM or VP), “What was the result of this test in my region or on my platform?”

Uggghh…

With a 1% control across the entire population, you frequently are “boxed in” when it comes to sub-populations because you lose significance (both perceived and scientific) as you drill in.  You may be OK on a couple of large scale events on large populations, but as we know, every answer begs another question and you can run out of statistically significant answers pretty quickly.  If you use a large control at the macro level, you are (as a rough example) 99% confident at the macro level, 98% confident one segment down, 97% confident two segments down, 95% confident three segments down, etc.

One way to handle this is to build the test from subsegments up to the macro level.  Let’s say at a minimum you want 3 subsegments in the test – retention, lifecycle or tactical / seasonal – and each of these you want to be 95% confident in.  Since some of these programs are triggered by behavior (lifecycle) and some by calendar (seasonal) I’d guess the sizes of the populations and number of executions could be vastly different.  Meaning, you may only need 1% control on the seasonal promotions but more like 5% or 10% control for some of the lifecycle stuff to be 95% confident on the outcomes of those.

When you sum all these segments up, you often end up with more like 2% or 3% of the entire population in control groups to always be at least 95% confident at all the desired subsegments, which means you end up with even higher confidence at the macro “all campaigns” level – a very good thing.

And much better than trying to explain why you can’t answer a subsegment question because you used 250K instead of 400K or 600K in the control group, if you know what I mean!  That’s when people forget the arguments about foregone revenue and start saying stuff like “Why did you not use a larger control group for this test?”

In the end, you will thank yourself again and again for using a larger than minimum required control at the macro level because you WILL come up with that unexpected “must know” question and be thrilled to find out you actually can answer it at a decent level of confidence.

Good luck with it, let me know what you learn!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Choosing the Size of Control Groups

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: You kindly clarified a few issues when I was reading Drilling Down earlier this year – so I hope you don’t mind the direct email.

A: Yes, I remember!

I am working for www.XYZ.com, a social networking / virtual world site based abroad but visitors are 85% US.

Our growth up to now has been mainly viral and in the summer we hit 1.2M UVs operating on the Freemium model with only 5% of our registered users converting to paying customers and a significant portion of our revenue coming from ads.  On average our customers are active on the site for something like 4 months making their first purchase around day 28. 

But to take us to the next stage we are embarking on some marketing for the first time using AdWords and various revenue share campaigns, and of course to do this sensibly we need to arrive at a reasonable estimate of LTV.

A: Makes sense!

Q: To calculate an adjusted LTV I removed all customers with a lifetime of less than 4 months but this gives a low estimate as this calculation ignores the bumper summer months and the extra paid for features put in place earlier this year.  Calculating LTV using ARPU and monthly churn (not sure how to calculate this in our environment) gives another different estimate.  Is there any help or advice you could perhaps give us?  If not in the US then perhaps you could recommend somebody abroad – can’t find anything in the literature relevant for start-up like us.

A:  It sounds to me like you’re trying to make this too complicated, at least for the place you are at this time.  Monthly churn and the “28 day” threshold are nice to know on a tactical level, but LTV is more of a Strategic idea that does not necessarily benefit from analysis at that level.  And you may not really want LTV, but a derivative that might be more helpful.

Let’s say the average user sticks around 4 months.  Say also that you generate revenues of $1 million over that period, and 1,000,000 users had some level of activity.  So your revs per active user are $1.  In terms of generating net revenue, you want to acquire  users for less than $1.

Now, we know that number is topline, and obviously there are expenses.  Companies like yours do not have very straightforward financial models because of the large amount of R & D that may be capitalized rather than expensed.  So you need to go to someone in Finance and determine what the right number is to use for looking at ROI.  Ask them, what percent of our revenues are left over to pay bills?  Or, what number would you like to see increased through a Marketing program?

Is it cash flow? Earnings before Interest, Taxes, Depreciation, Amortization?  Gross Margin?  Some other?  Then, what percent of sales does this number generally run at?

Let’s say it’s 40%.  In other words, 40% of revenues is actually available to pay bills and so forth.  So in the example above, .4 x $1 = 40 cents, which is the max you can pay to acquire a user, and anything less than that generates money to pay bills.

This method of course looks at all revenues. Not sure why you would want to look at it any differently, since even users that don’t “purchase” still generate ad revenues.

But let’s say you want to be more specific, you care only about buyers and only want to run campaigns that generate buyers.  In other words, the advertising revenue is “nice to have” but you want to build out the paid marketing model based on the acquisition of visitors likely to purchase.  You can run the same model above, but only look at the known the buyer group.

Take any 4 month period, find revenues from purchases and divide by number of people purchasing (not purchases, but individuals who purchase, revenue / user).  Then apply the same 40% flow through from the model above, and that’s the max you can pay to acquire a buyer.

When you aggregate known buyers, segment by source and you will find different campaigns generate different kinds of buyers; some will stick around longer than 4 months, some less.

Segment these folks by campaign and run the same model as above, purchase revenue for the period they stick around divided by purchases times the 40% flow through.  That’s the max you can pay to acquire a buyer in that segment.  So you end up with (just guessing) being able to pay 5 cents for campaigns that generate people who stick around 2 months, 15 cents for people who stick for 3 months, and 40 cents for people who stick 4 months.

You can make LTV equations very complex, but often the point of the exercise is not really “what is the customer LTV?” it’s “how much should we spend to improve cash flow?” or something similar.  This is a much easier question to answer and often what the company really wants to know.

Said another way, it’s very difficult and often dangerous to peg an LTV number in a dynamic environment because there are so many potential changes that will impact it; LTV is a number you may fully understand 2 – 5 years from now.  Until then, you need something “close” that drives the same kind of thinking and action, and the approach above will get this done for you.

As things evolve, this number (called “flow through”, recently have heard it called “Near Term Value”) will basically approach true LTV as you extend the number of months in the measurement period.

At the point where your business is somewhat static at an operational level, you can then look for true LTV by examining the revenue of actual defectors.  This is the only way to really peg LTV – after users have left and sufficient time has elapsed where you do not believe they will come back by themselves.

Until then, you are better off trying to figure out how much you can pay to attract high  quality user / buyer segments.

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Customer Value in the Freemium Model

Their are 2 primary challenges to implementing this idea.

1.  “Over time” is a concept many management folks have a hard time embracing; what matters are the profits this year, or this quarter, or this month.  Unless the whole company embraces an “over time” measurement approach it is difficult for Marketers and Analysts to drive towards programs and practices supporting the LTV outcome.

2.  The $2500 is an average figure.  Most customers are worth less; 10% or 20% are worth much more.

Most people I talk to embrace the general idea of LTV models intuitively.  It’s really a cash flow concept, isn’t it?

So Financial people get it right away, and if Marketers could align with it, there would be no conflicts and the Marketing budget becomes virtually unlimited.

In fact, many folks in the PPC world follow just this model – they have unlimited budget as long as each conversion costs no more than “X”.  Because the company knows if it spends no more than X on a conversion, it always makes money.   Marketers and Analysts involved with these “Cost < X” PPC programs love them, because Management loves them. 

And Management loves them, why?  Because the CFO loves these programs  Why?  Because they are based on Cash Flow analysis, which CFO’s understand very, very well.

So then, what will it take to get more acquisition budgets like these Cost < X  PPC programs?  We have to address the two challenges above:

1.   In the near term, forget about “LifeTime” Value, because most companies lack an incentive structure to support the LifeTime of anything.  Create a benchmark called “6 month value” or for a fast cycle business, “3 month value”, or “1 year value” if appropriate. 

At HSN, we mostly talked in terms of “90 day value” because the CFO wanted to link to the quarterly reports.  Challenging benchmark to achieve for a Marketer?  Sure, but the payback to Marketing was unlimited budgets for any program generating threshold ROI (90-day T-bill interest rate) in 90 days.  

Update:  Question - why did CFO choose this 90-day benchmark?

CFO’s have to invest Cash that comes in net of expenses going out.  They frequently put this Cash in T-Bills because it’s a reliable short-term investment, but they are under pressure to optimize these returns.  If they can get a better return investing in Marketing, they will do it – as long as they can prove these returns are “real”.   This is where Analysts play – proving the ROI of a Marketing effort beats the current 90-day T-bill rate.  How?  Use control groups.

Then score and rank the performance of programs – and the people and management responsible for them – using a simple customer value model that aligns the performance of marketing with the Cash Flow model - like this one.

2.  To optimize profitability, recognize the $2500 number is an average, and the benchmark value (6 or 3 month) of a customer is largely determined by the media / offer and product being offered in the acquisition campaign.

Segment by campaign and product to find your X month value, and use the Cost < X from PPC to determine what can be paid to acquire customers in each segment.

If the ”Cost < X” number in PPC is 15% of sales, then a $100 6 month customer can be acquired for $15, a $1000 6 month customer can be acquired for $150, and so forth.

This approach should make total sense to a CFO who loves the Cost < X PPC model.  And the interesting thing is, if you can get this X Month Value play right, LTV frequently (but not always) takes care of itself.  In other words, the Marketing risk is really in the front end; the back end value post “X Month” is profit gravy.

And this approach should make sense to you as well. 

Because longer term, once you start looking at your business in this way, you will have the confidence of management and the tools to measure the impact of just about any  product or program - including Service, Usability, Centricity - in the same financially-oriented, Cash Flow modeled way.

This methodology becomes a pattern, a part of the culture; a universal tool to measure the success of any Marketing or Service effort using a single, common-across-all-domains metric: ”X month” customer value.

Once that happens, you’re not just optimizing Campaigns, you’re optimizing the whole business.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

“X Month” Value

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q: I have recently purchased your book Drilling Down and going through the many interesting concepts.

A: Thanks for that!

Q:  I work for a membership Organization and we would like to conduct some analysis into who we may lose and approach them even before their membership lapses.  But the only problem here is that we carry data only on the purchases made (though many of our members do not purchase our products and stay a member) and web site visits.

A:  Are you *sure* that’s all the data you collect?  I once worked with a professional membership org that thought they only had one data source, but turns out they had 8 – from 8 different areas of the org – that nobody really knew about.

Q:  How do I know if a particular member is going to resign and lapse soon with this limited amount of behavioral data.  Recently it’s been a concern that we are losing members who have been with us for more than 10 years and who are in their mid career profession (aged between 30 to 45) and indicated no specific reason for resignation. 

This has been going on for the last few months and now we would like to strategically target these customers and approach them even before they react negative.  What concepts could help me to do this? Your guidance would be much appreciated.

A:  OK, my answer will be in two sections: if you (hopefully) find you have more data than you think, and if you really don’t have any other data to fall back on.

The first thing you should do is make sure you don’t have any other data sources.  Purchases and web sites visits might be the most obvious, but do you have:

Membership Campaigns
Conference registrations
Customer Service incidents
Local or regional meetings
Training sessions
Speakers
Orders for training materials

and so on.  In some orgs a lot of this is siloed  or even outsourced, but the data is still there – it’s a question of whether you can access it.   Think through the business model, and think about any possible interaction points with a member.  Then ask, Where would this data be?  You might be surprised at what you have.

Once you have more data, subscription relationship analysis of this type and predicting churn come down to three main thrusts:

1. Number of activities – similar to banking customer analysis, we often find that the number of different areas a member has tangible contact with is predictive of retention or defection.  For example, some members attend the annual conference and others do not.  Some members participate in local meetings and others do not.  Some buy training materials and others do not.  Some members do all three, some two, some one.

On average, the members engaged in all three activities are the most likely to remain members relative to those engaged in only two.  And members engaged in only two activities are the most likely to remain members relative to those engaged in only one.  This is a “run rate” kind of retention, an expectation.  From a Marketing perspective, this means you want to always be adding to the number of activities a member engages.

2. Change in number of activities – when you see a member drop from 3 to 2 activities, this is a clear signal that there is a problem of some kind, it’s a dis-engagement from the org.  You want to take action when you see these events, find out what is happening and if there is anything you can do to correct this.

The reason driving this downgrade may be a soft incident, say a content problem, or may be a hard incident, like payment problems. Either way, the org needs to find out if the issue can be addressed.  If you start to see “clustering” of these kinds of downgrades in relationship quality, it’s likely something more systemic is going on.  Often you will see certain segments who exhibit similar problems.

For example, members acquired through a certain publication may exhibit similar downgrade behavior at roughly the same interval from joining.  This is evidence of a systemic problem – something about folks from this source is unique, and for whatever reason, the org is not satisfying their needs.

3. Predicting change in activities – if you want to go further down this road and actually *predict* a change in the number of activities before it happens, you can look at the Recency within that activity.  A member who goes to conferences on a regular basis who then skips one is in danger of defecting from that activity – for some reason, the conferences are not providing the value they used to.

Or, something has changed with the member, their position in the LifeCycle has moved.  The conferences still provide the same value as they did before, but this value is shrinking for the member who (perhaps) needs more challenging or different content.

The same could be said for attending local meetings or buying training materials, etc.  For a known user of a specific activity, how long has it been since they used it?  Does this non-usage break an established pattern of usage?  If so, you have a triggering event for a marketing / membership intervention.

OK, so what if you really don’t have any more data?  You can create it.  One of the easiest and least intrusive ways to do this in a member org is with surveys.

Why?  Membership orgs have embedded permission to interact with members; it’s the nature of being part of such a group. What I mean by this is asking members to take part in a survey is not only quite natural, it’s often expected and perhaps even appreciated.  After all, what could be more aligned with a membership org than asking the members where the org should be headed and where they would like to see it go?

By thinking through and properly crafting such a survey, one should not only get a sense of potential friction points in the org overall but also get a sense at the individual level of which members are becoming dissatisfied and more likely to defect.  Implementation of a program like this means, of course, that the survey responses are tracked at the individual level so that action can be taken at the individual level. You can’t just do a random popup on the web site to make this work.

Moreover, in the out years, one can reverse engineer the reasons for defection.  Each year, analyze the population of members who defected in the previous year, and ask yourself, How are these people similar?  Do they come from similar backgrounds or industries?  Did they join in the same year? Come from the same marketing source?  You may already have such survey data and simply had not thought of using it for analyzing and predicting which members would defect.

Finally, I am fully aware that often, bringing these kinds of issues to the table can be painful for a membership org.  In my experience, many of these orgs are highly politicized and there is a certain “we’ve done it this way for 100 years” attitude.  When you present data that contradicts long-held beliefs there can be tension.  And this is fine - when the org is on board with the project.

So my final advice would be this – whether you have the data or generate it, the success of a project like this can revolve around the strong commitment of the org to actually do something about member defection. Ask this question of management first:

If we find members are leaving because of something we are doing or not doing, will we change this, even if the problem contradicts long-held beliefs?  The answer will show you how committed the org is to fixing member retention.

“It depends” is not the answer you want to hear from these senior players, because this means they don’t have a retention problem, they have an acquisition problem.

Unless they are willing to change, they need to recruit more members that are not like the members they have if they want to decrease defection in the membership.

Hope that helps!

Jim

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Member Retention in Professional Orgs

Want to see the answers to previous questions?  Here’s the blog archive; the pre-blog newsletter archives are here.

Q:  I am hoping you can help answer a question for our team.  By way of introduction, I am the CEO of XXXX.  We are a specialty retailer / restaurant of gourmet pizza, salads and sandwiches.  We would like to know  restaurant industry averages (pizza industry if possible) for customer retention – What percentage of customers that have ordered once from a particular restaurant order from them a second time?  I am hoping with your years of expertise and harnessing data you may be able to assist us with this question.  Look forward to hearing from you.

A:  Unfortunately, in those said years of experience, I have found little hard information on customer retention rates in QSR and restaurants in general (if anyone has data, please leave in Comments).  It’s just the nature of the business that little hard data, if collected, is stored in such a way that one can aggregate at the customer level.  The high percentage of cash transactions doesn’t help matters much; there’s a lot of data missing.

Over the years, sometimes you see data leak out for tests of loyalty programs, and of course clients sometimes have anecdotal or survey data, but this is not much help in getting to a “true” retention rate.  More often than not you discover serious biases in the way the data was collected so at best, you have a biased view of a narrow segment.  Often what you get is a notion of retention among best customers, or customers willing to sign up for a loyalty card, but not all customers.  And the large “middle” group of customers is where all the Marketing leverage is.

What to do about this predicament?  

There are really two issues in your question; the idea of using industry benchmarks when analyzing customer performance, and the measurement of retention in restaurants.

As far as industry benchmarking, two things:

1.  Annual reports for publicly traded eateries may be of help.  Customer loyalty info may be disclosed in these documents or conference calls with Wall Street.  Still, it will probably be of the quality referenced above – narrow in scope or behaviorally biased.

Sometimes you can put snippets of different conversations into an equation that allows you to guess at repeat purchase rate; hospitality analysts often want to understand repeat behavior and do this kind of forecasting.

2.  Ignore the industry benchmarks.  If you have the capability to track repeat rates, simply establish what they are now and use them as internal benchmarks to not fall below or create programs to improve against them.  

Frankly, I tend to discourage using “industry benchmarks” because the kinds of businesses that can really leverage repeat behavior and retention (customer-centric model) are usually *different* from the industry, so using a benchmark (say, from Domino’s) is probably low-balling your potential.  

Not that Domino’s is a “bad” operation, mind you, but they are what they are, they tend to be more on the operational excellence side of the game than customer intimacy (that’s what we called the customer-centric / social approach back in the early 90’s). 

Product leadership, the 3rd value discipline, is pretty much table stakes for anyone in the restaurant biz, and I assume from your business description you just might consider this a primary focus which you then leverage to create power in the intimacy area.  This is essentially the Apple Strategic model.  If the product is not great, the love will not come.

My point is this: without understanding the value discipline and Strategy of a competitor, you can’t know if any benchmark is something you want to compare to, because the business may have a completely different focus than yours.  Worse, using industry averages simply hides any real information you might gain that is actionable for your business.

For example, even though Walmart and Nieman Marcus are in the same business, I don’t think anyone would say they have the same Marketing Strategy or core value proposition.  Walmart is of course the poster child for operational excellence with the end result being value pricing, which flows to the advertising content.  There’s nothing “wrong” with this approach, it simply is what it is, and customer intimacy / relational / social marketing simply doesn’t really fit here.  You certainly can try to be as intimate as possible; but it must be done within the constraints of the model and not reduce operational excellence.  Importantly, this is a “mass” concept, so Push media is the most effective.

Sam’s Club is an example of how one might accomplish this mix.  A “membership” is certainly more customer intimate and allows customized communication, a key component of customer intimate execution.  Again, this flows into the advertising content.  Sam’s gets to leverage the Walmart infra, so they can at the same time maintain a decent level of operational excellence.  Remains to be seen if they could do so without Walmart.

Nieman Marcus on the other hand uses a customer intimate value proposition, and their execution reflects that.  Value pricing is traded off for a high level of customization and personal service, where repeat business is very important since the number of customers this proposition attracts is smaller than the “mass” approach;  you have fewer, but each more valuable, customers.  In this model, mass media is not very effective because the audience is not mass; instead, you rely on the intimacy to Pull customers in, and much more of the Marketing budget is invested not in Advertising, but on in-store (employees, fixtures, locations) and individual communication. 

This relational or customer intimate model is the root of  ”social marketing” and why any attempt to turn online social activity into some kind of mass media advertising opportunity is a complete Paradox.  A step by step example of optimizing the relationship marketing / social model is here: Marketing Bands Series.  To optimize the social model, you divert Marketing budgets away from Mass Advertising and Push into Pull areas like Usability / Store / Interfaces / Packaging, Customer Service, and Customer Retention.

Given the above, would Nieman Marcus ever consider using Walmart’s customer retention rate as a benchmark?  I think not; this approach would make no sense at all.  The mass model can’t leverage customer retention because it’s not intimate; if you can’t act on the metric, why measure it?  This is not to say Walmart “doesn’t care” about repeat business, of course they do.  But they can’t really lever it because it’s more operationally efficient for them to use the mass approach.

That’s a very long explanation for why I dislike using industry benchmarks but many, many people don’t realize how important this idea is; it’s why on a core business model basis some companies will not be able to realize significant benefits from “going “social”.  So on the whole, I would much rather use internal benchmarks that I can improve on that are aligned with the business drivers and are controllable through my own execution.

From looking at your web site, I’d judge you a Nieman as opposed to a Walmart, so customer retention can be a powerful tool for you.  So let’s talk about measuring retention.

“Retention” is a very time-specific concept – over the course of 3 months?  A year?  Five years?  A 20% retention rate over a 5 year period and a 60% retention rate over a 3 month period might both be stunning achievements, if you know what I mean.

So, if you are able to do the analysis, I would pick some marks – 3 month, 6 month, 1 year, etc. – and see what you get for repeat buyer or retention rates.  The slope of that curve will determine where any danger points are that you might take action on.  

For example, if retention falls dramatically from 3 to 6 months, then you know that you should be watching for people who have not transacted in over 3 months, and for  those people you should craft mail / e-mail promotions designed to bring them back.

As often happens with restaurants, there’s probably a good chance that if the person is still living in the area (more on this below), the reason they are not coming back is probably  controllable – they had a bad experience.  A promotion like “We’ve missed you” or “Give us another chance” that is tightly targeted to known defectors will usually pay back quite handsomely in both the short and long term. Defected customers not only visit once on the promo but also (hopefully) have a better experience and re-engage as a repeat visitor.  If your value prop is customer intimate / social, you absolutely must invest in superior customer experience so repeat experiences are rewarding.

If you see some success with this approach, you could then fine tune the analysis to find out if the dropout has a peak in month 3, 4, or 5.  This fine tunes timing of your drop; the closer you can get to the behavior with the message the more effective the campaign will  be.  There is a “peak profitability” timing in one of these months.  

Then the program can be automated, for example: if we don’t see a transaction from this person for 120 days, drop the message.  This way, you end up mailing every month but the audience is completely different and very highly targeted each and every time.  You will find this “right message, to the right person, at the right time” approach is much more profitable than mailing all customers because it directly leverages the customer intimate value prop.

Speaking of mailing all customers, the people who are still active within this 4 month time frame are probably still loyal and you can improve overall margin by not sending these special promotions to those people until they “slip” out of the 4 month window.  There’s no reason to discount to people who are highly likely to purchase anyway.  This is the Pull part of a relationship or social  execution.  What you should be really concerned about are the people who are dis-engaging, where there has been product or service failure.

In fact, in a relational marketing scenario, there is no real need to market to these people at all, you’re basically “preaching to the choir” (example) and doing so is a waste of resources (and often margin).  You will be far better off taking the money you used to spend marketing to the choir and allocating it to in-store, core value proposition ideas.

Many marketing people (especially of the Push variety) find this difficult to understand, but there no more powerful Marketing tool than your value proposition when communicating to the active customer base.  It’s why they are coming back, your Pull is already strong with them.  Why beat them over the head with messages when they are telling you by continued transacting that they like what you are doing?  Wasteful.  (more detailed example)

Finally, in a location-based scenario such as restaurants (and since you are the CEO and not running a single store), you might consider factoring in local uncontrollable churn into any metrics you create as internal benchmarks.  

Households in different areas have different natural churn (move) rates.  Since you have stores in different states, for example, one would expect a lower retention rate from stores that have a higher natural household churn rate.  These stores might be doing very well with controllable churn (product, service) but without the household churn adjustment, they could be unfairly benchmarked “bad”.  HH churn numbers are generally available free from city / state government or the Census.

Hope that helps!

Jim

Note to blog readers: Do you see the parallels above to a lot of what is going on in online publishing / advertising / marketing?  If not, see Jonathan Mendez’s Reaping the Ads You Sow for a more direct analysis of the same concept online.  The strength of the web is in Pull, in converting demand, not Push or creating it.  Use offline for Push; that’s what it’s good at, and synch the two to optimize the entire Marketing ecosystem.

Have a question on Customer Valuation, Retention, Loyalty, or Defection? Go ahead and send it to me here. If on the topic above, you can leave a comment on the post:

Relational vs. Transactional