As you said, there is at least one big difference between this group and test – you already know they can’t stand your company (if unmailable = unsubscribe) or are so indifferent to the company they didn’t maintain their email address (if unmailable = undeliverable).

I can’t think of any better way to ensure your performance looks brilliant than to use unmailables as a control group – since they are guaranteed not to respond ;)

Said another way, these “sequential touch models” imply a level of accuracy that simply does not exist, given we know there is so much missing data form technical issues, cookie problems, cross-device / platform usage, etc. So I question why people would want to spend a lot of precious time trying to quantify this kind of detail in the “process”.

I’m willing to say, Hey, I don’t know exactly how this works, but the outcome is consistent and I’d rather spend my time finding more of these consistent ways to make money than chasing down the details of exactly how it works. It’s much more efficient to work at the “mix” level testing different levels of inputs than to try to chase down the individual contact sequences, which in many cases are not accurate anyway.

After all, what would you *do* if you knew a certain specific sequence was desirable, how would you force it to happen?

If by “Matrix” you mean “media mix”, that’s beyond my skill set…

Some insights we found have stuck with me: those acquisition sources that generated the largest long term value per customer also generated the most value as an acquisition source compared to those media channels that generated customers with low long term value.

The conversion ratio was insignificant in determining if an acquisition media channel was going to generate big long-term returns or small return. For example we had one media channel in the top 5 ranked (based on long term value) with the second best conversion rate and another with the 61st best conversion rate

Thus I now know that long-term value of a customer is more important to judging an acquisition media channel than conversion rate.

The media Mix model sounds interesting but tough to implement. I like the metaphor with the targets but got confused when you said you go for precision rather than accuracy and wanted to clarify this with you. As with regards to outcomes I would, as a rule of thumb, focus on accuracy first and then precision after (if there was value in making the outcomes more precise). Does not mean this is correct in all situations though.

If I look at your target diagram it illustrates to me that while the outcome is always the same for the same inputs, it is off target. And it’s always off target by a similar amount. Whereas, The target that represents accuracy is more volatile in its outcomes (less precise) but overall the outcome will be, on target.

Why would I want a model that is less volatile, but off target when I can have one that is on target but more volatile (i.e. occasionally the shot is way off target – hits the outer ring)?

After reading further through your text you state the following

“If you are looking to build credibility and consistency in your analytical practice, I’d lean towards precision over accuracy; most management people really do not care about the details of how you got there anyway.”

“These models are quite precise in terms of predicting outcome. But they never attempt to be accurate if you are looking for “how” the outcome occurs”

Maybe this is the difference between transparency and encapsulation (hidden information) on how the model works (how it obtained the result).

Taking stock, I think you are saying, from your experience, that you will sacrifice the accuracy of the model results/outcomes to gain the advantage of reduced volatility (more precision): thus you are reducing Risk as the expense of reducing the quality of the model: its accuracy.

I guess this is fine so long as the accuracy off the model is not scarified too much.
Do you have any more info on the process of building he matrix model?

Glad to see we’re not throwing out direct response measurement completely; it may be an incomplete picture but at least it’s precise and reliable, leading to sound business decisions. As a decision maker, give me this alternative over guesswork every time.

What’s kind of surprising to me is an increasing willingness among analysts to substitute guesswork for facts when faced with complex analytical challenges. As we move up the complexity chain, it’s up to the analyst to recognize what can and cannot be measured properly, and speak up rather than being bullied into turning speculation into fact.

There’s nothing wrong with speculation and gut feel based on experience – unless this information is labeled as a fact. If an analyst can’t measure an idea properly, they should say so and offer an alternative approach or outline what resources are needed to do the job properly.

And, in reality, I have nothing to add to your blog post. I agree with all the points you’ve made 100%.

I never intented for my posts to state that “Direct Response Measurement overall is a Wet Dream”.

In The Dangers of Direct Response Metrics for Online Retailers my intent was to demonstrate that it’s dangerous to rely only on the simple direct conversion funnel (click > convert), which is still mostly used by (direct) marketers to evaluate media spend / campaign efficiency.

My key point was that the purchase decision process in most cases isn’t instant, and the tools we have available today do not make it easily possible to measure the impact of all the touch points leading to the purchase, and we therefore should not rely only on direct response when optimizing our marketing investments.

I’m a direct marketer by origin. I love direct response metrics. However, as stated in my post, they are not enough. We can no longer rely ONLY on direct conversion rates, cost per order etc.

Unfortunatelly, most still do that today. It’s not just the marketers. I’ve met many who try to do things differently, but are then blocked by “black & white data” oriented CEOs and CFOs.

In Moving Beyond Direct Conversions (1): Adapt to the Purchase Cycle I tried to expand on the idea, and proposed a) adapting goal KPIs to campaign goals (directly related with the purchase cycle stage they are primarily targeting) and b) expanding our basic direct response metrics “pool” with not only assisted conversions, but also engagement and other softer metrics.

But, the bottom line is, I completely agree with all of your points.

I might have been somewhat too aggressive in trying to bring my point home. A large part of that is due to the mistakes I’ve made myself in the past.

Almost a decade ago, when I was more or less starting in online retail and building my team, my focus was almost exclusively on direct response metrics. I was all about direct CRs, direct CPOs etc. All I cared about was direct campaign / media investment profitability (not counting lead generation, though). And I trained my team the same way.

My first real eye-opener was an in-depth analysis we did on the impact of our email program, finding that it has about a 3x stronger indirect impact VS direct conversion impact.

Later, with more experience, I started implementing more complex analytical frameworks, but a lot of the damage has already been done. I had already very successfully “converted” a lot of the people to the “wrong path” (yeah, I know, sounds a little too poetic:) — to such an extent, that it than proved difficult to correct my own mistakes.

So, much of my vigor comes from my own frustration over the mistakes I’ve made in my early days as a marketer/online retailer.

The second reason is that I’ve seen far too many marketers, CEOs and CFOs focus only on measuring and optimizing for the direct response, consequently blocking their customer growth.

Jim, thank you for the discussion. Enjoyed it as always!

The key to deriving this insight was deliberately starting the campaign as “social only” with no other media and extensive measurement capabilities already onboard. Then, gradually launching new layers of media, each with their own tracking so incremental value could be established and media interaction measured. Thanks to Shaina Boone at Critical Mass for a fantastic job managing the Leaf project with such measurement discipline.

As I recall, Old Spice had a very heavy TV sched which is not mentioned much when people sing the glories of how successful a “social” campaign that was…