Test planning and developing optimization strategies is a long, bumpy road of trial and error before you can reach a true discovery about your customers.

This is where I should also include an important caveat – even the experts can lose their way if they are not careful.

Today’s MarketingExperiments blog post features three insights from our research lab that we use to keep our test planning and optimization strategies on the right track.

Look for ideas beyond our own

For starters, we recognize no digital marketer – or scientist, for that matter – is an island.

We use peer review sessions to come together for open discussion and brainstorming on how we can develop testing and optimization strategies to help our Research Partners learn more about their customers.

These collaborative sessions range from fun to intense, but overall, they produce collaborative solutions to challenges that would have unlikely been discovered otherwise.

Use a proven methodology to turn those ideas into tests

Turning the ideas gained from brainstorming into tests that will help our Research Partners learn more about their customers is not easy – especially when considering a few of the important details of test planning:

• Should we use a radical redesign approach?

• What metrics should we us to track and evaluate tests?

• How do we balance risk and reward?

These are just a few of the challenges to consider in test planning. But, what about optimization?

Test planning is useless until we can understand how elements on a page influence conversion. While there are many ways to do this, one of the simplest – and likely most effective – is by learning how to see your marketing through the eyes of your customers.

To help us see through the eyes of a customer, we use the MECLABS Conversion Heuristic as a tool to help our optimization analysis.

Don’t overlook opportunities for value proposition development

A value proposition is the reason why your ideal customers should buy from you rather than any of your competitors.

So, if your value proposition is unknown or unclear, then you’re likely leaving revenue on the table as your ideal customers look to competitors for choices that provide them the most value.

Although test planning can help you build your customer theory, taking the time for value proposition development can help you build marketing that helps your customers move through your sales funnel with confidence.

Related Resources:

Analytics and Testing: 3 tips to optimize your testing efforts

Marketing Optimization: 10 resources to help your online testing efforts

Conversion Rate Optimization: Building to the Ultimate Yes

Numbers breed confidence. Sometimes, false confidence. So, it’s not only vital to understand what your test results are telling you, but the limitations of those results as well. This understanding should shape how you interpret and present results to your clients or senior management.

Consider the source

Your analysis is only as good as the data upon which it is based. Understanding the limitations of your data and the way it was pulled from databases will assist you in designing the most ideal analysis.

In the case of MECLABS’ website testing, this includes understanding how the various testing and metrics platforms define and record key metrics such as “visitors,” “visits” and “conversion.”

It’s not just what you’re saying…

One of the challenges in interpreting results in the role of a consultant is the constant struggle between projecting confidence in the reliability of those results and a scientist’s obligation to fairly portray the limitations that exist.

To help with this, when reporting results, I generally try to avoid any language that suggests total certainty.

Here are a few examples that come to mind:

“Always,” “Never,” and “Must”

I also try to avoid undermining confidence in my results by using words like these:

“Unknown” “Speculate” and the always present “But.”

The reason for this is while it is necessary to express healthy levels of doubt, there is a delicate balance in doing so without undermining a client’s or leader’s trust in your results. After all, they will likely be used as a basis for vital business decisions.

If there are key limitations to a particular conclusion, certainly it is important to mention alongside that portion of the presentation. However, detailed information on overall limitations including data validity issues and analytical methodology should be included as footnotes or an appendix, separating them from the main presentation of results.

How you phrase analysis also comes into play during presentations, when business leaders or clients usually ask difficult questions. You may have immediate answers for some of their inquiries, but not for others. Ultimately it is how you engage clients that makes a difference in how they perceive the results.

For example, if there is no way to directly answer a question with the data at hand, is there perhaps a proxy for that data or a directional indication given by other related data?

A response of “we have no way to know” is unlikely to satisfy doubts, yet a response of “we cannot measure that directly, but if I performed an analysis of X,Y or Z, we could perhaps learn related facts” is likely more reassuring.

In short, don’t forget it’s not just what you’re saying in your analysis, but how you say it that matters.

Know when to walk away

If the limitations of an analysis are so great you as the data analyst do not trust the results, it is your obligation to discontinue the analysis.

Presenting an unreliable or incomplete analysis misleads your audience. The trust they place in you as their data analyst is valuable and not to be taken lightly.

Certainly if an analysis is placed in context of its limitations so the audience understands how broadly or narrowly those results can be generalized, there may be value in presenting the results of an analysis based on incomplete data, but if the data is unreliable or internally inconsistent to the point where data cleaning cannot repair your trust in the dataset, there is no reason to continue analysis.

It may be just as valuable for your client to issue recommendations for improving their data collection and to recommend future analyses to be carried out on that future clean data.

Related Resources:

How to Predict, with 90% Accuracy, Who Your Best Customers Will Be

Marketing Metrics: Can you have one number to rule them all?

Marketing Optimization: You can’t find the true answer without the right question

Testing can be frustrating at times.

You have set objectives and reasonable expectations that your new treatment design will outperform the current champion.

And then it happens …

Your treatments are crushed by the control. Nothing seems to work. After a few test cycles of consistent underperformance, it starts to feel easier to just throw in the towel and call it quits than to weather another decrease in conversion.

Or … instead … it may be time to consider a radical redesign test approach.

Today’s MarketingExperiments blog post will share two common reasons to consider this approach and help you assess your testing efforts.

What does “going radical” mean exactly?

Now before I go any further, let’s get clear on what I mean by a radical redesign test approach.

• A radical redesign is a test approach in which a test variation that contains multiple variables (often unrelated) is manipulated in an unstructured manner.

In short, it’s a big shakeup on a test variation that equates to a roll of the dice.

And while you still may be able to isolate some common denominators between the radical redesign and the control, consider a radical redesign as an approach that strategically neglects experimental method best practices (isolation of variables, etc.).

Because of this, radical redesigns are often regarded as a last ditch effort approach (or a “Hail Mary”).

So, with all of that said, here are two common potential reasons to go radical we often encounter in test planning with our Research Partners …

Reason #1: Nothing else seems to work! (Or, “I can’t move the needle!”)

If you find yourself running multiple single-factorial and variable cluster tests to no avail, then maybe it’s time to go back to the drawing board and create a new radical treatment that seeks to rejuvenate the testing process and offer a new baseline for subsequent incremental tests.

Reason #2: Risk is not a concern for you (or your boss)

If you’re looking to make an impact and aren’t currently too worried about the potential for loss, then radical redesigns are often an exciting test approach with high potential for large impacts.

Sure, it’s a roll of the dice as I mentioned earlier, but the potential learnings you can gain from having a window of consequence-free testing is an opportunity for reward that is few and far between.

Ready to jump in?

Great, here are a few resources to get you started …

[MarketingSherpa Webinar] Optimization: A discussion about an e-commerce company’s 500% sales increase

Homepage Optimization: Radical redesign ideas for multivariable testing

Homepage Optimization: How sharing ideas can lead to more diverse radical redesigns

Testing: Go big, or go home?

What to test (and how) to increase your ROI today

A/B Split Testing — How to use A/B Split Testing to Increase Conversion Rates, Challenge Assumptions and Solve Problems

One of the common questions I receive from Research Partners focuses on what metric they should use to track and evaluate tests. The tendency is often to want a single metric that defines the measure of success.

While it is important to gather consensus on which key performance indicators, or KPIs, will be used to evaluate tests early, there should never be a reliance solely on a single metric as the gatekeeper of success given your secondary metrics can provide just as much – if not more – insight into your visitors behavior.

In the land of testing, the marketer with one metric is not king…

If you are only using one metric, you are not seeing a full picture. Each of your KPIs tells a part of the story of performance. Only relying on one alone can mislead marketers to make poorly informed decisions.

For example, let’s say you’re testing a PPC ad. As you know, the sole purpose of an ad is to get the click and let the landing page do the selling. For this reason, you determine  your KPI is clickthrough rate since that is what the ad directly affects.

Makes sense, right?

Now let’s say that your results come back and show that both ads receive the same number of clicks and that there is no statistically significant difference in clickthrough rate.

So what happens now?

Since clickthrough rate was the only metric measured, then you may draw the conclusion that both ads perform the same and that either could be used to achieve the same result and in some cases you may be right…

However, making this assumption is a big risk that flirts heavily with a similar risk of assumption derived from artificial optimization.

What metrics and methodology can tell you about your customers

At MarketingExperiments, the element of motivation is given the most weight in our methodology for a reason – it can have a dramatic impact on performance that is not always apparent at first.

So while both ads had the same clickthrough, one may have attracted a group that was more qualified or motivated to move through the remainder of your sales funnel. Had only the clickthrough rate been used to analyze performance, it’s likely that valuable customer insights would have been missed and ROI would have been left on the table.

The big caveat here – don’t fall into the trap of assuming variables in your sales funnel that remain constant will continue to perform at a constant.

Related Resources:

Analytics & Testing: 3 statistical testing methods for building an advanced customer theory

Analytics and Testing: 3 tips to optimize your testing efforts

Marketing Optimization: The 3 phases of evidence-based marketing

One of the most effective ways to increase conversion is by decreasing the amount of perceived resistance and aggravation your prospects experience in your lead capture process – or in short, dialing down the levels of friction.

In today’s MarketingExperiments blog post, we’re going to get our hands a little dirty by diving into some evidence-based marketing to learn how the MECLABS research team made two simple changes to a lead capture process that increased lead rates 166%.

What is friction?

Before we get started, let’s get clear on what friction is exactly.

At MarketingExperiments, friction is defined as “a psychological resistance to a given element in a sales or sign-up process.” In other words, it’s a psychological element present in your marketing that prevents prospects from acting on your offer.

It’s also important to mention here that your goal is not to eliminate all friction. Anytime you ask for information, there is going to be at least some amount friction present. Instead, you want test and optimize your way into identifying and mitigating as much friction in your lead capture process as possible.

Now let’s take a look at the research notes for a little background …

Background: A luxury home builder seeking to sell homes to families with a higher-than-average income level

Goal: To increase the number of leads

Primary Research Question: Which color scheme will result in a higher conversion rate?

Approach: A/B multifactor split test

Control 

The control featured a two-step lead capture process that started with a “request more information” call-to-action that redirected prospects to a second form field page requiring users to provide their first and last name and email address into the form fields.

Treatment

For the treatment, the team removed the second step completely, changed the form field layout to appear fewer in number, made providing information optional for users and removed the questions and comments field.

Results

What you need to know

By minimizing friction through reducing the number of steps and fields, the treatment outperformed the control by 166%. To learn more about how friction impacts conversion; you can watch the free on-demand MarketingExperiments Web clinic replay of “Hidden Friction: The 6 silent killers of conversion.”

Related Resources:

B2B Marketing: Value proposition discussion with Dr. Flint McGlaughlin

B2B Lead Gen: A/B split test helps increase quote requests 262%

Marketing Analytics: 3 steps to help Sales and Marketing improve productivity

When I was in college, I took a class on complex analysis and after all the lectures, studying and nerve-racking exams, I learned one important thing about customer behavior – some characteristics of a person will likely contribute to their future behavior.

In other words, my grandparents are not likely to start buying iPods, but at the same time my younger sister and her friends are not going to go out and start buying rotary telephones either.

Many times, variables such as gender, age, income, education and geographic location will likely play a role in why your customers say yes to your offers. This brings me to my point that selecting a test methodology robust enough to explore statistical relationships among variables is more important than ever to your marketing efforts.

In today’s MarketingExperiments blog post, we will simplify three basic testing models you can use to build an advanced customer theory.

Our goal is not to give you a Ph.D. in statistics, but rather, we want to provide you with a few test methods simplified and free of as much mathspeak as possible you can use to aid your team’s next discussion on test selection.

Test Method #1: ANOVA (Analysis of Variance)

Marketers can use the ANOVA testing method to understand if a statistical significance exists within or between groups. Landing page optimization is a good example of how ANOVA testing can used to analyze a customer’s response to different treatments based on variables of interest.

For example, suppose you’re testing landing pages and you want to determine if the income or education level of new and return customers has any statistical significance on the probability of conversion on the landing pages you’re testing, then ANOVA would be the optimal test method to consider using.

Test Method #2: Logistic regressions

Logistics is a testing method for prediction analysis. In other words, a logistic regression test can help you to discover the statistical likelihood of a conversion for customers in demographic A versus customers in demographic B.

With logistic regressions however there is only one catch …

Customers in both demographic groups A and B have to be known as significant contributors to the likelihood of conversions.

Test Method #3: Time series analysis

Time series analysis is a test method similar to logistical regressions in that’s it has a basis in predictive analysis, but time series analysis is focused on what you can learn from historical data trends.

Understanding the seasonality behavior of your website traffic is a perfect example of when you would use a time series analysis.

These are just a few of the testing methods available to help you learn more about your customers, but ultimately no marketer is an island. So, if you have a testing method that you use to build your customer theory, feel free to share it with in the comments below.

Related Resources:

Marketing Optimization: How to design split tests and multi-factorial tests

Marketing Metrics: Why all numbers aren’t created equal

How to Predict, with 90% Accuracy, Who Your Best Customers Will Be

Online Marketing Tests: A data analyst’s view of balancing risk and reward