To that end, here is the final look at advanced optimization paradigms:

No More Focusing on Test Ideas –

If we view optimization as a discipline, one that never starts and never ends, one that is about the constant changing and learning of a user experience, then there is no longer any need for individual test ideas. An idea naturally has a start and an end, as any hypothesis comes from a belief in a specific solution to an existing problem. People get so caught on their idea, be it from their own experience, some piece of data that they just know means they have the solution to all your problems, or just “best practices” that their brains shut down and they stop trying to find the best answer. The problem here is the entire process leads to a myopia that gives us the right to stop, the ability to prove ourselves “right” and the natural affinity towards a set path.

If the focus is no longer on test ideas, however, then the system is what you focus on. If we are instead treating things as a cycle: explore, learn, ideate on all feasible alternatives, execute, learn, repeat, then there really is no individual campaign or test. The path is never about what you think will win, or what you want to tackle, but instead only on where the casual data leads you and the evaluation of all feasible alternatives. Test ideas become the least important thing you can discuss, and should be viewed with high levels of skepticism. There is no such thing as a good test idea, only a concept that can be broken apart, challenged, and improved. Fear any expert trying to tell you a single great test idea, or any guaranteed set of steps to improve your site, as they are only playing to your own insecurities; the reality is any single idea can not hold up to scrutiny. This impacts your own beliefs as much as any other, you have to hold yourself to the same level of scrutiny, not allowing what you want to happen to be the path that you go down or the answers you seek. You have to be willing to step outside your own opinion and be able to focus on all feasible alternatives and only what the efficiency of changes are; any bias that you allow to limit what you test, be it because of experience or popular opinion, devalues the outcome that you can generate.

The biggest challenge most people have is the feeling of a losing of control. We often like to blame other groups for this behavior, but by far the most guilty group are analysts who are so busy trying to prove a point with their data that they fail to see the larger picture. They so want to prove a path using their analytics that they fail to factor in the need to change to an active form of data acquisition in order to move forward. You have to worry about your own biases before you can stop others. It is easy for all groups to get focused on what their experience or gut tells them is right, often to poor and inefficient outcomes. Make it clear that no idea stands alone. Put in place measures to insure that you are not limited to popular opinion or only what you think or want to win. This often means that you have to prioritize resources in ways that you are not doing today, but ultimately this is the only way to insure you are getting the greatest value and insuring your own continued education about what the value of actions are.

Free yourself from the cycle of defending and pushing every idea, instead creating momentum and a consistent pattern of action. Everyone is afraid of moving towards the infamous 48 shades of blue extent of this path, but the reality is it frees you. You no longer need consensus and you can push the boundaries of what you try. Once you have gotten to blue as the most important element, you may, depending on your feeling for the N-Armed bandit problem, want to test out to 48 different variations, but that is not an affront to you. People built the system, fed the system, and control the system. Once you get to the point that you know what you need to know, why not let the system provide the answer for you? The system is only as valuable as the people who feed it, yet we fear the system and we fear becoming lost to the system.

Moving down this path, of avoiding individual ideas or about trying to find the perfect solution allows you to re-imagine and recreate who and what you are on the fly, without massive redesign efforts. It allows you to avoid holding anything sacred on the site or about ever worrying about the entire concept of “right”. The user experience becomes a fluid thing, where the true value of data, your creativity, and the ability to move past your own biases, determines the magnitudes of growth that you will experience. The true value of the individual is in how well they feed the system. The system is only as valuable as what goes into it and by democratizating all ideas; by forcing the conversation away from ideas and towards feasible alternatives, you are giving more value to the creative freedom of the members of your organization.

Optimization powered Analytics –

Let me pose a theorem to you: Analytics, by itself, is completely worthless.

Let me challenge you by looking at the entire current practice of analytics as nothing more than hubris. That the current use of analytics, especially by those that perpetuate to be experts, is nothing but a newer accepted justification for what you were already going to do or where already thinking. Every new misunderstanding of moneyball, or of advanced statistical models, is a sales pitch designed to make you feel like you are making a much larger impact then you really are. This is not to say that analytics can not be powerful, only that the way that data is abused by the practitioners of the industry to propagate myths and bad practices is worse then worthless, it’s inefficient.

People have gotten so lost in their ability to collect information, the speed we can get feedback, and the need to justify their existence that they never take a moment to question what can you really get from pure analytics. Numbers have become the new shield by which we persuade others of our “greatness”, not to actually provide value, but instead using data to tell stories driven by ego and a want to be the one making the decision. We so want to target a group that we find one that stands out, or we so want to show our value that we tell someone they are doing something wrong, only to replace their “bad” decision with an equally biased one, taking credit for any result that comes from this use resources. In the rare best case scenario with analytics, you are left with probabilities and no clear direction, in the worst and most common cases, we are left with biased “insights” powered by everything but data.

In reality, we are no longer trapped by this use of data, like so many other industries before, because of our ability to interact directly and in an efficient and speedy manner. The data loses all value when we force a path on it or we forget what it can and is really telling us. Because we are a new field, mostly manned by people without real practical data discipline, we allow our own lack of understanding of the nature of data to allow our own biases to paint a picture that does not exist. There are hundreds of agencies, groups, and people who claim to have the newest way to repeat the same types of “analysis” without any newer insight into the value of that data. There are always new ways to corrupt statistics or different analysis techniques that are used to push an agenda, not to actually provide real value. We are so busy trying to run full speed down a path that we miss some really important and fundamental facts. Using only correlative data, we have no way to know the cost to change, the real value that anything by itself provides, or the actual scale of impact of any future change.

If efficiency is a measure of outcome over cost, then we have no way to have any insight into any piece of that equation. All the analysis in the world can not overcome the limitation of a one directional limited data set from a constantly changing and imperfect ecosystem. We find something that sticks out in the data, and then pretend that this is the thing that is more valuable than all the other pieces of data, simply because we can “identify” it, despite the fact that we have no idea of the value of that change nor what some other undiscovered optimization would bring. How does knowing that people from search spend half as much time as people come to your site in any way tell you the cost to change their behavior? Do not confuse your ability to derive value and efficiency with your ability to “discover” something in analytics. That you can even change that behavior? Or the relative scale of impact compared to other feasible alternatives? How is the anomaly any more efficient than the thing that looks like everything else? What do you really know from just identifying something from correlative data?

Why do we accept that limitation and why do we not try to give the context necessary to better answer those questions? Why do we perpetuate the myth of only passive data acquisition as a means to answer so many of the questions that we pretend to be able to answer today? Why do we pretend we can start with this magical data set and somehow arrive at the best answer? We are forced to use conjecture to make assumptions and then pat ourselves on the back when we get a result. We decide on what we are going to do analysis on, find a single answer, and then defend it because it is backed by data. Is that result a good result? If I have a 100 possible positive outcomes, and I get the 2nd worst one, who would tell me that is a good thing? Yet when we do not account for that context of our answer, we are constantly shouting our accomplishments from the hilltop. Do we congratulate the outcome we got or the 98 that we missed? If scored 2% on any test, you would think you failed miserably, but yet we hide this truth from ourselves to make sure that we all feel like we got an A. The truth is that we will never know any of the important contextual information we seek from correlative data alone.

Let me propose that testing, as a creator of causal data in a controlled setting is the only way to actually achieve all those value propositions that you have been promised. That causal data, the seeking and creation of it and the use of it as a transformative agent to power that earlier data collection, to move past so many of the limitations of online data collection, is the only true way to answer these important questions. That by “powering” your analytics, being willing to look past the myths and bad practices, and by breaking down what you really know from your data is the point where myth become real and where you can truly and dramatically impact your businesses bottom line. This is why machine learning is such a big deal, why we move towards optimization algorithms, and why it is so vital that you understand the value propositions of your various types of data. All of those methods leverage casual information as a building block to grow and learn. There is a better way, but it requires you to be humble and disciplined to reach that “nirvana”.

The core problem with analytics is that you are limited to linear correlative data. No matter how pretty a model and how much statistics you apply, you will never know the value of an action, nor will you know the efficiency to change it. We are trapped because the passive nature of the data you are trying to use only looks one way (towards the past) and has no way of accounting for feasible alternatives, or even the null assumption. You are stuck in the land of rates of action; you have 2.8% CTR on your other products model on your product page, but is that good or bad? Even if that is much higher or lower, how do you know that acting on it is any better than acting on the thing that looks just like all the others? If you removed it, where do those clicks go and is that more valuable than what is there now? Does increasing it help or hurt, or more importantly, what happens if it is not there, or what is the cost of changing it as opposed to the cost to change another module? Are people who purchase more likely to sign up for newsletters, or is it the other way around? All of those questions can be answered directly and efficiently through testing, and once we have created a number of interactions, we can start to see patterns from those causal relationships. We have the power with very little effort to start to really see the impact of changes, not just try and extrapolate them blindly.

What if you instead ignore all of that data in its passive form, and instead look for the active interaction of data to inform those decisions? What if instead of starting with correlative data, we ignore it until we have the context to make it valuable? What if we use the causal relations with an eye towards efficiency. What if you viewed data as an active measure, one that gains more value the more you eliminate unnecessary waste in the system, and one that only takes hold once you are disciplined in how you think about, what you measure, and how you actively change it? What if we stop allowing our biases and misconceptions of data dictate the start of our analysis, and instead allow the data to truly tell us what matters? What if you start measuring the value of your correlative data by its interaction with the casual data to allow for a much deeper connection to efficiency. What if you start looking for the value of an action, not the rate of an action? Testing is your active arm, to change all of that correlative data into causal data, if you are willing to go down that path.

This is the opposite of the myth of using analytics to power testing, but instead forcing yourself to accept that correlative data, with all the limitations that are inherent in online analytics, is not enough to make meaningful decisions. This is not about using testing as a means to prove one point right, but as a means to understand and value alternatives against each other. Changing correlative data into causal data presents you with information that is truly actionable and that truly gives you insight into the outcomes, value, and costs that we pretend we already have the answers for. This is the last step of the evolution of looking for the best answer and of stopping biases from leading you astray.

The challenge is that you cannot just take one test, or any single data point and pretend you have meaningful inference. Just as you can not pretend to know the direction of a correlation or the value of something from its rate of action, you can not just pretend to answer everything from a single test result. Diving through all that analytics data from a single test result is a dead end that leads to the same problem that plagues most uses of analytics. You have to be disciplined and can only reach this point after you have run a full series of tests. Think in terms of using this data to increase the efficiency of the system. You get real value only when you apply testing to power your analytics. We can measure the value of the items on the page, their very existence, and the costs to change them. We can quickly get tests live on multiple page types and measure the relative value. We can run a series of a tests on a page, and induce changes that allow us to see what segments are exploitable, or even what the influence is of various parts of a user experience are to those segments. If we are disciplined, we learn, and we never stop, then we can induce answers to achieve a positive result, while also answering those great unknowns that are ignored by analytics alone.

To make this even better, the act of acquiring the data also comes with the benefit of meaningful lift and improvement to your business. There is no zero sum game of only acquiring data or of getting lift, instead using testing to power your analytics allows you to meet the needs of change and growth while giving you all the promised panacea that so many claim analytics is providing by itself. It allows you to truly think in terms of efficiency and to be able to know the value of the different feasible options before you. It requires you to change completely how you think about analytics, to look at as part of a larger ecosystem by which you are informing the data, and then using that data to inform future action. It is not just pretending that the data is informed and then blindly using it to prescribe action. If you instead act to create casual information, use that to filter your correlative data, and do this with discipline, you can actually get those answers that we pretend we have today.

The sad truth is that most people who are in testing come from an analytics background. Just as many old school marketers struggle to stay current in the face of change, so too do many data “experts” who give new names to the same misguided techniques. They view everything through the analytics lens, and as such this makes them want to try and justify their analytics via testing, and to apply the same problematic disciplines to testing in order to bring it in line with current efforts. They so want to justify what they have done that they ignore its fundamental weakness and try to force new disciplines to conform to what they are doing. This leads to an entire marketplace full of people stuck trying to justify their existence, but very few willing to challenge its entire value proposition. I challenge you to avoid that black hole, be willing to challenge your own worldview and your own core beliefs about data, and to instead look at how you can best get and acquire meaningful data and how best to leverage it outside of what you are comfortable with. Very few people try and look at testing as its own discipline, or even better to see how that discipline can impact and change how you view other actions. There is a giant fishbowl of people who are in a race to the bottom justifying and preaching analytics as a feeding system for testing. I challenge you to be better than the current environment.

Let me instead suggest that you will only achieve real value if you flip that system, challenge yourself to think outside of that box, and to power your analytics via your testing. Testing is just one skill of many, but it deserves its own place at the table, not one that is a filter by which you justify other actions.

Conclusion –

The goal of these posts is to introduce new ways of thinking and to challenge your current mindset. I have shown the evolution from the most fundamental skill to paradigms that challenge your entire data worldview. It is only by changing what we do that we grow, and it is only by challenging our own core assumptions about what works that we are able to really make the dramatic impact to the bottom line that we all claim to want to achieve. You can not just accept that everything you hold true today will be the same in the future, nor can you expect to get improve if you refuse to change your own behaviors.

The reality is that there is no such thing as “right”; in the entirety of human history we continue to find better answers to all our questions. What I am proposing is allowing these new ways of thinking to interact with what you are doing and to see if you can then find a newer “righter” answer that brings your program to a whole new level. It is only through changing our fundamental building blocks of what we do that we achieve the scale and impact that we want to achieve. Change who you are, what you think, and let in other ways of thinking and try to be better than the water you are swimming in. Be willing to leave your current lake and find the diverse ocean of disciplines and ideas that are out there, and you will always be growing and getting better at what you do.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 - Part 1 / Testing 303 - Part 2

Timeline:

• News - 0:37
• Chris Parkin Interview about Genesis 3.0 - 1:20
• Support Question - 6:20
• Consulting Tip - 7:45

More information about processing rule

• Knowledge Base Article: 10655

More information about SAINTBernard Enterprise Edition

• SAINTBernard EE Solution Information
• Video Blog with SAINTBernard EE demo

Other Episodes:

• Episode 4 - Auditude? MissSpe1lings + Reduce Latency
• TMI? - Episode 3
• Advanced Solutions - Ep 2
• Advanced Solutions - First Video Blog

Other posts you might be interested in:

• Mobile Solution Series: Advanced Mobile Traffic Analysis [Advanced Solutions]
• Campaign Tracking Revisited – Part 1 (Overview) [Advanced Solutions]
• WARNING: Facebook Social Plugins Issue [Advanced Solutions]

We would like to get your feedback so we can improve. Please post comments or e-mail me at Pearce (at) adobe.com.

Have you ever played the game of Guess Who?  This is the game where you have a crowd of people and you have to ask your opponent questions to figure out who their name is.  This game, like site optimization, is all about asking the right questions.  Contrast these two players:

Player 2’s Questions	Player 1’s Responses
“Is your person Male?” “Does your person have a beard” “Does your person have a beard from ear to ear?” “Is your person David?”	“Yes” “Yes” “Yes” “Yes”

As you probably noted, player 1 was asking all the wrong questions.  The art of asking the right questions will not only make you a good Guess Who player, but it will help you be a strategic tester.  You can be the best test executer in the world, but if you ask the wrong questions you will never realize the full value of your testing program.

The best companies with the best testing programs don’t worry about test ideas, they focus on test questions.  Questions are important because they easily translate into very clear site learning.  When you test an idea verse another idea, you only know which idea was better—you are still stuck in “better” testing which is the wrong outcome.  In contrast, when you ask a really good question it helps you shape the strategic setup of the test, opens up the testing possibilities, allows you to understand why something worked, and guides you to where to test next.

As an example of this I was on a call recently and the client’s senior management wanted to test a left navigation verses a top navigation with the hope that they could get more links in front of their customer.  This is idea vs. idea and it is making a lot of assumptions.  They are assuming that showing more links is important, that the left navigation is the best place to show links, that the changing the navigation this way is better than other navigation tests we could run, and that the navigation matters in the first place.  The other problem with trying to run a test like this is that in the chance that you do end up with a better navigation, you still haven’t learned anything—you don’t really know why it is better.

Deconstructing an Idea

If we were to reframe their idea into a good question we might go through a process like this. The original idea was to get more links in front of the customer, but what are the links for?  The links are so that customers can find what they are looking for.  How do people find what they are looking for?  They use lots of things like navigation, search, internal campaigns, featured products and catalogs.  After we deconstruct the idea it is easy to see that the question we should be asking is “How do customers find products they are interested in?”

Once we have the high level question we would want to find out if the links and navigation even matter at all. To do this we would ask: How important is the navigation relative to other shopping methods (navigation, search, internal campaigns, featured products, product catalog, etc)? Depending on how this test goes we may learn that the navigation doesn’t matter at all and we could then test removing it all together.  This helps us use our time and resources to answer the questions that are most valuable and keeps us from going down rabbit holes.

Taking the time to think about the assumptions built into the idea can help refocus the question to one that eliminates assumptions and gives you the highest return on your resources investment.  If the navigation came back as the most influential then we would begin to run through the additional questions outlined below.

· What is the best location for the navigation (left, top, right, multiple, etc)?

· How should the navigation function (drop down, fly out, static, accordion, refinement, etc)?

· What is the best look and feel of the navigation (color, shading, text size, icons, etc)?

· What is the best size of the navigation (big, small, skinny, fat, etc)?

· How many options should we have in the navigation (Departments, subcategories, sub sub categories, etc)?

· What should be in the navigation that is missing?

· What do we have in the navigation that should be taken out?

· etc

You have to develop the skill of taking an idea and deconstructing it into something meaningful that will help you learn.  Often you will get an idea that needs a lot of translation to become a valuable test series.  You have to develop the skill of taking two or three steps back so you can see the big picture.   Like the Guess Who example, don’t get caught asking questions or testing ideas that don’t matter.  There are no perfect ideas, there are only questions that help you learn about your site.  Back yourself out of the weeds to ask the highest level question then work from there.

Creating a culture that focuses on asking the right questions so that you learn what works on your site can take time.  One way to do this is to create a form that anyone can fill out that.  Have them give you their test ideas in the form of a business question so that they understand that all ideas can be built on.  It is important to avoid getting worried about who is right, and instead have each person help inform the system to make sure that we get the best results and ask the best questions.

Remember that there are a lot of benefits to asking really good questions.  Good questions help

· You understand why something worked (clear learnings)

· Shape the strategic setup of the test

· Open up the testing possibilities

· Guide you to where to test next

Good luck

Rhett

Did you know that when someone is executed in the electric chair, the first jolt of electric current is designed to cause immediate unconsciousness and brain death?   This is interesting to me because the human body is composed of electric currents and energy.  I’ll spare you the 7^th grade proton, neutron, electron story, but suffice it to say that too much electricity and we die, no electricity and our bodies just wouldn’t work.

I have seen this same phenomenon in organizations that are testing.  Often companies will have too much or not enough of something important.  The end result is usually the same as too much or too little electricity in our bodies – the testing program stops working or suffers strategic brain death.  As an example how testing electrocution can occur, I’ve seen companies end up executing themselves to death by running lots of tests.  They execute so many tests that they don’t take the time before or after a test to strategize.  They literally become the executioner and the victim all at once.

When any one aspect of testing is over or de-emphasized the testing program can be put at risk. Clients often get hung up with test execution, but this same thing can happen when other important testing steps have too much or too little emphasis.  Another example is a client who was known to glorify results evaluation and had whole teams dedicated to interpreting test results.  They would run a test and spend a lot of time and resources pouring over analytics data, backend data, and putting together pages and pages of metrics.  In this case too, the whole program suffered.  The ironic thing about this is the same company couldn’t tell me one thing they learned in their whole previous year of testing—they had lots of results and numbers to interpret those results but that is it.  Fortunately this client has made the shift to give equal importance to the other important steps and their testing program is looking much more promising.

You might be asking yourself what those steps are.  What process can every company follow to make sure they are remembering the right things at the right time?  Let me introduce a simple testing methodology that outlines 6 repeatable steps that every testing program could follow to make their tests as strategic and successful as possible.

1. Question Creation

2. Question Definition

3. Question Evaluation

4. Test Execution

5. Results Evaluation

6. Action

Mature and successful testing companies know that testing takes a lot of work and a lot of discipline for continued success.  The discipline is adhering to a methodology that helps move testing forward and keeps the company doing the right amount of the right thing at the right time.  My goal is to help you see that there is more to testing than any one aspect—including running tests.

I hope that these steps will help you focus your efforts and will bring the strategic nature of testing to the foreground.  To that end, I will be exploring each of these steps in detail and what it means to have enough of each step to empower us, but help us avoid overdoing it and in turn keep us from strategic brain death or non-functioning testing programs.

Turning abandoned visits into revenue opportunities

HeBS Digital uses the Adobe® Digital Marketing Suite to measure the results of digital marketing campaigns. Additionally, we use the suite to measure user behavior on hotel websites, and use this information to improve the conversion process to generate more revenue for our clients.

The suite offers a variety of reports to measure user behavior. For instance, the ‘conversion funnel report’ in SiteCatalyst® provides a breakdown of all important conversion metrics, from point of entry to the completion of a hotel reservation. The ‘pathing report’, also found in SiteCatalyst®, identifies the points at which users abandon their reservation. Knowing where along the reservation process people abandon is of utmost importance, and this can be narrowed down to the availability page, the room selection page, or the payment processing page. In sum, the funnel and pathing reports provide us with practical insight on making the case for what we recommend to clients: the HeBS Digital Reservation Recovery Strategy.

Industry-wide, the conversion rate on travel and hospitality websites remains at a steady 2 percent. This means that of all visitors to the website only 2 percent will ultimately book a room. Even with multi-million dollar online marketing campaigns, the major hotel brands and OTAs still hover around a conversion rate of between 2 and 3 percent. Using SiteCatalyst®, HeBS Digital’s client portfolio over a two-year period resembles the overall industry averages:

• From visitor to the site to a booking initiated: 45%
• From a booking initiated to a completed reservation: 5%
• From visitor to the site to a completed reservation: 2% - on par with the industry average

A high volume of visitors initiate a booking but do not complete a booking. This further supports the notion that visitors tend to check rates and availability, and then leave the hotel website to compare these rates with other hotels, read reviews, watch a related video, chat with friends, etc. Many of these hotel website visitors never return to complete the transaction.

To discourage visitors from comparison shopping, hotels deploy strict rate parity policies and publish best rate guarantee programs. They hire digital marketing experts to target highly qualified site traffic and retarget whenever possible. They utilize attribution models to determine what points in the research process yield the highest ROIs. They optimize their sites according to the new Google Panda and ‘Freshness’ rules and attempt to keep customers engaged through social media. They participate in bid management to reach qualified search traffic, and so on. In sum, hoteliers are doing everything possible, except attempting to recover reservations from guests who abandon the booking process.

HeBS Digital recently completed a usability study for a major hotel brand. To answer the question: “What is your primary purpose for visiting the ‘Hotel’ website today?” the overwhelming majority reported to be looking, shopping, researching, and browsing (multiple answers were permitted.)

Response Percentage
Make a reservation 32%
View or cancel a reservation 5%
Research or compare rates 20%
Look for deals and specials 38%
Look for hotel information 18%
Find destination information 3%
Browse 11%

The range in responses demonstrates the variety of uses shoppers view as the purpose and function of a hotel website. There are many reasons a guest will visit and leave the site. The same is true for starting and not completing a reservation.

HeBS Digital’s reservation recovery technology enables hoteliers to stay in touch with their customers after they abandon the booking. The technology is designed to capture shoppers who have abandoned at the point of purchase, and then recover these shoppers by sending a post abandonment emailer that invites the shopper to return and purchase. Thus far the pilot program for the major hotel brand has recovered over $160,000 in revenue in the first two months since installation.

How Does the Reservation Recovery Strategy Work?

Once the visitor reaches the point of purchase stage, abandonment is likely (as noted in the industry data above). A percentage of people will enter their email address, first name, and last name and then decide to discontinue the booking for whatever reason (checking with a spouse, distractions, payment issues, continuing to shop around, etc). It is this population of abandoners that the recovery strategy aims to reach and gives the hotel another opportunity to sell to this customer. As seen in the case study below, this is a sizeable group of potential buyers.

Here is how the strategy works. After the shopper abandons the payment page, a branded emailer is sent with a message thanking the guest for visiting the website. Included in the emailer is a phone number and chat function (if applicable) that connects to reservations. The emailer also contains a very important deep link that will return the shopper back to the point of purchase. The deep link embeds the specific property (used in cases for multi-property companies), dates of stay, number of guests, and room type selected from the point of abandonment. This allows the shopper to pick up where left off.

In this case study the emailer was set to deliver within two hours of abandonment. The emailer did not offer any incentive or other sales message – just a simple, non-obtrusive message thanking the guest for their visit with the link to return to the point of abandonment. The phone and chat function were not tracked.

The recovery system allows for personalization as well as language capabilities. The emailer is personalized and addresses the customer by their first name (if provided during the initial visit), and the language of the message is based on the language specified during the initiated booking. In other words, the more information gathered the more personalized the emailer.

Reservation Recovery Case Study Results

Client: Major hotel brand with up to 20 properties in North America
Date range: 10/1/2011 to 12/6/2011

As mentioned above, as many as 45% of visitors to a hotel website initiate a booking. In this case study, only 17% were of reachable quality (they provided an email address before they abandoned).

Total Booking Attempts:
• Initiated bookings: 38,147
• Abandoned bookings: 19,538
• Reachable bookings: 6,793 (17% of all initiated bookings got as far as entering the email address)

We were able to capture 17% of all shoppers who provided their email address before abandoning the page. Shortly after launching the strategy, we moved the email address field to the top of the booking process and made it the first item to complete on the payment page. We feel the higher placement made a significant contribution to the number of reachable bookings.

Results:
• Reachable bookers: 6,793
• Total reachable bookers that returned: 838
• Total returned bookers that converted: 192
• Total revenue recovered: $160,871

The 192 converted bookings from the total reachable 6,793 reachable bookers above represents a conversion rate of 2.8%. This is consistent with overall conversion rates for the industry. Of the 192 bookers, it is unknown what percent would have come back to book without the abandonment emailer and the 192 bookers did not impact the overall conversion rate for this brand. Instead, we view these bookings as incremental, as another tool to fight the OTAs, and another method to stay engaged with a highly qualified customer.

However one feels about the practice of recovering business, it cannot be disputed that an additional 2.8% conversion rate from qualified visitors can amount to sizeable revenues. If these numbers continue, the brand should expect to generate approximately $780,000 by the end of the year from this initiative. Given the nominal fee to set up, create, and run the project, the annual return on investment will nicely exceed 7800%.

About the Authors:
Jason Price is Executive Vice President and Mariana Mechoso Safer is Vice President, Marketing at HeBS Digital, the industry’s leading full-service hotel digital marketing and direct online channel strategy firm based in New York City (www.HeBSDigital.com). HeBS Digital has pioneered many of the “best practices” in hotel digital marketing, social and mobile marketing, and direct online channel distribution. The firm specializes in helping hoteliers build their direct Internet marketing and distribution strategy, boost the hotel’s Internet marketing presence, establish interactive relationships with their customers, and significantly increase direct online bookings and ROIs.

HeBS Digital & Adobe Partnership
With HeBS Digital’s expertise in the hospitality industry and Adobe’s leading edge analytical and design tools, the two companies have forged a partnership in best practices in online marketing and distribution. To learn more about the partnership please visit us at http://www.hebsdigital.com/aboutus/adobepartnership.php or contact HeBS Digital consultants at (212) 752-8186 or success@hebsdigital.com.

In our introduction, we talked about disciplines that enable you to move faster and align on a common goal. In the second series, we went over disciplines to help you think about tests and testing differently to get more value from your actions. The third and final evolution takes us in new ways to view the world and our organization, and challenges us to go in new directions, to understand new paradigms that should fundamentally challenge some of the most common and fundamental beliefs about data and optimization.

One of the great quotes that I keep close at heart comes from John Maxwell, “If we are growing, we are always going to be out of our comfort zone.” With that in mind, I want to introduce these paradigms for your program and challenge you to take these and evaluate them outside of the fishbowl, as an idea on its own that can help you program get past its current plateau and to help you grow in your thinking about optimization.

Analytics and Optimization as very Different Disciplines –

There are many different ways that you look at and act on data in testing that are the exact opposites of analytics. Where so many programs fail is when they force one way of thinking onto their data, resorting back to what they are most familiar with. In the world of analytics, you have to look for patterns and anomalies, look across large data sets and try to find something that doesn’t belong, and that doesn’t fit with the rest of the data. You are constantly looking for outliers that show a difference and then extrapolating value from those measurable differences. In the world of optimization, you have to limit yourself from looking at anything but what you are trying to achieve, and to act on data that answers fundamental questions. It becomes extremely easy to fall back into more comfortable ways of thinking, because the data sounds and looks similar, but ultimately success is dictated by your ability to only look at the data through a different lens. You have to stop yourself from trying to dive down every possible data set and instead focus on the action from the casual relationship around the single end goal.

It is what you don’t do that defines you as much as what you do. It is about “did removing this module improve revenue performance“, not “did the CTR drop of this change increase CTR to the main image and where did those paths lead“. It is also about not allowing linear thinking to interrupt what you are doing. You have to focus on the value of actions, not the rate of them. You are looking at the value add from a user (RPV), not the amount of short term actions (CTR). Never look at how many people moved from point A to point B, but instead only look at the measurable impact towards your site goals. Just because you increased clicks, or got more people into a funnel, or even got more transactions, it does not mean that you increased revenue. Assuming there is a linear relationship between action and value can be extremely dangerous and myopic, many programs have been ran into the ground because they do not understand the difference between the count of and action and the value of the action. Analytics forces you to think in terms of rates of action, but optimization forces you to think about the value of actions and the cost to change a person’s propensity of action.

Think of your site as a giant system. You have an input of people, with each input type interacting differently with the system. The things you sell, the layout, the experience, all of it makes up a giant equation. When those people enter your site, they go through, and they come out the other end at some rate or some value. The numbers or rates associated with that one path is analytics. That inherent behavior based on the current user experience is their propensity of action. In testing, you have to solely focus on your ability to increase or decrease that propensity of action, not about the absolute value of that action. We care that we increased that behavior by some delta, some lift percentage, not that it was 45% and moved to 49%, but that we increased it by 8.9%.

In testing, the answers you might receive will be of the nature of “we got more of the high value and less of the low value populations” or “the system improved as a whole by 4%.” Ultimately “answers” matter far less then the changes observed and your ability to act quickly and decisively on it. What you won’t receive is why, or what each individual part of the system did to get you there. Those answers are stuck to the realm of correlation and as such have to be ignored because you at best have only a single data point. We are trying to move forward as quickly as possible in the realm of optimization, so getting lost in loops of trying to answer questions that are not answerable only hinders your efforts. No matter how much you analyze an individual test result, you will never have more than a correlation. This means you have to think differently in order to use that data. It doesn’t matter why, or which piece, or even which individual population (though dynamic experiences on outcomes is important) so you have to force yourself to not go down those roads.

It is also about the ability to hold yourself accountable for change. So many analysts fail because they view their job responsibility ends on the moment they make a recommendation. There is a revolution taking place in our industry, lead by people like Brent Dykes, that is changing the entire view of optimization away from the recommendation and data, but to the final output. In optimization, you are only successful if the results you find are acted on and made live. It requires you to view the cycle as one of action and not one of inaction. It is not that both don’t have their place, but you to be really successful you have to be able to step away from your analytics self and instead think differently and force yourself to act differently in order to get the results you need.

Testing Applied to Multiple Teams

Testing is something that has many core disciplines, but takes on a very different look and value for different groups. Your IT team may get a completely different “value” from testing than your merchandising team, as your design teams might from your analytics team. Many groups believe that because they have applied their testing from their landing pages to their product pages, that they have expanded the value of testing throughout the organization. Instead they need to rethink how optimization disciplines can interact with the different groups efforts on a fundamental basis. Testing is not just changing a marketing message, it is the evaluation of possible feasible altneratives, something that all groups need to do to improve their effectiveness. Testing is just as applicable to your SEM team, your merchandising, your product management, your IT team, your personalization team and many others. Each group has different needs and different disciplines, and as such you have to apply the disciplines of testing to them in different ways. A IT team can use testing to decide on which project to apply long term resources to. Your UX team can tie testing to their qualitative research to understand the interconnection of positive feedback to overall site performance. Your SEM team can use testing to measure the downstream impact of their various branding campaigns.

The reality is that applying all the unique benefits of testing to different groups, and not just increasing the space that you do the same things to can fundamentally improve your entire organization. While this might sound like a simple one, the reality is that most groups do the same type of testing or try to apply the same techniques across multiple parts of the site, not for different teams. Each group may be aligning on the same goal, but they do things in a very different way. Applying optimization to those groups looks and acts in very different ways, and such it is difficult for most groups to really apply these disciplines in a way that truly impacts the fundamental practices of more than one group.

Instilling this use of testing as a fundamental building block also allows you to get ahead of a large number of major problems. It forces organizations to test out concepts well before they decide on them as long term initiatives. One of the most common examples of this is in the realm of personalization, where so many groups are sold on the concept, but not willing to go through all the hard work of figuring out exploitable segments or the value and efficiency of various ways of interacting with the same user. Getting ahead of the curve and testing out the efficiency of the effort will save dramatically improve the performance of the effort. If you test out a complex idea in one spot against other feasible simpler ideas, and find the simpler idea is better performing, as it almost always is, you save massive IT resources while getting better results. It is far more likely that simple dynamic layout changes for firefox users are going to be magnitudes more valuable then a complex data feed system from your CRM solutions, and testing is the bridge to know that before you fall down that rabbit hole.

Each group tends to end up at the Nth degree of the same thing they bought the tool for. So often, the fear of the unknown or of challenging someone’s domain stops new groups from allowing testing in, but when you can overcome those barriers, you can have an exponential impact on the organization. When you start trying to apply optimization to multiple types of internal practices, and you are able to bring the results together in a real synergy, that is when you are able to really see optimization spread and to see the barriers drop throughout an entire organization. It also the point where those lessons you learn become three-dimensional and become universal across the entire organization.

Testing has No Start and No End -

Optimization is not a project. It is not something that is just one person’s job and it is most definitely not something you can just choose to end some random Tuesday. So why then do people view it as a series of projects, with a start and a stop? Why do they view it is only part of one person’s role or responsibility, or something that is done when they have the chance. There are functional reasons to have set people assigned to testing, and as programs grow to have a separate specialized team, but that is not the end of the battle. Why do we try to force artificial time constraint on it, with starts and stops and talk about it as something we did, or will do. It is either an action that you live, or it is not. If everything your organization is doing, be it some small tweak, or a redesign, or the release of a new feature is not viewed as part of an ongoing process, with lessons to learn and to be evaluated democratically through the system of optimization, then optimization has been allowed to have this artificial start or stop just to appease various members of your organization.

Optimization has to be something you live. You have to be thinking in terms of it every day, you have to view each task as something that can get better, you have to view each idea as just one of many, and that it is not up to the HiPPo or anyone else to decide on. It is a responsibility to not let projects, or holidays, or new CMOs or anything else stop you from this constant quest to improve, the site, the processes, the people. Do not confuse the actions of running or a test as the entirety of optimization. It is vital that you view the act of creating something new as just the very first step, and not the end point. There should be no point where anything is thought of as “perfect”, or “done” or that you can just throw something live and walk away. Optimization is part of every process, it is part of every job, and it is something that everyone works together to make sure that it is part of every action that the organization takes.

When you have finally started to incorporate testing into your organization, all projects will view it as another natural part of their evolution. Project plans will incorporate not only the concept of optimization as an ongoing basis so that it is part of your expected timeline, but they will also stop trying to get everything “perfect”. If you view your projects as never finished, then there is no need to have everything get perfect signoff, nor do you need to have perfect agreement on each and every piece. What is important is that you spend as much time and resources on testing out all those ideas that you have discussed, instead of just sitting around a room and compromising on a final version. You will no longer be so caught up on your pet project, as the entire concept is that it will and must change.

So much of what happens in organizations is about the politics of owning and taking credit for different initiatives. There are people’s reputations and egos on the line when they propose and lead dramatic changes, especially redesigns, for the site. If you can truly incorporate testing and optimization as a vital part of all processes, one that is not just a “project” but is part of the very existence of the site and the group, then you free people up to no longer being so tied to their “baby”. Treat all ideas as malleable and transient, to the point that everyone is really working together to constantly move the idea forward. It will can be a dramatic shift to organizations once they reach this point, but ultimately it is when groups really start to see dramatic improvements on a continuous basis.

So often we talk about not following through with each of the concepts I have brought forth, but the reality is each action is tantalizingly easy, but the real discipline, the ability to keep pushing 6 months from now is what really differentiates programs and people. Being willing to move past the barriers, put the pieces in place that make a difference, and being willing to change how you and others think are the real keys of a successful program. If you are always trying to do what is easy, or just listen to the pushers of magic beans and myths, then you can never really grow your program to the levels that are possible. Do the hard work, get out of your comfort zone, and you can continue to get better and can continue to see more and more value from your testing program.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 - Part 1 / Testing 303 - Part 2

The theme of the first group of disciplines for programs was all about organizational consistency and getting everyone to work efficiently and quickly towards one goal. Once you have achieved that ability to move forward and act as needed, the main ingredient for success comes from your ability to think about testing differently, to challenge yourself and others to focus on learning and on not making assumptions. Way too many groups end up leveraging their testing program to only look a the impact of what people want to do, instead of using as a tool to learn and go in new directions. Often times the least efficient part of any system are the people running it, and as such we have to come up with ways to break down that barrier and to allow the test data to really dictate where we go, what we talk about, and how we allocate resources. How you think about testing, how you challenge yourself to not fall into the many biases that dictate human nature, and how you can bring people to challenge their own actions dictates the scale of the impact of each test and of the program as a whole.

So many groups run into the problem of not realizing that they are sub optimizing. They run a test, and they get a winner, and so they assume that everything is golden. They report the winner, they move on. We get so caught up in the immediate return on our actions that we never take the time to understand what it means in context. The problem is that we are not treating each action with the respect it deserves, and that they are taking they are busy looking at today and not tomorrow. It is never about getting a result, it is about the ability to differentiate different results from each other, to make sure that we are going down the most efficient path for our business. If you get a 3% lift, that is more then you had previously, but what if the 3% is just one recipe in a test with a 5%, a 7% and a 10% winner. Would you then think pushing the 3% winner was the best course of action? The only way to escape that trap is by changing how you think about testing, and the practices that you put in place around that change, you are given the ability to measure the efficiency of your actions, to view the causal relationships of alternatives and to measure the scale of impact and the cost to achieve them.

In order to make sure you are getting the most from the actions you have enabled, here are 5 disciplines which will help you achieve the results you want from the program.

Best instead of Better Testing –

I have already explored this concept here, but it is important to understand the distinction. “Better” testing is the act of trying to figure out if one idea is better than another. It limits the playing field and is used to make an immediate decision about who is “right”. Best testing is trying to figure out what the value of each feasible option is, and to figure out what the best places are to put resources or to move the site. Testing is a system, one that only produces value based on the quality of the input. If you limit your input to only popular opinion or a few ideas, then you are dramatically hindering the output of that system. It is about using testing to not just push preconceive notions but to instead democratize ideas, so that the system is more important than the idea. If you are able to let testing tell you where to go, you will not only get better results from this test, but you will better inform future tests and stop yourself from spending resources in an inefficient manner.

Best testing is the first step to allowing your program to produce exponential growth in a way that just a test would never be able to to do. You are able to use the program to build, not just run tests. The entire goal is to increase efficiency and to facilitate learning, and the easiest greatest step towards that goal is forcing your team to think in terms of what is best, not just what idea is better.

Focus on Learning –

Every test you run is the chance to learn about your site and to get outside of your comfort zone. So many groups fail because they only test what they think will win, or what they focus on trying to get a consensus about recipes. If you spend the time and energy you waste talking about test ideas and focus it on creating all the options, you will spend less time and energy and will get better results. Stop arguing and move those resources towards creating. If you are really trying to open up your testing to feasible alternatives, you will constantly find winners that fly in the face of crowds. If you are focused on that outcome, you suddenly find all sorts of new lessons waiting for you, with the added benefit of getting magnitudes of value on top of what you learn.

There are so many assumptions, misconceptions, and faulty “best practices” that dictate the online world. Even worse, we make assumptions that something that works elsewhere works for your site and your users. We gain nothing if we are just proving ourselves right, but instead when we challenge those ideas, we start to learn about what makes your site unique and what works best for what you do. We start learning about the best places to efficiently change your site, and even who the most exploitable user segments are. Even better, those lessons seep into your other conversations, to inform product plans and senior management and all other groups about what you know, not just what you think or pretend you know.

The most obvious example of this is with multivariate testing. So many groups, especially agencies, push MVT testing as a tool to find a single answer by throwing a number of variants for multiple items on a page. It is a big mixing machine to reach a new version of the page faster. If you change that and use MVT testing as a learning tool, to focus on what section of the page or what factor of a section is most influential, then you are able to leverage your resources in a way to maximize the ROI and learn, accomplishing both tasks far better then just throwing things up to see what sticks. Anytime you are running a massive MVT, full factorial or not, you are sub optimizing, both in a resource and time perspective, but also because you have failed to leverage the opportunity to learn.

Living Knowledge Base –

As programs grow, and as you use testing as a vehicle to learn, the most important thing you will accumulate is not lift, but functional knowledge about your site and users. Storing and sharing this knowledge, based off of causal data, informs future decisions in a way that analytics or just “Best Practices” will never be able to do. You have to make this storing and sharing a function of the team, and make it accessible and meaningful to all groups, even those that were not part of the test that gained the knowledge in the first place. For most groups, this accumulation and sharing of knowledge has scales of impact far greater then just the individual test results.

Building a repository of that knowledge, and having it be an active breathing thing, that interacts with people and exists outside of individual tests is vital to achieving the results that programs want to achieve. What it is not is just a list of tests run and their results. What it is meant to be is something that shares lessons, successes, and failure, but is focused on the learning you have done from your program, not the minutia of the actions that got you there. Every version of this is different, as is each and every organization, yet they share the same characteristics: they focus on what has been learned across tests, they are easily accessible, they are used to start conversations and as a barometer to go, and they see what does and more importantly what does not work. It also allows you to weigh the various actions against each other, as just lift alone does not tell you the scale of impact. If you continue to try the same things that fail, you will never be able to leverage the exponential efficiency that learning should allow you.

Iterative Testing –

Iterative testing may seem like a no brainer, but so many groups fail to understand or leverage it as a discipline, instead talking about but failing to act on it consistently. It should be an organizational rule that no test is ever “over”, but only at a new stage ready for the next test. If you are using tests to learn, then you will know how to prioritize a page, which then needs to have the different sections explored, for not only what the feasible options are, but also what the most influential parts of those are and then what the best way to tackle that winning factor may be. The goal here is to make sure that each test that you run maximizes the efficiency while mitigating the opportunity cost, and the only way to do that is to constantly build off prior knowledge to insure that you are maximizing your placement of resources. In many cases, this can be the most difficult barrier for groups to actually overcome, as while there is a lot of positive talk around the subject, there are so many pushes and pulls for your time and for the resources, that it is easy to lose track or to just stop at the end of any given test. You have to force yourself and your group to maintain that momentum, and more importantly leverage all of that learning, to really drive where you are and where you are going.

Here is an example:

You take a product page, you challenge yourself to learn about what the most influential section are, so you test out what belongs and what doesn’t by removing all possible sections on the page. You learn that 2 sections don’t matter, the top navigation and the brand information. You then test out removing them together and find that you have improved the page towards your site wide single success metric. At this point you have a new page after pushing the winner, but need to dive deeper. You then use a small 3X2 MVT to learn that the button is the most influential element on the new page. You follow that up to look at the factors of that button to figure out what about it drives that influence. You learn that color is far more influential that copy or size so you then test out 5-6 different colors, and learn that purple, the color that everyone thought never had a chance of winning, actually does better than all the other colors.

What you have is an entire path that you never could have preconceived. You have forced a way to make sure that popular opinion did not drive what you test. The resulting page is not one that anyone could have predicted, but is by far the best performing. You have learned what the importance of elements are, the best way to change them, and you have used very little in the way of resources. You are free to continue, as you optimize the second most important part of the page, or you optimize the second factor of the button. The process continues, either in that same part of the user experience, or where you can see other opportunities to do the same process based on what you have learned.

Iterative testing is something that is constantly thrown around, agreed on, but then why is it so rarely done consistently? Why do so many groups think that talking about iterative action is enough? Groups get too caught up on the single winner that they miss that it is just one part of a very fluid user experience. You have to force yourself to follow this path, that is why it is a discipline, and when things become difficult to push through and do this, always, in order to reap the rewards you are seeking. Even better, if you are using segmentation at each point, you will end up with a page that can be dynamic based on the winning alternatives for exploitable segments. Each “test” is really just the next evolution of the same process. You have to stop yourself and your organization from viewing the test as the ends to itself. Doing this consistently, not once but always, really differentiates the value you will receive from testing.

Deconstruction –

The ability to have someone present you a test idea, and then break it apart to find the assumptions that lead to it so that you can learn as you grow is a vital skill for optimization. One of the most common mistakes that testing groups make is to take each idea at face value. Every idea comes from only one point of view, and is riddled with biases. You have to force yourself and others to get past those points to really discover the value hidden behind what sounds like a conventionally accepted concept. Treat the most important part of your program as the system by which you discover new things or challenge biases, and you will always be able to get greater results. We need to take any idea, and challenge every core part of it, so that we leave nothing to chance and so that we can really evaluate what works, not just what sounds like it works.

So rarely are you presented with the chance to optimize a page from start to finish as mentioned above, but that does not mean you are not still responsible for applying the same discipline to any starting point. People will always come to you with test ideas, either through internal debates, feedback, or just as they are going across the site. There is most likely value from each idea, but the real skill is to break down the idea, to make sure you are not presuming the path, and to learn. Is that item even needed? Does what you have been doing, is it positive or negative? Is that the best place to put resources? What else can be done with that space? Being able to take an idea and challenge the components of it is how you will arrive at the most important lessons you will learn.

The idea of targeting content in your carousel on your homepage based on what people have already purchased sounds like a great idea, but lets evaluate all of those assumptions:

Is purchaser even exploitable (can you change their behavior by changing the user experience)?
Is it the most exploitable way to look at the same user?
Is the homepage the right place?
Is the carousel the most influential part of that page?
What type of changes to the content are most influential, is it the wording, the presentation, the location?
Does content have the largest impact?
How does that entire path compare to other alternatives?

That is just the very first pass. It is actually far more likely that just changing the layout of your homepage based on browser is going to be both much higher in yield, but also more efficient of a much larger scale.

It takes practice and discipline to force yourself to challenge all of these ideas. It can often lead to discomfort at first, but pushing past that and assisting others in seeing their own biases and their own assumptions helps everyone grow. Testing is not about who had the best idea, it is about becoming humble and realizing everyone is “wrong” and about creating a system to push past that and learn and challenge common convention for the betterment of everyone.

Conclusion –

I am often asked how I define a “successful” program. When it comes to measuring a program, I measure it on how often they have learned something new and unexpected. Have they done something that makes everyone go “That can’t be right”, or “that goes against everything I have ever heard.” The only way to have those moments and to get the magnitudes of value both short and long term that those conclusions present you is to break apart ideas and to challenge yourself and others on the questions that you are trying to tackle. It isn’t about being sadistic or altruistic, but instead about looking at the discipline as a means to an end that helps everyone achieve their goals.

In the end, it doesn’t matter if you can run 100 tests a month if you are running sub optimal tests. The point is never what you did get, but what you got in relation to what you could have gotten with the same or fewer resources. So many groups get lost because they focus on the actions, and not the disciplines that define them and the larger picture of the program that they exist in. You did an action, that you might have done anyways, but it is over and you are left with nothing but the next idea.

You have to change how you think and how you act to get the results you want and to make testing a fundamental part of who you are as an organization. I talk in terms of disciplines, because that is what they are, they are core beliefs that only take hold when we challenge ourselves to live by them every day, not just when it is convenient or easy. Challenging yourself to these disciplines and asking the tough questions of others will allow you to move down a path away from the obvious to the world where you learn, grow, and have results that really move people.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 - Part 1 / Testing 303 - Part 2

All programs are built off of key fundamental disciplines that allow all other disciplines to grow and to be acted on accordingly. All groups start out by needing to grow past the initial challenges and limitations that face your program. Most groups find a point where testing has shown a few wins, you find places to test where it makes sense, but you haven’t been able to get the level of consistency or buying that you are looking for. You are trapped in a world of explaining why you want to run a test, or working with different groups who aren’t interested in what the rest of the organization is doing. Early development programs have many factors that unite them; you might have been testing for 3 days or 3 years, and yet if you haven’t really taken the time to grow and incorporate testing beyond a superficial way, you will never achieve the true value that testing can provide for your organization.

The first thing that differentiates programs is the organizational pieces that have been put in place to help them move at a speed that will truly impact the bottom line. With that in mind, here are 5 key disciplines that will allow your program to the next level. For any testing program to work long term and to grow, you must first address the organizational roadblocks that will without fail trip up your program. The first and sometimes most vital aspect of growing your program can be adoption and the realization that this is not a once in a while type of action that you take.

Single Success Metric –

Without fail, the single greatest determining factor in the overall success of your testing program is have you unified on what you are trying to achieve and are you optimizing for the end behavior you want. So many programs fail when they try to optimize for the next page or a small action, or even worse to try and optimize for multiple metrics to make everyone happy. So many programs are less the sum of their parts, and because of this they are never able to impact the fundamental bottom line of the business at the level that they are capable of.

It isn’t just about optimizing for the goal of the site and not focusing on a individual concept, you must be able to answer all questions of this format: If A goes up, and B goes down, what do we do? What about A and C? Ultimately, you will quickly discover there is one thing that you really have to go up to be successful. It isn’t about clicks, our bounce rate, or just moving people to another part of your site, it is about focusing on what all of those behaviors are trying to accomplish. In just about all cases, the metric you will end up using is directly tied to how you make money, something like RPV, or lead conversion rate, or an engagement metric. What is important is that is global across your entire site, not just the section you think you will impact. That is what you need to focus on. You need to have everyone in agreement on how to act on the test before it launches, so that you don’t lose momentum or get lost in the internal quicksand of politics.

Nothing allows your program to be more valuable and for you to bring people together then to have everyone pointed towards the same end goal. Nothing ends internal debates faster than not worrying about clicks or sign-ups or search versus organic then optimizing for revenue and giving all actions an equal footing.

When I come in to assist groups and make sure they are getting the most value, nothing is more vital and I will refuse to go forward unless we can get agreement on what defines success and how we are going to measure it. You are often defined by what you don’t do as much as what you choose to do, and this is the seminal moment when you can take a giant leap in your program. This is the moment when a new discipline, unique to optimization, comes into place. It also allows us to treat any idea democratically and let the numbers define success.

Consistent Resources –

One of the many failings of early programs it he lack of ownership of the process and tool. Does IT, marketing, product, merchandising own testing? Whose resources get assigned and who will run this specific test? Do you have a FTE, or just a few random hours assigned to someone who has 10 other tasks on their plate? It is vital that as you build your program, that you have consistent time and resources assigned, outside of any specific test or project. The skills of testing, not just the tool, but understanding how your site works, how to move things, testing disciplines and how to navigate internal politics, all of these take experience and time. Programs that have new people jumping on board, people who run a test once or twice a quarter, or who have not clearly identified who will set-up, run, and do reporting on tests have the hardest time creating the momentum and cadence that allow your program to really blossom and to help insure that you are getting the magnitudes of value that testing can provide.

A good first step for any program is to assign a set number of hours per week. Assume that you will always be testing, and such, this won’t be tied to any individual test. As a good next step, you can make sure that any test you run can be done with that amount of resources in one week. Assigning the same people, a set amount of time, and making sure that you use it to always be moving forward will help your program achieve the cadence and scope of impact that you are looking for.

Infrastructure –

Having a proper infrastructure on your site is vital to be able to have a real program. The pieces of your infrastructure has two main components, the site code and the organizational components. A good robust infrastructure on your site should allow you to get a new test live within a day on any important part of your site without IT involvement. If you are creating testing code for each test, or if you require a separate ticket and full QA process for each test, you will never be able to move at the speed that a successful program runs at.

The other component for infrastructure is the organizational components. Do the important groups in your organization understand how you will run a test? Why you run a test? What will define success and how best to think about testing to provide meaningful answers? Are they bought in to testing or do they see it as something that threatens their imminent domain? Whereas the technical infrastructure is something that may be done mostly at the start and then revisited from time to time, the educating and working with various groups in your company is an ongoing thing and is vital for long term growth.

Rules of Action –

Once you have figured out your single success metric, do you have a consistent set of rules to define how you call a winner? How about when you have a winner, how does it go live, who is responsible, what is the timeframe? How do you build off of a test result? It is vital that you answer these questions and get people in agreement BEFORE you launch a test. Success and failure of any test is not in the lift it provides, but your ability to act on that data in a clear and expedited manner.

Groups will often write these down in a document and get signoff from different leads of the organization, or create a wiki and provide access to everyone. Each group has their own unique version of this, but it is vital that before you get to the action phase, that everyone is clear on how and why you do what you do. Nothing kills a program faster than people desperately trying to stop a change because they were “wrong” or people not willing to act on data because they were not ready to go. Never start a test if you are not able to act on the results, and rules of action are your way to ensure that you will be able to do this.

Program Champion –

Outside of having a single success metric, there may be no greater determination of long term success than having someone with some power in your organization who sponsors and pushes testing. Having a person who can navigate the political waters to enable the disciplines listed above, and to help communicate and hold people accountable for results. This person may not be involved in day to day actions, but they will often hold regular review meetings, talk to different groups, and present results to senior leadership. It is not about “owning” testing, optimization is part of everyone’s job, but this person helps get testing the attention it needs and holds people accountable.

You can look at any of the best testing organizations, and without fail you can name this person instantly and clearly. That person has bought in and while the program is often not perfect, they have helped lead it in a way that there is no doubt who sponsors it, why they push it, and how much value it brings to your company.

These disciplines represent just the first steps to really becomes a optimization leader, but they are vital in order for you to be able to grow and to facilitate the adoption of more advances skills. In the second set of disciplines, we will explore ways to think about testing to insure that you are getting more value from each of the actions which you just helped facilitate.

To navigate the entire testing series:
Testing 101 / Testing 202 / Testing 303 - Part 1 / Testing 303 - Part 2

“Social media offers clear advantages for developing more positive experiences and lasting relationships with our viewers and for reaching new audiences,” says Blandon Casenave, vice president of Digital Media Research at NBC Universal. “Using Adobe solutions, we can gather rich, actionable data that we can use to continually provide viewers with more personalized, relevant experiences.”

With the help of Adobe solutions, NBC Universal found that if a video viewer “liked” a video, the viewer consumed more than two times more videos; the company also uncovered that if a video viewer shared a video, he or she accessed almost three times the amount of digital video and online content; NBC Universal also discovered that when viewers commented on a video, their consumption increased by approximately seven fold. The level of insight that NBC Universal is able to gather can be essential in finding better ways to help advertisers reach the best audiences.

Check out the full success story at the following link: http://omniture.com/go/43261

What social media activities are driving the biggest returns for your company?

How many times have you had to try and get sign-off from a new group only to have them push back or say that doesn’t “feel right”. How many times have you wanted to bring testing to new people only to have them shy away the first moment what they were sure would win, loses? The irony of this is that some of the most ardent supporters of testing in mature programs are the very people who were challenging the programs at the very start. Anyone that has built their reputation on their artistic talent, or by declaring “here is how we are going to do this” has to overcome their fear of losing that control in order to gain the power and efficiency that testing can bring.

How many times have you gone into a conference room with people to brain storm? Or gone to an offsite just to come back with a long list of who you are going to target? Or some written rules on color guidelines or the like. There is a lot of hard work that goes into those efforts, but the problem is that they are filled with assumptions and compromise. They are designed to either make everyone feel like they contributed, or to make the HiPPO happy. What testing does in the best cases is stop that cycle, so that you are no longer trying to figure out the one way to make things work, but instead have open discussions on what is feasible. It democratizes ideas and is agnostic as to the value of them. The entire point is to be able to measure the value of each idea against the others and figure out the best one to go with. There is a great deal of fear, what if you are wrong? Does this make me look bad? My way has always been “right”, and so on. People have built empires on this fallacy, often with no one holding them accountable to the actual value of those ideas.

So how do you fight this? The first thing you must do is to get everyone to agree on what you are trying to accomplish. This has to be a single thing that you can measure and that is universal across the site (this is not about group A versus group B, this is about everyone working together to improve the site). I have seen so many programs struggle, get no value, or end up in political quagmires simply because they refused this first step. It can be very difficult or very easy, but at the end of the day, the single greatest determination of future success for optimization is agreement on what you are trying to accomplish.

Once you have that measure, then it becomes about taking those ideas and measuring them against that goal. Remember you want to challenge the common theory, meaning you should include null assumptions and things that contradict what you think will win. It serves you no good if you align everyone on finding an answer to a question if that question is irrelevant or sub optimal. What is funny is that there is almost an inverse correlation between what people think will win, and what does win. Get opinions from multiple sources, especially from one or two from outside the group that has owned that concept or portion of the site.

Step three is simply test. But at the end of the test, don’t worry if you were wrong, and don’t make it about you versus me. This is about everyone working together to find what works. If everyone is working for the same goal, then it is easy for everyone to get the credit and for everyone to align. The entire point was that the better the ideas feed into the system, the more diverse and risky those ideas are, the more you learn and the better the results you will have. You have to stop worrying about who was right and instead encourage people to be wrong. Being wrong gives you so much more than being right, and it gives you new learning to share and bring value to other parts of the site.

If you do this enough, you will get to the point where you no longer need to have those large conferences or off sites, you just need to compile the feasible options and move forward with letting the test tell you where to go. It becomes less about trying to fit the square peg into the round hole (or in some cases, into no hole) and more about aligning to move forward with what you learn. You will not end up at the feared 48 shades of blue axiom, but instead you will end up where you treat all feasible ideas as valuable. It frees up your UX and creative teams to try new things and to not worry about upsetting their superiors. It allows them the flexibility and the ability to be “wrong”.

Everyone is fearful of the unknown and the risk of giving something up. What is important is to share the challenge and the reward and to make it about adding value to what they were already doing, and to not blame anyone when they were wrong. Testing an idea is not about the loss of control, it is about helping to make it as successful as possible. This can’t be about you versus them, or my idea versus yours, in order to succeed you need everyone to work on achieving the same goal. Any system is only as good as its input, and any input without a proper system to facilitate it will always lack value in the end. Encourage new ideas, encourage trial and error, as long as you have a system in place to mitigate loss, you have so much more you can gain from learning a new path or stopping a bad practice on your site.