My Analytics & Optimisation Blog

Optimising with a flamethrower

2017-01-24T09:51:00.000+01:00

A/B..N / MVT testing is many marketers' favourite pastime, however, if your company isn't structured to support optimisation then your test program can easily become a waste of time and money. Part of the problem stems from the amount of misinformation that makes people believe it's possible to generate big gains from small changes. But assuming you're running your tests correctly, you soon realise these minor changes don't make much (if any) difference. Unless you're testing tangible changes/differences then you're on a hiding to nothing, and quite often "tangible" means difficult/ time-consuming stuff. If you're limited to presentation testing (button, image, layout, headline), this can be like rubbing 2 sticks together in an effort to start a fire. You can't be constrained to presentation changes only, to step up a gear you also need to be testing business logic, now you can throw away your sticks, pick up a flamethrower and really start some fires!

Productivity

There's this debate about whether the digital revolution is really increasing productivity - Charles Duhigg, author of Smarter, Faster, Better

Although we have more efficiently tools than ever before, some believe that technology is making us less productive; meetings, email, chat, endless notifications, news ... although you may feel you're "getting work done", there's a good chance you're producing nothing. Being busy is easy, however, accomplishing positive business change requires some thought.

If your testing tool offers a WYSIWYG editor, this now means you can accomplish a lot more without those pesky people in IT slowing you down! Unfortunately, when things sound too good to be true it normally is, if you want to become productive with your test program then you'll need to be working across your company and getting buy-in for radical change.

Test less, test what matters

Our best performing customers are dead - study conducted by Fidelity

Analysis by the financial services group Fidelity found their best-performing investors/customers were the least active (dead). Living investors under-perform because they tend to be actively trading or busy being busy!

Contrary to popular belief, not everything needs testing! Adopt a common sense approach to what needs testing and what doesn't and monitor the stats when content changes are made outside of an A/B test. Alternatively, your testing program will be bogged down with never ending tests that don't gain significance.

Stop paying too much attention to optimisation blogs and reports with claims on how to make big conversation gains. Much of what is written will not apply to your business, unfortunately there's not a top 10 list of things to A/B test that will boost sales. It's you that best understands your business, customers and products and it's this knowledge that should be used for developing an optimisation roadmap.

Networking, org structures

Upgrading your optimisation toolbox to a flamethrower requires your company to be aligned around optimisation. Unfortunately, too often the optimisation team is out on a limb somewhere which makes purposeful experiments challenging. Being part of the Product team could make more sense vs. Marketing but if restructure isn't possible then networking with the right people is essential so you can trade ideas and get buy-in with those capable of making big changes to your website, shopping-cart, apps, campaigns, algorithms etc. Without this, you'll be limited to moving things around on landing pages all day or rubbing 2 sticks together in an effort to drive conversions.

Okay, that's it - thanks for reading. Let me know your thoughts, ideas in the feedback section below.

Digital marketing fallacies

2017-01-12T15:52:00.000+01:00

In this post we'll look at popular marketing activities that could be having a detrimental impact on your business and why looking at economic policies and the reason they fail can help us understand the reason why.

Henry Hazlitt was a philosopher, journalist and economist, he wrote the acclaimed book "Economics in One Lesson". In Hazlitt's book he reviews government economic policy and argues that many are fallacies, he states 2 fundamental factors are responsible for failed policy:

1) Self-interest. While certain public policies in the long run can benefit everyone, other policies only benefit one group at the expense of all other groups.

2) Failing to analyse. There's a persistent tendency to only look at the immediate effects of a given policy, or only to look at the effects on a special group, and to neglect looking into the long-run effects of a policy, not just on that special group but on all groups. It is the fallacy of overlooking secondary consequences.

It's clear to see how these factors don't only apply to economics, self-interest is a problem in many businesses and it's compounded with dysfunctional org structure that can create competing teams, bonuses and egos. Failing to analyse can be caused by system/data limitations, lack of expertise or not wanting to know; Upton Sinclair's quote can often be applied: "It is difficult to get a man to understand something, when his salary depends on his not understanding it."

Before we look at marketing initiatives fraught with mistakes, let's close this introduction by reviewing the one lesson from Hazlitt's book "Economics in One Lesson". Notice how we can just as easily swap out "Economics" with "Marketing":

ECONOMICS, as we have now seen again and again, is a science of recognising secondary consequences. It is also a science of seeing general consequences. It is the science of tracing the effects of some proposed or existing policy not only on some special interest in the short run, but on the general interest in the long run.

Discounting

Fallacy: discounting increases sales, so it must be good

Now, I'm not trying to argue that all forms of discounting are bad, however, when little thought is applied it's probably the worst marketing activity possible with severe financial/brand consequences. Discounting, is everyone's favourite marketing trick, sprinkle some here and there and if by magic sales 'pop' and everyone is a genius! With that being said I understand the short-termism of the business-world and the pressures of "making the quarterly number", therefore making a special offer can be the most effective lever in your marketing toolbox. However, it's probably fair to assume that the majority don't reflect on what the financial consequences of discounting can be, so let's look at some bad examples of discounting.

Discounting by device
Many of the examples I've seen to date will most likely suffer from market cannibalization. Unless intentional, cannibalism, which can be seen as competing against yourself, can often harm profitability. Here's a great (bad) example from match.com - one of the largest online dating and subscription services:

Prices shown to desktop devices:

Prices shown to mobile devices (*for the same subscription/product*):

At the time of writing, this is how pricing compares for the same subscription:

Device	1 month	Saving	3 month	Saving	6 month	Saving
Desktop	£42.99	-	£92.97	-	£113.94	-
Mobile	£29.99	30%	£59.99	35%	£59.99	48%

Strategically, I don't see how charging half the price based on your device can be a sustainable long-term plan, especially when you consider an individual can be exposed to both sets of pricing which must be off-putting. The mobile pricing will undoubtedly relate to "mobile not monetising" which is a common complaint amongst marketers, and I'm sure when the mobile pricing was dropped there was a jump in sales but would a jump be pure upside? or are there longer term ramifications? Hazlitt's point of failing to analyse would certainly apply or more comprehensively put: neglecting to look into the long-run effects and not just on a special group (mobile in this example) but on all groups (all channels/ devices). Now, I'm not suggesting that looking at the long-term effects of match.com's device specific pricing is straightforward and from a technical standpoint creating an a/b test with control group across devices would be challenging, so faced with the alternatives then sticking with price parity has to be the sensible approach vs. throwing against the wall and hoping something sticks.

Discounting by channel
A bad example of discounting by Channel is when affiliate sites offer a coupon/ discount code that applies a lower price compared to that found on the merchant's front-facing store. Have you ever gone to checkout and seen a "promo code" box and feel you're missing out, so you go search Google instead of completing your purchase? Yeah, it's annoying! But when our enthusiastic marketer opens his/her analytics report, what they see is "new" sales attributed to a different Channel, what a stroke of genius! Even crazier examples include paid media offering lower prices on special landing pages compared to going directly to the merchant's website, one can only assume to improve the advertising program ROI at the determinant of the company as a whole. These are all examples of self-interest and failing to analyse the whole group and not just a special group; sales go up in one place and down in another. Debunking such events can sometimes be as easy as looking at a first and last touch report for highlighting sales/channel cannibalization.

Discounting frequently
This article wouldn't be complete without mentioning the problem of discounting overuse, we probably all know a website or retail shop that always seems to have a sale - are sofas ever not discounted at DFS?? Your customers build a tolerance to your discounting just as you do to consuming sugary energy drinks, in the early stages you get a kick which helps with your cognitive/physical load but the effect lessens with usage, to the point your overall health deteriorates. Hopefully, that analogy made sense! Below is an anecdotal example from my inbox, I'm regularly exposed to "limited time" promotions, but if I ever decide to buy the product and it's not on "sale" then I won't make the purchase, I'll just wait a month or 2 for the next "special" event or maybe I forget and don't buy. There does seem to be a culture now with everyone expecting a discount, in such a scenario it could be worth adopting what Amazon do where everything has a discount all of the time, the prices are inflated so it allows for an always on discount.

Smarter people than me have comprehensively discussed the negative consequences of over discounting so if you want to read more I've included a link to an article by Jim Novo in the further reading section below. In summary, this is a cautionary note, discounting frequently is likely to cause problems and would require proper analysis on primary and secondary consequences on both the short and long-term (which is challenging) to be sure there isn't a detrimental impact on your business in the longterm.

A/B testing

Fallacy: winning a/b tests improve business KPIs

Moving away from discounting, another common fallacy is thinking we're improving our business with lots of a/b testing. This is probably every marketeer's favourite pastime, the problem is most people don't know how to do them; this results in losing a/b tests being called winners that have no or negative impact. With the mass availability of tools such as Google Optimize, the problem will only increase, more so with the WYSIWYG editor trend. It's great that running a/b tests has never been easier, however, this only glosses over the difficult aspects of running tests, which are:

1) Validity. There is a host of reasons why your test is flawed, caused by statistical and setup errors; common problems include calling the test result too early, this happens due to lack of understanding and/or self-interest as we want our test idea to be a winner.

2) Hypotheses. What are you testing and why? Many people's idea of a hypothesis translates into guesswork, which most of the time is a waste of time. Testing larger more tangible changes, generally have a higher chance of success compared to small copy and layout changes but these generally require more effort and buy in from the broader organisation.

Even when tests are run correctly there's still the risk our winning test idea is actually a fallacy; this is due to us tracing the effects of our test in the short and not the long term, sometimes this is due to our final topic, the novelty effect.

Novelty effect

Fallacy: conversions increased with the introduction of a new feature so this must be good

The novelty effect is the tendency for performance to initially improve when something new is introduced due to increased interest, quite often once the newness has disappeared then performance regresses to the mean. Do you remember those "punch the monkey" banner ads when they first came out?

As you may have guessed, the concept was to punch the monkey! Click through rates must have been off the charts when they first started appearing but it's easy to understand why there would be a strong correlation between time and deteriorating click throughs. If you work in Digital then you're frequently exposed to new best practises, tools and marketing tricks. A few that spring to mind and could suffer from the novelty effect are:

Exit overlays
Remember when these first came out? What a cool idea to grab an email address or offer an intensive to buy before leaving a website. But, they seem to be everywhere now, not just when exiting but on arrival too. I can't be alone thinking these are overused and annoying, as people become banner blind they must also become overlay blind. If you still think they're the coolest thing since sliced bread then here's a post on how to implement.

Cart abandonment remarketing
There are various tools that offer cart abandonment remarketing. Assuming you've entered your email then you receive communications that offer assistance or an incentive to complete the purchase. Is this cool or creepy? How many of the "recovered" sales would have purchased anyway?

Lapsed subscription discount
You have an annual subscription to a newsletter or some software and don't renew immediately, before you know it you're receiving emails with big discounts trying to win you back.

The 3 marketing activities above could bring incremental conversions, however, over a period of time the effectiveness can disappear or worse run negative. Although anecdotal, I have tech-unsavvy friends tell me how they add items to cart and intentionally abandon to see if they receive an incentive, I'll admit that I always let my subscriptions lapse and wait to see if there's a better offer coming. The more prevalent an activity is, the more expected it becomes, so previous gains can disappear further down the line. Because of this, it's important to run validation campaigns to vouch for previous winners, time can be a majorly influencing factor so don't assume a winning treatment will always be a winner, it's healthy to be a sceptic!

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.

Further reading
Economics in One Lesson
https://www.amazon.co.uk/d/cka/Economics-One-Lesson-H-Hazlitt/0517548232

When Does a Visitor Need a Coupon?
http://blog.jimnovo.com/2010/12/17/when-does-a-visitor-need-a-coupon

Conversion ranges and confidence intervals

2016-07-27T14:43:00.000+02:00

On the 24th June, the UK nation woke to the news that the country had voted in favour of leaving the EU. This was mostly a surprise. The financial markets and sterling had rallied the night before, confident that the UK would stay. But why the confidence, what did they know? The polls had always reported a close outcome. Part of the reason for the surprise was the media (no different to a digital marketer) misleading their audience by communicating stats without a margin of error or confidence interval (which is a range of outcomes). A YouGov poll published a day before the vote showed 51% of those surveyed supported staying in the EU while 49% supported leaving. The margin of error was 3 percentage points (not mentioned by the media). Those +/- 3 percentage points means the result could equally end up (as it did):

Leave (49 + 3) = 52%
Remain (51 - 3) = 48%

A common question when working in Digital Marketing is: "How's the test doing?" And maybe you reply: "It's up 20%". And maybe you present your results in a similar format:

If you're communicating absolute conversion numbers - here's the problem: it's misleading. Although I've never run such a survey, I'm sure a good proportion of people would read the proceeding stats as "we're 95% confident the variant outperforms the control by 20%". Of course, we know that's not true, right?? However, it's easy to understand why people are misled. Instead, we should communicate a conversion range and confidence interval. So now we'd say "the confidence interval for the variant is: 8.3% to 9.1%'" or "a range of likely values for the population mean is 8.3% to 9.1%, with a confidence level of 95%". Using the Control and Variant confidence interval we then calculate the conversion rate delta range. VWO (Visual Website Optimiser) reports this but I've struggled to find an online tool that does this, however you can use Ewan Millar's Chi-Squared Test tool: http://www.evanmiller.org/ab-testing/chi-squared.html for calculating the confidence interval between 2 samples and then calculate the delta between the upper and lower range, so using the stats from the preceding table we end up with:

upper range: (9.1 - 6.9) / 6.9 = 33%
lower range: (8.3 - 7.6) / 7.6 = 9%
which gives us a conversion range: 9 to 33%

And when documenting your results it can look like this:

So there you have it, start being less specific with conversion rate while adding absolute values for context. Alternatively, report on the mean value but add the margin of error. It may confuse some to start with but it's more accurate and makes more sense.

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.

Improve your experiments with a better control

2016-07-13T19:11:00.000+02:00

Client-side testing tools render content and URL redirects on the client (browser), which will cause a small deterioration in user experience. This deterioration will vary but as marketers, we accept this based on the advantages we gain from rapid execution compared to working with a less agile server-side implementation. Client side testing tools generally have 2 methods for displaying content:

1. Overlaying
Your original or control content is loaded and overwritten with new content using JavaScript. Depending on the testing tool some use CSS to hide the original content before the new content is added to prevent a flickering effect, such as:

"visibility:hidden;" or "display:none;"

2. Redirects
Your testing software makes a client side JavaScript redirect using a command such as:

 window.location = "http://www.yoururl.com/variant-a.html";

Redirects are useful when your variation code exceeds 200 lines of code or the layout you're testing is radically different, it could be you're testing a completely different website on a separate domain.

So what's the problem?
Both of the methods mentioned above could negatively impact your test result leading to an invalid result, this is how:

Your variant page takes too much time to load due to the amount of additional code being output.
Tracking fires less consistently with a redirect or overlay.
Page flicker caused by an overlay or redirect (probably the biggest issue to look out for, especially with redirects).

All these scenarios give the control an advantage and invalidate your experiment. Of course you could argue that all of these issues should be picked up in QA but intermittent and browser specific problems can be hard to detect. Another solution could be to run regular A/A tests but that aside it should still be a best practice to create a more 'apples to apples' environment.

Solution
The fix is simple. When using an overlay also do the same on the control but of course you overlay with the original content - so nothing changes other than we now mirror the rendering mechanism of the variants we're testing. The same applies for redirects, so we redirect the original URL back on itself - you need to be careful that you don't create a redirect loop, I wrote a quick post on how to prevent this with Adobe Target here. We now have a fairer comparison with a more consistent user experience.

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.

Preventing redirect loops with Adobe Target

2016-07-06T11:47:00.001+02:00

This is a quick tip for Adobe Target users when creating redirect tests/offers. It's possible (as I discovered) that one day you need to redirect to the same page you have an A/B test running. An example could be, adding parameters to your Control URL which is then used by another data capture system. Redirecting to the same URL becomes a problem as you end up in a redirect loop *unless* you kill the mbox on the redirect. I've been aware of the mBoxDisable=1 URL debugging parameter and found it useful for developers that don't want to be part A/B tests while working in staging, but it can also be useful if you want to prevent a redirect loop, here's what it looks like:

It's worth noting that although we're making a call to disable the mbox it doesn't impact reporting, so the redirected user with a disabled mbox is still part of the test and reported on. And that's it. Maybe you find this useful, maybe not!

Hotjar review

2015-09-02T19:33:00.001+02:00

I've been using Hotjar pro for a few months and felt it deserved a review. Before I get started let's take a quick look at what Hotjar does with some added commentary from my experience of using the product.

Hotjar feature set
> Heatmaps
Heatmap reports allow you to toggle between desktop, tablet and phone devices while viewing click, move and scroll data. The average fold and percentage of page scrolled are handy indicators that in the past I've had to calculate independently of the clickmap tool, so I've found this feature very useful. There's a "share heatmap" button that is great when emailing reports to people without Hotjar accounts. Probably the killer feature recently added is the capability of creating heatmaps on dynamic/session based pages (including password protected pages). This means you can look at heatmap data on shopping cart pages, we had looked at this in the past with another vendor but were restricted in having to implement a PHP API which wasn't practical.

An example heatmap from a form page:

> Visitor recordings
Visitor recordings are one of my favourite tools, you can't beat spending time watching how people actually use your website. What's really cool is you get to see what they're seeing, so if you have an A/B test running - you'll see the actual variant they're exposed to. I was quite surprised with how far this actually goes, to the point I've watched recording of users that have a browser infected with adware and you see the ads too. As with the heatmaps it's easy to share recording with a public URL (again, very handy) but the most useful feature is being able to create visitor recording on session/dynamic pages such as a multi-step shopping cart.

An example visitor recording taken from my blog:

> Conversion funnels
If you have a set of pages you want to analyse you can create a conversion funnel and watch visitor recordings of those that fallout at stages or those that go on to complete the funnel. I have only dabbled with this feature, I like the idea but to date I've mostly used the visitor recording on particular page/s that interest me.

> Form analytics
This feature allows you to see form field level abandonment rates, which fields take too long to complete and what fields are left blank. I've only briefly experimented with this feature as they don't support forms on dynamic pages yet, the good news - it's on the roadmap.

A form analysis report with drop off, time spent, left blank and re-filled data:

> Feedback polls
This has been my most used feature, it takes minutes to launch a poll (like the one below) and before you know it, highly actionable feedback start coming your way. You can trigger your poll using 4 different events:

Immediately after the page loads
After a delay of n seconds
When user is about to abandon the page on a desktop device
When user scrolls halfway down the page

These are perfect for high value funnels when the visitor is about to abandon.

> Surveys
The survey feature allows you to build fully-blown surveys using Hotjar's editor, once done they can be published via URL or just before your visitors abandon your website. The following options are available when adding questions:

Long text answer
Short text answer
Radio buttons
Checkboxes
Net promoter score (you should see this one when you scroll down this blog post)

More question types are on their product roadmap including "question logic" which might be important to some of you.

> Recruit user testers
Hotjar's recruiters feature allows you to recruit your own visitors for user testing. You have the same targeting options as with Polls and the following information is captured:

Full name
Age
City
Email
Phone number
Country (automatically captured)
Device (automatically captured)
Browser (automatically captured)
Operating System (automatically captured)

What makes Hotjar standout
Okay, so here's a run down of what's impressed me since using Hotjar:

> Unlimited
There are no restrictions with server calls, number of websites (assuming they're yours) or the number of users. This is a stark comparison with other vendors where we've had to conserve server calls by only adding tracking codes at the time of launching a heatmap, video recording or survey. This "always on" approach becomes a big time saver as we can now go live in minutes with initiatives compared to days/weeks when multiple teams would have been involved with adding tracking and QA.

> Free version
There's a free version available, this is a great acquisition idea and I'm surprised more vendors don't try this. I started using the free version on this blog and soon after we started to use the paid version at my work.

> Support
The support is fantastic, I've had issues resolved within hours of reporting them. Even the free accounts get support and all my emails have been responded to in a matter of hours or less. I'm not sure how this will scale as they get more users but for now it's excellent.

> Pricing
The pricing is very disruptive when compared to more established players in the marketplace such as ClickTale. For €89 per *month* ex. VAT (yes, monthly billing opposed to annual) we get a ton of features and insight that would cost significantly more when using multiple vendors.

> Roadmap
There's a publicly available roadmap for everyone to see and understand when new features are coming: http://docs.hotjar.com/page/roadmap

What can be improved
> Integrations
Being an Adobe customer I'm keen to see integrations with Adobe Analytics and Target but there's no mention on the product roadmap. This could be an issue further down the road as heatmaps, net promoter scores and visitor recordings are all good scenarios when web analytics data would be needed for further context.

> Technical
We've had an issue with a few heatmaps when the page is using unique div ids so click data can't be reported. Moreover click data reporting can be confusing depending on how your pages are structured as each div has clicks associated with it, so if you have a button with parent and child divs then click numbers will be associated at different areas of the button when I'd expect to report one number associated with the whole button. However, heatmaps are more about looking at a picture than reporting absolute numbers so this isn't a deal breaker.

> Security
I'd like better security options, this isn't a complaint specifically towards Hotjar as most SaaS platforms don't offer much. At a minimum you should be able to restrict login access by IP address.

Wrapping up
In summary the ease of getting Hotjar's features live has been invaluable for gathering qualitative data for actionable experiment ideas, fixing content and debugging. If you're haven't tried it yet give their free version a spin - you've got nothing to lose.

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.

Is Digital Analytics misleading?

2015-06-30T17:00:00.000+02:00

If you search on Linkedin you'll see that Digital or Web Analytics is the de facto title/department name for many of us, but is the phrase misleading? First off, what is Digital Analytics? Judah Phillips wrote the book Building a Digital Analytics Organization and describes it as:

Digital analytics is the current phrase for describing a set of business and technical activities that define, create, collect, verify, or transform digital data into reporting, research, analysis, optimizations, predictions, automations, and insights that create business value.

The activity of digital analysis, at the highest and best application, helps companies increase revenue or reduce cost.

And the definition from the Digital Analytics Association:

Web Analytics is the measurement, collection, analysis and reporting of Internet data for the purposes of understanding and optimising web usage.

There's probably little surprise with the above descriptions, so why the beef? Digital Analytics is being used to describe a multidimensional discipline but few will understand this and instead assume analytics and analysis are the same thing and that's how most dictionaries see it too. The technology research firm Gartner wrote an article called The 'Analytics' buzzword needs careful definition:

The definition of 'analytics' is a broad one, leading to confusion among enterprise users
...
When Gartner conducted an informal survey among users about what the term means, the answers ranged from online analytical processing (OLAP) to monitoring call centres to reporting and data mining.
...
Gartner had more questions than answers after the survey.

So what's a better name?
The endgame for Digital Analytics is consistently described as optimisation. We measure for reporting, we use reports for analysis and we use analysis to optimise. Good existing department names within the organisation include; Customer Acquisition, Customer Support, Customer Retention. A bricklayer isn't called a brick-picker-upper, a restaurant server isn't called a food-carry-outer. Moreover, neither role uses a buzzword to inaccurately describe a collection of tasks. Scrutinising semantics may seem pedantic but confusion can be caused by organisational fractures. The rising trend of companies with a separate optimisation department is evidence of this. Many working within these optimisation silos will baulk at the idea of using Adobe/Google Analytics, consequently leading to a less scientific approach to experimentation with hypotheses being swapped for opinions and guesswork.

Having a department name aligned with its goal will have a positive business impact (IMO) as physiologically employees are reminded of their endgame. Let's face it, when's the last time someone understood what you meant by telling them you worked in Digital Analytics? ...okay, web/digital optimisation might need further explanation, but a few might know what you're talking about!

What do you think? Is this petty-mindedness or it does make a difference?

Create an exit popup with Google Tag Manager & Ouibounce

2015-06-25T18:27:00.000+02:00

Have you ever wanted to add one of those popup windows that disable the page content and make you acknowledge an offer when you try and exit? The popups are called 'modal windows' and according to Wikipedia: 'A modal window is a graphical control element subordinate to an application's main window which creates a mode where the main window can't be used'. There are companies that sell exit intent technology, most notably BounceExchange (at $10K per month though it's not cheap). But fear not, this post will show you how to add much of the same functionality using the free and open source JavaScript library from Ouibounce and the free to use Google Tag Manager.

First off: I'm sure exit intent windows can help achieving a conversion goal in certain scenarios, however it's a balancing act between this and not annoying visitors, so you may want to test this over a good period of time and be sure returning visitors aren't negatively influenced. Anyway, back to our modal window example and how to display it when the visitor is about to leave.

Give it a spin!
[Note: this intentionally won't work on on a mobile device] Act as if you're about to leave this page and see it in action, and while you're at it; sign up for my newsletter! The modal window has been set to "aggressive mode" which means you can refresh this page and you can initiate the modal again, non-aggressive mode would use a cookie to only show the popup window once per visitor.

Okay, so now that you've seen it in action - on to the steps:

Step 1: Add Jquery & CSS
Make sure you have jQuery included on your landing page:

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js">
</script>

Now grab the CSS here: http://carlsednaoui.github.io/ouibounce/modal.css and add to your page, this can obviously be adapted to your website brand.

Step 2: Add your hidden modal window
Here's the modal source code taken from the Ouibounce website, you can format however you want however don't change the id or class names, some classes can be omitted, such as the title and footer - so experiment with what you need.

<div id="ouibounce-modal">
<div class="underlay"></div>
<div class="modal">
     <div class="modal-title">
          <h3>This is a Ouibounce modal</h3>
     </div>

     <div class="modal-body">
          <p>You can style your modal however you want.</p>
          <br>
          <p>Close this modal by clicking "No Thanks" or outside of the modal.</p>

          <form>
               <input type="text" placeholder="your@email.com">
               <input type="submit" value="learn more &raquo;">
          </form>
     </div>

     <div class="modal-footer">
          <p>no thanks</p>
     </div>
</div>
</div>

Step 3: Create the tag in Google Tag Manager
Go to 'Tags > New' and add the code below then hit Continue and on to the targeting.

// Use the snippet below for including the library or host yourself
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/ouibounce/0.0.11/ouibounce.min.js"></script>

<script type="text/javascript">
// By default "aggressive mode" is set to false, so the option
// below can be removed for one popup per visitor
var _ouibounce = ouibounce(document.getElementById('ouibounce-modal'), {
     aggressive: true});

// Hides the modal when visitor clicks outside of the modal window
$('body').on('click', function() {
     $('#ouibounce-modal').hide();
});

// Hides the modal when visitor clicks on the close / footer link
$('#ouibounce-modal .modal-footer').on('click', function() {
     $('#ouibounce-modal').hide();
});

//  jQuery stuff to keep modal window open when it's meant to
$('#ouibounce-modal .modal').on('click', function(e) {
     e.stopPropagation();
});
</script>

Step 4: Target the modal
URL
I only want this modal window to appear for this blog post so I've created a tag that's triggered on Some Pages which in my case is when the URL contains 'create-exit-popup-with-google-tag'.

Step 5: Create exceptions
DOM ready
We want the tag to fire only when the page is ready and our modal window is accessible so we add a blocking trigger to prevent our tag firing until the Document Object Model is ready.

Excluding mobile
Exit intent detection *isn't supported on mobile* devices and I'm not sure you'd want to do it anyway, to be sure our tag won't fire and to save page load for mobile devices we can exclude our modal window tag by adding some custom JavaScript as a User-defined Variable:

function()
{
    // detect mobile device
    var mobile = navigator.userAgent.match(/Android|BlackBerry|iPhone|iPad|iPod|Opera Mini|IEMobile|webOS/i);

    if(mobile)
    {
        return mobile;
    }
    else
    {
        return "desktop";
    }
}

Once that's saved it can be set as a blocking page view trigger. The above code returns a mobile name when it's a mobile device and the value "desktop" when it's not. So if the value is not "desktop" then this rule will fire and block our modal window code from being executed. Adding the blocking rule and referencing our User-defined Variable, looks like this:

That's everything done, you can now save the tag, QA and publish and you're good to go! Our saved tag with firing triggers looks like this:

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email, RSS or by trying to leave this page!

What to use for personalisation?

2015-06-07T17:03:00.000+02:00

Personalisation is up there with some of the top digital marketing buzzwords, but how do you do it? Let's be honest, unless you're Amazon it's unlikely people are logging into your website when they visit, so you have no idea on age, gender, purchase history, etc and you might not even have an account login, so what do you use?

Personalisation vs Targeting
Before we start it's worth being on the same page with what Personalisation is, as quite often marketers discuss Personalisation and Targeting interchangeably, here's how Wikipedia describes the 2:

Personalisation, also known as customisation, involves using technology to accommodate the differences between individuals.
Targeting is to make a thing or group of things a target, to select it or them to be acted upon.

You can think of Personalisation as a customer/visitor benefit and Targeting as a marketer benefit, however, there are situations when the line seems fine. A good example is when Starbucks use mobile proximity targeting to pull customers into the store, this is clearly a marketer benefit and not you having content personalised based on your location.

Outside of one-to-one marketing, much of what is termed as Personalisation could perhaps be better described as content customisation. With that in mind, let's look at some targeting/personalisation examples that don't require a logged-in state or account id:

Self-identification
Why guess? Just ask 'em! A simple example below from British Airways asking for your country and language as you visit their website; this is a far better approach compared to when Google has a guess and shows you the default language of wherever you're located. This could be expanded onto many levels, how about asking homepage visitors to identify themselves into customer segments (business, consumer, support, agent, reseller). Armed with this information your personalisation capabilities have significantly increased.

Technology
Browser
According to surveys, if you're using Internet Explorer you're above 40 and technically less savvy; Chrome user and you're younger and more "with it". I have observed staggering conversion rate difference between browser types. I'm sure this is why Adwords (& others) don't offer browser targeting as much of their ad revenue would vanish overnight. So if correlations between browser and age are believed, this can be powerful especially if you offer products that target certain age groups.

Operating System
While operating system conversion rates will most likely vary, my experience (to date) suggests it's less predictable and persona assumptions are harder to make, that being said if you're offering OS specific products it's a highly predictive variable; for example you offer Windows and Mac hardware/software.

Device
Responsive/ adaptive design is one of the most popular forms of content customisation, with content and sometimes the offer changing radically based on the device. Customisation needs to be approached with caution as users are often in research-mode on mobile and want to see the same content as a desktop user. If your analytics data shows that mobile users don't convert as well as desktop, maybe the user experience should be simplified so that mobile users can be educated easily with more selling options left to the desktop/tablet experience.

Onsite behaviour
When visitors are exposed to paid search landing pages, visit a product page or take an action such as download, make a purchase, search onsite they are self-identifying and this information can be used for returning visits, for example, a homepage that helps them complete their goal easier.

Location
Depending on what you do, this could be considered the low-hanging-fruit. Popular products can vary across geo and region so this can influence homepage offers. If you're a multinational company this could be an opportunity to adapt creative to the country or city so the experience feels more local and if you have a multi-language website then your personalised language should read like it was written by a native and not Google Translate.

Time
The big one here could be seasonality, based on the period of time and sometimes used in conjunction with location it might mean certain products are more popular and content can be adapted accordingly. Depending on your business this could be done at a granular scale for example weekends or even time of day.

New vs. Returning
How does visitor behaviour differ? Are new visitors less likely to convert compared to repeat, if so does research material need to be more prominent for new visitors while returning visitor content can be more focused towards conversion?

Referring URL
This could be an easy way for adapting content if you have some significant referrers where the referring content helps identify the persona.

Tracking always on personalisation
So how do you know if it's working and there's an ROI? Where possible I would recommend an always on control segment so that measurement is possible, as an example: you decide to offer around the world cruises on your travel website homepage to Internet Explorer visitors as this is an activity popular with the elder generation. Instead of personalising to 100% of Internet Explorer visitors you would reduce the sample to let's say 90 or 95 percent with the remaining ratio receiving the default. This makes measurement possible, things change so this is why you'd want to be measuring all the time and not to assume after an initial successful test period that the personalisation will work forever. Microsoft's new browser, codenamed "Project Spartan" could be a huge success and steal back market share from Chrome which would play havoc on your age based boat cruise personalisation.

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.

Optimisation analysis. Know your visitor

2015-05-18T22:00:00.000+02:00

If you had to sum up in a few words how to analyse for optimisation it would be this: know your visitor. Knowing nothing in an anonymous visitor environment would equal a spray and pray optimisation approach. Observing visitors and how they differ will facilitate the formulation of hypothesises for experimentation or recommendation for change. There's always an array of analysis that's possible, even in an environment that lacks user login data. So let's look at examples that are instantly available in your digital analytics system.

A good way to start making observations about visitor differences is to break down main KPIs and compare against dimension values. As an example if you had an online wine store, some key metrics would be: order rate, average order value, average order quantity and bounce rate. Some worthwhile dimensions for comparison are: geo, browser type, device, time to purchase, visitor frequency, checkout, traffic source and products purchased. Our wine store investigation would involve a thorough review of each KPI, so starting with order rate, we'd iterate through each dimension one level deep. When analysing ratios such as order/conversion rate, I would also look at absolute order numbers for giving context and gauging popularity, so you could think of this as an order rate and orders investigation. If we took 5 dimensions to analyse this would output 10 investigations:

Dimension combination	Potential questions
Geo-> traffic source	How do these vary? Does natural or paid search convert better in the UK? Is traffic distribution across channels even or skewed towards a poorer performing traffic source? Maybe email is The Channel and some countries have fewer email addresses?
Geo-> checkout	If there's order rate differences across location, why could that be? How about: a popular payment method missing in a particular country or suspect translations?
Geo-> product	What's the look-to-book ratio for the top 10 selling wines and how does it compare across geo? What do these differences mean? Does homepage content properly represent what's most popular in each country?
Geo-> browser type	How does the user's browser impact order rate? Are Chrome users more likely to purchase? Could this be useful for personalisation?
Traffic source-> checkout	There will most likely be differences here, is there anyway to make changes to the checkout experience per traffic source?
Traffic source-> product	Are certain traffic sources more likely to purchase particular wines and can this be used for personalisation?
Traffic source-> browser type	How does browser conversion rate vary across geo? Are Firefox visitors in Germany more likely to purchase compared to Internet Explorer visitors in the US?
Checkout-> product	If the checkout order rate varies based on the product, why could that be? Do the descriptions or thumbnail images need a review in the cart?
Checkout-> browser type	Are there browser incompatibility issues? IE 8 causing problems again?!
Product-> browser type	Are visitors of certain browsers more likely to purchase certain wines? If so can this be used for personalisation?

Once the above investigations are complete we do the same for the other KPIs and when exhausted we can go another dimension deep (when it makes sense), for example: Traffic source-> checkout-> geo. I find it helpful to export the data to Excel where you can apply conditional formatting, the average order value analysis below asks questions straight away; why are the French so financially frugal? Don't they like our fine British wine?!

You will have more dimensions depending on your business which all play a role in better understanding who your visitors are and how they behave. I find this systematic approach helpful otherwise it can seem overwhelming on where to start.

I hope this was useful and thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts on Twitter, Email or RSS.

How long should my test run for?

2015-04-18T08:38:00.001+02:00

AKA: How many conversions do I need? When working in optimisation or analytics this question can come up a lot. So opposed to frequently repeating yourself this is a good subject to have documented somewhere for referring your inquisitive marketing persons. Below is an article I've complied with the help of several sources to best answer this question.

A common misconception is to stop the test once a statistical confidence level of 95% or higher is reached. Well, consider this - 1,000 A/A tests (two identical pages tested against each other) and during the test:

77% reached 90% significance
53% reached 95% significance

So if you stop your test as soon as you see significance, there's a 50% chance it’s just noise. [3] Moreover data from Google suggests that 90% of content changes have no or a negative impact [1] so we need to be sure we're testing correctly!

Sample size
A healthy sample size (visitors, conversions) is at the heart of making accurate statistical conclusions. An A/B test reaches confidence when the observed difference is bigger than chance alone can plausibly explain. Imagine you are trying to find out whether there's a difference between the heights of men and women. If you only measured a handful of men and women you would risk not detecting that men are taller than women. Why? Because random fluctuations mean you might choose an especially tall woman or group of women. However, if you measure many people, the averages for men and women will stabilise and you will detect the difference that truly exists. That's because statistical power increases with sample size. [1]

Understanding how many visitors / conversions you need
There's a couple of things you need to know:

Your baseline conversion rate (what the conversion rate currently is of your page).
The minimum improvement you want to be able to detect, sometimes referred to as the Minimum Detectable Effect. You *could* think of this as the uplift expected from the experiment. Be warned: if you limit yourself to detecting uplifts of 10%+ you will miss the smaller wins that were out there.

The table below gives you a general guide for visitor and conversion volume *per branch* (an A/B test has 2 branches). Example: You're running an A/B test with a baseline conversion rate of 5% and you want to be able to detect a relative win of 5%, this would require 120K * 2 = 240K visitors which is approximately 12K conversions.

How to calculate The Minimum Detectable Effect
This value used will depend on what's being tested among other factors, from experience when little or no testing has been carried out then bigger wins are possible but when the low-hanging fruits have been picked off you could be looking more in the 2-7% range; this could mean longer and fewer experiments.

Time
In addition to having sufficient sample size (volume), experiments also need to run for at least one full business cycle, this is normally 7 days (although 14 would be better). Additionally experiments should not be stopped mid-cycle, i.e. on day 10, instead we should continue the test for another 4 days. Average order value and conversion rate can differ between each day of the week and results should reflect the full mix of visitor types which can vary between early morning on a week day and the afternoon of Sunday, moreover running experiments for sufficient time means we're less likely to be impacted by the novelty effect. [3] [5] [6]

Capping sample size
Now that we know how many visitors/ orders our experiment needs we can cap the volume. This means our experiment stops when we're completed a full business cycle and we have the minimum required sample size; at this point it's declared a winner or a loser.

What happens if your experiment is a mega-win?
Medical experiments use sequential experiment design for exactly this: sequential experiment design lets you set up checkpoints in advance where you can decide whether or not to continue the experiment. This means if the minimum detectable effect was originally set to 5% but the treatment creative is outperforming by 15% it's possible to have this evaluated at predefined checkpoints that would give the required significance level. [2]

References
Most A/B test results are illusory [1]
How not to run an A/B test [2]
How many conversions do I need [3]
Sample size calculator [4]
How long to run an A/B test for [5]
Novelty effect [6]

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts from this blog on Twitter, Email or RSS.

Creating an interaction analysis dashboard

2015-02-19T20:22:00.001+01:00

Interaction analysis is looking at how visitors interact with your website or other digital property. Examples could be click through rates on rotating homepage banners, % of page scrolled, online tool/feature interaction, video play rates and pathing reports. With so many interaction possibilities being tracked and spread across multiple reports, this feels like a good opportunity to create a consolidated view in a dashboard.

Why should you bother?
Over a period of time your website can become a virtual dumping ground for the latest fad marketing trend such as "Like" and "Tweet" buttons, moreover you may have many interactive gadgets and controls but does anyone know if they're being used or is it just unnecessary clutter? By creating a dashboard dedicated to interaction you can educate your UX team on what's working and what's not. And with a data engaged design team a Digital Analyst is more likely to get buy in for proposed optimisation and experiments.

Making the dashboard
For the purpose of this example we'll be using the following software (others will do the trick too):

Adobe Analytics (Omniture) + ReportBuilder
Excel
getPercentPageViewed plug-in
Snagit (for screen capturing and labeling)

And for our interaction analysis example we'll take Apple's homepage. As of this writing their UK homepage looks like this:

We'll be taking a screen capture of Apple's homepage and adding to our dashboard, it helps to include a visual representation of the user interface, without this the dashboard data is more challenging to interpret. In the example below I've added numeric labels with corresponding chart titles to make reviewing of the data super easy, regardless of analytic level. For some light analysis there's a country option (top left) that allows going one dimension deep. And here's what it looks like in Excel:

And that's it. Hopefully this gives you some ideas. Over a period of time you can build out a dashboard to cover important pages and funnels, some pages can take a while to fully document but the effort is worth it IMO. If you have any comments, questions or feedback please leave them below. And you can follow new posts from this blog on Twitter, Email or RSS.

Performance monitoring analysis. Who's doing it?

2015-01-26T19:00:00.000+01:00

Performance monitoring analysis across digital properties often falls through the cracks due to unclear ownership or the assumption someone else is doing it. I've read stories how analysts have uncovered shopping cart browser anomalies that have saved companies millions in revenue, that's all well and good however with a proper performance monitoring framework in place this should get picked up outside of ad-hoc analysis.

What is performance monitoring analysis?
Performance monitoring analysis also known as "is crap broken/working analysis" is the regular (daily/weekly) review of your important digital metrics. You'd expect your Technology department to be monitoring system health and performance, however, there'll be all kinds of things they can't see or aren't concerned with. If you're in e-commerce you can normally assume the top line metrics are being reviewed regularly so if revenue falls off a cliff then alarm bells start to ring. But what about everything else that's a dimension or two deep? It's quite possible that all sorts of discrepancies, bugs and blips are about and you just wouldn't know about it.

What should be monitored and by who?
It's not efficient to centralise the monitoring so I'd suggest a decentralised approach which effectively distributes analysis across the business with appropriate business/product owners. So the million dollar question - who are the owners and what are they monitoring? To understand the "what" it generally helps to split your business into commercial and platform products, let's take an online bank as an example and for brevity take a simplified view of their commercial, platform products and metrics;

Commercial products	Metrics	Dimensions
Bank accounts A primary product of a bank and online application forms would be a major source of new business.	Application form visitors, Completed applications, Form completion ratio	Traffic source, Account type, Nationality
Mortgages	Mortgage amount, Application form visitors, Completed applications, Form completion ratio	Traffic source, Mortgage type
Insurance	Insurance amount, Application form visitors, Completed applications, Form completion ratio	Traffic source, Insurance type
Platform products
Website Seeing as browser compatibility can be the cause of many anomalies you'd want browser versions available as dimensions with the main KPIs.	Visitors, Completed forms, Logins, Error pages	Browser type, Browser version, Operating system
Website accounts This is the account login for customers to check balances and make transfers.	Logins, Average time, Transfers	Account type, Location
Mobile accounts Same as the above but for mobile.	Install rate, Uninstall rate, Crash rate, Logins, Transfers	Operating system, Account type
Online forms I'd envisage this being an aggregate view of all the lead generation forms, the data being reported will have overlap with some of the previous products listed but I don't see that as an issue, there could be more detailed metrics being reported such as form validation errors. In addition, there could be non-commercial products such as mortgage calculators as part of this view.	Form Visitors, Form error rate, Form completion rate, Abandonment rate	Form type, Traffic source
Analytics All of our commercial and digital products are being monitored by our analytics software which means we need to monitor the monitoring system to be sure of the data integrity.	Unclassified and unknown KPIs, Completed forms, Visitors, Form completion rate	Browser type, Browser version

How to help
Identify who the metric owners are, sometimes this is easy and sometimes not - platform products are more abstract than commercial so if there's not already a product owner see where you get by nominating someone, quite often people are happy to be better informed with relevant data around an area they have a working interest. Once you have a list of products and owners you can get to work on developing the dashboards, this is where tools such as Adobe's Report Builder can really help; pulling data into Excel and scheduling reports is a breeze which means your product owners don't need to login/learn your analytics package.

Anomalies should be caught sooner with a good reporting framework in place, and with a distributed analytical network working across the business it's possible that relevant optimisation suggestions are generated too.

Right, that's it - thanks for reading. If you have any comments, questions or feedback please leave them below. And you can follow new posts from this blog on Twitter, Email or RSS.

Building an experiment dashboard

2015-01-13T19:30:00.000+01:00

This article will explain a cool way of monitoring and presenting your experiments that will save you time and make you look (more) awesome! Prerequisites are you're familiar with; Adobe Target, Analytics/Omniture, Report Builder, Excel and you're using the Adobe Target plugin ...that being said you can probably repurpose this for other analytic and testing tool environments.

What problem does this solve?
Building an experiment dashboard gives you a single view of all your experiment metrics without having to create multiple reports spread across browser screens. Moreover it can also become your experimentation documentation, and the dashboard format will copy & paste easily into presentations, so ultimately you'll be killing many birds with a single stone (sorry birds).

How to do it
For saving time in the long run we want to make our dashboard reusable so this means creating templates per experiment type. The 'types' are defined by KPI and number of variants. So if you're running an A/B experiment and your main KPI is orders - this would be a template, likewise so is an A/B/C experiment with orders as the KPI. You may have another A/B template where registrations/installs are your main KPI ....you get the idea. Once you've created the template types for your common experiments you'll only need to change a couple of parameters in your excel sheet and hey presto everything works! So let's get started with an example; here are the steps to create a dashboard for an A/B experiment where orders are the main KPI.

Step 1)
Plan the dashboard. When running an A/B experiment one of my favourite reports is viewing the conversion/order rate daily performance versus the control. This simply shows whether it won or lost and by how much and is a great way to see if there were any anomalies to investigate. Additionally we want to see the running totals so our first 2 charts look like this:

The next 2 are very similar but this time we're looking at revenue per visitor:

And the final 2 we're adding are for distribution (how many order and visitors have been part of the experiment) and average order value:

Step 2)
Next we fire up Report Builder and add our data blocks, we need to use the "Campaign > Recipe" dimension which allows us to select the experiment recipes/variants.

It's advisable to have a "settings" sheet in the Excel document where we can add; the running date of the experiment and the "Campaign > Recipe" values. The data blocks will then reference the cell locations (as it's doing in the screen captures above and below). For future experiments it means we just change a few Excel cell values and everything works opposed to editing a load of data blocks - which isn't fun.

Campaign > Recipe values:

Step 3)
Add calculated fields, conditional formatting, charts and make it look pretty. I'm not going to go into detail as I'll assume you know Excel and the template used in this example can be downloaded below.

Step 4)
Optionally you may want to add a section for describing the experiment and documenting the outcome. In a separate sheet you can also add screen captures of the control and variant content.

And once done the final output will look something like the below. It's worth noting that we should try not to go overboard with charts, our goal is to have a good overview. We can dig deeper if needed by going to our Analytics tool. That being said adding metrics such as visitors, orders and revenue trended seems perfectly valid.

Hopefully this was useful - the Excel template used in the above example can be downloaded here. If you have any comments, questions or feedback please leave them below. And you can follow new posts from this blog on Twitter, Email or RSS.

What does a Digital Analytics department do?

2014-12-01T20:56:00.002+01:00

So, what does a Digital/Web Analytics department actually do? Ultimately my belief is we're product owners of the "Optimisation Platform". The definition of this platform being a combination of; analytics, optimisation, tagging, survey and delivery systems. So for me this means; Adobe Analytics, Target, Dynamic Tag Manager, Qualtrics, Clicktale & Experience Manager.

Developing a roadmap

Strategy and developing a roadmap is at the heart of product ownership and for the "Optimisation Platform" this is a combination of implementation advancements and optimisation execution. Making experimentation, reporting and analysis easy should be a primary dimension of our role. Easy access to data means more insight, likewise easy experimentation should mean increased test volume and wins. My roadmap includes optimisation execution and analytics/optimisation/delivery implementation improvements. We as a department outline the implementation projects whereas the optimisation execution side of the roadmap is a collaborative effort with other stakeholders from the business, they have ideas for experiments, in addition we fill the roadmap with experiments and personalisation projects that are based on data and good ol' fashioned creative reviews. Ultimately our roadmap is about driving; measurement, analysis and optimisation.

Snapshot of an example roadmap document:

What else is there to do?

Analysis	Sound analysis should be the basis for much of what gets added to the roadmap for experiment execution. Other types of autonomous based analysis may not require being part of an experiment, in addition there will most likely be requests from the business for ad hoc analysis which sometimes requires vetting.
Quality Assurance	This is an interesting topic, there's the obvious need of validating tracking before release, however there's also the QA requirement for experiments and personalisation campaigns. Many companies will have centralised QA departments but getting them involved with A/B, MVT or personalisation campaigns could be challenging, so I believe this should be a task for the digital analytics department who'll fully understand the underlying technology, tracking and targeting. Ultimately this could mean some interesting conversations with your IT department!
Reporting	No surprises with this one. The burden or report requests can be reduced with well planned dashboards. Our department's reports generally focus on data integrity such as unknown revenue/goals, monitoring of live experiments, browser compatibility, server calls.
Tracking	With the advent of Tag Management Systems gone are the days of having to work with IT for changing tracking codes, so this pillar can firmly be entrenched within the Digital Analytics team. Maintaining the "Optimisation Platform" will often require technical know-how so having an individual/s that understands JavaScript, networking and how your systems work will be useful.
Training & Support	This covers all areas of the platform; identify the individuals/teams that will benefit from ongoing training and those that are more suited for top line dashboards.

Comments

How does this compare with what you're doing? Please add your thoughts and comments below, some conversations from Twitter so far:

"What does a Digital Analytics department do?" http://t.co/FsD6JpSndU Good article by @richardhayes28 #measure
— Jan Exner @ Adobe (@jexnerW4D) December 1, 2014

@jexnerW4D @richardhayes28 @jenkinsa But the comment "Gone are the days of having to work with IT for changing tracking codes" - huh??
— Peter O'Neill (@peter_oneill) December 2, 2014

@peter_oneill @richardhayes28 @jenkinsa I read that as "can finally decide myself that I want to use cid instead of ccid"
— Jan Exner @ Adobe (@jexnerW4D) December 2, 2014

@jexnerW4D @peter_oneill @jenkinsa Now we have a TMS and data layer, I can't remember the last time we contacted IT for tracking :)
— Richard Hayes (@richardhayes28) December 2, 2014

@richardhayes28 @jexnerW4D @jenkinsa Who creates the Data Layer though? I agree to adjust business logic but adding new tracking?
— Peter O'Neill (@peter_oneill) December 2, 2014

@peter_oneill @richardhayes28 @jenkinsa the perfect reason why when designing the DL, I'd go for a "give me all you got!" approach.
— Jan Exner @ Adobe (@jexnerW4D) December 2, 2014

@richardhayes28 @jexnerW4D @peter_oneill @jenkinsa so you a) were supreme beings during DL design; b) needed only minor changes since then.
— Lukáš Čech (@cataLuc) December 2, 2014

@peter_oneill @jexnerW4D @jenkinsa We define the DL. Yes, we control *all* tracking and do the QA. It works well.
— Richard Hayes (@richardhayes28) December 2, 2014

@richardhayes28 @jexnerW4D @jenkinsa @cataLuc Is the DL defined for both page load & visitor interaction, I haven't got it right 1st time b4
— Peter O'Neill (@peter_oneill) December 2, 2014

@peter_oneill @jexnerW4D @jenkinsa @cataLuc I think you're missing the point. Decentralised structural change can be a consideration now.
— Richard Hayes (@richardhayes28) December 2, 2014

@richardhayes28 @peter_oneill @jexnerW4D @jenkinsa forgive me my scepticisms. Experience says that it is possible, but so hard to achieve.
— Lukáš Čech (@cataLuc) December 2, 2014

@richardhayes28 @jexnerW4D @jenkinsa @cataLuc I have disagreed with TMS vendor statements that IT not required e.g. http://t.co/KLC0ucsVhF
— Peter O'Neill (@peter_oneill) December 2, 2014

@cataLuc @peter_oneill @jexnerW4D @jenkinsa Tools like DTM are taking work away from IT and consultants. Fact :)
— Richard Hayes (@richardhayes28) December 2, 2014

@richardhayes28 @peter_oneill @jexnerW4D @jenkinsa some tasks yes, but at the same time you are doing a couple of trade-offs, agreed?
— Lukáš Čech (@cataLuc) December 2, 2014

@cataLuc @richardhayes28 @jexnerW4D @jenkinsa Would be a great #MeasureCamp discussion session
— Peter O'Neill (@peter_oneill) December 2, 2014

@richardhayes28 @cataLuc @peter_oneill @jexnerW4D Marketing should b autonomous. IT as lynchpin for services is the problem which DTM solves
— Adam Jenkins (@jenkinsa) December 2, 2014

@jenkinsa @richardhayes28 @cataLuc @jexnerW4D Why marketing? What about product, UX, etc? Mkting can't write tags w good naming convention
— Peter O'Neill (@peter_oneill) December 2, 2014

@peter_oneill @richardhayes28 @cataLuc @jexnerW4D Because Digital Analytics is function of marketing n yes MKTG can maintain n write tags
— Adam Jenkins (@jenkinsa) December 2, 2014

Adobe DTM for improved Target QA

2014-11-21T20:22:00.000+01:00

In a previous post I described the options available for Adobe Target QA. Now with DTM there's a much easier/nicer/quicker way of doing this! (This post will take ~2 minutes to read.)

Step 1) Make sure you have the DTM browser plugin installed - it's available for Chrome and Firefox. The plugin allows you to see debug messages but also easily switch between DTM's production and staging environments which is what we want for the purpose of this post. It's worth knowing that you can recreate this functionality with some JavaScript bookmarks, so this can be used for IE or maybe you prefer using bookmarks opposed to the plugin.

Switch staging on bookmarklet:

javascript:localStorage.setItem('sdsat_stagingLibrary',true);

Switch staging off bookmarklet:

javascript:localStorage.setItem('sdsat_stagingLibrary',false);

Another approach is to go to your browser's console and add the above commands without "javascript:" ....like this:

Step 2) Create an *unapproved* page load rule in DTM for adding the Mbox. If you're not sure how to do this you can read this post (it's easy).

Step 3) Switch staging to "On" using your browser plugin or bookmarklet and go to the page with the Mbox - unapproved DTM rules are rendered when you have Staging switched on - this is the *magic*. Now hit refresh a couple of times so Adobe Target gets to see the Mbox.

Step 4) Login to Adobe Target and create a campaign using the Mbox that was just created and publish (make live).

Step 5) Go back to the page with the Mbox and start your QA. Once you're sure everything works - publish the Mbox page load rule in DTM.

Easy. Any questions or comments feel free to add below.

Who owns your digital content?

2014-11-13T12:16:00.000+01:00

Unclear ownership is a popular complaint amongst digital professionals and much of this pertains to content. So with that in mind I thought it would be interesting to think and discuss digital content ownership. So, who owns what?

Ultimately your website/digital property is a product and your product's presentation layer requires ownership and by ownership I mean; whoever "owns" the content should be defining the roadmap and giving final sign-off on creative. Before we can define ownership we need to list the reasons for producing and changing content:

Acquisition - mostly landing pages; search, display, affiliate and various content for SEO.
Conversion - scheduled and unscheduled (one-off) content for existing; customers, users, visitors. Some examples include:

homepage banners/pages (special offers, seasonal content)
email (encouraging first/repeat conversions + general communications)
app messaging (encouraging first/repeat conversions + general communications)

Products - new products result in new product pages. In addition (an important one), your website probably has digital products such as; shopping cart, account login, mobile apps etc - these would all have product owners.
Experimentation - A/B and multi-variant, experiments run for a finite amount of time and content changes when there's a successful result. Experiments span all content categories; 'conversion', 'acquisition' and 'products'.
Personalisation - these are targeted campaigns that run 100% of the time (no end date), examples would include; returning visitors, OS, device, browser. Personalisation would span all content categories.

So based on the above 5 content categories, here's my opinion of ownership:

Acquisition	Your acquisition/search team should own their content as they're responsible for paid campaigns and SEO. The alternative would be an internal blame game if targets aren't met.
Conversion	This team is creating most of your companies content, there could be designers as part of this team and they could also be the same team that's developing content for the other content categories; Acquisition, Experimentation, Personalisation and Products. The name of this department could be something like "Sales & Marketing" or "Campaign management". Ultimately this team owns most of your digital property content including the homepage. However an important caveat, this is ownership of the default/control content and important pages should only change based on the outcome of a test.
Products	Assuming your business has products of some description, I'd assign final sign-off of these pages with the corresponding Product Owner/Manager - ultimately these people should be 'on the hook' for their product's revenue. If this feels like an odd fit then it could also sit with the same people that are managing the conversion content. However your digital products such as; shopping cart, account login, mobile apps would have product owners and they would have final sign-off on content.
Experimentation	Testing hypotheses are usually based on data insight or visual reviews - the roadmap for this should be owned by the Digital/Web Analytics & Optimisation team (or something similarly named). Owning the roadmap involves adding tests and managing requests coming from; Acquisition, Sales & Marketing and Product. If the requests have weak/no hypothesis then it shouldn't go into your roadmap.
Personalisation	Same as above other than all personalisation campaigns are driven by data insight, other stakeholders in the business can have inputs however if data doesn't support the idea or the volume is too low it shouldn't go into your roadmap for testing.

So in summary your content is owned by 4 departments:

Do you agree or have a completely different idea? Let me know below if you have any comments or questions.

Using DTM as your campaign targeting layer

2014-11-01T07:06:00.000+01:00

DTM has a whole load of targeting capability which includes being able to target Mboxes. Put another way, DTM can replace your Adobe Target campaign level targeting. So why bother?
Adding targeting at the Mbox level through DTM will reduce your Mbox server calls/costs. In the past if we wanted to run a homepage test for Mac OS visitors we'd burn server calls for 100% of the visitors regardless of their OS, now with DTM as the campaign targeting layer, we're only using server calls for those that match our targeting rules:

Global Mbox strategy

If you're currently using a global mbox it does pose the question whether you'll want to continue as this will undoubtedly be more expensive. The primary reason for adopting a global Mbox strategy was to have the ability to add mboxes to any page through Target opposed to bothering IT, however you can now do this through DTM and it's *significantly* more user-friendly (a marketing person can do it) compared to adding them in Adobe Target. Here's how easy it is in DTM. So I'd recommend you disable the global Mbox in the Marketing Cloud before you download:

When to use Target or DTM

There will be situations that you'll need to define targeting in Adobe Target for example at Experience or success metric level, here's how my responsibility split would look across both systems when creating an optimisation campaign:

DTM	Adding Mboxes, Campaign level targeting (including scheduling).
Target	Outputting content, Experience and success metric targeting, defining success metrics.

Both Adobe DTM and Target are highly complementarity and I'd recommend you give them a spin if you haven't already. Any questions, feel free to ask below.

Multi-channel attribution coming to Adobe Analytics

2014-10-26T07:34:00.000+01:00

Good news! I was told a couple of weeks ago that multi-channel attribution is coming to Adobe Analytics Standard and will include various attribution models. Unfortunately it probably won't be available until sometime next year, so here are some options if you don't want to wait:

Option 1) Upgrade to Adobe Analytics Premium (formerly 'Insight'). Understandably not an option for most.

Option 2) Using the Cross-Visit Participation plugin you can concatenate visitor data, so if you concatenate your campaign evar value across visits and use the '>' delimiter you could end up with a report that looks like this:

PPC > Email > Display
Affiliate > Website > Display

A couple of caveats - this can become a nightmare to classify, so hopefully you have well structured keys to make this easier via rule builder/ Excel. You won't be able to detect natural search or direct traffic so these 2 channels may need to be rolled up together, in the example about it's been labelled "Website".

In addition you can store your campaign eVar in a traffic variable that will allow pathing your classification at visitor level in Ad Hoc Analysis (formerly "Discover").

Option 3) Using a combination of first, last and linear eVar allocation to observe how your campaigns and channels are contributing to success. Again this won't include direct or natural search Channels. However one major caveat is linear eVar allocation only distributes values within a visit. So for many this won't be usable.

In addition, according to the Ad Hoc Analysis documentation: "Ad Hoc Analysis has the unique ability to report different allocations for conversion variables: the default allocation, last allocation, and linear allocation. If Ad Hoc Analysis is your organization's primary reporting tool, having all conversion variables set to first allocation gives you all available allocation values." However my experience suggests this isn't correct with all values reported as identical across; last, linear and the default (first), so either the documentation is incorrect, there's a bug or I'm somehow misinterpreting the documentation.

Option 4) Working with Adobe consulting there's an option that involves a daily import using the web API to import data from data warehouse. There's 4 models available: First Touch, Last Touch, Linear, U-Shape. The attribution model is a Java application and will involve you processing the data on **your own hardware** this is taken from Adobe's documentation (not a joke!):

I'll personally wait for multi-channel attribution to be available in "Standard" while using a combination of option 2 and 3. Let me know if you have any questions or comments below.

Evolution of the web analytics department

2014-10-05T09:59:00.000+02:00

Most web analytics departments function as a 'thermometer' informing the business of what's happened opposed to it's original and often forgotten true existence as a 'thermostat' - helping to control and influence what's happening. The definition of 'web analytics' according to the Digital Analytics Association is: Web Analytics is the measurement, collection, analysis and reporting of Internet data for the purposes of understanding and optimising web usage. The 'optimising of web usage' is the most important aspect but normally completely missing or vaguely accomplished at best. An influencing factor behind this is companies siloed department structure; Analytics, Delivery and sometimes Optimisation are managed independently with different agendas and direction. To properly drive a web/digital analytics strategy you need analytics to be embedded within the organisation and not just a mundane reporting and tracking function. With the evolution and integration of tools such as; Adobe - Analytics, Target and Experience Manager the traditional digital roles and responsibilities have become clouded so it's about time for internal reform.

Analytics + Optimisation

The evolution started with tools like Adobe Target, now we can run tests ourselves based on data driven insights without the need of IT and it was our first foot-in-the-door of updating website content. Our department name had changed from 'Web Analytics' to 'Web Analytics & Optimisation'. In parallel Tag Management Systems allowed us to take control of tracking and adding mboxes, so we no longer needed to bother IT and wait weeks/months for deployment. IT were now aware of us having more control and we started to receive more calls, emails and meeting invites. Were IT getting worried that we no longer needed them to save the day?

Analytics + Optimisation + Delivery

Content delivery has always been technical - either through the production, the system itself or both. With the advancement of content management systems like Adobe Experience Manager the entire business is empowered with content change allowing for decentralised updates, the spreading of workload across many teams and content/component reuse. This now means the "Delivery" function is less about churning out content and equally weighted with system configuration and advancement. A well configured system encourages self-service and empowers business users outside of the Delivery team - the same philosophy holds true with a well managed web analytics implementation. Seeing as optimisation is the primary objective or web analytics, this should be the major motivating factor behind content change. These change requests become the driving force behind how components/content should function and where + what should be personalisable - all of this drives the furthered consolidation of responsibilities into the 'Web Analytics & Optimisation' team. But now we're called 'Digital Analytics, Optimisation & Delivery' and we own the 'integrated optimisation platform' and deliver a strategic roadmap with everyone aligned on objectives and priorities. And here's how responsibilities could look with a more integrated team approach (click to enlarge):

Understandably every business is different and the changes I've outlined might not be feasible for you, however if you've decided upon an integrated platform approach then at the very least you'll need a unified strategic direction. Let me know your thoughts.

Remarketing cart abandoners with DTM & Target

2014-09-28T18:05:00.000+02:00

I'm going to show you how straightforward it is to target visitors that have abandoned a shopping cart without purchasing, we'll retarget these visitors with personalised homepage content related to what was in their shopping cart, at the same time we'll run an A/B test to report on whether there's uplift. This article will demonstrate how quickly you can implement more advanced testing/targeting that prior to DTM would have involved passing parameters via mboxes and possible intervention from IT. Now with DTM we can do it in no time without having to change any page code!

Step 1) We want to target visitors that had iPhones in their shopping cart for future retargeting, to start with we'll create a 'data element' in DTM that accesses the iPhone product name from the cart, this could just as easily be a SKU/product id or other similar unique identifier. Here's what it looks like in DTM:

This is where we're grabbing the product name from:

Step 2) Now we create a Page Load rule that is triggered when our data element (from previous step) has the value 'apple iphone'. In addition we set a cookie for 30 days - this will be our retargeting window and we'll use this cookie for targeting our personalised homepage content.

And setting the cookie in the 'JavaScript / third party tags' section:

 _satellite.setCookie('cart-contents-iphone', 'true', 30);

Step 3) Next we create an mbox named 'homepage-remarketing-iphone' that will wrap the remarketing content, for this example we'll assume it's the main homepage banner. So we create another Page Load rule which targets the homepage URL and we conserve our precious mbox server calls by only outputting when visitors have the 'cart-contents-iphone' cookie value set as 'true'. Prior to DTM we had to burn mbox server calls on 100% of our homepage traffic.

Step 4) Now we login to Target and create a 'Landing Page Campaign' (this forces our targeting to be evaluted every time, A/B would probably work too though). We target the mbox location: 'homepage-remarketing-iphone' and we only want visitors in this campaign that have *not* purchased, so we target our campaign with a custom segment from the 'Marketing Cloud Audiences' target of 'Non-Purchasers':

We'll use default content for the control and personalised iPhone content for the variant. Once this campaign is live we can report on whether our personalised experience is yielding incremental revenue.

What the default content/banner looks like:

And our superduper, optimised & personalised version:

How easy was that?! These type of tests before DTM could have been more of a multi-department project and now it's more like BAU.

Adding mboxes with Dynamic Tag Manager

2014-09-25T13:25:00.002+02:00

I'm going to show you how to add Adobe Target mboxes to your pages using DTM. Combining Adobe Target with DTM can take your testing and targeting to the next level. Many organisations have areas that are difficult and time consuming to run tests/ add mboxes, such as the shopping cart. DTM removes all boundaries, in addition the complex and abstract scenarios for evoking an mbox become limitless!

Step 1) First off login to DTM and Adobe Target as a tool - this is available as a predefined option so set-up is a breeze, you have the option for managing the mbox.js file which means you now have control over updating this file. Once added your overview panel will look something like this:

Step 2) For this example I want to target an mbox in the shopping cart where the URL contains "cart".

Step 3) With the Adobe Target tool enabled we now have a new section for adding and naming mboxes. Mbox placement is done by referencing pages elements via the DOM, right-clicking on a page element and using the developer tools enabled with your browser makes this a straightforward process. In the example below I want to change the page title which can be accessed like this: h2.heading.

In total I'll change 3 elements on this page - see below how I'm referencing them:

Step 4) Login to Adobe Target and create a Campaign and Offer referencing the mbox we just created. Those with good eyes will see my new shopping cart title in the HTML offer box below: <h1>Richard's shopping basket - 100% money back guarantee</h1>

Step 5) Once we've created all our variant offers and set our TnT campaign live we can access the cart page again to see the highly optimised version:

When we inspect the page before and after we can observe the differences generated by DTM -

Before:

<h2 class="heading">Your shopping cart</h2>

After:

<div id="_sdsat_mbox_5290797208435833_" style="visibility: visible; 
display: block;"><h1 class="heading">
Richard's shopping basket - 100% money back guarantee</h1></div>

You may not agree however I think this is frigging awesome! Many organisations can wait months to get simple tests run in high volume funnels where small conversion gains can make a big difference. A Tag Management System utilised to it's full capability can become more of a "targeting layer" or a Targeting Management System - plain old "Tag" doesn't do it justice.

Testing Adobe DTM using Charles Proxy

2014-09-19T06:09:00.000+02:00

I'm going to quickly show you how you can start using all the cool features of Adobe Dynamic Tag Manager in less than 5 minutes. We're going to do this using Charles Proxy, for those of you not familiar with Charles Proxy, it's an HTTP proxy /monitor / Reverse Proxy, one of the coolest features is it's rewriting capability that allows you to "fake" making changes to your live website - this will allow us to add DTM in no time! Prerequisites are that you have Charles Proxy and Dynamic Tag Manager (if you don't have access yet - contact your account manager).

Step 1) Login to DTM and after you have added a new property, go to the "embed" tab to access the Header and Footer code. I'm only using the Production code. It will look something like this for the header:

<script src="//assets.adobedtm.com/cb442236e8fbb669a07de6df8af5f191b2203455/
satelliteLib-aff16ca4897413887b57a00e881397b3ebeb3f42.js"></script>

And the footer:

<script type="text/javascript">_satellite.pageBottom();</script>

Step 2) Next we open Charles Proxy and go to: Tools > Rewrite > Add. From here we rewrite the HTTP response we receive from your website which allows us to add the Header and Footer DTM code whenever we have Charles Proxy open. I've named the rule DTM and added the location which should be your website address.

Step 3) Within the rules section create a "Body" rule where we match the opening head tag of your page and replace with the DTM code and the head tag prepended:

Step 4) Create another "Body" rule and this time we match the closing body tag and replace with the 2nd part of our DTM code and concatenate the closing body tag:

5) If you already have Web Analytics tracking on your page such as a reference to your s_code file, you'll probably want to remove it. In this example we have Adobe Tag Manager (the version prior to DTM), so we'll break the URL and prevent the code from executing:

Finally your DTM rule should look something like this:

Now when we open up your website and as if by magic, when we view your page source we can see the DTM code in the Header and Footer locations we specified, which means hours of endless fun playing with Dynamic Tag Manger. Here's a great place to start learning: https://outv.omniture.com/

Optimisation & personalisation roadmap template

2014-09-10T19:32:00.000+02:00

It's a good idea to have all your planned experiments documented somewhere with timelines. Attached and screen shot below is the format I'm currently using. I'm breaking out the type of activity into:

Homepage testing
Price testing
Landing page test
Personalisation
Shopping cart tests

Feel free to download and adapt for your needs.

Cohort analysis - Adobe Analytics/ Report Builder

2014-09-06T12:08:00.000+02:00

This article will show you how to carry out cohort analysis using Adobe Analytics/ Report Builder. For the purpose of this example a fictitious company called "Acme software" has a 30-day trial application and we'd like to know when orders are taking place post the trial download. This will mean using download date as a cohort and comparing it to the order date. Although this scenario might not be relevant for you, the basis of this article can be adapted for whatever cohorts have meaning, for example; order, newsletter sign up, registration, install, etc.

1) First off we'll create an eVar for storing the download date - which we've called "First touch download date", we're storing the date in YYYY-MM-DD format (this is an Excel friendly format too). Allocation should be first touch (means it won't be overwritten), and we've set the expiry to 90 days. Adding classification might make things easier for you, so this should be a consideration - for example; week number, month.

2) Now that we have an eVar for storing when the download occurred we need another for comparing when the order happened, we use the same format as before; YYYY-MM-DD and we're calling this "Last touch date". This is simply the current date and will be attributed to the order or any other event we want for future cohort analysis.

3) Next we're going to build a segment, this won't always be required but I suspect it'll be a common requirement when building cohorts. So for this example I want a segment that only displays a particular downloaded product ("Acme software") and only visitors that have completed an order - this will really refine our data request. Our segment uses a Visitor container with our downloaded product and sequential order event.

4) Now to Report Builder - apply our segment, dates and apply aggregated granularity:

5) Next add our 1st dimension 'First touch download date'.

6) Click "next" and add our 2nd dimension 'Last touch date' and add the order metric. Our preview now looks like this:

7) Run the request and our Excel output looks like this:

8) We now add a 'days difference' column which is a simple subtraction of order and download date which will return an integer. Our excel sheet now looks like this:

9) The final part is to create a pivot table based on the 'Orders' and 'Days difference' column:

And Voila! With this data we can now create line charts that tell us when we receive orders based on the download date cohort: