Nike!

Most online ad effectiveness studies are designed to measure the branding impact of the campaign.  This is because while the internet is uniquely designed to directly measure transactions, it’s not well designed to solicit viewer engagement and/or opinions about advertising.  Many of the campaigns that we find ourselves measuring are specifically designed to understand who saw the advertising and whether that exposure turned into increases in brand funnel metrics.

Now for the fun part; without looking (I know it’s hard), what is the name of the brand I mentioned at the top of this post?  Go ahead and look… did you get it right?  This is a simple example (admittedly an exaggeration) of how most online ad effectiveness studies work.  As we know, most online ad measurements occur via a pop-up or even in a banner invitation.  Typically, the technology that triggers these invitations (JavaScript) is linked to the ad server that delivered the advertisement to be tested.

In practice, the invitation to the ad effectiveness survey is served immediately after exposure to the advertisement.  Technically speaking, it’s a little more variable than that; for example, a typical DHTML invite is triggered on the page following the one where the ad exposure occurred.  Or with short form research, the ad shows up in the ad space, effectively hiding the ad after a fixed length of time, usually 30 seconds.  And it gets even more complicated than that, as I’m sure plenty of people will point out to me.  However, suffice it to say that the event of inviting someone to take a survey is typically triggered within minutes or seconds of online ad exposure.  It’s really no different than my Nike example.

So now you have to ask yourself the question, “Is data collected immediately after ad exposure useful?”  I’ll save you the thinking and simply say yes. But that yes is a qualified yes, as is pretty much everything that gets said in market research.  Let me explain the qualifying factors, comparability and decay.

Comparability

If you’re measuring online in a vacuum it may be that assessing the immediate response to advertising is a decent approach when evaluated using a good experimental design. However, we rarely find a brand that runs advertising only online.  Typically, advertising is run across multiple media channels and most advertisers want to benchmark the advertising effectiveness across them to answer questions about creative efficiency for each media.  So when you compare online, which is evaluated within minutes of exposure, to TV, which is typically measured within days of exposure, you set up an apples and oranges comparison.

Most of the brands we work with dislike online measurement specifically for this reason, since to them it seems as if the methodology is designed to make online advertising look good.  Considering that the research is often funded by people who want online to succeed (media publishers), they look at the methodology and discount its value in assessing the impact of their campaigns.  As a result, even though there is technically nothing methodologically wrong with such a short timeframe on the measurement end, client advertisers are skeptical of the approach.

Decay

Our other qualifying factor is ad decay.  If you’re measuring the impact of advertising immediately after exposure, it’s impossible to understand how quickly or slowly the memories of that advertising fade.  Understanding the decay rate of your advertising is imperative to determining optimal frequency and implementing a flighting plan for your campaign.  Ironically, I don’t know of anyone that regularly takes ad decay into account in their online ad effectiveness measurement.

Late last year we hosted an ARF  a webinar on cookie deletion and ad decay and presented some of our initial findings on the topic of ad decay.  If you’re an ARF member, you can find a  replay of the webinar at their site.  But for the rest of you let me summarize what we found.  It may seem like a no brainer, but online advertising does indeed decay over time.  In the data we analyzed we found that over a 96 hour window a metric like ad recall (the key metric used in brand trackers to evaluate TV impact) decayed 12%.  This is a big deal, especially when you consider the implications on comparability.  Without measuring at differing intervals post exposure, it’s difficult to impossible to determine ad decay.

Falling Short

While short-cutting methodology for the sake of simplicity may be enticing, let me also point out that short form research can be plagued by technical glitches in how the survey invitations are delivered.  The best practice for short form invitations is to trigger the invite to cover the ad being measured.  By doing so, you effectively prevent the respondent from cheating by just looking at the advertisement on the page.  While this might seem like a simple task, believe me it’s more complicated than it sounds, especially when you think about serving that survey invitation across the hundreds of pages in a typical ad campaign.  Take, for example, the ad below for American Family Insurance that was run on Parents.com.  Notice the invite to the survey and how it’s not covering the advertisement.  While this wasn’t our study, I’m pretty sure I can say that based on this glitch the campaign was likely a success.

Taking a New Approach

Leveraging the Ignite Network recruitment approach puts InsightExpress in the unique position of being able to measure the immediate impact of advertising as well as the decay rate of that advertising.  This is a fundamental analysis that we believe needs to be part of how you evaluate and measure your advertising’s effect.  We consider ourselves pioneers in the study of advertising decay effects, and will continue to innovate and evolve what can be learned from this method.

So, would I recommend immediate measurement of online campaign effects?  Well of course, as long as you’re not concerned with comparability or decay.

Working in research, I often catch myself quoting stereotypes but only because they end up being true.  Minivans are more likely to be driven by moms, Neiman Marcus shoppers are higher income.  And in the world of research, survey takers are more likely to be women and more likely to be older.

In research speak, we refer to it as non-response bias or self-selection bias.  Essentially, certain segments of the population (e.g. men) are more likely to be non-responders to surveys.

It’s the reality of what happens when you invite someone to fill out a survey and,  typically, it’s not a big deal.  If I’m looking for a sample of 50 men and 50 women, I could take the proactive step of inviting more men than women to fill out my survey, understanding that fewer  men will be willing to take the survey and at the end the numbers will even out.

Type A Problems

However, taking that proactive step is not always possible.  Let’s assume we don’t know the gender of the people we’re inviting to our survey – something I’ll call a Type A problem.  In this case I just have to invite a bunch of people and look at the gender distribution after I collect the data.  And since women are more likely to fill out surveys than men, I might look at my results and find I have 60 women and 40 men.  Not exactly my 50/50 split.  Well, research affords another tool to turn this situation into the results we want, namely weighting.  I can weight down the women and weight up the men.  If each man’s opinion is scored 1.25 times I’ll have the equivalent of 50 men’s opinions, and if each woman’s opinion is scored 0.83 times I’ll have the equivalent of 50 women’s opinions.  So in essence, Type A problems are easily solved.

Type B Problems

But what happens if I don’t know the gender breakdown of the people I’m sampling, and I don’t know how many should be men and how many should be women – a Type B problem?  Let’s walk through that scenario.  I invite my sample to take my survey; 70 women and 30 men respond.  In this case, I don’t know if the final result should be 50/50 or 60/40 or 70/30 – that data is just not available.  Based on the realities of non-response, I can’t really trust the 100 responses I’ve collected but, then again, I’ve no way to correct the data.  This is the challenge with Type B problems: they can’t be solved and they don’t produce sound research results.  Yet clients are out there paying good money for this kind of data on a daily basis.

Online Ad Effectiveness = Type B Problem

Let me explain that last point in more detail.  Your typical ad effectiveness study is run using a pop-up recruitment method.  These invitations are triggered randomly at the time of exposure to the advertising.  It’s great because it’s random, but at the same time you have no visibility into who’s going to fill out the survey until your study is complete.  So what I have here is an example where I don’t have any advance insight into who’s going to get the survey invitation.

As I previously mentioned, this is not a big deal because we should be able to weight the data after the fact to account for any non-response bias.  But wait, in an ad campaign I don’t know who saw my ads in the first place so I don’t know how to weight the data.  I don’t know if I have too many men, or too many women, etc.   Essentially, if I have no visibility into the sample prior to the campaign and no idea how the campaign was distributed across demographic audiences, then I’m completely in the dark as to the efficacy of my results.  If you’re running studies using these kinds of invites (e.g. Vizu, SafeCount, etc.) then you have this problem whether you’re aware of it or not.

Let’s not forget to mention that targeted advertising exacerbates the issue.  If I have a campaign that targets men, yet most of my ad effectiveness results are coming from women, I can come to one of two conclusions: 1) my targeting didn’t work and I actually reached more women than men, or 2) women are more likely than men to fill out online surveys.  Now that we’re all up to speed on non-response it’s obvious that the answer is number 2 – women are more likely to fill out surveys than men.

What many online ad effectiveness solutions lack is any understanding of the actual audience being reached, and in some cases they also lack information about who is being reached.  Without those critical pieces of information, a researcher has no tools that can be used to adjust for biases in the data.  When biases exist in a dataset you can’t trust the conclusions you draw.

Solving the Type B Problem

As you’ve probably noticed, we do things a little differently here at InsightExpress.  First and foremost, we leverage our Ignite Network as a preferred approach to online ad measurement.  Since it’s panel-based, we’re able to determine a significant amount of information about the audience being reached by advertising.  In fact, prior to sending any surveys we can determine who’s being reached by the campaign.  So in those cases where an advertiser might be targeting men 18-34, we can quantify with the panel how many men 18-34 the advertiser is actually reaching.  This fantastic bit of information serves as the necessary data to combat the issue of non-response.  Even if our sample has some non-response biases (which all surveys have) we can correct those biases by balancing the survey results against our Ignite Network responses.  Voila, correct data!

Non-Response and Shortcut Form Research

Think about it: this is another reason why short form, one-question approaches are generating bad results.  How would you correct for non-response in your data if you have no demographic information to enable that correction?  Short form research suffers the biggest challenge of all:  they know nothing about the people they sample prior to sampling them, and they still know nothing about those people after sampling them.  Even if you had a calibration source for one-question questionnaires, you don’t have any demos on the respondents to correct for the non-response bias; they’re engineered to produce bad data.

Of course, the counter argument you’ll hear from short form vendors (specifically Vizu) is that their response rate is so high it makes weighting irrelevant.  While this line of reasoning might sound great, let me point out two fundamental flaws in their logic.  First, even if your response rate is multiple times higher than the average response rate, you’re still less than a 5% response rate which is far from being representative (check out my previous post for more background on response rates).  Secondly, non-response is non-response, which put another way is to say that people who don’t respond will still not be responding.  Women will still be more likely to take surveys than men, and older people will still be more inclined to take surveys vs. younger people, etc.  No matter what any company does, there will still be biases in the population where specific segments are willing to take surveys and others are not willing.  Nothing can change that fact, and the only tool you have as a researcher is data to adjust for and correct that bias.  However, when you run your surveys as one- question questionnaires, you’ve engineered yourself into a corner.  You can’t correct your data because you have nothing to use to correct the problem.

When in Doubt, Ask

If you’re buying ad effectiveness research from a vendor, make sure you ask them if they correct for non-response biases by weighting the in-tab results against an audience delivery report.  If they don’t, you can’t necessarily trust the data.  To that extent, even if they adjust their results based on an audience delivery report, make sure you understand the source of that report because it could also suffer from a non-response bias, depending on how the data is collected.

As I explained in an earlier InsightfulAnalytics blog post, what makes online ad effectiveness measurement work is the use of an experimental design.  I’ve also mentioned in earlier posts that while experimental design is a fantastic approach and one we recommend, for a variety of reasons clients prefer to run quasi-experimental studies.  One of the important aspects of putting together a good quasi-experimental design is to create a control cell that is as equivalent to the test cell as possible. Unfortunately, and if you’ve read some of my other posts you’ll realize this is a trend, that’s just not how things work online.

When I first started doing online ad effectiveness research in 1997 there was no such things as ad server delivered tags.  Everything we did for sampling a campaign was hard coded to a page, including the advertising.  This made for an extremely easy design.  Since there was no complex ad server to worry about, I could randomly redirect visitors to either the page with the test ad or the page with the control ad.  It doesn’t get much better than that – pure random assignment of the respondent pool.  However, with the advances in ad serving the survey sampling code moved into the ad server and thus began the era of the pop-up and the dreaded bonus inventory.

For those of you who don’t know how this works, let me paint a picture for you.  I negotiate a buy with a publisher for 100,000,000 premium ad impressions.  It’s pretty sizable and I want to measure the effectiveness of the campaign.  Since the test ad will be measured via pop-up (or more precisely a DHTML fly-over) triggered by javascript code on the page where the ad runs, the only gap in sampling I need to fill is people who didn’t see the ad.  But here’s the rub: I don’t want to spend any more money on premium impressions to run a public service ad, just to collect people who didn’t see my tested ad, so instead I ask my publisher for bonus inventory to run a PSA.  Now it gets tricky.  The publisher knows I want to evaluate the advertising and indirectly their site, so a measurement of the campaign could mean future business.  It’s also likely that the agency will be annoyed if the publisher doesn’t fork over bonus inventory, so the reality is the publisher has few options.  As you can imagine, there’s more downside for the publisher in this equation.  As a publisher, you’re forced into giving up bonus impressions for a campaign which means you’re giving away inventory that could be earning you money – so it’s a loss leader.

If you were a publisher and your client just bought 100,000,000 home page impressions and was now asking for bonus inventory for a control cell, just where are you planning to source that bonus inventory from?  Are you going to give the advertiser the most equitable sample of respondents, i.e. bonus inventory from home page impressions?  Or are you going to find the cheapest, hardest to sell bonus inventory and hand that over?  Obvious conclusion here but I’ll say it anyway, you’ll get the cheap stuff.  What this means is that while your test cell may have been recruited from home page visitors, your control could very well come from a niche section of the site.  Or more effectively illustrated, if you bought impressions and collected test cell sample from the NFL home page on a sports site, your control cell respondents could be coming from cycling or figure skating – not exactly the most favorable comparison.

Of course there are alternate methods of identifying control cell respondents.  First amongst them is to rely on a page node.  This is a javascript tag that lives on a website as opposed to in an ad server.  This gives you access to recruit from the premium inventory sections of a site without needing to be tied to a bonus ad impression.  InsightExpress deploys our own nodes, called iCompass, across a number of the comScore 250 sites, and Dynamic Logic has a similar system in their Safecount.net infrastructure.  While a node improves the comparability between test and control, it is by no means a comprehensive solution.

The other approach that one can take is to create a model to predict results for the control cell.  This is an approach that is commonly associated with comScore and their Smart Control methodology.  These models are often reverse frequency models that look at the impact of an ad campaign at a frequency of 1, 2, 3, 4, etc. and reverse forecasts that data to what an impact would be at a frequency of zero.  This novel approach wins kudos for being an innovative solution, but being a model it’s highly susceptible to errors.  Specifically, since these models forecast based on frequency, you need to be absolutely certain that two things are true:  frequency counts are accurate, and there are no differences in the site affinity across frequency buckets.

What this means is that if there is any cookie deletion present in the campaign you could have a number of people in your lower frequency buckets that actually had a higher number of exposures.  If your model assumes that someone had one exposure and in reality they had five exposures, you’re data is wrong.  When this happens you end up with garbage in, garbage out.  Ironically, the data gets even worse when you apply frequency caps (as clients often do).  With a frequency cap applied, viewers who don’t delete their cookies will only see the advertising as frequently as the cap allows.  However, those who delete cookies are unrestricted in terms of exposure and can end up seeing the ads more times than restricted by the frequency cap, and due to cookie deletion the server only ever counts them as a single exposure.  When frequency capping is employed on a campaign it is not unusual to see higher impacts in lower frequencies due to cookie deletion.  This certainly makes the data impossible to model.  To see how big of a deal cookie deletion can be in a campaign, check out my post on cookies.

Even more concerning is the error that’s introduced into these models when they’re applied at a site level.  Specifically, to understand how to forecast effect back to a frequency of zero (or the control cell), we need to understand the impact at various frequencies.  However, the fundamental audiences that make up site visitors can change dramatically as frequency increases.  If you think about it, this makes total sense.  A person who only goes to RoadAndTrack.com one time in the past month is very different than a person that goes to RoadAndTrack.com eight times in the past month.  The person who only goes once might be a random consumer following a link from a Google search for the best snow tires, while the person who goes 8 times a month might be a total gearhead and completely engaged in auto culture.  It’s natural that the heaviest consumers of a site are the folks who have the highest affinity with the site.  They’re also likely to interpret the advertising that runs on the site differently than those with the lowest frequencies.  Unless you can take into account the differences between these groups (which a model can’t), you’re reverse forecasting your data on an assumption that everyone is equal, and you get wrong data.  What’s concerning about the modeled approach is that there are no respondents, there is no easy way to refute the data, and often models produce positive results – exactly what everyone wants to see.

Most concerning about a frequency based model is that it assumes a linear relationship between no exposure and many exposures.  With these models, you’re inherently assuming that the relationship between one ad exposure and two ad exposures is similar to the relationship between 0 exposures and one exposure.  Of course this doesn’t make sense.  No model can predict the initial effectiveness of an advertisement.  Some ads might move brand metrics dramatically after the first exposure and some might move metrics only slightly.  The amount of that initial movement cannot be determined by a model, but only by empirical observation of the actual effect.  So if you’re looking for the truth you might want to try a different approach.

The final method of control cell collection that bears mentioning is our own patent pending Universal Control methodology.  Many people in this industry are heretical about promoting the use of random experiments.  This sounds good at first blush but it really misses the mark when it comes to what truly matters in this kind of research.  Sure, random assignment is great; I won’t dispute that fact. However, more importantly, studies need to be “blocked.”  Talk to anyone in the medical research field and they’ll tell you they run random block designs.  For some reason, most internet researchers forget the blocking part of the design.  Many of us here at InsightExpress believe that blocking is more important than random assignment (more on this in a later post).

For those of you who don’t know what blocking is, it’s a method deployed in experimental research to ensure/force equal representation across test and control cells.  If I’m studying the impact of an ad for a heart medication and recruit 1000 test and 1000 control, I need to ensure that there are an equal number people with heart disease in both the test and control cells. The incidence of heart disease is such that in a random sample of 1000 test and 1000 control I could end up with significantly more heart disease sufferers in either my test or control cell which would corrupt the efficacy of my design.  In fact, the same thing happens routinely for more mundane variables such as age or income.  Random designs are not a panacea, at least not without blocking or controlling for the audience.

As you can no doubt tell from this detail, here at InsightExpress we put a lot of faith into the practice of blocking, and our Universal Control approach does just that.  For every study we run, we’ve pre-assigned (just like in a true experimental design) control cell respondents for every one that’s exposed.  Each person exposed to a campaign has in effect a twin that serves as their control, and our testing has shown that this blocked approach produces much cleaner, and more comparable results between test and control cells.  We also love that this process runs off of our Ignite Network which means no more pop-ups, no more bonus inventory, higher response rates  and no more scrambling to find control respondents.  What’s even better is that, unlike the results of a model, our control cell contains actual respondent data that can be easily verified, and cross tabbed to understand sub-segments of an audience.

So, if you skipped the bulk of this post and jumped to the end here’s what I’d suggest you take away:

• Bonus inventory control cell collection is flawed,
• Nodes improve things, however are not universally available,
• Models are built on data that can create erroneous output,
• Blocked control cell collection most closely mirrors the true spirit of an experimental design.

Molly Elmore

Yesterday Molly Elmore, Vice President of Research here at InsightExpress (not to mention regular InsightfulAnalytics blog contributor) authored an article in MediaPosts’ Metrics Insider newsletter on a comparison of recall to Opportunity to See.  Like MediaPost, we think this is an important topic and hope you find the article of interest.

Below is an extended version of her piece, which includes several supporting charts unavailable in the Metrics Insider article.   If you have any questions about Molly’s analysis or would like to discuss her findings in more detail, she can be reached at melmore@insightexpress.com.

An Exploration of Advertising Effectiveness Methodologies: Comparing Recall to Opportunity to See

The Evolution of Advertising Effectiveness Research

Over the past decade, cross-media research has become increasingly important to advertisers.  Today’s marketers utilize multiple media channels to reach their target audience, and advertising research methodologies have also evolved to compare those channels on their ability to educate and persuade.  InsightExpress recently examined two popular methodologies to determine if and how their results differ.

Dueling Approaches

The first methodology is based on consumer recall of advertising. After a campaign launches, a respondent is asked a series of survey questions where one shows an advertisement that is part of the campaign being measured.  Those who recall seeing the ad are classified as “exposed” and those who do not are considered “unexposed.”  Comparing the two groups leads to a statistical determination of the effectiveness of the campaign at changing awareness and perceptions towards the brand.

The second methodology is based on the “opportunity to see” an ad  (OTS).  Here, respondents are asked about recent media consumption habits including specific TV shows, channels, and magazines.  In the digital world, cookie data is collected on advertisements sent to their browser.  This media consumption data is compared to the campaign’s media plan or a TV post buy report to determine who had the opportunity to see the advertisements.  Respondents who consume the “right” media but not at the specific time that the advertisements ran are assigned to the “unexposed” or “control” group to provide a baseline for comparison.  As with recall, the two groups are statistically compared to determine if any changes in attitudinal measures resulted from advertising exposure.

The Contest: Recall vs. Opportunity To See

To understand the benefits and limitations of these two approaches, we examined a random sampling of studies that contained the ad recall questions and the OTS questions.  They ranged in complexity and included a single site online advertising effectiveness study, a multi-site online advertising effectiveness study, and a multi-channel cross-media study.  Results were calculated using the OTS and recall methodologies, with data averaged across the various studies to determine how the results differed.

The data showed considerable differences between the two methodologies.  As seen below, the OTS aggregation revealed that, on average, the campaigns were moderately effective at boosting two of the exchange of information measures: awareness of advertising for the brand, and association between either messaging in the creative or sponsorship elements with the brand.

The recall-based approach led to very large increases in every measure, including the difficult to increase persuasion measures like favorability and intent.  But this was not surprising, since for Recall the exposed group just includes people who remembered having seen the advertisement.  Clearly, this approach would only be correct if 100% of the people exposed recalled that exposure.

The next step in our comparison was to understand the accuracy of each at assigning people to the correct group, (which can be difficult to do).  However, the digital channel tracking data provides information on which people were served the ads in the campaign allowing us to see if people correctly recall their exposures.

Two digital campaigns were used for this second part of the analysis.  The cookie tracking data was compared to respondents’ recall of the ads shown again within the survey.

Between 16% and 25% of those who were not shown the ads incorrectly recalled exposure to the campaign.  Only 52% to 69% of those who were shown the ads remembered the exposure.  Two campaigns may not provide enough data to definitely conclude what the false positive levels are across channels, but this analysis illustrates that people appear to have inaccurate memories for advertising recall.

While there may never be a perfect methodology when it comes to assigning people to cross-media exposure groups, using recall likely overstates the effect of campaigns considerably.  It may be tempting to use it because the results are likely going to be very positive, however, marketers and researchers looking for accurate findings should be cautious of the recall-based approach.

Recently, during several discussions about custom panels, I’ve been asked how I advise my clients to set up their panels for success.  Since there seems to be some misunderstanding about what panels are (and are not), I wanted to separate fact from fiction.

Myth #1:  A panel needs to be free-form communication to get the most out of it.

In my experience, I have found just the opposite to be true.  The most successful panels that I’ve been a part of have excelled precisely as a result of the structured nature of the engagement.  By mapping out a set of objectives, determining the proper mix of participants, identifying the ideal panel size and outlining panelist touch points well in advance, we were rewarded with a steady, but manageable stream of long-term, qualified insights.

Myth #2:  It isn’t a problem to ask the panelists recruited for qualitative research to participate in quantitative  research.

While this may seem like an efficient approach to conduct both quantitative and qualitative research, it won’t yield high quality data and it will do very little to generate goodwill among your panelists.  At InsightExpress, our InsightPanels approach manages each panel asset independently yet harnesses the unique benefits of exploratory qualitative interactions and high-quality quantitative research in a single, centrally managed, full service, yet affordable proprietary research solution.  By managing each panel asset individually but incorporating progressive (quant/qual) insights, we honor research best practices and offer a decade of quantitative experience in a closed loop, integrated learning environment.

Myth #3:  Honesty may not be the best policy when it comes to managing panelist expectations.

If you are serious about establishing a viable panel then you should definitely be upfront about what you expect from a panelist.  Successful panels require brutal honesty when it comes to setting commitments and communication frequency.  By establishing these expectations upfront, you won’t see high levels of attrition once you move full steam ahead which will save the all important time and money.

Myth #4:  A panel demands too much time from my already limited resources.

It can be argued that this myth has some truth to it since it’s hard to argue that panels don’t require a level of time commitment (especially during the planning stages). However, the time investment that’s required to establish a panel ties directly back to our earlier discussion about structure.  Panels that are purposefully designed to strike a delicate communication balance across a variety of engaging qualitative and quantitative touch points can reap informational dividends.   Moreover, with the support of a knowledgeable team on your side, any perceived burden is greatly reduced by the value and depth of the insights.

I would love to learn more about other panel myths that you may have come across.  Please post or send them to me directly at dkatz@insightexpress.com

Jerome, who holds the position of VP, Director, Data Sciences here at InsightExpress, was selected for recognition as a finalist under the “Innovation” category.  He was nominated by Marc Ryan, our EVP, Chief Research Officer, for his role in the creation of our patent pending Universal Control methodology for use in quasi-experimental design advertising effectiveness studies.

Universal Control is a new method for developing a control group for measuring advertising effectiveness with several key benefits.

• Universal Control is pre-calculated before actual ad exposure.
• This pre-calculated control group assignment is closer to the theoretical ideal experimental design and matches are available in real time at very high scale.
• Universal Control employs an intelligent twinning (multi-dimensional matching)process that improves stratification and the internal integrity of test and control group distributions

Clearly, the ARF – and Jerome’s colleagues at InsightExpress – are enthusiastic about these contributions to the industry.  There’s a lot more to lo learn more about Universal Control, so for further details please contact us through the InightfulAnalytics blog or at info@insightexpress.

Congratulations, Jerome!

Here’s a blurb from our press release, which you can read in full on the InsightExpress site.

“InsightExpress, a leading marketing research and data analytics firm, today announced that it has been commissioned by VivaKi to measure advertising effectiveness on tablet devices employing its Tablet AdInsights product as part of the most recent wave of The Pool, an ongoing research initiative that seeks industry alignment for engagement platforms of the future.  VivaKi is a division of Publicis Groupe and home to two of the world’s largest digital agency networks (Digitas and Razorfish) and two of the world’s largest media agency networks (Starcom Mediavest Group and ZenithOptimedia).

A first-of-its-kind initiative, The Pool was developed to examine the best ad models in various emerging media spaces by pooling the resources of clients and content providers to uncover human insights.  In October, The Pool launched its latest wave of research (or “lane”) focused on finding a scalable model that will allow advertisers to engage with consumers across multiple tablet devices, through both print and video content. This phase of the multi-year program unites publishers and technology companies including: ABC Television Network, Bonnier, Crackle, Digital Broadcasting Group (DBG), Microsoft Advertising, Mojiva, Rodale Inc., Scripps Networks Interactive, Tremor Video, USA TODAY, VINDICO, The Weather Channel and Yahoo!. Participating VivaKi advertisers include: Best Buy, Cadillac, The Coca-Cola Company, ConAgra Foods, General Mills, Goodyear, Samsung Telecommunications America, Walmart and others.

Tablet AdInsights is a pioneering tablet ad effectiveness research solution from InsightExpress.  Born from the company’s industry leading AdInsights® offering, Tablet AdInsights employs a proven test/control methodology to measure the brand impact that these hybrid devices have on campaigns for advertisers, agencies and publishers against metrics such as awareness, message association, favorability, consideration and intent….”

To learn more about the latest in tablet research from The Pool team, check out this post on their site.  Or, a more comprehensive overview of The Pool is available here.

Whether you’ve only read about the technique or consider yourself a diehard scanner ‘n scrammer, you’ll be interested in Kathryn Keogel’s piece, “Mobile Scan ‘N Scram: How Worried Should Retailers Be?” published in Wednesday’s Ad Age Digital.  We’re happy to have worked with Kathryn to contribute research on the reality behind the “scan ‘n scram” trend.  And despite what some industry experts see, you may be surprised (or not) to learn what kind of impact this behavior is actually having and who’s most likely to be scanning and scramming.

I understand this perspective, but the issue is certainly not so cut and dry.  However, before we get into more detail, let’s look at how the cookie is deployed on the average study.  The cookies used in online ad effectiveness studies operate in one of two basic ways: storage or identification.

• In the storage model, all of the information about what a viewer of the advertising has been exposed to is stored in a cookie on the viewer’s machine. This means that the record of ad exposures lives in a cookie on the browser, and when that cookie is deleted the information about ad exposure is lost forever.
• In the identification model, the cookie maintains a unique identifier and the ad exposure information is stored in a central database and keyed to the identifier in the cookie.  However, while the information about the ad exposure is maintained indefinitely in the database, if a viewer deletes the cookie containing the identifier then the linkage between the user and the database is lost. An example of the storage model is the approach used by Safecount, which stores ad exposure information in a cookie on your browser (http://www.safecount.net/yourdata.php).  On the other hand, InsightExpress employs an identification model where the cookie only stores a unique identifier. 

Knowing who was exposed to the campaign is obviously critically important because it tells us who to sample for our research study.  Yet it’s also important to know who wasn’t exposed to our campaign – those that don’t carry the cookie.  Obviously, this is where people get nervous.  If my friend Peter sees an ad for Bayer Asprin eight times, we can imagine the ad should have an extremely significant impact on Peter.  On the other hand, if he deletes that cookie, the next time we see Peter we’ll think he’s never seen the ad.  In fact, the next time we see Peter we might ask him to be part of our control cell as a respondent we think has never seen the advertising.  Clearly, he has seen the ad but since he deleted his cookies we no longer can associate Peter with exposure to the advertising. Moreover, if we include him in our control cell as a respondent, his opinion is likely going to taint the results since he’s supposed to represent people who are unexposed to the advertising.

While at face value this scenario may seem to destroy the purity of our experimental design, it’s important to understand that in the grand scheme of things cookie deletion  only has an impact on the assignment of test and control in some circumstances.  This is because cookie deletion only becomes an issue when a campaign has a high degree of reach amongst the measured audience.  This concept is easily explained using two examples.

Example 1: Low Reach Amongst Target

Let’s say I’m running a campaign on Yahoo! targeting a general audience.  In this example, I’m running 5.5 million impressions on Yahoo! with an average frequency of one.  Let’s assume that these impressions are run-of-site, so every one of Yahoo’s 145 million visitors has an equal chance of seeing the ad.  This means that 139.5 million visitors to the site won’t see the ad while the remaining 5.5 million will see the ad.

If I survey 10,000 unexposed people for my control cell, the likelihood of a cookie deleter showing up in my sample is extremely low.  Why?  Well, think about the population I’ll be drawing my control sample from: people on Yahoo! unexposed to my advertising.  There are 145 million of those individuals.  Because of cookie deletion I’ll think that number is actually higher.  Of the 5.5 million people that saw my ad, I can assume that 32% (the average cookie deletion rate) will delete their cookie, and as you will recall, once that cookie is deleted I’ll have no idea if someone has seen my ad.  So 32% of 5.5 million is 1.76 million which you can add to the 139.5 million that didn’t see the ad giving you the total size of the audience that I think did not see the ad.

Right away you’ll notice something important: our deleters are a very small portion of our total unexposed audience and, in fact, they represent just 1.3% of the total unexposed audience.  If true, then I can expect that of the 10,000 people I sample for my control cell 1.3% – or 130 of them – will have actually seen the ad but subsequently deleted their cookies.   Those 130 are the people I need to worry about because since they have seen the ad they’ll likely have a higher opinion of the brand, thus polluting the purity of my control cell.

Here’s the big question: what impact did the deleters have on the results?  Let’s assume the ad had a positive impact which resulted in 31% of people exposed to the ad wanting to buy the advertised product.  This is in contrast to only 25% of people not exposed to the ad who want to buy the product.  So if I split my 10,000 respondents into two groups – the 9,870 that didn’t see the ad and the remaining 130 who did – I can simply take a weighted average of these two numbers to see what influence the deleters had on my results.

In this omniscient example we know the truth to be that 25% of people who didn’t see the ad wanted to purchase the product.  Now a weighted average including the people who did see the ad but deleted their cookies results in a purchase intent score for my control group of 25.07%.  Barely an impact!  The odds of a lot of cookie deleters showing up in my sample are just too low to worry about.

Example 2: High Reach Amongst Target

If I take the same campaign as above and change my target market to Hispanic moms between 25 and 34, there may be only 7,000,000 of those individuals on Yahoo!  So my 5.5 million impressions could in fact reach a much higher percentage of the target market.  For argument sake, let’s say we reached 60% of the target audience, or 4.2 million target viewers.  In this example, only 2.8 million of the total target didn’t see our advertising.  But don’t forget that on top of that 2.8 million we need to add in the people we think didn’t see our advertising (the cookie deleters).  If 32% of the people that saw the ad delete their cookie there are another 1.3 million viewers of the campaign that I would think didn’t see the advertising.  So I’ll think that the total number of people that didn’t see the advertising is 4.1 million (2.8M who didn’t see the ad + 1.3M who deleted their cookies).  When I sample from this group I’ll inadvertently include in my survey sample a large number of people who did in fact see the advertising.  I know that approximately 32% of my control sample did in fact see the tested advertising.  That number stands in stark contrast to the measly 1.3% in the previous example.

By running through the same impact calculations, we learn that instead of the true purchase intent of 25% for this control cell, those cookie deleters will artificially inflate our control up to almost 27%.  That’s enough of a difference in my data to adversely impact the results of my research.

The Reach Variable

As you can see, when the reach of an ad campaign is relatively low amongst the research audience the odds of someone who deletes their cookie being included in the sample for control is ridiculously low.  And even if we accidentally included them in our analysis, the impact that those people have on the overall results is incidental.

On the other hand, a high reach campaign can be very adversely impacted by cookie deletion.  The truth is that the high reach audience is more common than the low reach audience.  It’s often the case that the structure of the analysis increases the reach amongst audiences we care about. This happens because we don’t really measure Yahoo! in aggregate with the research we conduct, we often measure defined audiences on small sections of sites (e.g. Yahoo! Autos).

As these examples illustrate, the impact that cookie deletion has on the measurement of your campaign is entirely dependent on your special circumstances.  If you’re doing a heavily targeted measurement or if you’re running a heavy reach campaign, you’re bound to encounter problems with cookie deletion polluting the results from your control cell.

Misattribution of Frequency

But let’s say that you’re running the ideal campaign (if one were to exist) where cookie deletion was not an issue.  Are you still safe from its effects?  Well, when it comes to misattribution of an exposed respondent into the control cell, I’d say yes you’re in the clear…with one big caveat.

No matter how big or small your campaign might be, your understanding of the impact of frequency on ad effectiveness is entirely flawed as a result of cookie deletion.  Our numbers show that, for an average campaign, approximately 50% of the cookies that are assigned to a frequency of one have indeed seen the ad multiple times.  That’s a massive misattribution of frequency.  This point is crucial, especially when trying to understand the impact of frequency on campaign metrics.  Cookie deletion within your exposed audience is resulting in data that suggests viewers see the advertising significantly fewer times than they actually did. 

When you think about it, this makes absolute sense.  Take the example above with my friend Peter.  We know he saw the tested advertisement eight times before he deleted his cookies.  The next time he sees the advertising we’ll register him as seeing the ad once.  Why?  Because as soon as he deleted the original cookie he became someone we thought had never seen the ad, so his next exposure has to be at a frequency of one.  Taking this example a step further, if Peter saw the ad eight times and deleted the cookie after every time he saw the advertising we’d think he’s eight different people who saw the ad once.  Each deletion turns him into a new viewer of the ad with no way of recreating that history.

Mending an Achilles Heel

So is cookie deletion the Achilles Heel of the ad measurement industry?  In short yes.  But the long answer is that cookie deletion is not necessarily a problem when your campaign has a low target audience reach.  However, if you plan to measure the impact of frequency (and all of our clients do), cookie deletion makes that analysis almost irrelevant.

Is there any good news here?  Well, of course there is. At InsightExpress we firmly believe in making sure you’re adjusting for cookie deletion,.  In fact we take it a step further and adjust not only for deletion but also for the fact that a viewer might be seeing your ads in multiple browsers or on multiple devices.  We employ a patent pending approach called a universal ID that doesn’t just apply an aggregate cookie deletion fudge factor as many competitors do, but specifically identifies when people are deleting and reassigns their exposures in our data warehouse.  This ensures that when our analysts are looking at data about exposure to an ad campaign they will know that everyone has the correct exposure assignment.   

This post is part of Marc’s series on “How to Evaluate and Buy Online Ad Effectiveness Research.”  To read other posts in this series, click here.

The promotion was featured on creative package elements for all three Coke Brands: Coke, Diet Coke® and Coke Zero®.  For Diet Coke and Coke Zero, the promotion was an enhancement of basic package design, featuring oversized flakes on cans and some labeling that drew attention to the campaign.

The most dramatic change came in the packaging elements that wrapped the signature Coca–Cola® product.  For the majority of shelf-facing packaging, the major change was the replacement of the Coke “wave” with a trio of polar bears on snow drifts that approximated the “wave.”

Arctic Home can

But the real surprise was for those consumers who purchased the 12 and 24 packs of 12 oz cans.  Inside the red boxes were (of all things!) white cans.  The cans were predominately white, with silver bears (it’s Christmas time in the city!) and Coca–Cola in red.  The logic seemed to be to let people purchase the product as they normally would, then when they got home and found the white cans, it would cause a double take drawing attention to the Arctic Home promotion.

Best laid plans went astray when Coke began receiving complaints from Diet Coke consumers who were purchasing six packs or individual cans and making the wrong selection based on the updated package design.  It got stranger when the company started receiving complaints from people who thought the Coke in the white cans tasted differently, or that they disliked the new formulation or variety.

As a response, Coke has since decided to stop producing the white cans, and replace them with red cans with white polar bears to match the scheme on the two liter and 20 oz bottles.  A special FAQ page was set up on the Coke website to help people diagnose whether they had regular or Diet Coke sitting in their fridge.  News outlets picked up on the confusion and the rest is history.

Frankly, I have no idea what Coke did or didn’t do to vet the impact of this package change and the perceptions people would have about the white can before it went to market.  But I wondered: Could this outcome have been prevented?

To answer this question, we decided to test the Arctic Home packaging with InsightExpress’ PackageImpact package screening and testing solution to see what we could find out.  We tested the standard 12 pack case to the Arctic Home 12 pack case, each in separate monadic cells.  We also fielded cells for traditional cans vs the Arctic Home cans.  After screening the sample to exclude those aware of the packaging fuss (~36%), we honed in on some top level findings:

• Brand Recall (confusion) was lower for the Arctic Home case compared to the traditional case,  where recall was nearly perfect.  This recall gap widened for the cans themselves.  Nearly 25% of respondents misidentified the brand on the Arctic Home can.  And 17% of respondents thought the Arctic Home can was packaging for Diet Coke.
• Top box purchase intent scores dropped dramatically relative to traditional packaging for both the Arctic Home case and cans.
• We also looked at the Value, Overall Appeal and Package Liking metrics.  For these, the Arctic Home case obtained a marginal increase, while the Arctic Home can saw marginal decreases relative to traditional packaging.
• Not surprisingly, the Arctic Home packaging dramatically moved the needle on the New and Different metric compared to standard packaging.
• In assessing fit for the Coke brand, the Arctic Home packaging lagged the traditional packaging creative by substantial margins.
• In looking at the ImagePaint heat map results, people liked the polar bears and the program.  However, overall engagement with the package was down.  On average, 28% of the surface area was highlighted positively for the traditional 12 pack package whereas only 16% of the surface area was highlighted positively on the Arctic Home 12 pack sleeve.

PackageImpact: ImagePaint heat map results for Arctic Home can

So what does it all mean?  Overall, people liked the Arctic Home program and they loved the polar bears (aren’t they cuddly?).  But stepping out on the color of the Coke cans created confusion for the Diet Coke drinker and was not considered a good fit for the Coke brand generally.

So given  these findings, why did Coke choose to make the bold decision to color their iconic red cans white?  Perhaps the answer is that they didn’t have this wealth of information.  Established methodologies for package testing typically involve focus groups and in person eye tracking.  Both methodologies have their place, but they don’t cover the spectrum of information needs vital to making the right packaging decisions.

It is easy to envision the focus group where a snow white can with polar bears is unveiled and the group enthusiastically greets the design with enthusiasm based on the creativity and the appeal of those lovable bears.  Even within an eye tracking experience researchers are likely to discover eye focus shifting to the “likable” portions of the can.

But in neither situation is it likely that key – vital – questions were asked such as “What brand do you think this is?” or “Is this design a good fit for this brand?”  These may not be the most important metrics in the equation, but they must be considered as minimum expectation items.  If a package can reliably establish who the brand is, then it has free reign to chase the ooh’s and ahh’s that might move the needle overall.

Package testing and screening requires a methodology that can capture a wide range of intelligence across a variety of creative executions.  Does it pop on the shelf?  Does it convey the correct brand and brand message?  Will people perceive value and purchase it?  Does it have creative appeal?  Does it create emotional touch points?  All information that needs to be taken into account before decisions are made.

Quantitative research collected with statistically reliable samples for a broad range of measures should be a part of every package testing program, but rarely is that the case.  If including this kind of information seems important to you, give us a call.

And yeah, I liked the polar bears.