Yet it is rarely followed.

We almost always present data.

Actually we don’t present data, we send out reports. With data. Lots of it. With 6 size font and some pies and stacked bar graphs thrown in.

Then we are frustrated that no one seems to pat us on the back, sing songs in our glory, give us more money.

We don’t truly tell stories because it seems like a lot of work. And it can be. But you’ll be surprised at how often it is simply a matter of framing things differently, letting your imagination roam free.

Last month I had to present to a group of executives in New York. One of the key things I wanted to communicate was the power of not doing random advertising but rather using freely available data to target the advertising on sites where relevant audiences exist.

Goals Summary:

Below is how I did it. . . . hopefully it will inspire you to look for stories in your data, stories that will hold interest and might even get you some smiles (and you know that a raise is not far behind!).

My first step was to try and tap into current events / pop culture. That calls for some research. I use Google Insights for Search as the best way to get a pulse on what people find interesting.

Specifically what I often do is run this query: Who are the most popular celebrities in New York in the last 30 days?

Turns out it is someone called Kim Kardashian. It also turns out I have no idea who this person is, an unfortunate side effect of not have time to watch television.

Quick Google search and I am caught up on why Ms. Kardashian is “famous”. She has some overlap with Paris Hilton in terms of the path to fame.

The key ingredient for any story is to have interesting protagonists. For this story due to their popularity it will be Ms. Hilton and Ms. Kardashian.

The plot: Your business has a need to market something related to Ms. Hilton and Ms. Kardashian, a perfume or a clothing line or a cd/dvd. Amongst other things you’ll want to make use of display advertising (banners / widgets etc).

How do you figure out who the right audience is, and where you’ll find them? As opposed to of course buying the main banner spot on www.yahoo.com were your ad might be a hit or a miss.

Tools for doing audience segmentation were quite expensive until recently. Google’s Ad Planner is free and makes this valuable data democratic. You can segment by demographic (age, education, income, gender etc) and psychographic (Extreme Sports Fan, Film Buffs, Fantasy/Comic Book Readers etc) data.

Perhaps its most cool feature is the marriage between all the above data with Google’s search data.

That’s where the analysis starts.

Question: What are the websites that are visited by people who have searched for the keywords “paris hilton” and “kim kardashian”?

Here’s the answer:

[Click on the image for a higher resolution version.]

Notice the I have typed the keywords on the bottom left. In the right frame are the sites that are visited by those who searched for those two terms. Some obvious sites, many surprises (good thing, now we know!).

I have a habit of sorting by Comp Index, just to check out concentration of the audience. For example a comp index of 990 means that you are approximately nine time as likely to find the same audience (paris, kim searchers) on wallpaperbase.com.

If you look at the higher resolution version (click on the image) you’ll easily find out how many page views are on the target site, what kind of advertising they accept, ad impressions/day and other data you need to create a media plan.

So far so good.

I have always believed that Men are more interested in the kinds of stories and “entertainment” value that Ms. Hilton and Ms. Kardashian generate.

The nice thing is I can validate that hypothesis. I simply open the Gender option in the left panel and choose Male.

You are looking at the top part of the segmentation panel. Notice the delta between UV (users) between the overall segment and just the Males.

Turns out I was not totally right. Males make up a bit less than half of the audience.

No worries. They are still a lot bigger than what many people think (and it is wrong to think it is overwhelmingly female).

My next believe, perhaps controversial, is that older males are more interested in Ms. Hilton and Ms. Kardashian than younger males. Now this seems odd because Ms. Hilton and Ms. Kardashian seem to be more cool and hip and more of a young generation cup of tea.

Well we can test my hypothesis, in addition to Gender I can also choose Age. . .

This data is still just for people, in this case Males, who searched for the key words paris hilton and kim kardashian.

It might have been a odd thing to say but it seems that 45 and older males are a lot more interested in Ms. Hilton and Ms. Kardashian. By almost two to one.

Surprised?

: )

Let’s prep for the punch line of this story.

I have identified a audience that is of value to my goal, marketing Ms. Hilton and Ms. Kardashian (or things connected to them).

I want to target the top end of this audience, Males 55 and older, how many of them are there and where can I find them (to ensure my advertising will be relevant for this audience and my ad dollars are spent wisely)?

Here you go. . .

[Please click on the image for a higher resolution version.]

How about now… surprised?

I was.

The top sites listed for this audience (older Males interested in Ms. Hilton and Ms. Kardashian) turns out to be bedrock sites, typically, for Republicans and the Conservative movement! Starting with a Comp Index of 1700 for impactguns.com. Other sites: weeklystandard.com, rushlimbaugh.com, nationalreview.com, worldnetdaily.com, and townhall.com.

Not in my wildest dreams would have I have expected that this audience would be so highly correlated with actual searches done for Ms. Hilton and Ms. Kardashian. It seems odd with the conservative moral values espoused.

Very Important: I am not judging them. To each unto his / her own.

For my marketing campaign one more valuable nugget of insight is in th above data (click above for higher resolution). Turns out they are also very rich. Note the prominent appearance of morningstar.com, pgatour.com, seekingalpha.com and ft.com.

So a bumper crop: right audience, lots of money to spend. That’s hot!

Now I have to go execute the campaign and I know where to target my ads, how many impressions/day I can expect and how many people I can hope to target.

Relevant audiences change with seasons, hot trends, shifting preferences. Repeat the analysis to ensure you have the most current data.

End of story.

Closing Thoughts:

Turns out this was a very effective story to tell, most people in the room were media buyers (especially offline).

They were impressed with the kind of data we have online, and how easily accessible it was.

They will never forget how wrong one can be about who the relevant audience might be (it would be impossible to guess the Weekly Standard, Rush Limbaugh audience might have any interest in Ms. Hilton or Ms. Kardashian).

Data Wins.

Ok its your turn now.

When you present data how do you tell your stories? How easy or hard is it? Got a favorite story to share with us?

What did you think of the above story? Methodology or conclusions? What did you link? What did I miss?

I would love to hear from you. Thanks much.

PS:
Couple other related posts you might find interesting:

• Make Your Web Analysis / Reports “Connectable”

• Competitive Intelligence Analysis: Google Ad Planner

• Competitive Intelligence Analysis: Google Insights for Search

Copyright © 2006-2009 Occam's Razor by Avinash Kaushik

Paris Hilton, Kim Kardashian & Telling Stories With Data

At that moment I think my reaction was something like “meh!”. Of course Craig was right.

Search Engine Marketing (Organic + PPC) continues to be a huge part of any company’s acquisition strategy on the web. Like other online channels it is targeted, it is effective and it is accountable. Perhaps its most unique asset being the ability to hyper-target relevant customer intent.

It would be fair to say that Paid Search it has also gotten very complex (Search Long Tail anyone?).

Your campaigns and ads are impacted by the many shiny buttons and pretty dials provided by search engines, complex algorithms that determine if your ad shows up (or not), and a bunch of cool things that search engines are doing (like Universal Search).

Given that context measuring impressions and clicks and click thru rates (CTR) and cost per click (CPC) and conversions are now merely price of entry, in fact you focus on those in the first couple days and then very quickly have to elevate your game.

Yet with our tools like Omniture or WebTrends or Google Analytics etc that is kind of all we end up focusing on as Analytics Professionals. Partly because of the limitations of the data available in the tools, partly because most Web Analysts don’t have the required deep understanding of what Paid Search is all about.

In this post I wanted to share five cool, “non-normal”, analyses that you can do to get a much better understanding of your Paid Search performance. The reports are all inspired from analytical principles I have advocated for quite some time on this blog.

All the analyses been created using the ClickEquations PPC Platform’s ability to do impressive deep analysis with its ClickEquations Analyst tool. [Disclosure: I am on the ClickEquations board of advisers.]

Here’s how I think of CQ Analyst: Take one part data from all search engines, one part unique custom metrics, one part custom database front end integrated into excel and the resulting ménage à trois combines to produce incredibly powerful insights.

Here are five that we’ll cover here:

Let’s go rock it!

#1: Identify Keyword “Arbitrage” Opportunities.

Most web analytics tools dutifully report on search engine data, which ones sending how many visits and what not. They also have a report for keywords. And surely by now you know that you can drill down from the search engine report to look at keywords for that search engine.

Now here is the reality : The ranking of search engines by traffic to your site is fairly well settled. Has been for a long time. It does not change month over month, or changes very little.

What does exist in your data is a different how one keyword performs across search engines! Thanks to the difference in algorithms for both organic and paid search in each engine.

Lookie here, my ClickTracks report….

See what I mean? The heatmap shows you how even my humble blog is optimized for different keywords in different engines. This allows me to look for “arbitrage” opportunities (focus on selective keywords for each search engine).

You can do this exactly same thing for your paid search keywords with ClickEquations Analyst, i.e. look at side-by-side performance rankings by clicks or conversions.

In this example, we see that the keyword dog boots is generating huge traffic on Google (1,309 clicks) but almost nothing on MSN or Yahoo. Why? Are we not buying this delightful keyword on those engines? Is our bid too low or our ad copy ineffective?

Questions you want to answer quickly before your once a year annual huge sale is over and you are left with tons of extra dog boots (heaven forbid!).

Sorting by clicks-per-keyword in another engine, Yahoo in this case, we see that our top traffic terms there are ones where Google is under-performing.

More opportunity!

Let’s bring this puppy home, switch to conversions for the same keywords across different search engines. . . .

It is pretty clear now where immediate opportunities exist (in case you did not notice, the frowny faces : ) and where you are doing ok.

In our case the keyword that is our second highest performer in terms of bringing home the bacon (conversions!) is doing great on Google but is we are getting eggs on AdCenter (or is it Bing now?) and YSM.

It highlights a possible Sales / Revenue opportunity and gives you marching orders to investigate. Remember you already know that this is working on Google, why not on the other engines?

#2: Rock Your World, Focus on “What’s Changed”.

While we obsess about our brand terms and our top ten key phrases the reality is that the long tail of search means that our organic and search campaigns focus on tens of thousands or hundreds of thousands of keywords.

One effective strategy to deal with this purely data problem is to focus on what’s changed.

No more data pukes. Just looking at things that need attention.

[imagine whip being cracked, politely]
You must do this to stand a decent chance of making ROI on your Search Campaigns.
[/imagine whip being cracked, politely]

The focus of your What’s Changed reports is to show campaigns, ad groups, keywords that are gaining more impressions, getting more clicks, producing more revenue (or not!) compared to a prior relevant time period.

Here’s the report:

[Click on the image for a higher resolution version.]

Here’s your schedule: Wake up. Take shower. Review major “rises” and “drops” in Gross Profit. Press buttons. Take action. Get coffee. Drive to work to receive praise for a job well done!

Notice the focus is not on Clicks or Visits (not that there’s anything wrong with that), its on money baby!

Here’s one more very useful example, focusing on Avg CPC’s, as you figure out what dials to move. . .

Every line in these reports begs a simple question: Why?

What is causing that ad-group’s revenue to soar 498% above what it did last month?

How come that keyword is down 74% in the number of impressions yesterday vs the day before?

Why did the cost per click for doghouse pets boulder suddenly shift that much?

The biggest problem with paid search, or web, analytics is that you don’t have starting points. These reports, and metrics, give you that. You can then go investigate the hot leads. Perhaps your competitors have sprung into action. Maybe your quality score has taken a dive. Who knows, the boys in the warehouse may have run out of stock.

The nice thing about the ClickEquations Analyst is that you can easily use the delta feature to set the am out of change between two time periods for any paid search metric.

You can create customized Top X modules of what’s changed and literally you have a living breathing dashboard that highlights the most important, and it never gets auto deleted by the HiPPO’s!

#3: Analyze Visual Impression Share, Compute Lost Revenue.

Did you read my post about Google’s Search Based Keyword Tool ?

One of the really cool things that SbKT offers is Ad/Search Share (8th picture in that post). It tells you how often your ad shows up when someone searches using a specific search query (for both paid and organic).

It is very useful in understanding what share of shelf you have (think shampoos in walmart). It is surprising that only the most hard core PPC folks seem to focus on this metric.

You can also get Impression share for your own keywords portfolio from the Adwords Campaign reports. You want to know how many people who are searching for a keyword that you are bidding on are not seeing your text ads (no see = no click = no honey!).

The CQ Analyst helps you visualize the impression share metric very efficiently, even your HiPPO will understand the story!

The impression share, Exact, report showing “share of voice” for queries matching your keywords. . . .

[Click on the image for a higher resolution version.]

Orange = Yea! Green = Ouch!

Need more motivation to work harder to improve your impression share? Here you go, missed clicks and revenue. . .

[Click on the image for a higher resolution version.]

Orange: Cause (ouch!). Green: Effect (ouch! ouch!).

The ClickEquations Analyst report is showing above the lost revenue in green, and the orange line shows why that is lost revenue (lost impressions).

It takes the average revenue per click from the impressions you are getting from your campaigns (so the real conversion data from your site) and extrapolating it out to show how much revenue the lost impressions represent.

In the above real customer example the lost impressions alone (without doing anything else) represents revenue growth opportunity of 30%.

Remember this is using your actual current clicks from ads when you do show up (impressions) and your actual current conversions.

Winning those missing impressions would require either a budget increase, or more likely some significant improvement in bids or quality score – but knowing that potential exists offers a clearer view of specific expansion opportunity than paid searcher usually see.

This is a superior understanding of the opportunity then what you’ll get out of standard web analytics reports. Superior because you are truly bringing deep AdWords data together with your site’s outcomes data.

#4: Embrace the ROI Distribution Report [Identify: Lovers, Friends, Losers].

If you are a Analysis Ninja prepare yourself for a minor orgasm. This is so cool.

Remember the 80/20 rule?

80% of ROI comes from 20% of your Campaigns. Replace ROI with revenue, profit, your favorite metric.

Step One: Specify your ROI goal and your minimum acceptable ROI.

Step Two: Run Report. :)

The report tells you how many of your campaigns, adgroups and keywords fall into three performance bands: Great (exceeds expectations), Good (meets expectations), Poor (sucks!). Or: Lovers, Friends, Losers.

[Click on the image for a higher resolution version.]

You’ll see the raw numbers, the cost vs revenue breakdowns, and the comparative percentage contributions.

So what’s happening here?

8% of the AdGroups are responsible for 54% of the revenue (!). Checkout the Avg ROI contrast.

Or more visual. . .

Its not even 80/20, more like 90/10!

It’s a ruthless way to determine which elements are hurting you, and which are actually helping.

This type of analysis allows you to take a very close / critical look at exactly how many of your keywords are really profitable, and which ones are sucking wind.

It should lead to a complete reevaluation of your keyword selections, match type settings, bid choices, ad copy, and campaign organization. Or in other words, the baby and the bath water.

This analysis is super cool because you end up with a prioritized to-do list of campaigns or keywords that require your full attention.

#5: Zero in on the Actual User Search Query (and Match Type).

While the entire business of Paid Search revolves around keywords, often there is not enough attention being paid to the role Match Types play in determining which search queries (the words the search engine user actually types) trigger your paid search ad.

We sadly make far too many decisions based on keywords we bid on.

In our campaigns we use Broad, Phrase and Exact match types to ensure we are showing up for relevant user searches. Hence most of our campaigns include key words which use all or some of these Match Types.

One of the simplest report you can create using the ClickEquations analytics platform helps you analyze the impact of these different match types on performance. . . search query by key word by match type. . . .

Column 1 is what was typed into the search engine. Column 2 is the keyword that was in your paid search campaign. Column 3 was the Match Type used.

And here is a report that will really solidify the Match Type concept in your mind: The match type report is for the keyword wellness cat food and the actual user search query. . . .

Impressed at the creativity of users who search?

The analysis you should do right away is to understand the performance of individual match types with the user search query for a clump of specific keywords you have bid on.

This analysis will instantly spark ideas for:

If you pause and think about it for a moment this search query report is perhaps the ultimate keyword research tool you could get your hands on. And it is free! Use it!!

The Analysis Ninjas amongst you won’t stop there.

No sirree, bob!

You will want to understand performance of various Match Types across your entire account! You’ll take three steps up and try to understand the forest view first.

A typical expectation would be that Exact Match keywords, which are very precise targets carefully chosen, will perform better than Precise and certainly Broad Match.

So is it?

The first report in ClickEquations Reporting gives you the top level analysis of your Match Type distributions.

It answers fundamental questions such as: What is the result of all your hard work in finding and promoting keywords to Exact Match? What about your plan to conquer the earth and moon and pluto to increase the % of revenue you get through Broad Match?

Here you go. . . . Happy Fathers Day. . . .

[Click on the image for a higher resolution version.]

In one nice view (come on you can’t argue with a pie and a bar chart on the same page!) the information you need at a glance to understand performance, all the way down to Revenue / KW.

Let’s drill down on that to look at it more clearly. . . .

Revenue Per Keyword from Exact Match is much higher than Broad and much much higher than Phrase. That validates our hypothesis and it provides nice support for all the effort we have put into creating highly targeted Exact Match keywords

Do even more of that, while you shift to leveraging Phrase and Broad more strategically.

My favorite view is perhaps to look at the bang for the buck of keywords based on match type. Outcomes sweetie, outcomes!!

The complete picture, Cost -> Revenue -> Gross Profit -> Net Profit by each match type. . . .

Niiiice?

For this specific client it turns out that every dollar spent on Broad Match is yielding a much lower return when compared to Phrase or Exact Match types.

Immediate work needs to be done on negative keyword expansions to further optimize spend.

Each company is unique, your picture might look like this. . . .

By focusing on Cost in addition to the four layers of Outcomes (revenue, gross profit, net profit) you are able to see a multi dimensional picture.

Digging deeper, as in the first part of recommendation #5, will help you then make changes to your match types and campaigns to optimally utilize the opportunity in front of you.

Happy?

Five sweet recommendations you can follow tomorrow, or today!

Paid Search has changed dramatically over the last few years. Yet we tend to report and analyze like we did during the days when hotmail was cool (!).

The stories we have shared today show that taking a different approach to the data already available gives you super awesome insights that lead to immediate cost savings / revenue improvements.

And it is not that hard is it? You need a bid management software like ClickEquations with built in advanced analytical capabilities, you need some awareness of AdWords / YSM / AdCenter and you are on your way.

Ok now your turn.

What do you think overall of this approach? Have you done any of the above five types of analyses? Got any other favorites that you would care to share? What do you use to do your Paid Search Campaigns analysis? Any tips or horror stories you want to share?

I would love to hear from you. Thanks.

PS:
Couple other related posts you might find interesting:

• Paid Search Analytics: Measuring Value of “Upper Funnel” Keywords
• Competitive Intelligence Analysis: Google Insights for Search
• How Thick is Your Head and How Long is Your Tail?
• Excellent Analytics Tip#3: Turbocharge Your SEM/PPC Analysis

Copyright © 2006-2009 Occam's Razor by Avinash Kaushik

PPC / SEM Analytics: 5 Actionable Tips To Improve ROI

1] Turns out the readers of Occam’s Razor are exceptionally gifted, they understand the challenges of web data

2] They are deeply motivated to do something about it, just not totally sure what.

This is a special unplanned post just for you, to help with issue #2.

My last post, Web Data Quality: A Six Step Process To Evolve Your Mental Model , unleashed an unusually exceptional set of comments from you all (sweet!). Today I want to share a cogent set of “next steps” ideas.

Practical strategies to deal with the problems you highlighted, nuances you can exploit, things you need to give up on, things you might consider doing more, bosses you need to ditch (!!).

Here’s my core premise:

You there? Then let’s go rock this thing.

My recommendations, mostly in the order of importance:

Curious?

The rubber meets the road now. . . .

#1: Give up. Pick a different boss.

Did you think I was kidding?

There is a entire generation of leaders in place today that don’t get it. Many of them, sadly, will never get it. I don’t blame them. They have seen the world in one way and they can’t change now.

We simply have to wait that generation out. For now we have to wait for them to get promoted / take on other life challenges.

When I have found myself in situations where there is just no chance of movement in the HiPPO’s mental model, I try to switch bosses.

Life is too short. There is too much money to be made. There are too many customers to be satisfied. Why waste your time?

If you can move on.

Find someone who is open to accepting the new data quality mental model. Someone who will take actionable recommendations and action them (even if they agree to just try one or two things first, perfectly ok).

One I have that small opening I work really really hard to make my new boss a hero. When I have done my job well the impact of that is huge. For me, for the boss and in turn on the company.

I realize that if you work at a small company this is a non-choice, you have one boss and She’s all your company’s got. In that case try to see if any of the things recommended below work.

Meanwhile remember to polish up your resume so you can find a better place of employment in case it simply does not work out. [The economic climate is bad right now, but it won't always be that way.]

#2: Educate them about the “perfect” source they love.

[Important: I am NOT saying that it is every a good idea to say: "look I am better because your favorite child is not perfect either!".]

More than once I have gotten a more open mind after I detailed to my executives the (irrational) faith they put in other sources of (what they don’t know are) imperfect data.

Take TV as an example (a fav of med-large companies, sorry not small ones).

Nielsen uses a few thousand people (between 18k – 30k) to measure the viewing habits of 200 million plus Americans. I am sorry but in this world fragmented consumption (tons of choice) it does not matter how much sophisticated math you put on that data to account for anomalies, you are left with high grade non representative “data”.

Consider this: Even Big 3 network CEO’s have been forced to put back shows that they canceled because of “low” Nielsen ratings only to be astonished by massive fan rebellions or huge DVD sales.

Just imagine what happens to a 18 to 30k dataset’s capacity to measure the non-major networks or the really long tail. CurrentTV anyone? : )

Yet those ratings and GRP’s are taken as God’s own word.

Even with 30% inaccuracy and the third-party sub optimal Omniture’s 2o7.net cookie your web analytics data is better than that.

Or here’s another one. Try to really understand the impact of a 180k panel data set from ComScore that monitors a couple hundred million Americans (in a even longer tail and more fragmented than TV world of the web). Contrast that with data that comes from HitWise (15 mil). Or is in the Google AdPlanner. Both substantially better (for similar data).

Yet the former is accepted as the truth. The latter are not. Because your HiPPO does not know any better.

Start a revolution:
1) Solve the major problem: Educate yourself. This is often the key flaw.

2) Present a dispassionate and non-personal education of each data source and its value.

3) Highlight how Web Data is less imperfect (if that is what you find) and how it provides more information (missing in other sources).

4) Ask for implementation of actionable insights (small at first) from web data.

[Big PS: Here's what I am not saying: I am not saying Nielsen (or ComScore) is not trying hard enough. I am not saying they are not applying the best mathematical algorithms Humanity has created. The problem is not either one of those issues. It is the core data they collect and how much of it. No amount of pretty Math can now accommodate for the new world order of content consumption on TV in their old word data set.]

#3: Distract your HiPPO’s from data quality by giving them actionable insights.

Dazzle them with your intelligence!

Like you distract a baby by jingling your key chain.

This is what I am talking about:

Change the focus from silly unactionable aggregated numbers like Visits or Avg Page Views Per Visitor etc. Instead you can find key sources of traffic. You can run controlled experiments to measure offline impact. You can figure out how to get existing website customers to buy more or more frequently or abandon carts less.

Because your Senior Management does not know what the heck to do with total Visitors and the caveats associated with Unique Visitors they send you back to the data quality torture chamber. If you can distract them by giving them interesting insights they’ll focus on the value.

[I have the privilege and the good luck to speak to lots of C-level folks at conferences or 1:1 meetings. I want you to know that I never lose an opportunity to educate them about the data quality issue and why they MUST look past it and focus on taking action. I am doing this every day. Every week. Every month. My tiny contribution to the Cause.]

#4: Dirty Little Secret One: “Head” data can be actionable in the first week/month.

I don’t know why many people wait for 18 months to implement Omniture completely. Or WebTrends. Or NedStat. Ok ok ok, or even Google Analytics! : )

Yes the implementation has to be “complete” (translation: never going to happen). But there are things that are “big enough” (head) in the first week and getting complete data for them is irrelevant because it won’t change your decision / insight.

Some of your data is good enough very quickly (dare I say even if not all your pages are tagged or you are still using third party cookies or have minor implementation issues).

You job during Week One is to look for the “head data”, places with big numbers / happenings.

Say your imperfect data shows that 60% of your traffic comes from Google and the keywords “Avinash rocks”, “Michelle is awesome” and “HiPPO’s stink” account for 40% of that traffic.

You can start taking SEO / PPC action right away because marginal improvements in those big numbers won’t really change what you do.

Or say you find, surprisingly, www.nytimes.com is sending you huge traffic to a part of your website that is related to porn (what!). You can start moving on that now.

Or the bounce rate on your home page is 65% (kill me now!).

Some things you don’t want to know with full confidence before you start moving.

I recommend your web analytics approach have a more nuanced approach.

Tell your boss: “We have to start moving on these things because the numbers are large enough and they indicate we need to monetize opportunity x / we need to fix problem y. But as to how many people look at your bio on our website, I am afraid we might have to wait a little while on that “tail data” until after we complete our audit.”

: )

#5: Dirty Little Secret Two: Data precision actually goes up lower in the “funnel”.

What funnel you say?

Here’s the one I am thinking about:

As you go deeper into the “funnel” you are dealing with fewer and fewer people / visitors / sources / keywords / pages / vagaries of nature.

The implication of having done all normal things (tagged your site completely, are using first party cookies and the right ecommerce tag on your thankyou.html page) is that there will be few things that could mess up data at the end of your “funnel”. The dataset is smaller, impacted with fewer vagaries of nature.

So when you start your web analytics journey start at the bottom of the funnel and not the top. You won’t find yourself mired in quicksand on day one. And it is easier to reconcile data at the bottom of the funnel.

Compare your orders in IndexTools with your ERP system. Compare your leads in Google Analytics with Salesforce. They won’t match, but it will be a million times easier to discover why (when compared to reconciling sources of data or average page views per visitor).

Here is the other psychological beauty: You know my utter devotion to measuring Outcomes. You start at the bottom of the funnel and you are starting with measuring Outcomes (inc rev, reduce cost, inc loyalty). Guess what? All HiPPO’s LOVE Outcomes.

By the time you get to the top of the funnel 1. You’ll actually be smarter and 2. Your management will be significantly more evolved in their thinking.

#6: Realize the solution to your problem is not implement one more tool!

Talk about compounding your problem.

I know bigamy, on surface, sounds really attractive. It is not. Monogamy rules.

I know. I know you prefer the former. : )

You believe data collected by WebTrends is of bad quality and so you implement Omniture (believe it or not I ran into two companies that have done exactly this!). Or you think Omniture is not working right so you implement Google Analytics as well.

You are just compounded your problem.

It is hard enough to follow the Six Step Decision Making Mental Model with one tool. It takes a lot of effort to understand one tool, get it right, move on to making decisions (remember your job is not to collect 100% accurate data, it is to find actionable insights!).

Two tools means reconciling a lot more, it means understanding sub nuances of two or three tools, it means chasing two vendors, it means more confusion, it means minor hell.

Remember there is nothing particularly magnificent about how Omniture collects data. Google Analytics does not have patent pending exclusive CIA techniques in its tags. WebTrends does not have any secret sauce.

Just use tags. Have ‘em on all the pages. Use first party cookies. After this all tools are pretty close in data collection.

It is ok to date many, it is even ok to get engaged to a couple of ‘em (hopefully at different times), marry one, then try to make that person perfect!!

I am going to get killed for that last one aren’t I? :)

[PS: If you can please don't use multiple paid tools. A. You are wasting money. B. These tools come with so many svars and eprops and variables and massive customizations in implementation that reconciling data between them will make finding life on Mars look like a cake walk.]

#7: Pattern your brain to notice when you’ve reached Diminishing Marginal Returns.

I have come to love and adore this classic principle.

You should work to improve data quality (especially if you find problems :)). But realize that after a certain point it is simply not worth it.

You can improve quality by another 3% but is the effort you put into that worth the ROI you’ll get?

The fact that you’ll feel good does not count.

Data quality seems to be such a holy crusade that it is hard to consciously walk away. The wise know when to walk away.

Remember your job is not to collect perfect data. Your job is to: Increase Revenue. Reduce Cost. Improve Customer Satisfaction/Loyalty.

To me the principle of Diminishing Marginal Returns is lovely because it both says you should work really really had to do the best you can but realize that beyond a certain point it is simply not work the effort.

Be rigorous about realizing you have reached that point. Then move on!

#8: If you have a small site, you have bigger problems than data quality.

You are a part time analyst, or a GAAC, hired to do Omniture analysis at a company and you find that even a 3 – 5% error turns out to be a big deal (because of small overall numbers).

Yes true. Realize that if you are a small company and a small number of people on your site then you have bigger problems than data quality.

For one perhaps focus on doing SEO to get more free traffic? Perhaps mine your existing customer data to find new ideas for product or customer sources? Maybe as an Analyst spend three weeks doing Marketing?

My point is: Is the best use of your time chasing the 5% error or getting an additional 150 people to your site (data be dammed!)?

Sometimes in life data does not become a problem until it becomes a priority.

My advice: If you are a small site focus on recommendation #4 above.

#9: Be Aware of two upsetting distractions: Illogical customer behavior. Inaccuracy benchmarks.

This will drive your bonkers but a lot of data accuracy challenges stem from the clash of the logical tools with illogical customer behavior.

Web Analytics tools expect and work on the basis of a set of logical rules.

The internet is fundamentally illogical. Because we, the inter-dweebs exhibit illogical behavior.

Now like all mostly rational beings we only behave illogically x% of the time (quickly bouncing between sites, changing our minds constantly, never seeing obvious buttons, missing relevant results etc etc).

I have never seen a case where with enough work and experimentation I could not explain even the most illogical behavior. In most of those cases at the end all I had was a regret that I did not focus on doing better things!

Second, if the data is not perfect why aren’t there benchmarks for how “bad” the data is?

In asking for benchmarks you are asking for what Donald Rumsfeld famously called the Unknown Unknown. The impossible.

The web is such a complex ever evolving beast that getting ranges for “inaccuracy” is just not possible right now. The huge difference between how sites are built, experiences are created, technologies at play, needs of each tool for each site does not make life easier.

You know a lot of known knowns in web analytics. Take action on that. Try to identify the known unknowns (do audits using tools like maxamine or observepoint or wasp), try to fix them. Then take action.

Benchmarks can become crutches / excuses. I am kinda sorta against that.

#10: Remember you can fail faster on the web.

The greatest gift the web gives you is the ability to fail faster. At low cost.

This translates into a insanely awesome ability to take higher risk. It also means you can move fast with less than 100% confidence and in the worst case that you are 1000% wrong that you can control the amount of damage.

This is not a privilege that exists in the offline world.

If I have only 80% confidence in the data I can send a small, 1,000, email blast and test the waters to see what will happen. I can send 3 different offers to different geo’s to validate my hypothesis.

I can try 5 versions of the home page and see which world because I am not designing the “you can only try once” cover of the catalog or newspaper ad.

If you had 100% confidence in the data you would commit to spending $500k on affiliate marketing. But if you only have 98% confidence you can commit to a four week pilot program with a budget of $50k. Lower risk, still the possibility of high reward, and a near 100% possibility of making a more confident decision about the remaining $450k.

Don’t wait. Just go.

Ok now its your turn.

What techniques you have used in improving data quality or simply getting around the nagging problem of data quality? What was your most successful “lets all get over this and move on” tactic? If you have come close to web metrics data perfection what did you do?

Which of the above ten strategies is your favorite? Which one do you think is simply baloney? It’s ok. Be honest. I can take it. :)

Thanks much.

PS:
Couple other related posts you might find interesting:

• Data Quality Sucks, Let’s Just Get Over It
• Web Analytics Technical Implementation Best Practices
• The Ultimate Web Analytics Data Reconciliation Checklist
• Convert Data Skeptics: Document, Educate & Pick Your Poison
• The Great Web Data Capture Debate: Web Logs or JavaScript Tags?

Copyright © 2006-2009 Occam's Razor by Avinash Kaushik

Slay The Analytics Data Quality Dragon & Win Your HiPPO’s Love!

Generations of Analysts, Data “People”, Decision Makers have grown up with the principle of GIGO. Garbage in, garbage out.

It made a lot of sense for a very long time. Especially because we used to collect so little data, its lack of even a little quality crapified the decision a lot.

GIGO also fueled our every expanding quest for data perfection and data quality. There are entire companies built around helping your “clean up” your data. Especially if you look at the offline traditional business intelligence, erp, crm, data warehouse worlds.

The web unfortunately threw a big spanner into the works.

Couple important reasons.

First, it is important to realize that we collect a lot of data on the web (type of data, elements of data, what not).

Second, our beloved world wide web, remember still a little baby, is imperfect at every turn. We use data collection methodologies that reflect our efforts to do the best we can, but they are inherently flawed. Just take javascript as an example. It is good at what it does. But not everyone has javascript turned on (typically around 2-3%). Zing: imperfection.

A lot of data. Imperfect data collection system.

Here is the most common result of this challenge: The “Director of Analytics” spends her meager resources in the futile quest for clean data.

Money is spent on consultants (especially the “scarady cats” who deftly stir this issue in their business favor). Everyone tries to reconcile everything across systems and logs. Omniture gets kicked out and WebTrends gets put in, supposedly for it “far superior” data quality (!!).

Makes me sad.

In the debate for perfect data is is important to realize that the reality is a lot more nuanced.

No Possible Complete Data on Le Web.

I humbly believe that the world of data perfection (”clean auditable data”) does not exist any more. It did for a long time because life was cleaner, mistakes were human made, sources were fewer and there wasn’t enough data to begin with (sure terabytes of it, but of what 300 fields? 600?).

On the web we now have too many sources of data. Quantatitive, qualitative, hearsay (sorry, surveys :), competitive intelligence, and so much. [Web Analytics 2.0 ] But these sources are “fragile”.

Sometimes because of technology (tags / cookies / panels / ISP logs). Sometimes because of privacy reasons. Sometimes because we can’t sample enough (surveys, usability tests). Sometimes because it is all so new, we don’t even know what the heck we are doing and the world is changing too fast around us.

Killing the Holy Cows.

The old people who did BI (me for sure, maybe you?) and moved to the web have had to come to the realization that the old rules of making decisions are out of the door. Not just because that mental model of what now counts for “data” means but also because what counts for “decisions” has changed, the pace at which those decisions need to be made have changed. It took companies a long time to die in the past. That process happens at “web speed” now.

Given all that if I don’t change, I’ll become a hurdle to progress. If I don’t change, I can’t help my company make the kind of progress it should.

You need to fundamentally rewire your brain, like I have had do rewire mine (it was painful): The data is not complete and clean, yet it is more data of more type and it contains immense actionable insights.

If you would only get over yourself a little bit.

So how to do this if you really do want to be God’s gift to web analysis?

The Six Step Soul Cleansing Process.

Based on my own personal evolution in this space I recommend you going through this five step cleansing process to ensure that you are doing this right, and you move beyond the deeply counter productive data obsession.

1) Follow best practices to collect data, don’t do stupid stuff.

2) Audit your data periodically to ensure you are collecting as complete a data set as possible (and as accurately as possible, #1).

3) Only collect as much data as you need: There is no upper limit to the amount of data you can collect and store on the web.

4) Ditch the old mental model of Accuracy, go for Precision (more here: Accuracy, Precision & Predictive Analytics). It might seem astonishing but your analysis will actually get more accurate if you go for precision.

5) Be comfortable, I mean really freaking comfortable, with incompleteness and learn to make decisions.

6) [In context of decision making] It used to be Think Smart, Move Fast. I think the next generation of true Analysis Ninjas will: Move Fast, Think Smart. Remember there is an opportunity cost associated with the quest for perfection.

An example of #1 is if you are using third party cookies in your web analytics tool like Omniture or CoreMetrics or WebTrends etc then you deserve the crappy data you are getting. For #2 use various website scanning tools for ensuring complete implementation, each vendor has their own, just ask. #3 is the reason more attempts to data warehouse web analytics data end up as massive expensive failures, or why you then get trapped constantly “mowing the grass”.

You are not going to believe me but in #4 if you actually go for precision your analysis will actually get more accurate over time (whoa!).

#5 is the hardest thing for Analysts (and for many Marketers) to accept. Especially those that have doing data analysis in other fields. They are simply not comfortable with 90% complete data. Or even 95%. They work really really hard to get the other 5% because without that they are unable to accept that they could make business recommendations. Sometimes this is because of how their mental model is. Sometimes is is because the company is risk averse (not the Analyst’s fault). Sometimes it is out of a genuine, if misplaced, desire to give the prefect answer.

Of course the net result is that lots of data collection, processing and perfection exercises happen. The business is starved for any insights to make even the most mundane decisions. I have had to layoff Analysts who simply could not accept incompleteness and had to have data that was clean and complete. Very hard for me to do.

#6 is a huge challenge because it requires an experience that most of us don’t possess. Of having been there. Because of working in companies that plug us into the tribal knowledge and context. Because we work in massively multi layered bureaucracies in large companies. In my heart of heart I believe, sadly, that it will take a new generation of Analysts and a new generation of leaders in companies. Still we must try, even as I accept the criticism that the 10/90 rule is not followed and that we don’t have enough Smart Analyst.

So: Best practices that collect as complete a data set as possible precisely allowing you to look beyond the incompleteness resulting you in moving fast while thinking smart.

Before You Jump All Over Me and Yell: Heretic!

Notice what I am not saying.

I am not saying make wrong decisions.

I am not saying accept bad data.

I am not saying don’t do your damdest to make sure your data is as clean as it can be.

What I am saying is that your job does not depend on data with 100% integrity on the web. Your job depends on helping your company Move Fast and Think Smart.

I am also not saying it is easy.

Reality Check:

We live in the most data rich channel in the universe, we should be using data to find insights, no matter how a little bit off the perfect number they might be.

Just consider this.

How do you measure the effectiveness of your magazine ad? Now compare that to the data you have from doubleclick. How about measuring the ability of your TV ad to reach the right audience? Compare that with measuring reach through Paid Search (or Affiliate Marketing or …..). Do you think you get better data from Neilsen’s TV panel of between 15k – 30k US residents to represent the diversity of TV content consumption of 200 million tv watching Americans?

There is simply no comparison. So why waste our life trying to get perfect data from our web sites and online marketing campaigns? Why does unsound, incomplete, and faith based data from TV, Magazines, Radio get a pass? Why be so harsh to your web channel? Just because you can collect data here means you won’t do anything because it is imperfect?

Parting Words of Wisdom:

Stuart Gold is a VP at Omniture. Here’s a quote from him:

“An educated mistake is better than no action at all.”

Brilliant.

The web allows you to make educated mistakes. Fast. With each mistake you become smarter. With each mistake your next step becomes more intelligent.

Make educated mistakes.

EOM.

Ok now its your turn.

What do you think of the web data quality issue? What are the flawed assumptions I have made in making my recommendation above? How do you ensure your data is as complete and as precise as it can be? Got tools or horror stories to share? What is the next data collection mechanism on the horizon that will be our salvation on the web?

I look forward to your comments and feedback. Thanks.

PS:
Couple other related posts you might find interesting:

• Data Quality Sucks, Let’s Just Get Over It
• Web Analytics Technical Implementation Best Practices
• The Ultimate Web Analytics Data Reconciliation Checklist
• Convert Data Skeptics: Document, Educate & Pick Your Poison
• The Great Web Data Capture Debate: Web Logs or JavaScript Tags?

Copyright © 2006-2009 Occam's Razor by Avinash Kaushik

Web Data Quality: A 6 Step Process To Evolve Your Mental Model

The result was this post: Top Web Analytics Questions, Twitter Edition.

Those 16 questions (!) were just one part of the story.

My twitter account is linked to my facebook account , so my tweets get posted as my status updates.

That means I got a bunch of questions on the facebook account as well. . . .

Here is a summary of the 9 questions / topics that are addressed in this blog post:

1. Twitter’s impact on bounce rates.

2. Does complete information translate into absolute action?

3. The most important business questions addressed by Web Analytics.

4. How to judge someone’s talent/ability in being a Web Analyst?

5. The mystery of “Returning Visitors” having 1 Visit to Purchase! <- Important.

6. Reliability, and effectiveness, of Predictive Web Analytics.

7. How to measure impact of Branding activities?

8. Metrics / Key Performance Indicators to check Daily (!), for any site (!!).

9. Tips and best practices for Filters and Expressions in Google Analytics.

So here we go, replies to my facebook friends, things that keep them up at night. . . .

#1: Dror Zaifman:

How do you think Twitter will effect bounce rates on web sites ? Meaning do you think that someone reading a Twitter post will get more excited due to the heighten hype on Twitter and therefore might be disappointed with the end result increasing the bounce rates?

Twitter will no more increase your bounce rates than say a digg or a stumbleupon or pick your favorite “hot right how” web 2.0 thingy .

In the sense that each of these channels tends to bring new traffic to your site, perhaps a higher percent of them might not be totally relevant for you. But I am not sure that the traffic from Twitter has any higher levels of ADD. :)

As to weather they should be disappointed or not, that’s your call. If you/others just use Twitter to hype what you do or push sub optimal content then you lose credibility, followers and more. So the system is “self correcting”.

#2: John Quarto-vonTivadar:

True or False? If you had 100% metaphysical certitude analytics coverage and could know anything you wanted to know, would some companies still be unable to increase their conversion rate? I depressingly suspect the answer is True. Remember I am not a rocket scientist. You need to dumb this down for me! :)

Let me rephrase John’s question (he is a rocket scientist!): Even if we had all the data in the world would some companies still stink beyond belief in their ability to improve conversion rate?

I am afraid the answer, as John predicted, is a depressing True.

This is not a data problem. It is a people problem. Or perhaps better put it is an Organization Behavior problem.

I think most of the time we underestimate two things:

1] Data is just data and you need to invest in analysis (and hence people) and most companies just want tools (or as you put it “acquire the solution”). At some point tools will move from simply puking data to giving insights with no human requirement. That day is not today. Or tomorrow. Or 2010.

2] It takes a lot to get over oneself (in this case the HiPPO’s), we can present data and win arguments yet people have deeply entrenched opinions that they are unwilling to set aside to actually implement what the data says. And of course I am not even going to touch on politics and solving for vested personal interests.

For example I am dealing with someone now who is doing the worst possible thing for the long term simply because he/she can get a promotion in the short term. And that’s not even the worst of the problems that “data” has to deal with every day.

Result: Lower Conversions.

Sad.

#3: Eric Werner:

What are the most common important business questions addressed by web analytics? – I find that a lot of marketing managers who are newly introduced to analytics say this is great – so what should I measure? I tell them it depends on the business questions they want answered and then they ask what questions should I want answered?

The single greatest root cause of failure with web analytics is the unwillingness or inability to understand what the site is trying to do, and hence defining goals.

While the real answer to your question is: it depends, let me try to see if I can help.

First tell them that Web Analytics can help measure three specific Outcomes from a website (more in the twitter analytics post):

Your question to them is: “Which of these are you working on? I can help you measure each or all of these if you tell me what you are doing.”

That should help focus them a bit and secondly get you started with the most perfect start in Web Analytics: Tying numbers to Outcomes (leads, conversions, loyalty, phone calls, downloads, whatever).

If they refuse to tell you which of the above they are solving for. . . first submit your resume at Monster.com and start looking for a job, the company you are working for is going down. . . then tell them that web analytics can help answer these questions:

Hope this helps Eric. More on this post if you are interested: Tips for Web Analytics Success for Businesses.

#4: Tal Galili:

How can you (and where can’t you) quantify a persons talent/ability in being a web analyst? How could I judge my own performance as a web analyst?

I look for:

1] Critical Thinking
(Interviewing Tip: Stress Test Critical Thinking. Please.)

2] Business Experience
(If all they have is button pressing / report publishing experience they might be very young in their career and that is ok, but if not I am looking for people who have business / marketing / finance experience, if they are a Marketer they get bonus points from me.)

3] High EQ (emotional quotient)
(Wikipedia: The ability, capacity, a self-perceived ability, to identify, assess, and manage the emotions of one’s self, of others, and of groups. Me: One person, no matter how high on the IQ scale can rarely change organizations alone.)

4] Flexibility in thinking, an openness to new information.
(You might think this is obvious, who is dumb enough to stay dug in when faced with new information. You’ll be surprised. I also look for people whose core thinking is not rigid, they realize the world is not perfect, they realize data is incomplete, they realize web Analytics, as in clickstream, is not panacea.)

5] Knowledge Seekers
(I mentioned in a recent interview that I spend three to four hours a week learning something new about our field. Trying new tools. New types of analysis. Reading non-pompous-only-theory-gossip blogs that share new methods of thinking. Attending free webinars in broad fields. I feel I am not doing enough. Find people that invest atleast that much a week.)

If you are doing these things I think you are on the right path. Certainly learn to use more tools and what not. But enrich your mind, keep it open and flexible, think like a marketer.

Good luck Tal.

#5: Robert Patterson:

Can you please explain how the Visits to Purchase Google Analytics report segmenting new and returning visitors can show returning visitors making a 1 visit purchase? Wouldn’t that make them a new visitor purchase and not a returning visitor purchase?

Robert’s question is one of those that you dig into and discover something deeply sub optimal. I am especially sad because this is one of my favorite pan session report.

Here’s what Robert is asking:

See that red arrow? If someone is a “Returning Visitor” why would the report say they purchased after one visit?

Why do you think that is?

Think about it….

Take a guess….

Aw come one…..

I’ll give you one more try…. come on Justin you know this one….

Got the answer?

Its not what you thought.

This report is wrong in Google Analytics. Well that’s not entirely right. Technically the label on top of the report is wrong. And what it actually measures then makes it useless.

I am getting ahead of myself.

What this report actually measures is: Visits to Purchase from the last Campaign.

Only it does not say that either in the label or in the in page help.

Take three scenarios:

Angie: Visit from paid search campaign. Direct Visit. Visit from email campaign -> Purchase.

Jennifer (Angie’s bff): Paid search visit. Direct visit. Organic visit. Bookmark visit. Direct visit. Direct visit -> Purchase.

Judith (Angie’s part of the time bff): Affiliate visit. Direct visit. Bookmark visit. Direct visit -> Purchase.

In summary, Google Analytics will only count the number of visits after a campaign (and campaigns in GA are email, affiliate, paid search, organic search,.. literally everything except direct/bookmark) and show that on this report.

That means this report, another one I love, is also wrongly labeled:

The correct label for this is Days to Purchase from the last Campaign.

I am sure the team at Google will fix the label.

The challenge of course is that while the name change will mean the report will have the right description, it will essentially be useless.

Just look at the above three scenarios. If they are all in the Days/Visits to Purchase from the Last Campaign what actionable insight do you get?

You are still ten million miles away from understand how long does it take for someone to convert.

The fix is not a change in the label, the fix is scraping the report and actually creating a real Days to Purchase and Visits to Purchase reports. If I want to know how many days/visits it takes someone to convert from a organic or paid or email campaign I can always segment that data and view the clean report. Here I don’t even know what the “last campaign” was.

Sorry Robert. And to all of you as well.

#6: John Stansbury:

Based on performance through yesterday, how reliably can I predict where we’ll end up EOD today? (Initial promising results using Holt-Winter adaptive forecasting, but time- and effort- intensive.) Additionally, how granular is too granular for actionable analysis? Is that determined by the agility of your site to adapt?

I wish there was a easy answer to this, sadly no.

Both your questions can be very specific to the business, the goals of the website, seasonal factors unique to you, the overall business strategy (and sub components of that applied on the web), yada, yada, yada.

But regardless of your business you’ll face these six challenges in your attempt to do “predictive analytics” on your web data:

All the details are in this post: Data Mining And Predictive Analytics On Web Data Works? Nyet!

As to your second question, how granular is too granular for actionable analysis, you’ll typically work with a portfolio. As you execute your analysis train yourself to recognize when you are reaching the point of diminishing margins of return. Then you stop, move on to the next thing.

More in this post, see #3: Bounces, Abandonment, Visitor Ratios & Data Drops!

One last tip, always seek to balance what you can do (analysis/insights) with what your company/site/HiPPO can actually action. What they can action might not be the top nine powerful actionable high impact things, they could only do ten through fifteen. Then forget the top nine.

Sucks. I know.

#7: Martin Leblanc:

How do you measure the effect of branding activities?

This is a complicated issue and I might not do it complete justice in a short reply, but let me outline some broad brush strokes.

I believe that branding is a worthy Marketing goal. It gets people to associate, hopefully, positive attributes with your products and services. Here’s one of the masters at branding:

Abercrombie & Fitch . The image above is not their website, it is their complete email campaign. The minor text at the bottom is the opt-out and their address. No call to action (!).

It certainly evokes an emotional reaction, perhaps a brand attribute they would want associated with them.

Measurement?

I firmly believe that every marketing activity has to drive outcomes. It can drive it now, it can drive it in 30 days, it can drive it in six months.

I believe that if you do “branding” you need to define an outcome, increased store sales, more people to the site, more leads for a future concert, newspaper stories from your out of the world campaign, something else.

If there is an outcome you can measure it. My favorites for measuring impact from branding campaigns, for the web:

• Increased Visitor Loyalty and Recency measures post campaign.

• If related to a product, increased sales (even if latent conversions).

• Improved “likelihood to recommend” scores, during / post campaign, as measured by exit surveys.

My favorite way to measure impact of branding campaigns is to do rigorous controlled experiments. They can prove anything, trust me.

For more on this check out #6 here: Multi Channel Analytics.

#8: Claire Devereux Thompson:

I have to check many client sites every day to make sure that things are going smoothly – what’s the one thing that I need to look at if I only have a minute for each?

I was stumped so asked Claire for a bit more.

Claire clarified (say that five times :) that her clients include non-profits that use the web to raise money, a small art school, a large regional furniture site, a online only gift store.

I am still stumped!

The real answer of course is: It Depends.

Fat good that does Ms. Thompson. So let me try to pull a rabbit out of the hat.

My first stab at this would be to look at Outcomes (Goals).

The above data is for a non-ecommerce, not for profit website. It has four goals, and for each quantified goal values.

It is easy to see daily progress (if that’s important), certainly weekly, by looking both at the sparklines next to each goal and also the two sweet numbers at the bottom.

Bottom line: Bottom line is important. Ask each biz to do this, add it to your dashboard. [Ideas for goals for different sites here: Measure Macro AND Micro Conversions.]

My second bunny, sorry tip, would be to focus acquisition, and the idea of the What’s Changed Report.

After you install the Enhanced Google Analytics plugin from our friends at Juice Analytics you’ll be able to something like this:

In your referring sites report you can click on the Who sent me unusual traffic? button and it will show you sites that have increased by 50% in traffic, or dropped 50%.

In the Keywords report you’ll see the same thing but with search keywords.

Both help you get away from the top 10 reports, that rarely change, and help you identify big shifts in keywords and referrers which should in turn help you know if something needs your attention.

I hope the above two sets of ideas help, but what I want you to focus more on is the philosophy I am advocating: 1) Start with Outcomes, always. 2) Focus only on what changes, that mining will help find gems.

Oh and it is a bit of work, even every day. No insight worth monetizing is ever free. Hmm… that’s pretty profound. No? :)

#9: Robert Kennedy:

How to get the most out of your filters/ expressions in Google Analytics. I am always pushing that angle of analytics, sometimes I mix up some wild concoctions :). Seems you are only limited by knowledge and imagination with no floor or ceiling. What is the best resource for filters and expressions?

Can I fess up that I never use them, mostly because perhaps I am not doing the same kinds of analysis.

There is one other reason. I have this constant hyper filter on: what’s the marginal value of me digging a bit more, doing this fancy filters/expression magic?

For me Advanced Segments suffices most of the times.

All that said three resources for you:

1. Robbin “I am the queen of GA expressions” Steif: Regular Expressions Part XII: Bad Greed. Yes that’s part 12!

2. EpikOne’s Regular Expression Filter Tester. Its really good for QA’ing things before you put them in the wild.

3. Not owing Justin Cutroni’s Google Analytics Short Cuts book is now considered a felony in 49 states (all except Utah for some reason). So get it!

I think that should get you going Robert, perhaps it is time for you to start tweeting your favorite expressions and filters? :)

There you go, nine questions that were top of mind for people who ran into my request for inspiration on Facebook.

These are very broad and complex questions, very difficult to answer in a short Q&A, but I hope you all find the answers to be be of some value.

Ok now your turn.

How would you have answered any of these questions differently? Did I miss something in one of the answers? Agree, disagree, shout with joy, cry with pleasure. . . do please share your thoughts.

Thank you.

Copyright © 2006-2009 Occam's Razor by Avinash Kaushik

Dear Avinash: Web Metrics & Analytics Questions, Facebook Edition

Guess what?

It’s true!

All web analytics tools have a smattering of metrics and key performance indicators that were created just because someone decided it would be cute to add / subtract / multiply / divide some numbers.

Many of these don’t pass the first sniff test and when if they do you are still left wondering: “What in God’s name and all that is holy in this world am I supposed to action based on this metric?”

The answer?

Nothing.

With that gloriously upbeat set up let me tell you what we are going to cover today: Three metrics that are available in pretty much all “adult” web analytics tools. Daily, Weekly, Monthly Unique Visitors.

They are so common yet most people don’t understand them well enough and fewer still realize how harmful these can be to your health even in day to day use.

So in this post we try to understand the most basic of the web analtyics basics, the Unique Visitor computation.

What’s a Unique Visitor?

It is simple really. . . .

Technical Definition: Count of all the Unique cookie_id’s during a given time period.

English Definition: The first time someone visits your site a first party persistent cookie is set in their browser. This cookie lasts any where from several months to several years. Each time that person visits your site that cookie identifies them as the same browser.

Notice I said browser, not person. It is likely, but not always true, that each a unique visitor is a unique person.

You can learn a lot more about Visits and Unique Visitors in this post: Standard Metrics Revisited: #1: Visitors.

Very predictably every 18 months or so the blogosphere goes wild with how accurate, or not, the Unique Visitor metric is. Much mud is thrown around. Indignations are foisted on the world. Name calling ensues.

Regardless of that Unique Visitors remains a valuable metric that used correctly, in place of Visits, measures success of your online marketing efforts.

Oh and your best weapon against ignorance? Education. See above post on Visitors. And this one: A Primer On Web Analytics Visitor Tracking Cookies. It covers cookies and deletion rates and other such yummy stuff. Read that and you have my word you’ll be the smartest cookie in the room.

See what I did there? :)

Daily, Weekly, Monthly Unique Visitors:

In many web analytics tools (say Yahoo! Web Analytics, Omniture, WebTrends etc, but not in Google Analytics ) you’ll also see Daily Unique Visitors, Weekly Unique Visitors, Monthly Unique Visitors and, sometimes, Absolute Unique Visitors.

Each is trying to tell you something about Unique Visitors, yet if you pause and think about it, I mean really pause and think about it, you’ll realize two of these are really bad for your health, and the third should be used with caution.

The core reason is that what looks attractive initially becomes progressively worse as you extend the time period. The Daily metric, so to speak, does not even last in value beyond two days!

So let’s spend a second understanding this slightly yucky phenomenon.

Here’s the data, from omniture.com, where WebTrends is used for tracking Visitors. . .

Now let’s go measure the complex set of metrics that’ll stare at you, let’s say when you crack open Omniture or WebTrends (or pretty much any other competitive web analtyics tool).

The Web Analytics Unique Visitors Story:

Before that realize that what you see will depend on the time period you are looking at. [Arrrh!]

And before I really really jump in… you’ll see a metric called Absolute Unique Visitor. I am going to use that as a proxy for how unique visitors should be computed correctly, regardless of what time period you are computing it for. Keep an eye on that number.

Looking at Month 1 and Week 1 at the end of Day One:

If you ran your reports at the end of day one here is what your analytics tool will report to you, with some delight and joy I might add. . .

Makes sense right? Do a happy dance, high five someone next to you, heck give them a hug and a kiss.

Now let’s make this more “complicated”.

Looking at Month 1 and Week 1 at the end of Day Two:

If you ran your reports at the end of day two here is what you’ll see. . .

Slow down the happy dance a bit.

Note the silly effect on Daily Unique Visitors, even though it was the exact same folks, Dennis and Matt, from the earlier day who visited on day two. They get counted twice.

Life lesson: Daily Unique Visitors is a useless number if you are looking at a time period of more than one day!

Let’s keep going.

Looking at Month 1 at the end of Week One:

Crack open your analytics tool, it has been a long week, look at the metrics, here’s what you’ll see. . .

Note the continuing uselessness of the Daily Unique Visitor number (and even if you trend it over time, as in the blue graph above, analyze what it is actually showing you? what’s the insight?).

In your Web Analytics Tool you might see a report that looks like this:

By know you know why there is a sad frowny face in that last Total row. Right?

Repeat: Life lesson: Daily Unique Visitors is a useless number if you are looking at a time period of more than one day!

Looking at Month 1 at the end of Week Two:

Gather everyone in your close proximity in the office, form a circle, hold hands, close your eyes, say a quite prayer, now open your analytics tool. . .

The Weekly number is wrong because it counts: Avinash, Dennis, Matt, Matt again, Ian and Jim. It counts Matt again because he visited during both weekly time periods.

Life lesson: Weekly Unique Visitors metric is useless if you are looking across multiple weeks. We’ve covered above why Daily Unique Visitors is, to put it mildly, sub optimal.

Ok only two more scenarios left, hang in there, it gets better.

Looking at the end of Month 1, for the whole month:

By now I am sure you are 100% up to speed on what you are going to see. . .

There is now triple or double counting happening in both the Daily Unique Visitors and Weekly Unique Visitors numbers.

Life lesson: Both Daily Unique Visitors and Weekly Unique Visitors numbers are useless when you look at a time period of a month.

One last scenario, not to make your brain hurt but rather to ensure you reach the state of maximum Analysis Ninja enlightenment!

Looking at the end of Month 2, for the two months:

Tingling with excitement. . . here’s what you’ll see. . .

There is now triple or double counting happening everywhere, the Daily Unique Visitors, Weekly Unique Visitors and Monthly Unique Visitors numbers.

The correct measure of unique is the Absolute Unique Visitors metric because it de-dupes the unique visitors across the entire time period you are reporting on.

Life lesson: Both Daily Unique Visitors and Weekly Unique Visitors numbers are totally really useless when you look across months. Use Monthly Unique Visitors with caution, knowing it is only de-duping for each month and then summing the number for each month.

If your tool provides Absolute Unique Visitors you are in luck because then you are getting true unique visitors across whatever arbitrary time period you choose.

Google Analytics provides you with the Absolute Unique Visitors metric.

It will do that across set time periods, like the month of March (or any number of months). . .

or across arbitrary time periods, as Monday March 9th through Thursday March 19th. . .

It will dedupe the numbers when it reports to you, rather than adding the totals of each day, week or month.

Complex but bonus for Ninjas: Depending on which graph you look at, daily, weekly or monthly, it will intelligently compute the number for each time period and also show you the aggregate deduped number for that time period.

Fly in the otherwise rather healing ointment?

Google Analtyics does not compute Absolute Unique Visitors when you segment the data, when you use the Advanced Segmentation feature. Those of you who read the blog know my utter infatuation with segmentation, so you can easily understand how sad this makes me.

You can get Absolute Unique Visitors for segments by using the “create a filtered profile that just data for the segment” method and that works if you have forethought. But it is sub optimal, just like some “enterprise” web analytics vendors telling you that you can only segment if you tell them before the fact what you might want to segment later.

Why do Web Analytics Vendors torture you with Daily, Weekly, Monthly Unique Visitors?

I knew you were asking yourself this question!

Good on you Mate.

If these metrics are that sub optimal, why do web analytics vendors put us through this torture?

Simple: Compute power (translation: cost, for them).

It is very computationally intensive to calculate for you the true real (Absolute) Unique Visitor number across any arbitrary time period or across multiple weeks or months.

Increased computational intensity for the vendor means more processing time and higher costs.

So doing Daily, Weekly and Monthly counts (and then summing them up) is cheaper for them.

After the first vendor decided to do this, and there were no major outcry from Web Analytics Users (or even Ninjas!), others quickly followed.

For the more prevalent vendors in the space Google Analytics is one the rarest that provides the truly de-duped Absolute Unique Visitor metric (in aggregate, not segmented, boo!). Only time will tell when Google will buckle under the computation/cost weight and stop providing it true Absolute Unique Visitors.

[Update: Both NedStat and Xiti, two wonderful European companies do allow for computation of Absolute Unique Visitors out of their standard packages, no additional payment or gyrations required. Add Unica's NetInsight to that list as well! Hurray!!]

There are some vendors that will tell you that you can buy their more expensive data warehouse solutions (at an additional cost on top of what you pay today) and then compute Absolute Unique Visitors yourself. True. Ask for the cost. Ask if its really Absolute. If prudent, pay more. Regardless, be informed.

Long lesson.

But now you are truly at a Analysis Ninja black belt level of proficiency!

Now your turn.

Please share your comments / feedback / critique / hugs / non-hugs about this post. What does your tool do? How do you think we should improve things? What would you eliminate? What would you add? What did I miss?

PS:
Couple other related posts you might find interesting:

• Standard Metrics Revisited: #1: Visitors
• A Primer On Web Analytics Visitor Tracking Cookies
• Five Rules for High Impact Web Analytics Dashboards
• The Ultimate Web Analytics Data Reconciliation Checklist
• Excellent Analytics Tip#5: Conversion Rate Basics & Best Practices
• Excellent Analytics Tip#4: Make Your Analysis/Reports “Connectable”

Copyright © 2006-2009 Occam's Razor by Avinash Kaushik

Standard Metrics Revisited: #6: Daily, Weekly, Monthly Unique Visitors.