• A dashboard, by definition, fits on a single page — this is straight out of Stephen Few’s book Information Dashboard Design: The Effective Visual Communication of Data; I was skeptical that this was really possible when I first read it, but I’ve increasingly become a believer…with the caveat that there is ancillary data that can be provided with a dashboard as backup/easy drilldowns
• The dashboard must include logic to dynamically highlight the areas that require the most attention.

The second principle is the focus of this post.

Actionable Metrics

It’s become cliché to observe that data must be converted to information that drives action. I’ve got no argument with that, but, all too often, the people who make this statement would also see this statement as blasphemy:

Most metrics should drive no action most of the time

Any good performance measurement system is based on a structured set of meaningful metrics. Each of those metrics has a target set, either as a hard number, as a comparison to a prior period, as a comparison to some industry measure, or something else.

Here’s the key, though: most of those metrics will likely come in within their target range most of the time! That’s a good thing, because it is rare that a business is equipped to chase more than a handful of issues at once.

A Conceptual (If Not Realistic) Dashboard

At the end of the day, when your user looks at a dashboard, here’s what they really are hoping to get:

This is as actionable as it gets:

• Only the areas that are performing well outside of expectations are shown
• What’s actually happening is stated in plain English
• The person viewing the dashboard has a concise list of what he/she needs to start looking into immediately

Will your users ever tell you this is what they’re looking for? No! And, if asked, the reasons why not would include:

• “I need to see everything that is going on — not just the stuff that is performing outside targets (…because I’m just not comfortable trusting that we designed a dashboard that is good enough to catch all the things that really matter).”
• “My boss is likely to ask me about her specific pet metric…so I need to have that information at my fingertips, even if it’s not going to drive me to take new action.”
• “I need to see all of the data so that I can identify patterns and correlations across different aspects of the marketing program.”

Arguing any of these points is an exercise in futility. Between the explosion of data that is available, the fact that not a day passes without a Major Marketing Mind talking about how important it is for us to leverage the wealth of data at our fingertips, and the fact that humans are inherently distrustful of automation until they have seen it working successfully for an extended period of time, all mean that a dashboard, in practice, has to include a decent chunk of information that will not drive any new action.

But the Concept Is Still Useful

I believe the conceptual dashboard above is a useful guiding vision. At the end of the day, we want to provide, and our users want to receive, information that is clear and concise, which the dashboard above certainly is. if we morph the concept above just a little bit, though, we get a dashboard that is only slightly less impactful but heads off all of the concerns listed earlier:

Get the idea? The same highlights pop, but additional data is included, and it all still fits on a single page. Obviously, the real dashboard would be one step further diluted from this by presenting actual metrics — numbers, sparklines, etc. But, by working hard to keep all of the on-target data as muted as possible, some clever use of bold and color through conditional formatting can still make what’s important pop.

Parting Thoughts and Clarifications

Any dashboard, whether it’s managed through an enterprise BI tool, through Microsoft Excel, or even through PowerPoint, should be designed so that the structure of the dashboard does not change from one reporting period to the next — the same metrics appear in the same place week in and week out. BUT, within that structure, there should be a concerted effort to make sure that the metrics that are the farthest off target (usually the ones that are the farthest off target in a bad way, but if something is off the charts in a good way, that needs to be highlighted as well) are what the user’s eye is drawn to. And, furthermore, those are the metrics that warrant the first pass of drilling down to look for root causes.Similar Posts:

• The “Action Dashboard” — Avinash Mounts My Favorite Soapbox
• A Simple Process for Establishing Corporate Metrics
• Dashboard Design Part 3 of 3: An Iterative Tale
• Social Media Measurement: A Practitioner’s Practical Guide
• Four simple rules for identifying a good metric

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | No comment | Add to del.icio.us
Post tags: dashboards

Feed enhanced by Better Feed from Ozh

Outputs are simply things that happened as the result of some sort of tactic. For instance, the number of impressions for a banner ad campaign is an output of the campaign. Even the number of clickthroughs is an output — in and of itself, there is no business value of a clickthrough, but it is something that is a direct result of the campaign.

An outcome is direct business impact. “Revenue” is a classic outcome measure (as is ROI, but this post isn’t going to reiterate my views on that topic), but outcomes don’t have to be directly tied to financial results. Growing brand awareness is an outcome measure, as is growing your database of marketable contacts. Increasing the number of people who are talking about your brand in a positive manner in the blogosphere is an outcome. Visits to your web site is an outcome, although if you wanted to argue with me that it is really just an aggregated output measure — the sum of outputs of all of the tactics that drive traffic to your site — I wouldn’t put up much of a fight.

Why Does the Distinction Matter?

The distinction between outputs and outcomes matters for two reasons:

• At the end of the day, what really matters to a business are outcomes — if you’re only measuring outputs, then you are doing yourself a disservice
• Measuring outputs and outcomes can help you determine whether your best opportunities for improvement lie with adjusting your strategy or with improving your tactics

Your CEO, CFO, CMO, COO, and even C-3PO (kidding!) — the people whose tushes are most visibly on the line when it comes to overall company performance — care that their Marketing department is delivering results (outcomes) and is doing so efficiently through the effective execution of tactics (outputs).

Campaign Success vs. Brand Success

Avinash Kaushik wrote a post a couple of weeks ago about the myriad ways to measure the results of a “brand campaign.” Avinash’s main point is that “this is a brand campaign, so it can’t be measured” is a cop-out. If you read the post through an “outcomes vs. outputs” lens, you’ll see that measuring “brand” tends to be more outcome-weighted than output-weighted. And (I didn’t realize this until I went back to look at the post as I was writing this one), the entire structure of the post is based on the outcomes you want for your brand — attracting new prospects, sharing your business value proposition more broadly, impressing people about your greatness, driving offline action, etc.

Avinash’s post focuses on “brand campaigns.” I would argue that all campaigns are brand campaigns — while they may have short-term, tactical goals, they’re ultimately intended to strengthen your overall brand in some fashion. You have a strategy for your brand, and that strategy is put into action through a variety of tactics — direct marketing campaigns, your web site, a Facebook page, press releases, search engine marketing, banner ads, TV advertising, and the like. Many tactics are in play at once, and they all act on your brand in varying degrees:

And, of course, you also have happenstance working on your brand — a super-celebrity makes a passing comment about how much he/she  likes your product (or, on the other hand, a celebrity who endorses your product checks into rehab), you have to issue a product recall, the economy goes in the tank, or any of these happen to one of your competitors. You get the idea. The picture above doesn’t illustrate the true messiness of managing your brand and all of the other arrows that are acting on it.

Oh, and did I mention that those arrows are actually fuzzy and squiggly? It’s a messy and fickle world we marketers live in! But, here’s where outcomes and outputs actually come in handy:

1. In a perfect world, you would measure only outcomes for your tactics…which would mostly mean you would actually measure at some point after the arrows enter the brand box above, but…
2. You don’t live in a perfect world, so, instead, you find the places where you can measure the brand outcomes of your tactics, but, more often than not, you measure the outputs of your tactics (measuring closer to the left side of the arrows above), which means…
3. You actually measure a mix of outcomes and outputs, which is okay!

Tactics are what’s going on on the front lines. Their outputs tend to be easily measurable. For instance, you send an e-mail to 25,000 people in your database. You can measure how many people never received it (output — bouncebacks), how many people opened it (output), how many people clicked through on it (output), and how many people ultimately made a purchase (outcome). Except the outcome…is probably something you wildly under count, because it can be darn tough to actually track all of the people for whom the e-mail played some role in influencing their ultimate decision to buy from your company. The outputs  can also be measured very soon after the tactic is executed (open rate is a highly noisy metric, I realize, but it is still useful, especially if you measure it over time for all of your outbound e-mail marketing), whereas outcomes often take a while to play out.

At the same time, if you ignored measuring the tactics and, instead, focussed solely on measuring your brand, you would find that you were measuring almost exclusively outcomes (see Avinash’s post and think of typical corporate KPIs like revenue, profitability, customer satisfaction, etc.)…but you would also find that your measurements have limited actionability, because they reflect a complex amalgamation of tactics.

So, What’s the Point?

Measure your brand. Measure each of your tactics. Accept that measurement of the tactics is heavily output-biased and measurable on a short cycle, while measurement of your brand is heavily outcome-biased and is a much messier and sluggish beast to affect.

Watch what happens:

• If your brand is performing poorly (outcomes), but your tactics are all performing great (outputs), then reconsider your strategy — you chose tactics that are not effective
• If your brand is performing poorly (outcomes) and your tactics are performing poorly (outputs), then scrutinize your execution
• If your brand is performing well…cut out early and play some golf! Really, though, if your tactics are performing poorly, then you may still want to scrutinize your strategy, as you’re succeeding in spite of yourself!

The key is that tactics are short-term, and driving improvement in how they are executed — through process improvements, innovative execution, or just sheer opportunism — is an entirely different exercise (operating on a different — shorter — time horizon) than your strategy for your brand. Measure them both!Similar Posts:

• "If you can’t measure it, don’t do it"
• Performance Measurement vs. Analysis
• Sometimes, the Data DOES Paint a Clear Picture
• Death to “Marketing ROI is Return on Influence”…Please!!!
• So, You Think Measuring Marketing Performance Is Hard?

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | One comment | Add to del.icio.us
Post tags: Avinash Kaushik, outcomes, outputs, ROI

Feed enhanced by Better Feed from Ozh

As for the event itself, we welcomed a new sponsor — Resource Interactive. The topic for the event was search engine optimization (SEO) with a little bit of search engine marketing (SEM). It wasn’t the first time that we relied on Dave Culbertson of Lightbulb Interactive to present, and it likely will not be the last, as his knowledge and enthusiasm about SEO, SEM, and web analytics is both entertaining and informative!

Dave attended SMX East in New York the week before WAW, and he agreed to pull together the highlights of the sessions that he attended. One of my favorite tweets from Dave while he was at the conference was this one:

“Ended up leading a lunchtime discussion on web analytics at #smxeast. Web analytics and SEO – like peanut butter and chocolate!”

Partly because Dave is one of the organizers of Columbus Web Analytics Wednesday, and partly because, well, SEO/SEM and web analytics really should be integrated, “search” is a frequent cornerstone of our WAW topics. Dave’s presentation was titled SMX East 2009: The Spinal Tap Wrap-up. At least half of us (myself included) didn’t get the reference, while a solid quarter of the attendees immediately got it and thought it was quite clever and amusing. There were 11 slides in the deck, so:

The presentation focussed primarily on SEO tips, although there was some SEM here and there. An incomplete list of the nuggets/surprises that jumped out the most to me included:

• PageRank sculpting — this is when you try to gently influence the Google PageRank for pages you control by making subtle, behind-the-scenes tweaks to both that page and other pages that you control that link to that page. Apparently, a somewhat common way to do this has been through the use of the NoFollow tag. While this may have worked at one point, Google now pretty much ignores the tag when it comes to assessing PageRank
• rel=”canonical” — this is a biggie, especially when it comes to web analytics and campaign tracking; this is a tag that can be added to a page to specify the exact “preferred” URL for the page. It’s important because many pages get linked to or arrived at with one or many extraneous parameters tacked on to the end of the URL: campaign tracking parameters for the web analytics tool, link tracking information for the e-mail engine from which a user may access the page, session ID or user ID information for the application that is rendering the page to enable it to make subtle tweaks in the content, etc. The full adoption of this tag by Google, Yahoo! Search, and Bing should go a long way towards removing the tension that exists between the SEO person pushing for the removal of these parameters in links (to avoid link dilution) and the web analyst who pushes to add them (to improve tracking capabilities). Google put together a nice write-up and video on the canonical tag after SMX West.
• keywords – this is “keywords for SEO,” rather than the SEM usage of the term. A lot of information was presented about studies as to where the appearance of a keyword had the most/least impact. Having the keyword in the domain name itself was great, but, of course, you’re not going to be able to do that for too many keywords! (I couldn’t help but thinking of Clearsaleing’s http://www.attributionmanagement.com/ site, though!) Even better is to have the keyword in the domain and in the directory path (i.e., http://www.keyword.com/keyword). Having the keyword in a subdomain (http://keyword.company.com) is apparently not very effective (there was a quick side discussion about an online shoe retailer — and I can’t remember which one it was and, ironically, can’t seem to put together the right Google search to figure it out — that tried creating a subdomain for very type of shoe they sold…which then helped trigger Google to make this not effective; I’m fuzzy on the specifics, obviously!) Another point here is that there is both the “what the search engine algorithm puts weight on keyword-wise” and the “how user behavior — which links users follow — is affected by keywords showing up in subdomains, domains, query parameters, etc.” factor — it’s hard to tease out which is which, so the studies have focussed more on “what actually happens” rather than “why it happens.”

At the end of the day, search engine optimization still comes down to providing great content in a way that users can easily navigate to it and consume it. Google’s algorithms are geared around making the same recommendations that a human being with an infinite knowledge of what content was where on the web would recommend in response to a question from another human being. SEO efforts need to focus on helping that theoretical human out — not trying to fool him/her!

I also distributed copies of the deck that Laura Thieme of Bizresearch presented at SMX East. That presentation was primarily SEM-focussed, but it also had some great nuggets in it. Unfortunately, Laura wasn’t able to attend WAW (see the first paragraph of this post!) this month. Laura presented at WAW back in July and really knows her way around SEM, so we missed having her there!

All in all, it was a good event!

Similar Posts:

• Win a WASP v1.09 for Analyst license at Web Analytics Wednesday
• Inventing a Metric
• A Record-Setting Web Analytics Wednesday in Columbus
• Monish Datta Stays on the Vegetarian Wagon (Sort of) at Web Analytics Wednesday
• Data Visualization — Few’s Examples

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 2 comments | Add to del.icio.us
Post tags: lightbulb interactive, Search Marketing Expo, SEM, SEO, SMX East, WAW Columbus

Feed enhanced by Better Feed from Ozh

With the well-warranted increasing use of sparklines, which are inherently trend-y representations of data, I like to be able to put a meaningful trend indicator that complements the sparkline. Throughout this post, I will illustrate trendlines, but I’m really focussed on trend indicators, which are a symbol that indicates whether the trend in the data is upward, downward, or flat. Although there are a few minor tweaks I’d love to make once Excel 2010 is released and allows the customization of icon sets, I’m reasonably happy with their 5-arrow set of trend indicators:

They’re clean and clear, and they work in both color and in black and white. And, with conditional formatting, they can be automatically updated as new data gets added to a dashboard or report. While I won’t show these indicators again in this post, the trendlines I do show are the behind-the-scenes constructs that would manifest themselves as the appropriate indicator next to a sparkline or numerically reported measure.

I’ll use a simple 12-period data set throughout this post to illustrate some thoughts (not as a sparkline, but the principles all still apply):

Trends are slippery beasts for several reasons:

• Noise, noise, noise — all data is noisy, which means it’s easy to over-read into the data and spot a trend that is not really there
• The aircraft carrier vs. the speedboat conundrum — the more data points you use, the more stable your trend, but the longer it takes to collect enough data to identify a trend, or, worse, to determine if you’ve truly impacted the trend going forward

Let’s start this exploration by walking through some of the common ways that “trend” judgments get made and point out why they’re troubling. I will then show an alternative that, while only marginally more complex to implement, works better when it comes to specifying trend-age.

Trending Approaches of which I’m Leery

Trending Based on the Change Over the Previous Period

The most common way I see trends reported is on a “change since the previous period” basis.

In this example, the trend would be an “up” because the data went up from the prior period to the current period. The problem with this is that, if you look at the longer pattern of data, you see that the data is pretty noisy, and it’s entirely possible that this “trend” is entirely a case of noise masking the true signal.

Trending Over an Extended Period

Another way to trend your data, which Excel makes very simple, is to add a trendline using Excel’s built-in trending capabilities (converting this trendline to an indicator would require some use of a couple of Excel functions that I’ll go into a bit in my recommended approach later in the post).

With this method, the trend would be indicated as “slightly up.” While this may be a valid representation of the overall trend…it seldom seems quite right to use it. The trend gets impacted heavily by any sort of big spikes (or dips) in the data. These keep the same upward or downward trend for a very long period of time. I had a blog post during March Madness one year that wound up driving a big spike in traffic to my site. While it was legitimate for that spike to show an upward trend when I looked at my traffic that week or month, that spike has now wreaked havoc on the macro trend indicator that Google Analytics has shown ever since — for several months that spike kept my overall trend up, and, then, once that spike passed the fulcrum of the tool’s trend calculation, it caused the reporting of a downward trend for severals subsequent months. Through the whole period, I had to mentally discount what the trend indicator showed.

Year-Over-Year Trending

Because seasonality wreaks havoc with trendlines, it’s not uncommon to see trend indicators based on year-over-year results — if the current reporting period is a higher number than the same period a year ago, then the trend is up. For trending purposes, this combines the worst of the two prior examples — it takes a very small number of data points (subjecting the assessment to noise) and it uses ancient history data in the equation.

This isn’t to say that comparisons to the same period in the prior year (or even the same period in the prior quarter, since many companies see an intra-quarter pattern) are bad. But, the question those comparisons answer differs from a trend: a trend should be an indication of “where we are heading of late such that, if we continue on the current course, we can estimate whether we will  be doing better or worse next week/next month,” while a year-over-year comparison is more a measure of “did we move positively from where we were last year at this time?”

Trending Approaches I Feel Better About

I’ve spent an embarrassing amount of time thinking about trending over the past four or five years, but I’ve finally settled on an approach that meets all of these criteria:

• It balances the number of data points available for the trend with the sluggishness/timeliness of the results
• It’s reasonably intuitive to explain
• It passes the “sniff test” — while a trend indicator may initially be a little surprising, on closer inspection, the user will realize it’s legit

The last bullet point is really a combination/result of the first two.

My Failed Exploration: Single Point Moving Range (mR)

Because of criteria above, I’ve discarded what I thought was my most promising approach — using the single point moving range (mR). A light bulb went off last spring when I took an intermediate stats class, and, although the professor glossed over the moving range formulas, I thought it was going to be the answer that would allow me to solve my trendline quandary — it would look at the “change over previous period” and determine if that change was sufficiently large to warrant reporting a measurable trend. After noodling with it quite a bit… I don’t think that it works for the purposes of trend indicators. For chuckles, a moving range chart for the example in this post looks like the following:

If you want to read more about moving ranges, the best explanation I found was on the Quality Magazine web site. I’ll just stop there, though. We’ve already lost on the “reasonably intuitive” front, and I haven’t even calculated the control limits yet!

And Another Failed Exploration: the Moving Average

There’s also the “moving average” approach, which smooths things out quite a bit:

I always feel like the moving average is some sort of narcotic applied to the data — it makes things fuzzy by having a single data point factored into multiple points represented on the chart. But, I’ll grudgingly admit that it does have its merits in some cases.

My Approach to Trending (At Last!!!)

There are two key elements to my trending approach, and neither is particularly earth-shattering:

1. Break the data into smaller components than the reporting cycle
2. Trend only over recent data, rather than over the entire reported timeframe

Going back to the original example here, let’s say that I update a dashboard once a month, and that the dashboard primarily looks at data for the prior 3 months. In that case, the 12 data points each represent (roughly) one week. IF I simply reported the data on a monthly basis, then the chart would look like this:

That shows a clear upward trend, regardless of whether I look at the last month or the last two months of data. It would be hard not to put an upward trend indicator on this plot. But, we’re relying on all of three data points, and we’re going back three full reporting periods to draw that conclusion. Both of these are a bit concerning. Invariably, we’d want to go back farther in time to get more data points to see if this trend was real…and then we’re falling into the aircraft carrier dilemma.

Instead, though, I can keep the granularity of the reporting at a week, but only trend over the last four periods:

I don’t actually plot the trendline shown in the chart above. Rather, I calculate the formula for the line using the SLOPE and INTERCEPT  Excel functions. I then calculate the value of the 4-weeks-ago endpoint of the line and the most-recent-week endpoint of the line and look at the percentage change from one to the other. I actually set some named cells in my workbook to specify how many periods I report over (so I can vary from 4 to 6 or something else universally) as well as what the different thresholds are for a strong up, weak up, no change, weak down, or strong down trend.

In the example in this post, the change is a 16% drop, which usually would garner a “strong down” trend — very different from all the upward trends in the early examples! And, even somewhat counter-intuitive, as the most recent change was actually an “up.” If the entire range has been trending upward, as shown by the 3-point plot as well as by a close inspection of the raw basic data (think of it as a sparkline), then you already have that information available as the longer term trend, but, of late, the trend seems to be somewhat downward.

A Note of Caution

This post has gone through what works for me as a general rule. As I read back over it, I realize I’m setting myself up for a case of, “Yeah, you CAN make the data say whatever you want.”

I’m less concerned about prescribing a universally-effective approach to trend calculation as I am about putting out a cautionary tone on the various “obvious” ways to calculate a trend. The sniff test is important — does the trend work for your specific situation when you actually apply it? Or, have you adopted a simplistic, formulaic approach that can actually provide a very clear misrepresentation of the data?

And…a Nod to Efficiency and Automation

The prospect of introducing SLOPE and INTERCEPT functions may seem a little intimidating from a maintenance and updating perspective, but it really doesn’t need to be. By using built-in Excel functionality, these can be set up once and then dynamically updated as new data comes in. I like to build spreadsheets with a data selector so that the dashboard is a poor man’s BI tool that allows exploring how the data has changed over time. The key is to use some of Excel’s most powerful, yet under-adopted, features:

• Conditional formatting — especially in Excel 2007 where conditional formatting can make use of customized icon sets
• Named cells and named ranges — these are handy for establishing constants used throughout the workbook (thresholds, for instance) that you may want to adjust
• Data validation — using a cell as your “date range selector” that references a named range of the column that lists the dates for which you record the data
• VLOOKUP — because you used data validation, you can then use VLOOKUP to find the current data based on what is selected by the user
• Dynamic charts — these actually aren’t a “feature” of Excel so much as the clever combination of several different features; Jon Peltier has an excellent write-up of how to do this

If set up properly, a little investment up front can make for an easily updated report delivery tool…with meaningful trend indicators!Similar Posts:

• Social Media Measurement: A Practitioner’s Practical Guide
• Dashboard Design Part 3 of 3: An Iterative Tale
• 2009 Predictions for CRM (not mine — Forrester’s)
• Dashboard Design Part 1 of 3: An Iterative Tale
• Guy Kawasaki (Almost) Says 3-D Graphs Are Evil

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 4 comments | Add to del.icio.us
Post tags: dynamic ranges, INTERCEPT, moving average, moving range, SLOPE, trend indicators, trendlines

Feed enhanced by Better Feed from Ozh

Bryan Huber from huber+co. interactive and Jen Wells from TeamBuilder Search

Todd Ehlinger from Nationwide, Mike Amer from DSW, and Elaine F.

The Erics — Goldsmith from AOL and Diaz from Diaz & Kotsev Business Consulting (not shown: the THIRD Eric — Eric Moretti from Quest Software)

The picture that didn’t come out well was the one of Laura Thieme of BizResearch with her daughter, Melina — hanging out on her mom’s shoulder…and ‘nary a peep the whole evening (why couldn’t I have had one of those kids?!)! And (cliche warning) cute as a button! As it turned out, Melina wasn’t the only kid who made an appearance — Dave Culbertson’s sons were in attendance on the periphery for the first part of the evening as well.

Rather than a formal presentation, we did an interactive, get-to-know-each-other, have-a-chuckle activity — conceived of and coordinated by Dave Culbertson from Lightbulb Interactive. Unlike my attempts to photograph Melina and Laura — where I only took one shot and then figured the flash was just cruel — I kept clicking the shutter at Dave until he struck a sufficiently expressive pose:

What Dave walked us through was a segmentation exercise: he had a list of questions, each with four possible answers, and we had to segment / re-segment ourselves after each question by going to the area of the room designated for how we would answer that question. An incomplete list of the questions:

1. Where did you go for your undergraduate degree? a) Ohio State, b) not Ohio State, but another school in Ohio, c) not in Ohio, but in the U.S., or d) outside the U.S.
2. Which of the following most describes your opinion of social media? a) revolutionary, b) evolutionary, c) nothing new, or d) what’s social media?
3. If you were going to read only one book this month, what kind of book would it be? a) non-fiction business, b) non-fiction non-business, c) fiction non-science fiction, d) science fiction (or something like that)
4. If you took only one vacation this year, where would you most like to go? a) the beach, b) the mountains, c) a large city, d) Disneyland
5. What kind of car do you drive? a) American, b) European, c) Japanese, d) Korean

After we’d segment ourselves, Dave would ask a few follow-up questions of the group. It really did turn out to be a lot of fun (and, if you’re reading this post and recommended a book on that question, please leave a comment with the book you recommended! There sounded like some excellent reads there, and I wasn’t taking notes!)

For my part, I enjoyed getting folks’ take on the Omniture acquisition by Adobe. And, Bryan Huber mentioned what sounds like a pretty slick tool for <social media buzzword>online listening</social media buzzword> that factors in the influence of the person who commented about your brand as well as what they said — another part of the evening where I wasn’t taking notes (but, come on, the pictures ARE fabulous, right?).

So, that’s the hasty recap of the evening. By the time this post publishes, I’ll be on my descent into Boston for a lonnnnng weekend with Mrs. Gilligan:

(And, for you, Eric G., none of the photos used in this post were subjected to post-processing other than cropping. There’s no way I’m going to be able to stick with that, though!)

Similar Posts:

• Columbus Web Analytics Wednesday — Jan 2009 Edition
• Monish Datta: “It was the best WAW yet!”*
• Another Great Web Analytics Wednesday in Columbus
• Columbus WAW July 2009 Recap — Bizwatch and More!
• Columbus Web Analytics Wednesday: A Speedy April

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | One comment | Add to del.icio.us
Post tags: WAW, WAW Columbus

Feed enhanced by Better Feed from Ozh

One of the comments on the blog came from John Grono of GAP Research, who, while agreeing with the post, pointed out:

You refer to this as “web analytics”. I also know that this is what the common parlance is, but truth be known it is actually “website analytics”. “web” is a truncation of “world wide web” which is the aggregation of billions of websites. These tools do not analyse the “web”, but merely individual nominated “websites” that collectively make up the “web”. I know this is semantics … but we as an industry should get it right.

It’s a valid point. Traditionally, “web analytics” has referred to the analysis of activity that occurs on a company’s web site, rather than on the web as a whole. Increasingly, though, companies are realizing that this is an unduly narrow view:

• Search engine marketers (SEO and SEM) have, for years, used various keyword research tools to try to determine what words their target customers are using explicitly off-site in a search engine (although the goal of this research has been to use that information to bring these potential customers onto the company’s site)
• Integration with a company’s CRM and/or marketing automation system — to combine information about a customer’s on-site activity with information about their offline interactions with the company — has been kicked around as a must-do for several years; the major web analytics vendors have made substantial headway in this area over the past few years
• Of late, analysts and vendors have started looking into the impact of social media and how actions that customers and prospects take online, but not on the company’s web site, play a role in the buying process and generate analyzable data in the process

The “traditional” web analytics vendors (Omniture, Webtrends, and the like) were, I think, a little late realizing that social media monitoring and measurement was going to turn into a big deal. To their credit, they were just getting to the point where their platforms were opening up enough that CRM and data warehouse integration was practical. I don’t have inside information, but my speculation is that they viewed social media monitoring more as an extension of traditional marketing and media research companies that as an adjacency to their core business that they should consider exploring themselves. In some sense, they were right, as Nielsen, J.D. Power and Associates (through acquisition), Dow Jones, and TNS Media Group all rolled out social media monitoring platforms or services fairly early on. But, the door was also opened for a number of upstarts: Biz360, Radian6, Alterian/Techrigy/SM2, Crimson Hexagon, and others whom I’m sure I’ve left off this quick list. The traditional web analytics vendors have since come to the party through partnerships — leveraging the same integration APIs and capabilities that they developed to integrate with their customers’ internal systems to integrate with these so-called listening platforms.

Somewhat fortuitously, a minor hashtag snafu hit Twitter in late July when #wa, which had settled in as the hashtag of choice for web analytics tweets was overrun by a spate of tweets about Washington state. Eric Peterson started a thread to kick around alternatives, and the community settled on #measure, which Eric documented on his blog. I like the change for two reasons (notwithstanding those five precious characters that were lost in the process):

1. As Eric pointed out, measurement is the foundation of analysis — I agree!
2. “Web analytics,” which really means “website analytics,” is too narrow for what analysts need to be doing

I had a brief chat with a co-worker on the subject last week, and he told me that he has increasingly been thinking of his work as “digital analytics” rather than “web analytics,” which I liked as well.

It occurred to me that we’re really now facing two fundamental dimensions when it comes to where our customers (and potential customers) are interacting with our brand:

• Online or offline — our website, our competitors’ websites, Facebook, blogs, and Twitter are all examples of where relevant digital (online) activities occur, while phone calls, tradeshows, user conferences, and peer discussions are all examples of analog (offline) activities
• On-site or off-site — this is a bit of a misnomer, but I haven’t figured out the right words yet. But, it really means that customers can interact with the company directly, or, they can have interactions with the company’s brand through non-company channels

Pictorially, it looks something like this:

I’ve filled in the boxes with broad descriptions of what sort of tools/systems actually collect the data from interactions that happen in each space. My claim is that any analyst who is expecting to deliver meaningful insight for his company needs to understand all four of these quadrants and know how to detect relevant signals that are occuring in them.

What do you think?Similar Posts:

• WAW(T) Columbus / Social Media Tools for Web Analysts
• Columbus WAW — ExactTarget, CRM, Web Analytics, Google…Coupons/Chrome/Ad Manager, and More!
• 40 Million Reasons Your Customer Data Isn’t As Current as You Think (or Hope)
• Columbus Web Analytics Wednesday Meets #fiestamovement
• Social Media ROI: Stop the Insanity!

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 3 comments | Add to del.icio.us
Post tags: Crimson Hexagon, CRM, Cymfony, digital analytics, GAP Research, J.D. Power, John Grono, Nielsen, Omniture, SEM, SEO, SM2, Techrigy, WebTrends

Feed enhanced by Better Feed from Ozh

• status quo — the way things are, as opposed to the way they could be
• inertia — The property of a body that resists any change to its uniform motion
• cognitive dissonance — a conflict or anxiety resulting from inconsistencies between one’s beliefs and one’s actions or other beliefs

The first two of these can be applied to any sort of technology or process change being introduced to an organization — entire careers and companies are built around trying to figure out how to effectively drive change within organizations. In the case of data management, the third defintion — cognitive dissonance — comes into play as well.

As a brilliant and phenomenally handsome man* once said, “Customers are people, and people are messy.” Customer data is inherently incomplete and imperfect. Any process or system that captures and stores customer data stores flawed data as soon as it rolls out for two reasons:

• It is not reasonable to add to any process all of the overhead required to rigorously capture and validate all attributes of a customer — it’s a balancing act between the efficiency of the process and the quality of the data captured
• Customer data decays, and it decays a lot more quickly than we like to admit; customer data maintenance tends to be an afterthought that gets addressed only after time has degraded the data to the point that it starts causing the company real problems

Once we hit the point where we really need to tackle our customer data management head on, we have two options, of which one option is completely inviable:

1. Throw out all of our customer data, customer data processes, and customer data systems and start over, but “do it right this time”
2. Identify the most broken parts of our processes and start fixing them — going after the lowest cost and highest benefit ones first and then working our way down the list until we hit a satisfactory point (which is, typically, never)

Clearly, the first option is not an option. No company would survive if they tossed out their customer base, barred their doors, and conducted no business for a year or two while they rebuilt their process and technology infrastructure.

That leaves us with the second option (technically, “do nothing” is an option as well, but that’s only an option if the pain hasn’t reach the point where it’s not an option!), and, thus, we reach a cognitive dissonance conundrum:

We know our customer data is dirty — customer service reps complain about the number of duplicate records in their systems, sales reps complain of the incomplete pictures they have of their customers (which hinders their ability to prep for and conduct customer visits), marketing complains that they can’t effectively segment and target their database because the customer data is bad, customers complain because the company keeps screwing things up in one way or another…

BUT

…as we start to explore and design replacement processes, we realize that these processes are going to be inherently imperfect, too. We may accept that the new process will be better (even significantly so), but we obsess about the flaws.

We don’t want to repeat the mistakes of the past and roll out something that is not bulletproof — a chink in the data management armor is a chink, no matter how small. So we obsess about the chinks. We propose process changes to accomodate the identified gaps. Even for the gaps that are purely theoretical (”yes, I see, but what if the poles reversed at the exact same point that pigs learned to fly — the process would break!”) We’re trying to do the right thing. We’re aiming for perfection — for a flawless process.

But we’re talking about customer data, and customers are people, and people are messy.

We find ourselves (and/or the people who will ultimately need to adopt the process) paralyzed, caught in an endless cycle of Visio vetting and process rework, perpetually getting halfway to the “perfect” process, but never actually getting there. At some point, due to impatience or frustration, someone stands up and yells, “Enough! Just build what you’ve got!”

And then we realize we’ve designed a process that is so complex and unwieldy that the cost to implement it would wipe out any hope of the company having a profitable year for the ensuing decade.

Of course you’d like a more tangible example:

Let’s say we’re trying to clean up our customer’s mailing addresses (which, thankfully, is now an exercise from my past, but that’s more a digression for a discussion over drinks than for a blog post!). Let’s say that, for any 1,000 customer addresses, we have conclusively demonstrated that at least 50 of them are bad — the postal service is going to struggle to deliver mail sent to them, and the postal service is going to fail more often than not. Now, let’s also say that we’ve demonstrated that, by introducing some automated cleansing processes, we can: 1) identify those 50 addresses, 2) “fix” 30 of them, and 3) flag the remaining 20 as being known problems that need some sort of manual intervention. Let’s say that, rather than 1,000 records, we’re talking about 10 million.

“Hurray!”

“Sounds great!”

“Awesome!”

“Gimme some of that!”

Ah…BUT…

…we have also  determined that, as part of those automated cleansing processes, we might actually take 1 of the 950 addresses that were already good…and make it worse.

Logically, the project should still be a go. We’re making 30 addresses better and only might be making a single address worse!

Ohhhhh…that single address. That molehill that eats its Wheaties, regularly applies cream provided by a shady character, and injects itself in the buttocks with a substance its cousin purchased over the counter in the Dominican Republic. The molehill grows. It grows quickly. Suspiciously quickly…yet no one seems to notice. It becomes a hill, and then a big hill, and then a mountain! The project manager is left scratching his head and wondering how a theoretical aside in a meeting three weeks ago has now become a virtually insurmountable issue that has put the entire project at risk of ever being implemented!

Cognitive dissonance — simultaneously recognizing that things are bad and must be fixed, but also accepting that the status quo is “right.”

The answer? I’d like to say it’s just a matter of putting the dissonant perspectives side by side and forcing objectors to reconcile them. That should work, right?

Alas!

As it happens, the current debate about healthcare reform in the U.S. prompted James Surowiecki to right a column on Status-Quo Anxiety in The New Yorker a couple of weeks ago. Surowiecki discusses the “endowment effect:”

“…the mere fact that you own something leads you to overvalue it. A simple demonstration of this was an experiment in which some students in a class were given coffee mugs emblazoned with their school’s logo and asked how much they would demand to sell them, while others in the class were asked how much they would pay to buy them. Instead of valuing the mugs similarly, the new owners of the mugs demanded more than twice as much as the buyers were willing to pay.”

Surowiecki goes on to relate this effect to the healthcare debate: “What this suggests about health care is that, if people have insurance, most will value it highly, no matter how flawed the current system.”

The same applies to customer data management all too often — we know we have a flawed system, but it’s the system we have, gosh darn it, and I don’t want your new system if I can find any imperfections in it!

This really has been a farewell post of sorts. Rambling, yes. Academic, yes. Lacking any prescriptive solution. But, hopefully at least a little entertaining, and maybe even with an insight or two that may come in handy to you. Look for a topical shift to measuring digital media going forward.

So long, and thanks for the fish!

* Dramatic license — I said that in this post, and “brilliant and phenomenally handsome” is perhaps a bit of an overstatement.

Photos courtesy of Olof S and bobster855

Similar Posts:

• Type I vs. Type II Errors in Customer Data Management
• Web Analytics Wednesday: A Segmentation Experiment
• Data Visualization — March Madness Style
• Reporting vs. Analysis — not just me making the distinction
• Quick Excel Tip: The FASTEST Way to Sum Numbers

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | No comment | Add to del.icio.us
Post tags: cognitive dissonance, intertia, James Surowiecki, status quo

Feed enhanced by Better Feed from Ozh

• I changed jobs — left Nationwide to become Director, Measurement and Analytics at Resource Interactive — which is 1000% the “right” move, but meant for a hectic/stressful month
• Back-to-school time, which was more than just getting our kids ready — my wife ran our two sons’ elementary school’s entire supply sale…and my “I’ll show you a few tricks in Excel to help you stay organized” offer morphed into a full-blown custom ERP system built in MS Access; August was the month when all the supplies arrived (think almost 10,000 no. 2 pencils…) and had to be divvied up; I did no divvying, but there were a number of late-breaking report requests; at last count, the database had over 20 tables (it’s almost a fully denormalized database), over 40 queries, 12 forms, and 20+ reports; AND…it’s now been extended to also handle the production of the school’s student directory; gotta love MS Access!
• Company, company, company — two visits from friends in Texas, two visits from my parents, a visit from my in-laws, and my mother-in-law moved in for six weeks to convalesce from surgery…all in a 3-week period in August

I’ve got one more good customer data management post in me that needs to get written, at which point I expect to be shifting over to more web analytics-y, social media measurement-y posts going forward.

And…as I played around with Drupal for a couple of projects over the past couple of months, I realized that the theme that I settled on after weeks of experimentation on this blog…is one that was built for Wordpress to mimic one of the Drupal default themes! How embarrassing!

Please be patient! My life will settle back down soon (I hope). In the meantime, if you’re going to be in Columbus in the middle of September, consider stopping by this month’s Web Analytics Wednesday on September 16th!Similar Posts:

• Getting Started with Master Data Management (MDM)
• Google Analytics = Strawberry?
• A Little Bit of Data Can Be a Time-Consuming Thing
• Dashboard Design Part 1 of 3: An Iterative Tale
• Columbus Web Analytics Wednesday — Jan 2009 Edition

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | 3 comments | Add to del.icio.us
Post tags: Drupal, MS Access, Nationwide, Resource Interactive

Feed enhanced by Better Feed from Ozh

Let’s Start with Definitions

Wikipedia has a nice write-up that goes into much deeper detail, but here are my quick and crude definitions:

• Type I Error = α (Alpha) Error = False Positive = you mistakenly judge something to be true that, in fact, is false
• Type II Error = β (Beta) Error = False Negative = you mistakenly judge something to be false that, in fact, is true

It’s easy to get mired in statistics-speak and make a snap judgment that this distinction between types of data errors are solely of interest and use to statisticians. That’s not the case. Depending on the situation, one type can be critical while the other can be barely consequential.

Customer Data Example: Automated Customer Merges

Most companies of any size battle duplicates in their customer data. I have yet to work at or with a company where the sales force and call center staff don’t complain about the number of duplicates that exist in internal systems. In this situation, it’s pretty common to try to automate a deduplication routine of some sort (some would say that this is the biggest benefit of a customer data integration, or CDI, initiative).

Unfortunately, customer data is messy. For every “identical match” in the system, there are generally 5 to 10 “probable matches.” After all, if the data was truly identical, then the likelihood that a duplicate would have gotten created in the first place would have been greatly diminished! That means that the automated matching system has to apply various business rules to determine which matches are true matches and which ones are not — the system makes an educated guess. Many CDI systems apply a scoring system — the higher the score, the more likely the two records are a true match. The lower the score, the less likely. It’s then up to the user to establish the threshold above which the records will automatically be merged (and, possibly, a lower threshold that will trigger a manual review of the records by a human being).

Where that threshold gets set is purely a Type I vs. Type II error decision:

• The higher the threshold gets set, the greater the likelihood of a Type II error — a false negative where two records could have been merged because they were, in fact, duplicates, but they were not identified as such
• The lower the threshold gets set, the greater the likelihood of a Type I error — a false positive where two records get merged, even though, in reality, they represent two different customers

It would be wonderful if the matching logic was such that a threshold existed that eliminated all errors, but no such threshold will ever exist. The question then becomes: which type of error is “more” acceptable? That will influence where you set your threshold. It’s a situational question! For example:

• If your customer data includes extremely sensitive information (think medical records, social security numbers, credit card numbers), then you will want to err on the side of Type II / False Negative errors — deal with the messiness of more duplicates and put the burden on your call center / sales force to trigger merges if and only if they have confirmed the merge needs to happen through a manual inspection and/or an interaction directly with the customer
• If, on the other hand, you are only using the post-merged customer data to send marketing promotions to the customers, then the added cost of the direct mail may push you to err on the side of Type I / False Positive errors — some of your customers may miss out on some marketing promotions, but very few of them will receive multiple copies of the same direct mail piece, and your overall postage costs will be lower

This is a very real example — a single company may use the former mentality as its core matching logic, but then maintain a separate data store with more aggressive matching for purposes where Type I errors are not critical.

Type I and Type II in the News

A friend and former colleague shared a recent story in The Washington Post about the Social Security Administration running into trouble when trying to deny social security benefits to certain felons. It sounds like there were some technical gaffs, for sure — when people with minor offenses on their record got caught up in the denial of benefits. Overall, this reads like it was a case of an unacceptable number of Type I errors — people being denied benefits because they were mis-identified as “fleeing felons” (the article reads like some of the stories of people being banned from flying because they had the same name as someone on the terrorist watch list).

Both types of errors will occur. As important as being clear on which type are “better” given the situation (and developing your processes accordingly), ensuring that you have processes in place to fix the errors when they do occur and are identified is just as critical — something that the federal government was apparently missing in this case! The federal government? Lacking in customer service? Shocker!

So, there you have it — more thought than you ever wanted to see on Type I and Type II errors. I also promised in my earlier post that a future post would cover my observations on cognitive dissonance and the status quo when it comes to customer data management, but that’s going to have to wait for another day!Similar Posts:

• Fear vs. Convenience — The Customer Data Conundrum
• Harvey Balls: A Good Way to Ramp Back Up
• 40 Million Reasons Your Customer Data Isn’t As Current as You Think (or Hope)
• More Data Is Better
• Performance Measurement vs. Analysis

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | No comment | Add to del.icio.us
Post tags: CDI, customer data integration, false negative, false positive, Type I error, Type II error

Feed enhanced by Better Feed from Ozh

From Monish’s tweets, a handful of us knew he had gone vegetarian a few weeks ago, and no one (Monish included) knew how he would fare at a barbecue joint. As it turned out, he ordered fish, which he said was good enough to count as staying on the veggie wagon. Laura Thieme asked if Monish had actually tweeted about what he was eating. Indeed, he has!

Although I wasn’t cognizant of it as I was reading his tweets, Monish has provided fodder to Twitter critics who equate “tweeting about what you eat” to public navel-gazing. The question is: will he now become self-conscious about it, or, rather, will he go to the other extreme and provide detail at every meal? I’m sorta’ hoping for the latter.

Yes, There Was Actually a Topic Beyond Monish’s Diet and Twitter Usage

This month’s event was sponsored by IQ Workforce. Corry Prohens made the trip to central Ohio to present on what he is seeing on the jobs/careers front for web analysts and search marketers.

Highlights of the presentation included:

• The Good News: internet retailers and online advertising have both continued to grow throughout the current economic downturn
• The Bad News: there are more people chasing fewer full-time web analytics jobs, and there are fewer remote/virtual office positions and less willingness/need on the part of companies to relocate candidates to fill positions
• Many people find their way into web analytics as a complement to another role: SEM, SEO, digital media analytics, offline marketing analytics, research/qualitative analysis, BI, etc. Having this complementary skillset clearly identified and articulated can be useful in a job search
• Contracting has its pros (high demand, usually more money, more flexibility, less politics) and its cons (the need to always be selling, limited “depth” with any project/company, travel often required, administrative headaches, and it can be hard to go back to a non-contracting role)
• When trying to hire a web analyst, work directly with the recruiter (don’t use an HR intermediary), write the job description yourself, be clear as to “needs” vs. “wants,” and choose a recruiter with expertise in the area you are hiring

A handful of resources that Corry provided for anyone who is looking to more effectively manage their career in web analytics:

• Job board consolidators such as Indeed and Simply Hired
• IQ Blast — the excellent e-newsletter that IQ Workforce produces
• The Web Analytics Association
• Web Analytics Demystified (Eric Peterson’s company)
• Newsletters (besides IQ Blast):
  • Internet Retailer
  • WebGuild Searchnomics (Dave Culbertson was floored that he and Corry were the only two people in attendance who were subscribers to this one)
  • ClickZ

My one addition would be WebSight, Stratigent’s monthly newsletter.

Attendees from Far and Wide

As I wandered around the room chatting with attendees, I realized we probably had our widest geographic reach of any of our Columbus WAWs to date:

• Portland, Oregon — Noé Garcia from Webtrends (a Columbus WAW sponsor several times over)
• Cincinnati, Ohio — Cindy Rabkin from Web Tech Analytics
• Philadelphia, Pennsylvania — Steve Colon from Webtrends
• Long Island, New York — Corry Prohens from IQ Workforce
• Durham, North Carolina — several attendees from McKinney

Locally, we had had attendees from Acappella, AOL, Bizresearch, Cardinal Solutions, Cisco, Highlights for Children, JPMorgan Chase, Lightbulb Interactive, Nationwide, Ohio Historical Society, Real Estates’s SEO, Resource Interactive, Victoria’s Secret, and a few others that I probably missed (note to self: a sign-in sheet would really be helpful!)

We had a good set of mingling/mixing before and after:

Cheng Deng and Todd Greene

Dave Culbertson and Steve Krause

Monish Datta, Karen Schneider, Bill Carey, Jen Elliott, Andrew Blank, and Steve Colon

Noé Garcia and Eric Moretti

Franklin Gbenah and Carol Fleming

What’s Next?

Our next WAW is tentatively slated for September 16, 2009, but I have yet to secure a sponsor. Let me know if you can help on that front — I’ll pursue any lead!Similar Posts:

• A Record-Setting Web Analytics Wednesday in Columbus
• Web Analytics Wednesday: A Segmentation Experiment
• SEO Tips and Thoughts at Web Analytics Wednesday
• Another Great Web Analytics Wednesday in Columbus
• Monish Datta: “It was the best WAW yet!”*

© Tim for Gilligan on Data by Tim Wilson, 2009. | Permalink | One comment | Add to del.icio.us
Post tags: Barley's Smokehouse and Brewpub, career management, IQ Blast, IQ Workforce, WAW, WAW Columbus, web analytics wednesday

Feed enhanced by Better Feed from Ozh