Web Analytics Analyzed

An Alternative to the Web Analyst Superman

pstrupp — Sat, 31 Oct 2009 13:33:17 -0700

An Alternative to the Web Analyst Superman
Seems like the job description for a web analyst continues to inflate and is now almost beyond realistic expectations for most mere mortals. I haven't seen a criteria describing command of antigravity forces, or the ability to turn oneself into a gigantic killer robot with fire shooting appendages yet, although it will probably happen soon.

However, it is commonplace to see expectations for technical command of web analytics tools, data mining capability, search engine optimization, statistical analysis, ability to find insights in web data, and business acumen and leadership to drive optimization of user experience and web design for bottom line results. Whew. No wonder it's hard to find and retain this type of talent.

In the Web Analytics group I manage we have taken a somewhat different approach. Regardless of the incredible individual talent and hard work of the folks I work with, we have seen most success with the team approach. One person can't possibly have the depth and breadth of knowledge, visibility, and influence across the business to be the superman (or woman). The alternative is to partner up and leverage everyone's talent to achieve the goal.

The key to making this work is the pairing of the web analyst with the business owner. The business owner knows the goals and the specifics and has the power or leverage to make needed changes. The web analyst knows how to engage with the business owner to understand what they really need to know, and can then go turn the knobs to get the data and interpret it and communicate it in the right context. Or if the data or technology is beyond the analyst's individual capability, he/she knows what other specialists to call on, and can translate the need into yet another sphere of specialty.

So rather than the superman, the web analyst becomes the super-translator.

Now if I can just find that gigantic killer robot to help me get a better parking spot in the morning...

Cookie Retention Rates

pstrupp — Mon, 24 Aug 2009 10:02:03 -0700

I've decided that I no longer care about cookie deletion.

I now care about cookie retention. It's more valuable from a business stand point to know how long users keep their cookies than to know how frequently they delete them. After all, if you are trying to say something about long term conversion rates, or other customer behavior that drags on for a while, it's easier to think in the positive. At least, the cow playing the ukulele in my head has been much happier since I started looking at the data this way.

Below is a chart that shows what percent of users returning after X days on the same computer still have the same cookie.

The data shows a very sharp drop off after just a few days (72% retention after 7 days) with a slowly decreasing rate of decay as the time span gets longer. After a year the cookie retention rate levels out at around 20%.

This means that if a user is going to delete their cookie they do it fairly quickly. After about 30 days the rate of decay becomes much slower suggesting a different mechanism is at play. One can suppose that the daily, weekly, and monthly deletion mechanisms are more deliberate, probably cookie deletion software and manual deletion. The longer term decay could be less deliberate, nonsystematic mechanisms, like accidental deletion or deletion for some isolated specific cause like fixing one's browser settings to repair a technical problem.

This data is important to understand when considering business processes that take a long time. Consider, for example, a B2B conversion process that takes six months from initial click through to closing a deal. The highest conversion rate you could hope to measure would be around 35%. In other words, if 100% of the users who clicked on your ad on January 1, made a purchase six months later on July 1, your measured conversion rate would be around 35% because that's how many users still have the same cookie.

Conversely, if your measured conversion rate after 180 days was 10%, then the actual conversion rate was more like 10%/0.35 = 29%.

Now, you would have to be pretty brave to stand up in front of a room and actually scale up your conversion rates based on cookie retention rates, but the point is that you can at least point in the right direction as to what your real conversion rates might be.

This finding is also important for optimizing intelligent systems which deliver personalized content to users based on anonymous cookie (non logged in) behavior. Basically, any learning mechanism which takes more than 30 days to fine tune recommended content for the end user is largely shooting in the dark.

How Did We Make this Calculation?

The data is based on a first party metrics cookie on a software developer forums site. It is the same data that I've been discussing in my last few blog posts.

Each data point in the chart is calculated as follows.

For 1 day elapsed time, we compared data from January 2 to January 1. We identified logged in user names who had logged in on Jan 2 and Jan 1. We kept just return user names who had the same IP, OS and browser to try to limit the sample to return users on the same computer.

Then we compared for the two dates, the visitor IDs for each return user name and counted how many were unchanged.

We repeat that calculation for another 1 day elapsed time increment of Jan 3-Jan 2. Repeat again for the segment of Jan 4-Jan 3, and so on to create 364 one day segments for the year.

We sum the total number of return user names and return user visitorIDs, respectively, of the 364 individual segments. Cookie retention for 1 day elapsed time is then calculated as the ratio of the number of return visitor IDs divided by the number of return user names.

Can you see where this is going?

We start over to calculate the retention rate for 2 day elapsed time. Take the data from Jan 3 and compare to Jan 1. Count return users names and visitor IDs. Take data from Jan 4 and Jan 2. Count. Repeat this 363 times, add up the segments and take the ratio. Record that as the cookie retention for 2 day elapsed time.

Etc.

Sounds like a lot of calculations, and it is. But that's what student interns are for. I was fortunate to have a brilliant young intern majoring in applied math, Garrett Clark, who did this work and deserves most of the credit for the results in this posting as well as the earlier cookie deletion data I posted in my blog. He tells me that he was up past midnight every night doing these calculations by hand, but I think he's lying. I'm pretty sure he used some clever programming, MySQL, and spreadsheet tricks to automate the calculations. He may have indeed been up past midnight every night, but it was more likely at the local brew pub.

As we (well, Garrett) iterated the calculation for longer elapsed times, the sample sizes decrease which leads to increased scatter in the data. For example, for 1 day elapsed time the sample size was 3372 return user names, but after 180 days the sample size had dropped to 19.

The sample size of return users gets smaller as you analyze longer elapsed times because fewer users are likely to return for the longer elapsed times. In addition, there are fewer segments available to provide data. A 10 day elapsed time has 365-10=355 data segments while the 360 day elapsed time only has 365-360=5 segments.

You'll also notice that the fit line superimposed on the data does not look like a smooth function. It isn't. We weren't able to find a single function which fit the data well across the entire time frame, so the line is mainly there just to guide the eye rather than imply a true fit. It makes sense, though, that more than one function would be needed to fit the data because, as supposed above, there are likely different mechanisms corresponding to different user behavior modes underlying the data.

It would be interesting to see how this curve might be different for different audiences. This audience is extremely technical. It's often been supposed that technical and consumer audiences behave differently and this method would be a good way to compare them.

Effects of Moving to a First Party Cookie

pstrupp — Mon, 27 Apr 2009 19:15:54 -0700

Picking up where I left off yesterday (okay, three months ago, but who's counting)...

We took the plunge and made the switch from the 2o7.net cookie to a first party sun.com measurement cookie. Other than the self-serving benefits of improved data, it seemed like an improvement to have a browser image request go to metrics.sun.com rather than the mysterious 2o7.net.

The outcome was an improvement in several data quality metrics.

The correction factor for monthly unique users improved from 78% to 83% (see last entry for what this means).
Percent of users who delete their measurement cookie at least once per month decreased from 20% to 14%.
Percent of users who block the measurement cookie dropped from about 5% to less than 1%.

As part of this analysis we also backed out what percent of our users use more than one computer in a month to visit our site: 30%. Of course, this was unaffected by the cookie change. (It would have been remarkable if it had been. Its consistency was a bit of a control in our calculations.)

Accurately determining the cookie deletion rate and multiple computer use is an inexact science based on comparison of IP addresses, user O/S and browser version profiles for a given visitor ID. So, I don't bet my life on the accuracy of these breakdowns. Only the correction factor can be accurately measured; the constituent mechanisms behind it are slipperier to divine.

We've watched these numbers over nine months now since we made the change, and although they do vary a little from month to month, we have not seen a definite trend either up or down in any of them.

My assessment is that it was worthwhile to make the switch from third to first party cookies, although I would characterize the improvement in quality of data as modest. The observed difference between deletion rates for first and third party cookies is not dramatic. Add to that the fact that much of the error is driven by multiple computer use, and the bottom line ends up being that the change is favorable, but not eye popping.

Nevertheless, I'm much more confident now having evaluated these parameters for myself on our own site. I no longer have to rely on published data from other sources which don't transfer to our situation, and I am in a better position to defend the level of accuracy in our data.

Unique User Measurement Error

pstrupp — Mon, 26 Jan 2009 08:12:53 -0800

My father used to tell me, "Son, never confuse monthly unique users with people."

Well, actually, he never said anything remotely like that as he was a baker by trade and never touched a computer in his life. But he did used to tell me never to mix yeast and salt, if that is of any use to you.

Unique users, of course, are not necessarily people. They are simply counts of unique cookies (more precisely, unique visitor IDs) who visited your web site. If people tend to use multiple computers to visit your web site, or delete their cookies, your unique user count will be inflated relative to the actual audience size. (Stop me when I say something new!)

So, I figured it was high time we go measure this on our own Web sites and see how wrong our data is. Fortunately, the analysis process is fairly straightforward. You need a site that requires user log in. You count the number of unique login IDs in a month, and divide it by the number of monthly unique users from your Web analytics tool for that same month. This ratio, unique login IDs/unique users, is what I like to call the unique user correction factor.

To be precise, I'm ignoring the effect of multiple people sharing a single login ID, or a single person having multiple login IDs. So, sue me.

On one of Sun's developer sites I measured a correction factor of 78%, meaning that I should take my monthly unique user measurement and multiply it by 78% to get a better estimate of my true audience size. This, by the way, was for a site which at the time was using the 2o7.net cookie, but more about that later.

I was a bit surprised that the error was not bigger. The well known comScore study of a couple years ago found correction factors more like 40% on the sites they studied, i.e. a much more serious matter.

The thing is, of course, is that you can't really extend the comScore results (nor my results) to your own web site. This is because the correction factor is a complicated function of cookie deletion, multiple computer use, and visit return frequency for users on your site. These factors will likely be different for your web site than other web sites.

Just to illustrate, if your user population deletes their cookies every day, but visit your web site only once per month, your monthly unique users measurement will be unaffected. Or, if your audience visits your site every day, but never deletes their cookies, your data will also be accurate.

The error gets big when users delete frequently and return frequently. (Similarly, if they use multiple computers and return frequently, your error will also be larger.) Imagine one guy who visits your site every hour and deletes his cookies between every visit. He is only one person, but will single handedly rack up erroneously huge user counts.

The message here is that you need to measure this for yourself so you can defend your own data within your own company.

The next question is what happens if we switch from the 2o7.net cookie to a first party cookie? I'll discuss that in my next entry.

MVT That Matters

pstrupp — Mon, 8 Dec 2008 07:44:08 -0800

I'll admit that the extent of my research on this topic consisted of stumbling across a blurb in Esquire Magazine somewhere between articles about the Best Steakhouse in America and the Sexiest Woman alive. Nevertheless, learning about the research of Esther Duflo of MIT brought into interesting context for me the hot topic of MVT in Web Analytics.

Esther Duflo is a professor of economics at the Poverty Action Lab at MIT. At the risk of oversimplifying her research, basically what she does is apply multivariate testing to economic aid for impoverished countries. Paraphrasing Tim Heffernan's Esquire article, say you spend a million dollars in Africa supplying textbooks, or treating students for parasitic worms. Which investment is more successful at improving academic performance?

Turns out the answer is treating for worms so that students attendance improves and thus they can learn more, even with outdated or no text books.

To quote the Esquire article summarizing the aid problem, "When you apply for funds to keep your project going next year, all you can say is...money helps. So please give us more of it." To address this problem, Professor Duflo and her team are applying data analysis to large scale, complicated economic and societal issues to cut through arguments over where funding for economic aid should be spent.

Sound familar?

So, if they can apply MVT for big problems like this that really matter, seems like running some simple A/B tests to improve my calls to action on my Web site should be something I can pull off.

Emetrics Summit October 2008

pstrupp — Thu, 23 Oct 2008 15:23:38 -0700

I was fortunate to attend the Emetrics Summit this week in Washington, DC. As always, it was a great conference. Here are my general impressions.

An unavoidable theme was concern over the economy. Attendance seemed a bit down, and many conversations included some sighs and rolling of eyes about upcoming budgets and staffing. Other than those happy thoughts, there was much of interest.

Major themes I picked up on were:

Where are the case studies and success stories of major business decisions being driven or influenced by Web Analytics results? Plenty of examples of web optimization based on WA data, but has this work become a significant driver of the core business?

The "Web" in Web Analytics is fading with the topic being more simply "Analytics", i.e. Marketing Analytics or Business Intelligence, or whatever you want to call it. It is getting harder to put a fence around "Web Analytics" as it is just part of the marketing data ecosystem.

There was an interesting panel discussion about privacy issues: pending and potential litigation; inconsistencies between privacy in different environments (e.g. people willing to share everything on their Facebook page, but also deleting cookies to preserve their privacy); privacy vs. value to the customer (no problem if my grocery store tracks everything I buy as long as I get 50 cents off mayonnaise, but don't you dare give me a metrics cookie as I get no value from it). Policy and standards development here is slow coming, but still seems like the elephant in the room.

Seems to me to be a flood of A/B and MVT vendors and solutions. Seems everybody was touting their version of a solution. Yet, I did not hear too many examples in conversations about successful MVT testing. Maybe I was just talking to the wrong people, or went to the wrong talks. Or maybe it is harder than people are willing to admit.

I noticed more attendees from the public sector (e.g. government) than in the past. Individual conversations with these folks is always mind shifting to a person in the private sector as they have special challenges around quantifying and measuring their goals which are not as simple as, say, counting revenue. How exactly does one measure, for example, if one's Web site has contributed to responsible policy formation for United States energy resources. I would like to see a pubic sector topic raised to keynote presentation status at a future Emetrics Summit.

More talks than I've seen in the past about artificial intelligence. Still seems a bit on the forward edge, but quite a bit of interest at the talks.

Speaking of forward edge, went to a talk on metrics in the virtual world, e.g. Second Life. For starters, I would probably be corrected for calling it a virtual world, but my ignorance aside, the talk made some interesting points about the ability to build brand with avatars, challenges this space is having as it matures, and an assertion, probably true, that this will be an environment which will require involvement from the marketing and measurement community as a matter of course before long.

Please Enter Though the Garage

pstrupp — Fri, 11 Apr 2008 14:25:01 -0700

Quick! What percent of your web traffic enters though your home page?

Wrong.

Take a close look at the entry pages to your web presence and see what percent of your visitors actually enter on your home page. I bet it's a lot less than you would assume before looking at the data, especially if your site is for a large enterprise.

Why is this? Because home pages are like the front door of your house. It's where strangers and people who have never been there before probably go. But friends, frequent visitors--they come to the side or back door because they know that's the best way to get inside. The quickest way to the kitchen where you keep the beer.

Your front enterence should make a good impression, of course, and be welcoming to visitors.

But it's an easy pitfall to put too much emphasis on your home page, and overlook what most of your customers (your best customers) are probably doing.

Are You Smarter Than Homer Simpson

pstrupp — Thu, 7 Feb 2008 10:22:57 -0800

My favorite episode of The Simpsons has a scene in which Homer goes into the witness protection program and the agents have to coach him to learn that he now has a new name. After many frustrating hours of them trying to get him to respond to his new alias, Homer leans over and whispers to the agent sitting next to him, "I think he's talking to you."

Sometimes I'm not sure I catch on a lot quicker than Homer.

My group and I spend tremendous effort in Sun trying to enable the latest and greatest Web analytics measurements-- custom events and special variables and bells and whistles to try to make campaign measurement more insightful or efficient. Marketers and web area owners are constantly asking for more and increasingly complicated measurement features so they can optimize their area of the business better. We try to please. Sometimes we do.

But then the CEO pops out a 10 second email asking a simple thing like how many downloads of X we had over the last two years. Frenzy! Panic! Fire drill teams spring into action! I can readily tell you the clickpaths and fall out rates of users coming from search key word "xVM" last month in Germany vs. Kazakhstan. But I'm not ready to answer a simple question from the CEO.

Thus, the lesson that I am as slow to learn as Homer was his new name. Do the simple stuff first. Do it well. Make sure it works. Make sure it's right.

Actionable Metrics

pstrupp — Fri, 16 Nov 2007 08:34:41 -0800

We're working on redesigning our monthly Emetrics report.

Every so often it needs to get pruned down, then built back up with the latest, greatest metrics. (I think that's a mixed metaphor.) Corporate and department goals and priorities change and evolve, and metrics reports need to do the same.

As part of the redesign I've asked one of my senior analysts to propose some new metrics. Using best practices she's been very conscientious about including only "actionable" metrics. However, as we progress through the process, and think about the audience that will be reviewing the report (Directors and VPs) I wonder what "actionable" really means at that level?

Online revenue is a great example. I can't imagine not having this metric in the report, but how actionable is it really? If revenue is down, you certainly want to take action, but I suspect the action would be "Paul, go figure out why revenue is down!".

I think the right answer will be that some metrics are "bottom line" metrics and need to be included, but they should be surrounded by actionable metrics which tell me what to do if the bottom line (or top line in this case) slips.

Flouting My Ignorance

pstrupp — Thu, 27 Sep 2007 09:51:11 -0700

I was interviewed by the Web Analytics Association a few months ago about Sun's efforts to measure Blogs.

You can find a podcast of this interview at

http://www.webanalyticsassociation.org/en/art/?433

in which I think I manage to say a few insightful things, a lot of maybe not so insightful things, and hopefully no outright bone headed things.

At any rate, if we were to run into each other at a conference and you asked me what Sun is doing about measuring Blogs, this is how the conversation would have gone.

Many thanks to Jennifer Day who conducted the interview and was patient and fun to engage with, and Jim Humphrys of the Research Committee of the WAA for inviting me.

Cold Fusion Moments

pstrupp — Wed, 29 Aug 2007 15:58:51 -0700

I can't count how many times in grad school I discovered the equivalent of Cold Fusion.

It was a fairly regular habit of mine to slip under my professor's door late at night a chart displaying freshly acquired data that would change the world, complete with an elaborate explanation, lots of exclamation points, and the opening sentences of my Nobel prize acceptance speech.

I'd then walk home, blurry eyed and exhausted, but proud of my great discovery and looking forward to being showered with praises the next day.

The next day I'd arrive back at the lab and look at my notes from the previous night and shout "Oh shoot!" (or something very similar to that) realizing the night before I had a cable disconnected, or had misplaced a decimal point by three orders of magnitude. I'd retrieve the chart from the professor saying "Never mind!" which he never did because he had already figured out my mistake.

It is very easy to discover Cold Fusion buried in Web Analytics data. The key is knowing when to publish it, and when not to. In other words, if you find something very startling in your data, good or bad, as an analyst you need to be sure of your finding before you blow the horn and get everyone in the business worked up.

To that end, here are the rules of thumb I have learned to follow:

1. Find at least one other way to make the same measurement. Run collaborating reports to test and support your finding. The first comment will always be "There must be a problem with your data", so you better be prepared to respond to that.

2. If possible, explicitly test the measurement. Go through the user's experience yourself on the Web site and validate that the data is being produced as you assume it is. (Many times it is not. Surprise!)

3. Once you are confident in the data, anticipate the next two questions and start answering them. Once others believe your data they will want to know "Now what?", "Why is that?", or "How did this happen?". Pursuing these follow up questions not only helps the business take action, but also helps you build confidence in your original finding.

Bottom line is you need to do enough to be confident you're right, but not so much that you never share your results and take action. Discovering that balancing point is key to being a successful analyst.

Inaccurately Precise

pstrupp — Thu, 16 Aug 2007 07:31:59 -0700

Sometimes this job makes me crazy.

I was discussing some web analytics reports the other day with a colleague who was fairly new to analytics. He wanted to know what pages were referring traffic to his web page.

"Do you want to know what external sites sent you traffic or internal sites?"

"Well, both I suppose!", he responded.

So I explained that there were different reports for each.

We then went on to discuss external referrers that sent traffic directly to his page as an entry page, or referrers that sent users to his page after entering on another page and navigated to his page. And that search engines were a different question altogether. After a bit of going in circles I paused to explain to him that half the battle with analytics is precisely determining what you want to measure. There are many subtleties.

"Gee, it's amazing how accurate these tools are." he commented.

To which I had to explain , that, no, they are actually rather inaccurate given issues around JavaScript being disabled, blocking of third party images, blocking of cookies (but not all cookies), deleting some cookies (with some unknown frequency and probability), surfing with multiple browser tabs and windows and computers, improperly tagged sites, dropped tags, non-html content, RSS syndicated content, and on and on.

"We inaccurately measure very precise things!", I boasted with a heavy degree of schizophrenic pride and distain for my own profession.

I haven't heard from him since.

Quantum vs. Classical Web Analytics

pstrupp — Fri, 13 Apr 2007 14:48:39 -0700

Let me drag you through your science lesson for the day. I promise you won't need your white lab coat or safety goggles.

There are two basic scientific models of the world. The older one, dating back to Sir Issac Newton is commonly referred to as "Classical Mechanics" and describes the world in terms of continuums. That means that measures of the physical world can be infinitely and arbitrarily broken down into smaller and smaller units. A unit of energy, for example, can be divided in smaller and smaller pieces with no limit to how you divide it or how small you make it.

This model worked fine to describe things for hundreds of years, up until around the early 20th Century when scientists began observing phenomenon that the Classical model could not explain. They found that in reality, the world can not be broken down into arbitrarily small units, but rather, was better explained by a model consisting of small, discretely sized units, or "quanta". This became known as quantum mechanics.

So, what the heck does this have to do with Web Analytics?

Web Analytics is going the opposite direction. We are transitioning from Quantum Web Analytics to Classical Web Analytics.

The paradigm that Web Analytics has been based on is a "quantum" model of page views. The user experience has been conveniently described as a series of discrete steps, from page view to page view. However, this model is starting to break down. With the introduction of new rich internet applications (AJAX, videos, etc.) the user experience is no longer a herky-jerky succession of steps from one page to the next. It is becoming a smoother flow from one activity to the next--more of a continuum rather than a series of steps.

How this change in paradigm will ultimately impact the way we think about web analytics is yet to be seen. However, we might be able to draw upon hundreds of years of scientific models to provide guidance.

Metricspalooza Part 3

pstrupp — Thu, 12 Apr 2007 16:29:53 -0700

Number 5 of things I've learned managing an Emetrics Group.
The Three Headed Monster. My experience has shown me than an Emetrics Group has three major internal clients: Web Marketing, Web Design, and Web Management. To fully add value you need to effectively serve each of these clients. If you ignore one, the whole value proposition breaks down. Here's why.

My simple minded view of a Web business is like this. Step 1: Bring in lots of qualified prospects. Step 2: Convert them. Step 3: Count your money. (I should teach at Wharton.)

The internal clients, obviously, map to these three steps.

Web marketing's job is to bring in gobs of potential customers itching to buy your products. But doing so is worthless if when they get to your site, the Web design is so lousy that they can't accomplish (convert) their goals. And the accomplishments of both of these organizations is diminished if Web Management doesn't have proper visibility to the bottom line, and the things that lead up to it. If you put too much focus on one area while neglecting the others, you might as well not bother with any.

So, those are five things I've learned so far. Hope you've found them useful.

Metricspalooza Part 2

pstrupp — Fri, 30 Mar 2007 08:52:05 -0700

Was the suspense killing you? Here are two more things I've learned about managing an Emetrics group.

You Need a Customer for Your Analysis: I think there are different schools of thought about how to choose topics to investigate. One is to just start exploring the data and see what you can uncover. Identify those big new opportunities that had been unknown and then advocate for change based on this new intelligence. The other approach is to identify internally where the current business focus, existing resources and funding are and dive into that area to provide guidance and insight to make these existing efforts more successful.

Maybe I'm just too pragmatic, but I've seen more value come from analytics by supporting existing initiatives than trying to create and advocate new ones. I know that sometimes you need to search out the new ideas, but there is always (around here anyway) so much near by opportunity to contribute to and see immediate value that I tend to steer my group to working with people who are ready to take action and accept help.

Web Analytics is a Profession, not a Project: I think this fact has become rather obvious in the last few years with the creation of the Web Analytics Association and the boom of the industry.

My point, however, is to bring people into your group who have internalized this and are committed to the field. I like to tell people that "Nobody ever said Web Analytics is easy" which is rather ironic because people actually say it all the time. It's just that they're wrong. It's not easy and it takes a real commitment to learn how to be good at it. And if the people in your analytics group are not viewing this as a career, they are unlikely to last and be successful.

Metricspalooza

pstrupp — Sun, 25 Feb 2007 13:10:17 -0800

I was going to write an entry listing the top five things I've learned so far about managing a emetrics group, but I'm feeling kinda lazy, so I think I'll start with two. I can add the other three next time just to increase the suspense (and after I think of three more).

Punchin' Out Chryslers: The beginning of the month is a very busy time for my group. We have numerous reports that we need to produce, each one targeted for a different audience and containing mainly different data. And everybody wants their reports done right away.

Of course, there's only so much you can do all at the same time, and only so much you can automate. Sure, you can create pretty automated dashboards from web analytics tools, but not every piece of data spits out of such a tool, plus the real value is in taking the time to review the data, interpret it, and boil its essence down to a one sentence business friendly sound bite. No analytics tool will do that for you.

Thus, you need to think in terms of metrics production, like you were running a factory. Reports need to be production friendly, there needs to be schedules that everybody knows they need to meet, and all hands need to pitch in to spread the work around so it can be done quickly. And you have to keep looking for ways to make this all run like a better oiled machine because requirements keep changing and new metrics requests keep coming. Which leads me to point number two...

Prune The Tree: It has become apparent to me that people are good at asking for more metrics, but rarely volunteer to let you stop producing any.

There is a constant desire to know the next thing, to know more, to follow the latest new project or initiative. Quite quickly a five page report becomes fifteen and too much for any reasonable person to read (and a production burden...see Point One above).

So, you have to prune back the charts and data that have lost their novelty and impact. Alas, attention spans are short and charts that show an essentially flat trend line just don't garner much interest or action. Identifying these less useful metrics and trimming them away is a necessary and sometimes delicate action you have to take.

Limits of Web Analytics

pstrupp — Wed, 3 Jan 2007 09:02:12 -0800

I had a discussion the other day with a colleague who was somewhat frustrated that I couldn't tell him why the conversion rate through a particular process had slipped after he launched a new design.

It was clear from the data that there has been a fall off, and we were even able to point to a particular page where users fell out of the process. Click path reports told us where the users were going.

"Thanks, Paul, but why are they doing that? That's not what we expected them to do."

"Well, Joe, Web Analytics are voyeuristic, not omniscient."

I'm not sure he liked that answer, but it was true. We can observe what people are doing, but we can only guess at why.

This is the stage, of course, where team work and collaboration kick in. We looked at the page and postulated a few possibilities, but really the experts in design and usability take over here. I offered to help run some A/B tests to validate design tweeks, but beyond that, I had to accept the limits of my abilities.

Two Types of People

pstrupp — Tue, 12 Dec 2006 11:45:57 -0800

One more comment on a similar topic to my last entry. I guess I've always known this, but have never digested it quite this way. And that is: there seem to be two types of people in the world: those who can take a fact and make it more complicated, and those who can take a fact and make it simpler.

Consider again the issue of "new visitors". When this data was presented at a meeting an audience member said "So, this is just the number of people coming to our web site who do not yet have our cookie?" Well, yes indeed. Well simplified.

Somebody else could have looked at the same piece of data and asked, "If a user is using tabbed browsing, will he show up as a new visitor if he visits from two different tabs? What if he is using two different browser profiles? Then what?" Good questions as well. I suppose we ought to understand that.

So, let me ask. Is a web analyst more like the first person or the second?

White Lies

pstrupp — Mon, 11 Dec 2006 08:44:25 -0800

I lied to my VP the other day, and if he ever finds out...

I think he'll thank me.

So, why would I be such an unscrupulous wretch, deliberately passing untruths to the person who has so much power to make my career successful or painful? Because that's my job.

I better start explaining. I was preparing some web data for him to use at a presentation he needed to make to his peers and his executive VP. Part of the materials included data about new and returning visits as well as visit number (i.e. is this the user's first, second, Nth visit?), and as I was assembling the slide I found it to be horribly complicated, burdened with nuance and caveats Were cookies deleted? What if the user used multiple computers? How long had we been capturing data for that web site compared to the other we sites? How do I explain the difference between a return visit and a return visitor? (I'm still struggling with that last one!)

So, I simplified the whole mess and gave him a simple sound bite.

The next day I was fortunate enough to be in the meeting when he presented this information. It was received by a smattering of nodding heads which subsequently proceeded to some valuable discussions about marketing strategy, rather than a rat hole trying to figure out what in the world that chart was trying to say. Thus, I successfully removed some trees so that they could see the forest.

So, am I advocating fudging of web data because it is too complicated? No! In fact, if an analyst on my staff were to pull the same trick with me I'd run him up the flagpole for not understanding the data (even if he actually did). My point is that as analysts you need to remember that it is your job to translate web data into business information and that process sometimes requires a judgment call if some details can be safely left out.

Simple Ideas

pstrupp — Tue, 21 Nov 2006 15:35:00 -0800

Did you ever notice how the best ideas are often the simplest ideas?

We recently spent some time on vacation in Tuscany, exploring the beautiful country side and hill towns, visiting churches and museums, and of course, sampling the great food and wine of the region.

One day we found ourselves at lunch in Montepulciano at Osteria Acquacheta (Via del Teatro, 22). If you're ever fortunate enough to be there, don't miss the opportunity to dine there, assuming you can even get in. It's a real authentic Italian experience; family run, over flowing with locals, raucous and bursting with abbondanza. The kind of place where you slam your fist on the table and shout "Bene!" when they drop off your steaming plate of eggplant Parmesian. The kind of place where they serve a small miracle of simple bruschetta drenched in local Tuscan olive oil and enough garlic to deter Mussolini, but to try to recreate at home results in a humbling waste of bread and hope. The kind of place where the wine comes by the carafe rather than the bottle.

The kind of place where the darn waiter won't give you a wine glass for your vino. You have to use your water glass.

Wait! What's wrong with this picture? Why would such a perfect place be too lazy to give you a glass for your wine? They even carefully explain to you that it is a tradition at their restaurant that you only get one glass. Cheap skates!

But as you progress through your meal and juggle the competing priorities of drinking water and wine, you are forced to alternate between glasses of one or the other. And thus, you end up getting less inebriated as you slow your wine consumption and switch off with water in between. Brilliant!

It turned out to be a natural regulating mechanism and we left the meal perfectly delighted with the food, comfortably soothed by the wine, and well hydrated.

Gray Privacy

pstrupp — Sat, 18 Nov 2006 13:03:37 -0800

Privacy continues to be a more and more complicated issue when it comes to web analytics.

As I see it there are two fairly clear levels of privacy on the web. If a user anonymously visits a web site, then they should be treated anonymously. This seems to be the equivalent of window shopping.

That said, if I own a conventional bricks and mortar store, I feel I have the right to watch how users look in my windows and observe what displays catch their attention the most. I distinguish one person from another by an anonymous attribute. The guy in the green sweater is interested in the golf clubs at 10% off. The woman in the dress likes the brand of Italian shoes I'm offering.

The other situation is when a user comes to a site and logs in, telling me who they are. Now I know that Joe Smith is in my store and is interested in the sale on golf clubs. I might even give Mr. Smith an extra discount because I know he shops at my store often.

But there is this gray area on the web I'm not sure what to make of.

Let's say at one point Joe Smith told me he wants me to send him emails when I have some kind of promotion. I send a personalized email to Mr. Smith and he clicks on a link that brings him to my web site. He has not logged in but I could easily know it was him on my site. That seems a bit sneaky to me if I were to track him by name.

Let's make it even grayer. (Hmm. Can things actually be more gray or less gray?) I send Mr. Smith an email which he clicks to come to my site. I direct him to a personalized portal customized for his needs. The top of the page says "Welcome, Mr. Smith!". I have let him know that I have identified him by name. He didn't really ask me to, but he went along with the attraction to come to his personalized site. If I follow his actions now on my web site, I know exactly who is looking at what.

But what is his expectation? Does he have an expectation of privacy in this case? I really don't know. I kinda think he does. Until he takes an explicit action to log in and tell me who he is, he has not taken the initiative to identify himself. Yes, he chose to come to the personalized portal, but did he choose to become no longer anonymous?

So, I'm full of questions at this point and not so many answers. I welcome your thoughts on the matter.

Building an Emetrics Group

pstrupp — Mon, 18 Sep 2006 07:54:12 -0700

The good news is that our Web VP made the commitment to form an emetrics group. Now we have to build an emetrics group. The difference being, of course, going from rearranging lines on an org chart, to delivering tangible value that is more than the sum of the parts.

Our first deliverable has been to corral the herd of metrics reports roaming the business plains. (Alert: Heavy handed metaphor!) We have had plenty of data getting produced around the organization, but it has not been collected in any meaningful way, nor has it been regularly reviewed nor discussed. So, our first order of business has been to produce a monthly department emetrics report which gets reviewed each month with senior web management.

That is a great first step and, for now at least, we have the ear of management who actually seem to pay attention at the monthly staff meeting as I pour over charts and graphs and try to point out trends that somehow predict where the business is going. It's a fine start and has been successful at stimulating discussion. But I doubt that the interest will last as trend charts are slow moving creatures and for the most part are either pretty flat or so noisy that you can't tell what is going on.

But what is coming from this are questions. A trickle at first. But one question leads to another and each month I get to come back with some analysis following up on last month's questions. And thus the snow ball rolls.

I can see the wheels turning in the directors heads. We are making emetrics part of the conversation of the business. Now to just keep that conversation going!

Where Can I Get My Pointy Haired Toupee?

pstrupp — Wed, 16 Aug 2006 07:54:31 -0700

I'm moving to the dark side. After several years of a "Program Based" Web Analytics effort at Sun we're taking the next step and forming a full fledged emetrics group to collect the like minded data and analysis nerds together. And I get to manage it. Give me a few months and I'll let you know if congratulations or condolences are in order. Should be very exciting either way.

We have made enough progress with our efforts here that the company is willing to make a regular organization of us. Coalescence and critical mass ought to take us to the next level of helping to drive the business with analytics.

I can already see that insights I'll be able to offer in this blog will be changing. Formerly I worked directly with marketers and other first level practitioners to optimize their sites and campaigns. Now I'll be dealing a lot more with management, engaging them directly in the conversation of emetrics, seeing what kinds of data and messaging register, figuring out how to actually motivate organizational action based on emetrics.

If I actually make any sense of it I'll let you know!

Measuring Conversion Latency

pstrupp — Tue, 6 Jun 2006 09:19:09 -0700

Here's something practical for a change.

I've written a couple entries below on dealing with conversion latency on a web site which is indeed a tricky problem. I don't pretend to have all the answers, but to provide some insight for myself I have instituted a customized measurement that is proving to be rather interesting. And because it is all already visible on our web site for anybody with the where-with-all to find it, I'm not really sharing any secrets here. (That should stave off the lawyers, I hope.)

My business question was "How long is the latency period between a user viewing our product information and purchasing a product online?"

Note that at least for now I am not asking about quantity, i.e. what percent of users convert from a product view to a purchase. I just want to build a histogram of the time lag (latency) between the two steps.

I have no choice but to explain this in the language of our vendor, Omniture, so bear with me. I suspect there are analogous ways to do this with other vendors but you'll need to do the translation yourself

The approach I took was to set an eVar on the product view page to capture the current date. I have linear allocation set for the eVar, an expiration of six months, and renamed the eVar "Product Research Date". I also set a custom event that I renamed "Product Research" and the s_products variable so that I can segment the data by product.

The code on the page looks like this (I X'd out the product name below so I don't reveal too much):

<script language="JavaScript">

<!--

var s_events="event11";
var s_products=";XXXX Server";
var s_eVar22=new Date().getFullYear()+"-"+(new
Date().getMonth()+1)+"-"+new Date().getDate();

//-->

</script>

The outcome of this is that I can run a report for product XXXX, broken down by Product Research Date where my selected metric is "Purchase".

The trick is that the report must be run for a single calendar day at a time, say June 1. Then I see a list of all the days on which a user who purchased XXXX had previously viewed the product research page.   The report looks kind of like this:

Research Date      Purchases
2006-6-1                156
2006-5-31               34
2006-5-28               27
etc.

Then in a spreadsheet I subtract the research date from the day of the report and determine the latency:

Latency      Purchases
0 days         156
1 day         34
4 days         27
etc.

Then, repeat the report for another day, determine the latency, and average enough data together until you have enough to build a histogram.

Showing real data would get me in trouble, but I can say that this leads to interesting analysis. Two teasers for you are that different products show clearly different latencies (like a server vs a simple software product), and that histograms of products with significant latencies often show bumps at 7 days, 14 days, and 30 days.

Thoughts on Multivariate Testing

pstrupp — Tue, 2 May 2006 07:24:03 -0700

I recently got back from the eMetrics Summit in Santa Barbara. Always a great metrics geek fest and lots of fodder for blogging.

One theme that stood out to me this year was a lot of talk about multivariate testing to optimize a web page. There were some good case studies presented and just the general buzz seemed to be that this next year would see MVT becoming common place rather than leading edge.

But as I listened and learned it became more and more clear to me that as useful of a tool as MVT is, it must be used appropriately. After all, when you run a MVT to optimize a page you are optimizing the page for a particular, explicit outcome. Often the targeted outcome is to increase conversion of a particular call to action. But while optimizing for this outcome, you may be de-optimizing other outcomes. And these de-optimized outcomes might not be obvious.

For example, you might optimize a page to be a lean, mean revenue generating machine, but inadvertently de-optimizing a softer metric like customer satisfaction. The user experience might become very efficient and utilitarian but end up leaving a bad impression. You might increase the likelihood of the current sale while negatively impacting your brand.

An interesting study would be to run a parallel MVT study with two outcomes, one being a practical conversion outcome like a click through, and the other being customer satisfaction or brand impression measured with a survey tool. Then compare to see if the same recipe optimizes both outcomes.

Never Mind the User

pstrupp — Fri, 7 Apr 2006 08:23:18 -0700

There was lots of talk last week at a web analytics conference in Park City, Utah about the future of web analytics, especially around the integration of various emarketing systems with web analytics platforms. The message was that web analytics was becoming an arm of emarketing, or vice versa, but either way all these systems are bound to converge so that you can more effectively measure the ROI of your emarketing efforts and optimize your emarketing dollar.

I think this technology convergence is inevitable and a good thing, but I left the meeting with a nagging feeling that I hadn't heard the phrase "user experience" nearly often enough in the last two days. Surely the user experience is core, but where does it fit into this cold language of emarketing and web analytics?

Here's one way to think about it.

A Web business breaks down to this simple model:

Demand Creation -> Conversion -> Business Results

which is influenced by these underlying factors:

eMarketing -> User Experience -> Customer Success.

eMarketing effectiveness is usually described in terms of ROI, but at its root should be viewed as a function of demand creation (both volume and quality) and conversion. And conversion is driven by the user experience. You can have a great emarketing program driving lots of well qualified customers to your site, but if they have a lousy experience when they get there, they are unlikely to convert.

Taking this dissection one step further, the user experience is comprised of the objective and the subjective. The objective piece being pure utility in that did the user accomplish what he wanted to, and the subjective piece being did the user enjoy the process. The objective, utility piece is well measured by traditional web analytics (conversion), and the subjective piece is measured by customer satisfaction (usually via a survey).

So really, although emarketing and web analytics are converging, I hope the discussion does not become obsessed with optimizing emarketing effectiveness as the end goal while forgetting that at the root of all of this is the user experience.

Duh? Or Not Duh?

pstrupp — Fri, 10 Mar 2006 14:48:51 -0800

"Duh? Or Not Duh?" Maybe that can be a new game show.

In Chapter 13 of Eric Peterson's book, "Web Analytics Demystified", he discusses conversion and defines conversion as

COMPLETIONS/STARTS = CONVERSION RATE.

Question: Duh or Not Duh?
Answer: Not Duh.

Although this definition seems obvious enough, the subtleties come out a few pages later...

"...when you are making a conversion rate measurement you want to use 'like' metrics, that is, visits for both the numerator and denominator, and not visits for the numerator and page views for the denominator."

"...also be careful not to mix and match metrics, that is, do not use 'completions' divided by 'respondents' as the former is a non-unique metric, the equivalent of a page view, and the latter is a unique metric, the equivalent of a visitor."

These points are worthy of discussion. Conversion is always expressed as a percent. This is a dimensionless quantity, a ratio of identical units which cancel, like visits/visits, not like submissions/visits or registrations/page views. (Having algebra flashbacks, yet?)

I bring this up because recently I had a discussion with some pretty smart marketing folks who got themselves all confused working through a conversion calculation. The calculation went something like this:

Impressions
    Click throughs 1%
Visits
    Response rate 1%
Leads
    Qualified leads 25%
Proposals
    Successful proposals 30%
Sales

They thought they needed to figure out how many unique users they needed to hit their sales target, but couldn't figure out how to work users into the equation. The thing is, unique users don't really factor into the equation.

Falling back to Eric's definition of conversion, you would deconstruct the above model like this...

# impressions completed (i.e. clicked)/# impressions started (i.e. served) = impression conversion = 1%
    1 visit/clicked impression
# visits completed (i.e. successfully)/# visits started = visit conversion = 1%
    1 lead/successful visit
# leads completed (i.e. qualified leads)/# leads started = 25%
    1 proposal/qualified lead
# proposals completed (i.e. sales)/# proposals started = 30%
    1 sale/proposal
# sales

When looked at this way, it becomes more clear what is being converted at each stage, that is

# "somethings" completed / # "same somethings" started.

The catch was to break out the assumptions about the transformations between dimensions--the change in units when the user moves from step to step. In the example, we are assuming that one clicked impression corresponds to one visit, one successful visit corresponds to one lead, and so on.

Of course, you must consider if the transformation assumptions are sensible. For example, might a user click on two banners and refer himself to the same visit twice? Probably not. Will a user submit two "contact me" requests to generate two leads in a single visit? Doubtful, but not out of the question. Could a single lead produce two proposals? Maybe, but not likely.

The bottom line is to know what you are converting (starts of something to completions of that same thing), and what assumptions you are making about the dimensional transformation from step to step.

Latent Conversions, Part 2

pstrupp — Fri, 3 Feb 2006 17:10:57 -0800

Let's talk some more about latent conversions (and no jokes please about the latency in my blog entries).

Same Visit Conversions
Assume a process which has four significant steps A,B,C,D. And for this example let's say that the same visit conversion from step to step looks like this:

A --90%-> B--5%-> C--90%-> D

or viewed as the composite

A --4.1%-> D

Granted these are exaggerated step to step conversions, but it helps make the point.

What is going on here? Well, it could be that Page B is poorly designed and really stinks at converting users to the next step. Or it could be (because we are talking about latency here) that Step B naturally leads to latency due to its place in the buying cycle (e.g. maybe it is a quoting page).

Let's say that it is indeed a quote generating page and the fact that users are ending their visit here seems reasonable, even necessary, if you were in their shoes. Is saying your conversion from A to D is only 4.1% have the business meaning that is useful to you? Let's look at this situation a little differently.

Population Snap Shot Conversion
In a previous blog entry I discussed the possibility of measuring "conversion" by taking the ratio of traffic at Step B to that of Step A. (Give that a read if you haven't because there are some significant assumptions to keep in mind here. ) Below is some example data that one might see:

A B C D
10K 9K 5K 4.5K (visits)

Taking the ratio of traffic at each step gives a "conversion" like this:

A --90%-> B --56%-> C --90%->D

This ratio approach I call the "Population Snap Shot Conversion". At any given instant you are taking a snap shot of how many users are at the different stages of the buying cycle. It is a way to gloss over the latency problem that affects the traditional "same visit" conversion, and provided that you are clear in communicating what you are measuring, can provide more meaningful business insight to how users are converting through your buying process when you have a big latency gap. The example above suggest that rather than 5% of your users converting from Step B to C that actually 56% who visit Step B eventually come back to visit Step C at some point. Of course, these are different users at B and C at this instant, but by making some assumptions about the population you get a different view of your buying cycle conversion.

But we're not done with the caveats yet! One of the assumptions to this measurement being somewhat meaningful is that you didn't just send a boatload of people to Step A because of a new promotion you launched.

Buying Cycle Equilibrium
The Population Snap Shot Conversion assumes that at any step in the buying cycle there is not a significant transitory spike due to some perturbation such as a promotion. The concept is that the buying cycle is in more or less equilibrium, meaning that there is not a bubble of users at any particular stage. Such a bubble would temporarily inflate the number of users at a certain stage and throw off the ratio of users who make it from stage to stage as motivated by the value of your products and usefulness of your web site. That unperturbed ratio is what you care about.

Thus, this approach is unsuitable for certain businesses. But for businesses where effects of promotions are damped out by stronger factors that drive the natural buying cycle, Population Snap Shot Conversion can help you look past latency and give insight into what percent of your users are coming back to move to the next step of the process.

Still more to discuss on this. Stay tuned.

Latent Conversions Part 1

pstrupp — Mon, 28 Nov 2005 10:08:46 -0800

conversion I wish we just sold socks. Then I think the concept of conversion on our Web site would be easier to understand.

What is Web site "conversion"? It is typically simply stated as the percent of visits which progress from one step in a Web process to another step. For example, if the shopping cart gets 1000 visits and 100 of those visits proceed to the "Thank You" page, that's a 10% conversion rate.

But what I never hear stated when conversion rates are discussed is the time frame of the conversion. It is usually assumed to be within the same visit (which could be any time frame as long as there is not a 30 minute dead time to terminate the visit.) However, such a simplification is not always warranted, especially if your business is complicated enough that it requires your users to return to your site multiple times over multiple days to complete the process. In this more complicated case, the notion of "latent conversions" must be invoked, and a time frame must be specified.

Latent Conversions.
Consider the case where customers generate and save quotes for a product on your web site, then need to go and use this quote to get a P.O. approved by their management, and then come back a few days later to the Web site and proceed from the saved quote to completing the purchase. How do you correctly measure and specify this conversion?

The stated conversion only makes sense if you specify a latency time frame.

Simplifying the example above, say 500 users save a quote on a given day, 100 of them return to make a purchase, and by some miracle every user who comes back to make a purchase does so in exactly seven days. Then you can state a "seven day or less latent conversion" of 100 divided by 500 which is 20%. The traditionally stated "same visit" conversion would be 0% because nobody progressed from Step A to Step B in the same visit.

Measuring Latent Conversions
Assuming you really want to measure latent conversions, how would you actually do this? Modern Web Analytics packages actually allow for this kind of measurement by letting you store previous activity of a user in a variable that is tied to a cookie, so that if a (unique but anonymous) user takes several visits to convert, the tool records this. Of course, this is all only as accurate as the user's cookie, but that is a whole other topic.

Another approach is to make some assumptions about your users. If you want to measure the latent conversion of your users from Step A to Step B, you could simply on a given day measure how many users completed Step A and how many (probably totally different) users completed Step B. Then divide the number of occurrences of Step B by the number of occurrences of Step A and call that the "conversion".

The big assumption here is that the users who are at Step A on that day are statistically indistinguishable from the users who are at Step B. Although the users certainly are distinguishable, you could choose to make the approximation that their characteristics and biases and motives are not distinguishable and thus the statistical assumption is defensible. How accurate this assumption is certainly depends on your Web site.

(Note that this approximated conversion is different from the "traditional" definition of web conversion which considers contiguous paths of specific users progressing from Step A to Step B. If your latency is seven days, the same visit conversion is zero!)

Other than the major assumption of indistinguishability of your users, this approximation approach has the major drawback that the latency period is unknown and unspecified. You would need to state clearly the assumptions you are making in this conversion approximation.

There is more to consider on the topic of latent conversions, such as multi step processes and distributions of latency, but I'll save that for another blog entry.

Who's Your Daddy--er, I mean, Customer?

pstrupp — Thu, 3 Nov 2005 08:42:50 -0800

Yesterday I had the opportunity to give a guest lecture to students at the University of Colorado on an Introduction to Web Metrics as part of their User Interface Design Class. In addition, after the lecture we have a hour workshop practicing developing a metrics plan. I've been giving this lecture to this class for four years now, and they keep inviting me back, so I must be saying something worthwhile. At a minimum, it gets the professor off the hook for a week.

And like any teaching experience, the teacher typically learns as much as the students do. I really enjoy working with the students each year (generally seniors and graduate students in the computer science department) because they ask questions that I would never hear around my normal work environment. The questions are often so fundamental that they come like a slap in the face--challenging concepts that I have long since taken for granted. And I'm refreshed by having to scrape the crust off of thoughts that I have considered settled and closed.

Each year is different, but a theme that arose last night went kind of like this.

"Business metrics? My Web site is not about business."
"I'm not sure who my customer is?"

A little background. These students are working on senior projects that revolve around redesigning a user interface of some sort. They pick their projects from proposals submitted by organizations or individuals outside the class who have a web site that needs improvement. So, it's a win-win. The sponsor gets a spiffed up web site, and the students get real work experience.

The twist is that many of these projects tend to be outside of regular "ecommece" sites. A few class project examples are: an application that displays geographical location of sensors worn by park rangers doing wilderness work to help with search and rescue if they go missing; a site that provides interesting weather data and maps for teachers to use in the classroom; a tool that makes interpretation of earthquake data easier to interpret. Really cool stuff! No boring old sites trying to coerce some chump into buying whatever widgets you happen to be selling today.

Thus, as I proudly instructed the students to (1) Identify a business goal, (2) Identify the customer web activity that demonstrates this business goal is being achieved; and (3) Determine attributes (measures) that quantify or characterize that activity, a lead balloon came in for a crash landing.

You see, many of these students had never really considered the fact that their projects represent a business, and that even though they are not selling anything, they indeed have customers.

The result of the evening, I realized, was not so much that the students created a comprehensive metrics plan for their web site, but that being forced to think about appropriate metrics drove them to ask and answer the most fundamental questions needed to be successful in their business.

And I, as the teacher, learned that metrics is not always about optimization; it is often about something much more basic. Metrics is often just about forcing clarity in how we think about our business.