Reach and frequency and the core concepts of traditional media planning and advertising. For a given site, program, channel, radio station, billboard, newspaper section, a target audience (the reach) is exposed to a certain number of occurrences of the media (the frequency). On the web, these concepts manifest themselves in metrics collected and reported from a number of recognizable services. Audience measurement firms, like comScore and Nielsen, web analytics firms, like Omniture and Unica, to companies somewhere in between, like Quantcast and Google, all have reach and frequency data. Many new media metrics can be used to proxy frequency- from time-based measures, espoused by audience measurement firms, to concepts like visitor retention or the repeat visitor rate cited by web analytics firms. On the reach side, companies refer to concepts like “unique visitors.”

These data, of course, available in free tools or in for pay tools are certainly helpful for planning campaigns. But reach measures can be dirty (cookies, unduplicated unique users, estimates from panels, coverage error). Frequency measures can be just as dirty (problems recording time in single page visits or visits on the last page, do page views really matter with AJAX and rich media, cookies again, and so on). We all are aware of the challenges.

Thus using basic reach and frequency measures for planning or evaluating a campaign does not suffice. So advertisers and agencies target demographics, like gender, age, income, education, and job title. It’s a given that advertising in the Robb Report reaches a different audience segment than advertising in Popular Mechanics. 

These brave new days we have “behavioral” tracking too. By taking into account visitor activity across sessions, such as past actions taken on a site or a roster of previous purchases, we can attempt to deduce what a person or segment responds to or is interested in based on their behavior.

Even with reach, frequency, demographics, and behavioral data to help guide advertising and media buying, we are missing an important attribute for maximizing the potential success of our campaigns. We do not have an available tool, whether free or paid, for advertising or buying media on or across sites according to measures of past performance. Such measures include ad clickthrough rates, conversion rates, goal completion rates, delivered impressions, and perhaps even harder to quantify financial measures such as ROI, ROAS, and ROMI.

Sure, historic, tacit knowledge of campaign performance exists and is used by agencies or publishers. However, there is no shared industry source that can help us answer “how has a site for display advertisement historically performed toward goals based on the reach, frequency, demographic and behavior of its audience segments?” Interestingly, a company minting money right now, named Google, can masterfully demonstrate performance in paid search campaigning and help advertisers unify it with segmented reach, frequency, and demographics.

Outcomes based performance measurement unified with reach, frequency, demographics, and behavior is what is missing in audience measurement tools, not frequently reported externally by web analytics tools or ad serving tools, and not available in ad planning tools. When advertisers can target display ads, or even video ads, to desired audience segments by reach, frequency, demographics, behavior in the context of known performance, media planning will be more effective.

Yes, I know…I’m just like that. Anyway, here I’m picking things up at Dr. Geertz’s February 5th, 2008 at 11:20 am comment.

Enjoy.

The Good Dr. Geertz, 5 Feb 08, 11:20am

Since Dr. Geertz published this comment he and I have had some rousing and wonderful phone and Skype conversations, one of which I extracted and published in My Easter Eggs Critiqued. Dr. Geertz was also kind enough to read through NextStage’s patent (several times. Somebody buy that man a beer!) and lend his expertise to our internal discussions. It’s wonderful when one finally learns the identity of one’s advocates.

Dr. Geertz offers that I was suffering from some consternation in my remarks above. I thank him for the thought and suggest that I was biting my tongue until I knew we had the patent more than anything else.

And also and for the record, I do not object to anybody else’s definition of engagement and their use of that definition as a description of some metric or tool they’ve developed, I only recognize that their definition is different from the one NextStage uses. I also believe that NextStage’s definition is more realistic and calculable when it comes to determining future acts by a given (visitor) population across the broadest possible number of platforms throughout time.

Theo Papadakis, 5 Feb 08, 2:43pm

Hello,

I read through your post on Avinash’s blog a few times and applied both simplified logical calculus and Alexsander&Dunmel’s Logical Calculus of Consciousness to it. I think there’s a flaw in your original proposition. You write “If x is engaged with y, x is related to y.” This translates to something like “If y = f(x) then (x,y) are in some space S” which is true. Prior to that you write that what you’re defining is a one way relationship and that’s not valid. This inconsistency demonstrates itself later on in a logical analysis of the steps involved in the hypothetical demonstration you offer.

You write “two kinds of engagement with an object, positive and negative, by which I meant that someone can be engaged with an object.”

Hmm…I think I explained this in Back into the fray comes Joseph!, Online Engagement: What Exactly Is It?, Meet Online Engagement’s Little Friend, Satisfaction and elsewhere. I’ll also be following these up with an AllBusiness.com post (it’ll go live on 21 Jul 08, 6amET, and is called “The Money Is Where Engagement Meets Satisfaction Online” for those who’ll go looking).

Engagement is the same because the same parts of the brain are active. You can be engaged by pain or pleasure. Your response to the pain or pleasure will (probably) be positive or negative and the parts of your brain telling you “Pay attention!” — those parts that dictate whether or not you’re engaged — don’t really care about your response, only that your attention is focused sufficiently such that the response can be effectively and adequately acted upon.

You then write “Engagement is not itself a psychological state but involves a mixture of rational beliefs and psychological states…”. I’ll admit to having some challenges with this. I think my definitions of “psychological state”, “rational” and “beliefs” are inadequate to your use of those terms. Beliefs, for examples, stem from a part of the {C,B/e,M} matrix that doesn’t rely on rationality.

<ASIDE>

I’ve written about the {C,B/e,M} matrix in several places such as Guest Blogger Joseph Carrabis Answers Dave Evans, CEO of Digital Voodoo’s Question About Male Executives Wielding Social Media Influence on Par with Female Executives, Responding to Christopher Berry’s Vexing Problem, Part 3 post, Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 2, my comment to this post and several other places. I can also offer a bibliography of how the {C,B/e,M} is derived for those with an interest.

</ASIDE>

A simpler version of the {C,B/e,M} matrix is described in Reading Virtual Minds and deals with the different ego states of Core, Identity and Personality. Where ever they come from, beliefs may be informed by data but they are not bound to rational thinking. They are, as described in my exchanges with Christopher Berry, based on anecdotal thinking and subject to all the pitfalls inherent therein. You have a good primer on this material in Schacter and Scarry’s Memory, Brain, and Belief

Likewise “rational” is not an adjective I’d use with “beliefs”. “Psychological states” I recognize as “a mental condition in which the qualities of a state are relatively constant even though the state itself may be dynamic”. Some worthy albeit diverse reads on this subject include Tomasello’s The Cultural Origins of Human Cognition, Gazzaniga’s Mind Matters: How the Mind&Brain Interact to Create Our Conscious Lives and Allison’s Minds in Many Pieces (sorry, couldn’t find those last two online).

You also write about the “degree of engagement” and tie your definition to “psychological investment/involvement”. I’m quite sure I don’t understand the use of these terms in these contexts. This is not to indicate they are invalid, only that I don’t understand them.

Lack of understanding, however, is much the central problem that needs to be addressed in these discussions. Any metric is meaningless unless the language of what is being measured is specific. It might make good business sense that some group has a proprietary definition but when that definition no longer applies the business suffers. It’s probably better (opinion warning) that a common definition be used and that businesses work at being more accurate in providing a metric based on that common definition (just as a point, NextStage uses common definitions that anybody can use. It’s our methods and technology that are proprietary, not our definitions).

If I understand your hypothesis correctly, you suggest that your definition of engagement can’t be measured, only inferred. I don’t think I’ll argue that point, only question it’s utility. Breaking down the logic I come up with

a) Hypothesize some phenomena

b) Define the phenomena such that it can’t be measured using standard and readily accessible tools

c) Take measurements using standard and readily accessible tools

d) Develop a formula that takes the measurements from the above

e) Claim that what is measured, via the formula, is the phenomena.

There are some logic holes here (to me, anyway) and I won’t go through them in detail. I will offer that there must be an unbroken chain of physical connection between how something is measured and what something is in order for that measurement — hence the resulting metric — to be valid.

I’m happy to continue this discussion if there’s interest. I do want to point out that the lack of precision in both definition and logic will probably be impedimentory. Do also note that I don’t recognize problems with “measuring degree of engagement” based on the definitions I use and the measurement methods applied (the three “real questions” in this thread’s original post).

Eric Peterson, 5 Feb 08, 8:28pm

“Save asking every person who comes to a web site “are you engaged?” (which I would assert is A) impractical and B) just as imprecise as my calculation, if not more so!), how would you propose we ground truth engagement and test the hypothesis?”

I can’t speak for Dr. Geertz and this is where that nap-of-the-earth flying thing comes in, I guess. I’ve often described Evolution Technology (ET) as doing exactly that, asking every person to a website “Are you …?” and then responding via whatever business rules are in place. Hence this is not impractical. Is it imprecise? That depends on how much precision you’d like. In more tests than I care to remember NextStage’s ET averaged 83% accuracy predicting outcomes (what people would do, when they would do it, etc.).

Re “satisfaction”: Yes, agreed. This is something I mentioned during one session at eMetrics SF 08 and also in Back into the fray comes Joseph! and Meet Online Engagement’s Little Friend, Satisfaction. I mentioned a future AllBusiness.com column post on this subject I’ll share some highlights here.

First and for explanatory purposes, I’ll shade in the figure in Back into the fray comes Joseph! so that it looks like the one on the right.

<ASIDE>

Those shaded areas actually do have meaning. People may have heard me mention that ET makes decisions and offers suggestions based on solid probabilities. The combination of those shaded areas (and their extensions) creates a bell shape. Imagine that bell filled with metal. Now you have a solid bell. Imagine every atom in that solid bell representing a probability that something will or won’t occur, that a visitor will do this or that, will respond one way or another, will think this way or some other way. The accuracy of the prediction — the likelihood or probability that something will occur — is based on the atom’s height within the bell and its distance from the bell’s surface. Things closest to the center of the bell are more likely to occur than things at its edge.

Thus something midway up the edge of the bell is less likely to occur than something at the bottom of the bell but dead center.

Hey, this is my world. I’ve learned to live in it and I’m not claiming anybody else has to or should.

</ASIDE>

What happens when you map those quadrants onto a standard Engagement chart is something like on the right. The money (if you will) is in the periodicity. The periodicity can depend on several things, most of which are business dependent. What the periodicity gives you (via linking to some standard (at least what I think are standard) web analytics values) is a near surgical ability to recreate optimal satisfaction-engagement periods at will.

Thus ends the 5 Feb 08 comments

Again, whoosh! At least I got a column out of this. Next time I’ll start with Dr. Geertz’s 6 Feb 08, 11:56am comment.

And thanks for everyone’s patience.

“When the only tool you have is a hammer, everything looks like a nail, and your living at the bittersweet motel.”

Bittersweet is an apt assessment when it comes to producing this type of research as an analyst: non-vendor clients love the insights, vendors hate the comparisons, and all-in-all the results often fail to shed any truly new light on the market. John should be complimented because despite publishing two somewhat poorly-resolved constellations, his work makes a few incredibly important points about the state of the market today.

I know that Stephane and Anil have already discussed the report, and nobody really asked me, but here are a few of my thoughts on John’s work.

If I’m Omniture, I’m not very happy about this constellation

Despite hundreds of millions of dollars of investment — including the acquisition of three of the company’s former rivals (WebSideStory, Visual Sciences, Instadia) and the roll up of Offermatica and TouchClarity — in the large Enterprise John’s assessment has Omniture in a three-way tie for “first” with Unica and Coremetrics. Compared to my assessment in 2004 and Greg Dowling’s work in 2006 (published by David Daniels in February ‘07), John’s work shows that Coremetrics and Unica are actually gaining ground on Omniture from a business value and market suitability perspective.

This is important because it reinforces both John’s central thesis and one of the most important caveats in all of web analytics: it’s not the tool that matters, it’s how you use it! Omniture’s own consultants make this point when they remind us that we need to work hard to take advantage of the systems we already have in place, and the reality of the situation is that you’re not going to be any more successful with Omniture than any other application until you invest in people, process, and technology with a realistic and well-considered business strategy.

Don’t get me wrong, I think that Omniture has brilliant technology and are in a great position in the market today — if they manage to actually integrate analytics, testing, targeting, and bid management in a truly meaningful way they will solve a bunch of real-world problems. But despite the hyperbole, hype, and braggidacio, Omniture’s competitors near universally have a similar opportunity and thusly I agree with Lovett’s asessment that there is no single market leader in web analytics today, Omniture or otherwise.

If I’m Coremetrics, I am pleased as punch!

Coremetrics is in a funny position in the web analytics market. Despite all of their competitors declaring them “done” and “yesterday’s news” they continue to rank well in both the JupiterResearch ranking and the Forrester Wave. Maybe the reason is that Coremetrics is actually still very competitive and able to provide the level of functionality and service that their clients are looking for at a competitive price. Could that be it?

In fair disclosure, I do some work from time to time for Coremetrics and I really like their team, but given their recent deployment of Coremetrics Explore and the expansion of Coremetrics Connect, I think Lovett’s work validates the observation that the only real difference between Omniture and Coremetrics is their general approach towards marketing and sales, not their technology.

Furthermore, despite having been long considered a high-end solution with a substantial price tag, Coremetrics actually takes first place for overall business value in the SMB sector beating not only Omniture but also Google Analytics and IndexTools which are free! I commented as much in the press release Coremetrics issued for this report, mostly because this type of market expansion is no mean feat given the quality of the competition. And to be fair, Lovett’s business value dimension encompasses more than just cost and includes flexibility, scalability, usability, and feature sets.

If I’m WebTrends, I’m bummed out!

Living here in Portland, Oregon I am perhaps more acutely aware of the challenges facing WebTrends. Last week they lost their CFO to another local firm, they already had to part ways with their VP of Client Services, Kory Kimball, who was only appointed in January of this year, and they are still looking for a replacement for Kathleen Brush who was brought in by Francisco Partners as an interim CMO. Now, to be fair, these staffing issues are offset by the fact that they still have some pretty bright folks on the team, guys like Barry Parshall and Aaron Gray, but leadership in this marketplace has to come from the top and right now, the top is looking kind of thin.

My advice to Dan and the Board at WebTrends is basically this: get someone who knows web analytics inside and out in a senior position ASAP and get them out there talking about the company, products, and market in general. On this point I disagree with my good friend Jeff who says that “business is business” and executives don’t necessarily have to be domain experts. When I look at the market I see folks like John Pestana from Omniture, John Squire and John Payne at Coremetrics, Akin Arikan at Unica, Dennis Mortensen at IndexTools, Brett Crosby and even the great Avinash at Google out there evangelizing for both their products and the entire field of web analytics.

Call me old school, but I think the same key insight that it’s not the technology, it’s how you use it applies everywhere. WebTrends is not going to be able to compete on a feature/function level because, according to John, the feature/function war is over and done. The competitive differentiation is going to have to come from somewhere else … and historically that “somewhere else” has been guys like John Squire, Akin, Dennis, and Brett working their butts off to help people understand that despite web analytics being hard, great gains are possible when everyone is invested in being successful.

Surprise, surprise, I was right about IndexTools

When I broke the story about Yahoo! acquiring IndexTools and pointed out that most people who have seen both applications consider IndexTools to be every bit as good as Omniture, Omniture complained. Brent Higgleke, their VP of Strategic Marketing, commented on Julien Coquet’s post about IndexTools:

“This move by Yahoo! was done to compete with Google. IndexTools does not compete “toe to toe” with Omniture. The majority of their customers are small businesses (80% of IndexTools customers are SMB according to CMS Watch.) This is great news for small businesses that use Yahoo advertising. However, mid-market and enterprise customers demand advanced functionality, deep domain expertise and specialized services.”

Sounds good Brent, except you’re basically wrong. Don’t hate me, but I’m gonna recommend that people go with Lovett’s assessment instead:

“[IndexTools] provides a profoundly capable framework for advanced analysis and offers flexible segmentation built on the premise that segment creation is best facilitated through exploration of data. Although currently available only through certified partners, the new free pricing model of IndexTools (a Yahoo! Service) makes it suitable for businesses of all sizes that seek a flexible interface and possess in-house staff looking for insights within data.”

The notion that IndexTools is somehow inappropriate for the large Enterprise, is feature poor, or is otherwise unworthy of consideration in an RFP process when available is just plain silly. John said as much in his blog post, commenting:

“It turns out that IndexTools does have nearly 80 percent of Omniture’s standard off-the-shelf functionality (77 percent to be exact).”

Now, I think we all wish that John would have published his list of “basic” and “advanced” features so we could better quantify the “missing 23%” in IndexTools. My suspicion is that the gang at Yahoo! are pretty conscious of John’s assessment and working diligently on the next generation of IndexTools, much like the gang at Google did with Google Analytics.

So I will state again, Yahoo’s acquisition of IndexTools is a long-term game changer. Yahoo! has still not given a time-line for making the application freely available to all, but an entire network of partners is already out there ramping companies up at a rumored rate of over 200 accounts per week! Obviously if Yahoo! becomes so distracted with their current business problems and never releases IndexTools then my assessment will change, but everything I hear is that my Christmas 2008 prediction is more or less correct.

Despite proclamations otherwise, people still care about data accuracy

Avinash Kaushik is perhaps most loved for his exclamation, “The data quality sucks, get over it!” which to those of us tasked with explaining the unexplainable resonates like crazy. The problem with “getting over it” is that those crazy wonks over in the business, and especially the gang in the corner offices, still want us to produce accurate reports that can be trusted over time. If you’re not sure about this, go down the hall and tell your VP that the unique visitor counts you have been reporting all year may be off by as much as 30% in either direction, you’re not sure, and see what he or she says …

Uh huh.

Lovett’s report seems to validate that nobody is getting over it and that accuracy is still important, especially in the vendor selection process (number three factor, following “flexibity of reporting options” and “ability to service business needs”.) I do disagree somewhat with John’s assessment regarding what to do about the problem, he seems to focus on the need for annotation capabilities in the product, but at the end of the day companies deploying web analytics solutions need to have defined business processes to account for tag coverage, data filtering, cookie deletion as well as a data collection validation process that is actually followed on an ongoing basis.

The next big battle will be about data integration

This is something that John and I have discussed on-and-off for some time, the idea that data integration capabilities are key as increasingly “internet marketing” is giving way to capital M “Marketing” (and, because of this, “web analytics” is likely to give way to capital A “Analytics.”) In light of this observation, John seems to be predicting that web analytics vendors will continue to build out functionality to allow them to be more deeply integrated into the business, while at the same time the existing Enterprise analytics vendors will enter the digital market via acquisition.

Hmm.

I think the problem with this is that these strategies have largely been tried and, for the most part, have failed to produce expected results. John predicts that web analytics vendors will build or acquire content management and relevance engines, which we have already seen with WebSideStory’s original acquisition of Atomz which included the Atomz Publish platform (among other examples, mostly CMS vendors building out analytics capabilities but Interwoven’s acquisition of Optimost is tangentially relevant I suppose.) Same for Omniture’s acquisition of Offermatica and TouchClarity.

Now I suppose it’s too soon to say if Omniture will succeed with “Test and Target” but there is no case to be made for WebSideStory + Atomz Publish being successful. Perhaps this was a problem of execution, but I rather believe that most true Enterprise shops either A) already have CMS in place or B) are unlikely to purchase CMS from a web analytics vendor given the otherwise complicated-but-entrenched landscape. It does look like Omniture is still supporting Publish so perhaps they will get traction that WebSideStory did not. Still, I’m not going to hold my breath, especially given the recent upgrades that Lyris has launched around their Lyris HQ product and the integration of ClickTracks, mostly targeted at the same SMB market and available at a tiny fraction of the Omniture’s price.

Similarly, the big Enterprise software players have all had the opportunity to invest in web analytics for years now and none have taken the plunge. Oracle and others were widely known to be looking at the sector but the only thing that came of all that was A) Microsoft buying DeepMetrix (nee Gatineau nee AdCenter Analytics) and B) Oracle buying Moniforce which I’m not sure really counts. In the meantime, SPSS has stopped supporting NetGenesis as of February 28th of this year and only SAS and Unica are still out there looking at deeper Enterprise integrations as far as I can tell.

Now, I have my own thoughts about the future of data integration and how web analytics will be levearged in business, but hopefully you’ll come see me talk at IMC 2008 in Vancouver when I talk about “Competing on Web Analytics” and hear what I have to say in person.

What, are you still reading this post?

In summary, for the three of you still reading this exceptionally long post, I think John has written a great report on the state of the industry and the vendor landscape. Every JupiterResearch client reading this blog should read it and give John a call to discuss. Or, you could come to the X Change conference in San Francisco and talk to John in person, or you could come see both of us at Shop.ORG in September and watch us fight like cats and dogs about which one of us is right about data integration. Up to you.

As an example of the kind of content published at YouTube, here is my good friend and Web Analytics Assocation Board member June Dershewitz and some of her thoughts about the conference. Cool, huh?

There are 15 different videos at YouTube from brilliant folks like Anil Batra, Dan Shields, Manoj Jassra, Marshall Sponder, June, Phil Kemelor, Gary Angel, and even yours truly who has never been known to shy away from the camera (although perhaps I should have that day, what the heck was up with my hair!)

Don’t know what the X Change is? Read all about it at the official conference web site.

Omniture’s point that Visitor Engagement is a bad idea because it has subjective components fails to understand the work that folks like Jim Novo, Steve Jackson, Theo Papadakis, Joseph Carrabis and others have done; it makes me wonder if the author bothered to read anyone’s work on the subject.

Worse, it makes me question Omniture’s long-term commitment to Visual Site customers since Visual (= Omniture Discover OnPremise) is, at least for now, the industry’s leading solution for creating derived measures and experimenting with visitor-level data. The point seems to be that simple measures of success, such as those provided by SiteCatalyst, are all that are required.

Hmmm …

We pretty much had this same debate a year ago when Avinash Kaushik disagreed with the use of calculated metrics to measure engagement, and I can see that Steve Jackson has already commented as such. I wouldn’t normally have written about this except the author said one smart thing when he commented you shouldn’t “try to build a better mouse trap, when you’re not taking advantage of the one you’ve got today.”

Agreed.

If you’re thinking about trying to leverage any measure of visitor engagement, regardless of which measure you choose, you should definitely make sure your web analytics house is in order first. Despite Omniture’s assertion, most people believe that web analytics is hard and requires a sometimes intense focus on people, process, and technology. If you’re not staffed appropriately, if you haven’t defined your key performance indicators, if you haven’t established core web analytics business processes, and if you haven’t worked to optimize your web analytics implementation then trust me, Visitor Engagement is not for you.

A good analogy is the one provided in Tom Davenport’s book “Competing on Analytics” where he describes how baseball teams like the Oakland A’s and my friend Judah’s beloved Boston Red Sox, and football teams like the New England Patriots have used new and innovative metrics to evaluate the performance of players, concessions workers, and the entire fan experience. Visitor Engagement is a new measure in web analytics, and thusly it will take a special type of analytical competitor to recognize the opportunity that this “uber measure” can potentially provide. And just like some teams have shown that they are not ready to adopt new measures to evaluate their business, some companies are simply not ready to explore complex key performance indicators in an effort to “Compete on Web Analytics.”

If you’re like most companies doing web analytics today, it is likely that you will benefit more from focusing internally and learning more about how to leverage people, process, and technology more effectively, rather than look externally for new metrics of success. You could get a good book on the subject of fundamental key performance indicators and spend a great deal of time implementing what you learn.

But if you’re interested in learning more about an innovative metric that describes the behavior and opportunity that exists with the 97% that don’t convert, a measure that you can apply to your advertising, content, B2B, marketing, or lead generation site that will compliment your otherwise robust key performance indicator suite, and a calculation that describes the level of Attention that visitors are paying to your site, your content, your testing and targeting, etc… well then I guess you’ll have to keep reading my blog (and Jim, and Steve, and Joseph, and a whole host of other people’s work who are committed to working these ideas out rather than just saying “balderdash!”)

If you’re not content to just keep reading and want to know more about my thoughts on Visitor Engagement, know this: I have been exceedingly clear that my measures of Visitor and Audience Engagement are new, and in their newness there is risk in the level of insight they may be able to provide you. I am not promising you better skin, new hair, or more friends, despite the validation that the measurement of engagement recently received when NextStage was granted a patent for their work on the subject. But, unlike some people, I have done my homework on the subject, and I continue to have conversations with some of the best companies in the world about how they can use new measures to improve their overall use of web analytic technology.

In the meantime, I guess I’ll put on my parachute pants, grab a glass of “Miracle juice”, and bust out the ol’ Snake Oil.

The idea is that once you’ve fixed your site, everything will be OK. But it doesn’t really work that way, does it? A site redesign is usually a major undertaking. Often, it is so overwhelming that it never gets done or it is not fully implemented. Is there a better way?

You bet there is. The Japanese call it Kaizen. Tony Robbins calls it CANI - Continuous and Never ending Improvement. I just call it continuous improvement. It works like this. You simply make improvements to your website on a continuous basis. Nothing dramatic, really. Just make small improvements on a daily basis. There’s a great book out there called “Web Analytics: An Hour a Day” written by Avinash Kaushik — perhaps you’ve heard of it. Basically, the book details small steps you can take every day to improve your website. 

Here are some of my suggestions for using analytics to make small changes to your website:

• Look at your top keywords. Are all of them being advertised with PPC?
• What are your worst PPC campaigns? Do they need negative keywords or alternate ads?
• Are you advertising all your top pages?
• Click map study. Can you re-arrange your navigation to be better matched with actual clicks?
• Study your popular pages. Are they optimized?
• Meta tag study. Do all pages have proper meta tags? Any duplicates?
• Tagging Study. Have all pages been properly tagged?
• Referrers Study. Who are our top referrers? Can you do better? Who are the new referrers? Can you advertise on them?
• Advertising Study. How are your ads doing. Find new places to advertise.
• Product feed study
• Best performing vs. worst performing pages. Both bounce rate and exits on path.
• Shopping cart analysis. Fallout.
• Affiliate study
• 404 errors review

What are some of the small things you do for your website? I’d love to hear your suggestions.

Okay. Into the deep end first.

My time away has been due to busyness. Perhaps some readers have heard, NextStage Received its first patent on its Evolution Technology. For years we’ve been intentionally below the radar, now we seem to be becoming a recognizable object rapidly approaching from the far horizon. Now that we’ve left nap-of-the-earth flying I’m able to discuss things more openly, me thinks, hence some of my responses now and in the future.

Are the visitors happy?

One of the things I did while I was away was talk with a few people (about 100 so far) about what I’ll call The Purpose of Web Analytics. I did this research because of something I wrote in this thread above, “…all these analytics are worthless unless they create happy, satisfied visitors, yes?”

I’ve talked with upper management in education, politics, at national telecoms, financial institutions, transportation, recreation, … a pretty diverse group. Most of them were involved in marketing products or services or some other form of gaining marketshare. None of them were web analysts or involved in web analytics except that they received reports and were expected to act upon them. None of them were particularly happy about being made accountable to a system that (they believed) wasn’t measuring … and here’s where the challenges really made themselves known.

What was being measured? Lots of money was being spent and lots of people were being told that the measurements mattered and as one fellow explained, for the amount of money they were spending they expected some consistency.

“What do you mean by consistency?” I asked.

He pretty much didn’t know. He and those with him said lots of things and it could be distilled to a general dissatisfaction that there wasn’t a single model that they could consistently use and derive actionable meaning from. The dissatisfaction grew geometrically when the discussion got into executives making decisions based on sales presentations rather than a given product’s specific informational abilities.

At one point I leaned towards a speaker and quietly said, “Remember, Joseph friend,” and everybody laughed because the tension in the room was broken.

I reference these anecdotes because one of my original hopes for this platform was an increase in understanding and acceptance of some mutual goals regardless of discipline or tool platform.

In the end, doesn’t it all come down to “…all these analytics are worthless unless they create happy, satisfied visitors…?”

If I can’t act on it, it doesn’t exist

The next item I wish to thread into this discussion comes from an online conversation I had with Critical Mass’s Christopher Berry about why web analytics seems to be a harder sell in Canada than in the US. You can follow my side of the conversation in Canadian Based Business Differences — Responding to June Li, Christopher Berry and Jacques Warren, Responding to Christopher Berry’s Vexing Problem, Part 3 post, The Language of Web Analytics - The Hard(er) Sell in Canada, Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 1, Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 2 and Communicating Science to Business and Vice Versa and links are provided to Christopher Berry’s side on the conversation in those posts. I’ll invite people to pay particular attention to Communicating Science to Business and Vice Versa because (and as Mr. Berry noted) the summation is what counts, “Business is different. Business (me thinks) tends to be more ‘Tell me how to use this’ hence most business proposals and reports start with Christopher Berry’s nuggets then go into explanations.”

My research is convincing me that (what I recognize as traditional) web analytics is going to be losing its authoritative power in the coming years. I think web analytics (and yes, this does go back to my original hopes for this blog) will evolve (just as anything will if it is going to survive in a given changing environment). What will it do and look like? I have some ideas, of course. Just ideas at present, though. More things to research before putting down on paper (or in a blog) at present.

This does tie into my comment re Avinash Kaushik’s “…we shouldn’t use ill defined engagement metrics as a proxy for something solid like a sale.” I’ve been an oft-times unwilling father-confessor to businesses frustrated by ill-defined metrics of any kind and wanting something that is justifiable a) financially, b) scientifically, c) arithmetically (forget mathematically) and d) produces some kind of “do A, get B”, “this-equals-that” link between action and outcome.

The comment I love about this is “If I can’t act on it then it doesn’t exist”, ie, it’s noise, a distraction at best and something best ignored. This was a wonderful statement used in a business practices discussion.

I’d really enjoy being involved in a web understandability/measurement/future usability discussion that has as its theme “If I can’t act on it then it doesn’t exist.”

“To measure and analyze on and offline behavior and then try to predict who to market to by figuring out what they think is not doable with one tool or one metric.”

I responded earlier to this comment. People who attended either the Toronto ‘08 or SF ‘08 eMetrics conferences are probably well aware by now that NextStage has patented a technology that can determine how someone is thinking through any programmable device. I won’t go deeper into the topic here except to offer a comment I posted on Jim Novo’s blog about the {C,B/e,M} matrix and its use in marketing and analytics.

Picking up where I left off with Jim Novo’s comments in this thread…

I finally had an opportunity to read Jim Novo’s Measuring Engagement and its related Framework for Engagement posts. I truly enjoy Jim’s writing style and the points he makes.

I especially enjoy and appreciate his referencing Relationship Marketing because it places people center stage. Understand people and you can both understand and predict what they’ll do. Watch only what people have done and you can only understand their actions in a specific historical context, you can only predict what they’ll do when the confluence of events that led to their original actions repeats itself. Exactly (and don’t hold your breath). Relationship marketing works at the question “…all these analytics are worthless unless they create happy, satisfied visitors, yes?”

Jim writes “The challenge with this model - and probably why it isn’t more widely known - has been the data, it’s a very analysis-intensive model…”. Yes. Agreed. If Jim (or others familiar with these concepts) is reading (or perhaps at the next conference we meet at), I think this is where being able to substitute cognitive heuristic models makes sense (see Liberation and Heuristics or Responding to Christopher Berry’s “A Vexing Problem, Part 4″ Post, Part 1. I’ve also written elsewhere that I often wonder why more businesses don’t make use of cognitive heuristic models).

For example, I’ve recently been applying heuristic models to helping adult second language learners increase their language acquisition abilities. That’s a traditionally very tough nut to crack and (so far, anyway) I’ve been able to isolate neural activity that tends to make adult language acquisition challenging. Example 2, using heuristic models in the above grew out of learning which heuristic models are used (non-consciously, of course) by which personality types in their decision making processes. This non-conscious heuristic model selection process is being integrated into NextStage’s Rich Personae. These and some other areas of my studies are intensely data-rich models that can be reasonably simplified via cognitive heuristics.

I also strongly like your concept of dis-engagement, although I tend to use a methodology that incorporates “satisfaction” into the scaling system (see Meet Online Engagement’s Little Friend, Satisfaction. I shared that the complete form of this during a discussion at the SF ‘08 eMetrics. It looks something like the figure on the right.

Some definitions to help in understanding; the x-axis is Engagement and is a measure of the amount of pleasure or pain an activity is giving you. If something is giving you either pleasure or pain to any degree your attention is focused on it, hence you are engaged by it according to the definitions documented in Attention, Engagement and Trust: The Internet Trinity and Websites. The y-axis is Satisfaction and is a measure of acceptance and rejection of some internal state and/or external event.

I believe what you are referencing as “dis-engagement” is what we recognize as the slide from high acceptance to “0″ acceptance. Note that this is not rejection (as rejection is an active negation of acceptance) it is a lack of acceptance. I appreciate that the difference might be subtle and I believe that difference is significant. Rejection — the active negation of acceptance — can be thought of as someone pushing something away. Zero-acceptance is the point where one can “take it or leave it” and the internal state and/or external event does not have any value assigned to it, hence doesn’t register strongly in the mind/brain.

Mapping this figure to real world experience, you always want visitors/consumer/etc to be in the first quadrant (where the green curve is). People are both positively engaged (they like what’s going on) and positively satisfied (they accept it gladly). Depending on what you’re selling you may or may not want people in those other quadrants. The second quadrant (bottom yellow curve) indicates someone focusing on painful experiences or information, the fourth quadrant (top yellow curve) indicates someone who finds pleasure in painful experiences or information. The third quadrant (red curve) is where visitors/consumers/etc often end up and marketers/businesses don’t want them to be — the former are actively psychologically and physically moving themselves away from a business/product/service.

I’ll offer that the above is also a reasonable representation of your:
1. Define / Measure Engagement – any way you want to, as appropriate for your business; whatever activity or combinations of activity you feel appropriate
2. Measure dis-Engagement – the absence of Engagement, as in the visitor / customer stopped doing whatever it is you define as Engagement for your business model

I think where the image above (and the math behind it) adds real value is with your “3. Take some kind of Marketing or Service action to slow or reverse the dis-Engagement with dis-Engaging folks” because it provides enough information to know how, exactly, visitors/etc are “right now” interacting with your marketing information.

I also agree whole-heartedly with your statements about predicting “dis-engagement”, etc. I would love to see the data you used in your example and apply it to the above. I’m willing to bet that satisfaction/acceptance was the real driver (and I won’t get into the depths of group satisfaction/acceptance states here (really, Joseph? You’re going to leave something out? Whatever for?)). I did get a kick out of your graph of email response rates falling over time. It was very similar to the results we found in our research on how to design an effective email newsletter. Bravo! I always love it when our findings match others’. Gives me hope we’re doing something right.

<ASIDE>
For what it’s worth, much of the rest of what you’ve written in your post is so close to what we learned in our email newsletter research that the overlap is astounding. Not surprising, I guess, as you’re listing an email-based experiment. It would be interesting to learn what else the rules we discovered pertain to. Let me know if would like to explore this.
</ASIDE>

You also list an implication about sending different messages to different segments. Yes, agreed. I believe the above allows for much more targeted and action-driven messaging (based on much of what I’ve shared above).

Perhaps, in the end, we’ve derived nothing more than a simplified mathematical model (complete with suggestions for better outcomes) of Relationship Marketing?

Whoosh!

Took me two days to put the above together folks. Sorry for the delay. More to follow. Soon.

Promise.

If you missed the first year in Napa, you gotta head to San Fran this year! Allow me to explain how X Change differentiates as I see it:

• Conversational. You don’t sit in a room and listen to people drone on in front of their powerpoints. People sit in Socratic circles and talk about a topic of interest in “huddles.” The huddle leader will bring up a topic, perhaps riff on some hard-learned experience or data point related to the topic, and ask for commentary from the participants. The conversation then flows, like Jazz, until there’s a cadence, then the huddle leader phrases a few more notes and progression begins again… Its atypical format depends on participants for success. No one is going to sit there and read you slides and provide one-sided opinions. You won’t just be sitting there listening (unless you want to). The best huddles are interactive and encourage active participation in the pursuit of shared knowledge, not passive reception of an individual’s knowledge.
• Focused. The huddle topics are highly specific and deeply relevant to the real world practice of web analytics today - from attribution to mobile measurement to integration to privacy to team structure, the huddle leaders selected topics that interest them to share with the participants. The focused conversational format should lead to symbiotic exchanges of information directly relevant to your job.
• Small. 100 people, 20 huddle leaders. You get to make meet interesting people and build working relationships with them. Cool folks like Bob Page, Rachel Scotto, Marshall Sponder, John Lovett, Jared Waxman, “Bob” Dylan Lewis will be leading huddles and hanging out. The Web Analytics Tuesday event will probably be bigger than the whole X Change conference!
• Exclusive. The huddle leaders were hand selected. In attendance will be industry leaders, corporate executives, industry analysts. All of the attendees work with analytics. And for gosh sake, it is at the Ritz in one of America’s most beautiful and eccentric cities. 

I think X Change is a unique experience and a worthwhile event where you get to really connect, and well, exchange (!) expertise with your peers and go home with new knowledge. At least I did last year. I’ll be leading a couple of huddles, one of the web analytics team and one on knowing when you’ve outgrown you analytics tool, so say hello when you see me. 

Make sure you check out the official web site at Semphonic and sign up today. The event will sell out soon. 15% discounts are available for Web Analytics Association members. 

• Jim Sterne, yes … that JIM STERNE, will be joining us as a participant! I’m not sure if I have seen Jim at an event where he is not the absolute Master of Ceremony and running the show so it will be nice to see my old friend (hopefully) relax and have the time to share his own experiences in our industry
• Many of the most influential vendors in the digital measurement ecosystem are coming, including Aaron Gray (WebTrends), John Squire (Coremetrics), John Dawes (Tealeaf), Eric Hansen (SiteSpect), Mark Treschl (OpinionLab), Eric Head (ForeSee Results), and more I am surely forgetting.
• We’ve also invited some of the best and brightest consultants in the business including Josh Manion and Bill Bruno (Stratigent), Jacques Warren (WAO Marketing), Aurelie Pols (OX2), Andy Fisher (Avenue A/Razorfish), Matt Jacobs (Digitas), Dan Sheilds (Wicked Science), Craig Danuloff (Commerce360), and of course the team from Semphonic!

By design, the X Change conference will be about 1/5 expert practitioners (the huddle leaders, who are amazing), 1/5 experts from the vendor, agency, and consulting community, and 3/5 practitioner attendees. Given that the huddle leaders are practitioners themselves, we feel that we have a pretty solid (and extremely rare) ratio of experts to attendees, and that this arrangement will provide amazing value to attendees.

It doesn’t hurt that the experts we selected are among the best in the business.

Yeah, there are some folks that are missing, to be sure. It would have been great if someone from Google Analytics could have come, and the guys from ZAAZ, and folks like Steve Jackson from SATAMA and others across the U.S. But there is always next year, right!

Anyway, if you’re thinking about joining us in San Francisco the time to register is now. Check out the official web site at Semphonic and get signed up today.

Please read what Jared has to say and please take 10 minutes to complete his survey!

————————————————————————

Shortly after Eric Peterson invited me to lead a few of the X Change conference huddles this year, I realized something slightly ironic … We collect tons of data about our customers, but we still have far less cold, hard data when it comes to ourselves. In other words, what are the KPIs by which we can benchmark our own analytics practices against similar organizations?

If you’re not sure why this should concern you, ask yourself the following questions:

• When is the last time you received special recognition from your boss for how cutting-edge your analytics program is?
• How much of your last analytics budget proposal got approved?
• If resources were constrained, do you know which one analytics project would give your org the biggest leg-up?

The missing puzzle piece behind all these questions is: the right benchmarking data.

Chances are your boss wants to give you praise, but isn’t sure just which of things you’ve accomplished merits awarding you that mounted, singing Bass fish trophy (stuffed with cash, of course). And the budget? Management never wants to hear they’re about as cutting-edge as an animated gif. Armed with the right data, you can lay out for them what it’s going to take to catch-up, or take the lead.

If you really knew how your programs and capabilities compared to other companies in your space, you’d have the foundation for a pretty solid case for both the praise you deserve, and also the resources to beef up areas you’ve been slower to ramp on. This isn’t news, of course, and Tom Davenport and Jeanne Harris’ book, “Competing on Analytics: The New Science of Winning” stresses the point that it’s not just about what you are doing on your island, but it’s about being faster and smarter that your competition in leveraging data and analytics.

So how do you know how you stack up? For a start, fill-out this 10-Minute Benchmarking Survey that I assembled to help your organization get a read on where you stand. It was challenging to come up with every use-case or application of web analytics in practice today and still have it be short enough that you can complete the survey while on-hold with your mortgage company. Ok, it was impossible. So I settled on a sampling of valuable activities that are components of many successful analytics practices. I hope you’ll agree it is sufficient for a strong directional read within key areas: Tracking, Competitive, Testing, & Predicting.

So I humbly present you with the Benchmarking Survey, and will reveal the results at X Change, and of course share your score back with you for your participation, once the results are compiled:

Start the Web Analytics Benchmark Survey

I hope to see you at the X Change conference. I’ll be leading a huddle on Testing Strategies alongside Dylan Lewis. At Intuit we have a sort of competition going for who’s group can take testing the furthest. We each have such different organizations and challenges, and that has led to very divergent strategies. Yet we are constantly learning from one another and looking forward to learning from the participants about what their challenges and strategies have been.

I’ll also be leading a huddle on Using Real-Time Surveys to Improve the Customer Experience. By building our own Voice of Customer apps, we’ve been able to provide an immediately tailored site experience for the customer and equally as important, effect more lasting improvements by orchestrating the right influence within the broader marketing organization.

Finally, I’ll be leading a huddle on Competitive Intelligence Tools & Methodologies. Sounds clandestine and exciting, huh? Truth is, I’d tell you more about this huddle if I could, but then I’d have to make you sign three different NDAs and learn the Intuit secret handshake … So just sign up and I’ll see you in person.