Online Applications and highly-interactive web sites written in full-on programming languages are becoming the norm on the web, not the rare exception. Measuring these applications is challenging for a whole host of reasons. In my last post, I talked about the importance of automated testing to bring measurement fully into the application development life-cycle. In today’s post, I’m going to cover an issue that plagues application measurement, video and most other rich media measurement – how to balance cost vs. coverage when it comes to measurement.
The problem of cost in measurement springs from our basic model. With SaaS implementations, each server call you make to your measurement system costs money. Not a lot of money, of course. And for web sites, there has been little doubt that the knowledge gained from page measurement more than offsets the per-page cost of collection. On most web sites, a typical visitor might view six to ten pages. A really engaged visitor might view twenty to thirty pages. In an online application, however, the number of user interactions (every click, zoom, and setting) can be much, much higher.
Without a clear paradigm for a unit of measurement (such as a page view), it’s unclear which of these interactions should be measured. Capture every one, and you’ll likely find that the cost of measurement outweighs the benefit.
As I mentioned, we’ve seen similar issues with video measurement. Many of our media clients went from measuring almost nothing about video to detailed measurement of how far viewers penetrated into each and every visitor played. But to get that measurement, they usually had to make multiple server calls per video – sometimes as often as every 10 seconds. For a 2 minute video, that means a full-play results in 12 server calls. That’s a lot of measurement expense relative to the value of tracking abandonment per 10 second interval.
In fact, a lot of that more detailed measurement ended up getting pulled.
We’ve seen similar pull-backs when clients over-measured flash experiences – tracking roll-overs for example. After awhile, most companies find they aren’t using that level of detail about user interactions very effectively and the cost of measurement isn’t justified.
In an earlier post, I described a framework for measuring applications that included four core concepts: units-of-work, application state, rest-states, and performance. Part of the reason I settled on these concepts is that they provide a reasonable balance of measurement vs. cost.
A unit-of-work is a function within the application. The unit-of-work would typically encapsulate a single use-case (like getting driving directions for a mapping program or paying a bill in an online bill-pay system). The measurement system would make a call whenever a unit-of-work is initiated. With this call (and all calls) it would pass an application state.
The application state is a set of variables that capture all the important settings a user might have (like sort-order, filters, zoom level, tab open, etc.). The goal of application states was to be able to tie application settings to important actions (units-of-work) without having to pass measurement calls every time a state is changed (like a sort).
Now I’m going to add a new concept – milestones. Milestones would be way-markers within units-of-work. A milestone could be named and sequenced if desired. Whenever a milestone within a unit-of-work occurs, the measurement system would fire off a call. Every unit-of-work would have at least two associated milestones – an abandon milestone and a completion milestone.
Using the milestone concept, developers could control how much measurement is being done within a unit-of-work.
When a milestone is reached, the measurement system would automatically get a new application state and it would generate and pass a performance measurement for the previous milestone.
With this type of framework, there would be little need to track every user change or setting with a server call. Each measurement call would capture some significant action along with the associate states. On the measurement end, comparing states between milestones would make it easy to produce reports about what settings users changed when performing a unit-of-work.
The unit-of-work/milestone combination provides developers with a very granular way to control the level of measurement in the application (that’s important) and, potentially, to be very sparing about the cost of measurement.
It’s quite possible to duplicate some of this functionality in existing systems today. In Omniture, for example, you can code units-of-work using an s.prop variable. By turning on pathing, you can track movement between units-of-work. By using a combination of unit-of-work and milestone in a second prop, you can enable pathing within milestones. You can use a list-prop to capture application state or you can use a carefully formatted and delimited string in a normal string s.prop.
If you correlate that application state variable with the units-of-work s.prop and the milestones s.prop, you have basic reporting on how application states changed across milestones. Of course, you have to do most of the work to use those reports since the application isn’t going to really help you any. And you also have to do ALL the work to setup and capture this stuff.
Of course, that’s the whole point of this series. You CAN measure applications right now, it’s just not nearly as easy or powerful as it ought to be.
With that in mind, my next (and final) post in this series will tackle the last and probably the most difficult part of measuring applications – figuring out how to analyze the information to make it useful!When we talk with the development teams these days, I’m often the one who has to break the bad news that there is no reasonable automation strategy for testing measurement. Not only that, but the testing is extraordinarily difficult to integrate into basic development cycles. Someone has to run through a test plan by hand, then go to the web analytics tool and try and shift through the data to isolate the test results. It’s kludgy, manual, slow and error prone. A perfect storm for testers.
This is especially frustrating because measurement isn’t “critical” functionality. Nobody wants to slip deadlines or delay release for measurement.The changes I’ve talked about so far – especially in the measurement interface – would greatly reduce the burden on programmers to get measurement INTO the application. But they really wouldn’t do much to make it easier to test.
I’d like to see Omniture and Microsoft and other application and measurement vendors integrate testing functionality directly into the measurement system. Ideally, developers should be able to configure “testing” as an overlay on the existing measurement interface. In other words, I wouldn’t add any code to test, I’d simply set a variable or an object that controlled the “testing” state.
When testing was turned on, every measurement call would be mirrored to an additional destination. Since online applications typically run locally, I see several ways this might be handled. First, the application could potentially save the measurement in a flat file on the local system (given permission). Alternatively, the application could save the measurement calls to an interface object in some easily parsed format. Finally, the application could pass the calls back to a secondary server that simply recorded them in a flat file on the server side or was part of a testing automation suite.
With this type of setup, testing automation tools could drive through a series of application steps which would produce a measurement file that could be quickly “diffed” with previous versions to establish correctness. This would dramatically reduce the amount of time new testing cycles took while greatly improving accuracy. Even if each test cycle had to be done manually, programmers would only have to worry about checking the measurement file once and getting a correct version.
Even without the automation aspects, this type of system would make testing applications in secure and mobile environments much easier. We often do the bulk of our testing using HTTP sniffers like Firebug, Charles and Fiddler. But these sometimes don’t work or conflict with the application on development networks and in secure environments.
As I think about this, I realize that in some ways this “testing” capability mirrors a universal tag. If the measurement library makes it easy to reflect measurement calls to multiple locations, you have the basic infrastructure for universal measurement.
That seems like a pretty good idea – something that would be almost trivial to add into a measurement framework of the sort I described in the first two posts. Pegging testing on a universal tag capability has the additional benefit that people may pay more attention to it that way. Testing is one of the most time-consuming, manual, and expensive tasks in software development. Unfortunately, as the least sexy task is a discipline with very little sexiness in general, measurement testing gets less real-world attention than the “best script” honors at an adult video awards program.
But if you’re going to build a really good measurement interface, building support for automated testing right into the system would go a long way toward making the incorporation of measurement into online applications routine instead of esoteric and painful.It’s obvious that online applications don’t easily fit the page view paradigm of static web sites. What’s less obvious is what should replace that paradigm. I’m not prepared to give a complete answer to that question. However, in our work actually building measurement into applications, there are four themes that have emerged and seem necessary to any good framework.
Measuring the Application StateWhen we build application measurement into traditional measurement systems, one of the most important concepts we’ve evolved is the idea of an application state. The idea behind an application state is that it captures all of the “settings” that are in place while a user accomplishes some action. Typical elements of an application state are things like “sort type,” “filter-type,” “zoom-level,” and “view-type.” In my last post, I used the example of a mapping application to illustrate some of the concepts involved in a good programmatic interface. It’s a useful example, because it’s a type of online application that virtually everyone is familiar with. In a mapping application, the application state is likely to include variables like “map type,” “overlay-type,” “zoom-level,” and “route-type.”
The idea behind an application state is simple. It allows you to track key combinations of interface elements by application function. It can also be used to reduce the number of calls you need to make to the measurement service (a critical factor for any high-volume application). You don’t need or want to capture every single mouse click in an application.
If a user clicks on zoom or sort, sending a measurement click is generally too expensive. Instead, we capture the entire set of application variables whenever key actions happen. In the map example, this would allow us to say what the most popular zoom levels are when viewing a route or adding a restaurant overlay vs. a hotel overlay. That’s incredibly useful information for GUI designers and its captured much more effectively in an application state than in any other fashion.
When we build an application state into a current measurement system, we generally pass it as a List Prop or a simple string variable. However, it takes some work to get it to tie to application functions in an intelligible manner. I’d like to see an application framework capture the state as a type of thing and allow it to be viewed and counted and analyzed in conjunction with different application functions.Rest States
One of the trickiest aspects of application measurement is what to do about applications where the interaction isn’t discrete. In our mapping example, panning and zooming are a good example of what I mean when I say not discrete. It’s easy to fire off a measurement call when someone clicks a button to change from a street view to a topo view. But when do you fire off a measurement call if someone is zooming? How do you know they are done?Ignoring these non-discrete interactions works for some applications but cripples measurement for others. So what I’d like to see is a measurement framework that includes a threaded callback mechanism that can be attached to various interactions and basically just checks to see if the interaction has stopped for some period of time. I call this stop-time a rest state.
Using my map example, when a user clicks on the zoom control of a map, it would fire a measurement function that started a thread (call it zoomwait). The function would be passed three things – the action it’s checking for, the wait period required, and the measurement function to call when the wait period has been realized. Let’s say we are checking zoom and we have a 3 second wait period. The measurement thread would check to see if a zoom had been done in the last 3 seconds. If it had, it would until at least 3 seconds after the last registered zoom time and try again. It would do this until it detected that 3 seconds had passed since a zoom. It would then call the measurement function specified which would make the actual call to the measurement system and pass the new application state.
The idea behind rest states is to make it possible to measure the application state AFTER the user has finished some set of actions. This makes it possible to capture many key application states without necessarily trying to measure every application state when a user is zooming, panning, or doing some other non-discrete interaction.
Measuring Performance
The current static web site measurement framework largely ignores performance. That’s a serious drawback, but one that’s hard to solve given the limitations on measurement with our current infrastructure. But for applications, measuring app performance is really, really important.
Application frameworks like .Net already include some low-level calls designed to make performance tracking around key functions possible. But not every development framework has this (many do not), and I think it’s a natural part of a measurement system. So I’d like a measurement framework to include this functionality.To measure performance, the system should have a way of starting a unit of work and defining it as being open-ended (not terminated by another start) or closed (terminated automatically if another unit of work starts). Units of work would be named and could be closed explicitly by the application sending a close call with the corresponding name.
The measurement system could use this concept in a variety of ways including measuring total time in application (by firing off an open ended unit of work named application start that is only closed on app clean-up), measuring total time in app areas, and measuring specific application times for functions like “sort” or “synch.”
Depending on how this was used, it could just as easily measure how long a user takes for an application (like filling out a form) or how long an application takes to do processing (like placing an order). In the current web environment, these two numbers are always conflated into page times. Providing the capability to break them out would be significant win in application measurement.
Units of Work
In my discussion on performance, I touched on “units of work,” the last and, in some ways, the most important of the application measurement framework themes I’d like to see a measurement system address.
When we measure static web sites, we often employ a methodology that we call functionalism. It works by assigning specific functions to each page and then analyzing their performance relative to those functions. It doesn’t work well for applications, however. The functional layer in an application isn’t page based at all.
I’d like to see this functional concept extended directly into the measurement framework. The idea would be to encapsulate all of the major application use-cases as units of work. The framework should make it easy for the developer to kick-off a “unit-of-work” and, as with performance, control the interactions that can occur (in other words, to be able to write logic that determines what to do when one unit of work interrupts another or takes place inside it).
Instead of page-based tracking, I think the basic level of tracking in an application should be at the unit-of-work level. Functions inside the unit-of-work could be tracked as “steps” or “tasks” depending on whether the unit-of-work was ordered or unordered.How would this work? Going back to our mapping application, a unit-of-work might be triggered when a user does an address search.
This “search unit” would persist as a user pans, zooms, and adds an overlay for hotels, and then clicks on a hotel or two. The unit would end when the user enters a new search or exits the application (and possibly on other actions).
The measurement system would track how often a visitor did each type of unit of work (how often the “search unit” was triggered, how long the user spent in the search unit, how long key actions (like the overlay fill) took to perform, the use-state at each point in the unit or work, and what the visitor did to exit the unit.Summary
I’m not sure that these four concepts (unit of work, performance, rest state, application state) are a complete description of an application measurement framework, but they would go a long way to providing logical, rich, and appropriate measurement to applications. Some can (and should) be done even if you are coding in current measurement systems – though all require significant hacks to do right now. But if Microsoft and Omniture are working to build robust application measurement, creating a measurement framework that embedded these concepts would truly advance the state-of-the-art. In my next post, I’ll take up one of the true pain-points in application measurement – testing.
Most measurement systems now work similarly – they rely on a small javascript program (the tag) that allows a coder to customize the values for certain variables and then call a function that assembles those values, adds a set of environmental variables (like browser and referring site), creates an image request and then executes that image request. All of the custom and environmental variables are passed as parameters in the image request and are decoded by the vendor when the request is logged on their end.
Think about it – all the fuss and muss of tagging – just to make a call to another server and pass a few values. In the world of applications, there are many ways of handling this basic task, by far the most common being to use a web service. Web services are vastly more convenient for the application developer than the current method (which is well suited to static HTML). Indeed, it's hard to imagine a more primitive interface for web communications than passing parameters in image requests.The virtues of web services include the fact that they are directly supported within every significant programming language (including direct support within the programmers development environment), they provide methods that allow for back-and-forth communication and error-handling when necessary, they provide more control over the nature and amount of information passed, and they typically encapsulate the information passing into much more logical and understandable programmatic units.
It’s true that some measurement vendors provide data insertion APIs. Omniture, for example, has one. But the Omniture Data Insertion API is not a real web-service. It’s just a ”post” based mechanism for sending data. It’s might be the right method to select if you’re coding SiteCatalyst measurement in an application right now – but it’s hardly a robust interface.
But to me, the most important part of building a good interface between application system and measurement system isn’t the handshake mechanism, it’s the structure of the programmatic interface.
Here’s some sample SiteCatalyst code that might get attached to a click-handler when an event is firing:
var s=s_gi('rsid');s.linkTrackVars='eVar34,prop8,eVar8,prop9,eVar9,prop32,eVar32,eVar50,events';s.linkTrackEvents='event3';These 13 lines of code create the basic SiteCatalyst measurement object (s), tell SiteCatalyst which variables to pass (linkTrackVars), which events to pass (linkTrackEvents), assign seven different variables and one event, and then call the s.tl function that generates and fires the image request.
Strewing chunks of code like this – even if it’s translated into C# or java - all over your application makes for an incredibly messy situation: it’s hard to document, hard to code, hard to test, and very hard to maintain. And like most poor coding mechanisms, it means that if you want to change measurement systems, you’re faced with a significant re-wiring task.
If this is the wrong way to integrate measurement into an application, what’s the right way?
Well, it isn’t to give the programmer a function call like this:
doPageView(“pagename”, “events=1,3,5”, “evar1=test;evar2=test2”, “prop1=red”);In truth, even this would be better than what we have now (though it’s easy enough to create this sort of wrapper). But this simple function library approach doesn’t create a structure that embeds measurement logically inside the application.
What I’d like to see is a system where you started with a basic measurement object. This object could be instantiated once by the application (though applications could use more than one) and would contain, at minimum, the target account information and the configuration of environmental variables (what needs to be pulled and passed).
Ideally, environmental variables would only be passed once on invocation of the application (since apps aren’t stateless like the web, the challenge of inferring sessions and the necessity for re-passing environmental information with every request doesn’t exist). This would dramatically reduce the amount of information required with each call and the corresponding network bandwidth required.
Underneath this global measurement object would exist one or more measurement objects that inherit the global state and allow for custom mappings of application variables. The custom mappings would let the developer populate measurement variables from object properties (like Name, ID, Value or isVisible), application variables, static values or special function.
In this way, a measurement variable could be populated with the Name Property (for example) of any object to which to which it was attached. If attached to a Form Object, it could automatically create something like a pagename variable for an application by mapping the Form's "name" property to the measurement pagename variable.
The measurement object could also be configured to support any or all of the default or specific methods that objects in the development environment have. In other words, a measurement object could be configured with a specification for “load”, “unload”, “click”, etc. This would allow the developer to automatically fire a measurement call in response to either specific functions (like a search) or general functions (like a form load).
With this approach, the developer can take advantage of built-in UI constructs and also manage a small set of application specific variables that will automatically be translated into the measurement system equivalents whenever an actual measurement call is made.
When I create an object in the interface, I would then attach my measurement object to it. When the object is loaded, any handlers for the measurement object would be fired if present. It would automatically set its values, give the programmer an opportunity for a call-back (which should be rare) and then fire the request.
This is rather abstract, so let me show how this work in a more concrete fashion.I’d start with a measurement object in which I would set the global information about the account (and maybe the vendor) I’m sending information to along with the environmental variables I want to pass on invocation.
Next, I’d create a custom variable object that described the specific variables I want to pass. Let’s suppose that I’m working on a Mapping Application and I want to collect the viewed area (the region/country/city shown in the Map Title), the map view (road, topo, satellite, etc.), the map state (overlays, zoom level), the language of the map, and the visitor identity. I would map each of these application variables to a specific measurement object variable. Next, I’d assign the click-handlers I want the measurement to fire on (I may want it on every re-draw but I might also want it on a more limited set of actions such as onSearch and onMapChange). Finally, I’d attach the measurement object to the underlying UI object.
That’s pretty much it. Once the object is attached, the programmer is done. When the object to which the measurement is attached fires an appropriate handler, the object would inherit its default state, populate the variables (like viewed area, map state, etc.), and then fire the measurement request.
Depending on my application, I might need only one custom measurement object or I might need quite a few. But with this type of system, the measurement objects would be embedded into the underlying GUI objects and their handlers. The measurement layer would deeply in-grained, it would provide inheritance, and it would provide configurability. It would be much, much easier to maintain such a system. The measurement objects themselves would be compact and discrete and they would be directly attached to the objects in which they reside.
Having a measurement integration at this level inside an application development framework would make the development process dramatically easier. Indeed, it would be possible to achieve a very high-level of measurement with a simple default configuration of the measurement object that might simply “fall-out” of a single drag-and-drop of the measurement object into the application. Such a default configuration might capture the object name and title of every form on load and unload, the object name and title of every “play” movie action, and the selected value from any Combo-Box. But it would also provide a high-level of control within the application for measurement at a very fine-grained object level.
As I mentioned in my first post, there really aren’t any serious barriers to building this type of system. There are probably different ways of achieving a deep measurement integration or designing the measurement objects – and I’m not confident that what I’ve suggested is the exactly right answer. But some form of deeply embedded measurement at the object level within the app framework seems both inevitable and right. Having an embedded measurement system won’t solve all the problems in application measurement, but it would be a big step in the right direction.
In my next post, I’ll take up a thornier issue – the right measurement paradigm in a world where page views and link clicks aren’t the ultimate measurement objects!When I blogged on the deal, one of my main points was that while the strategic rationale makes sense, capturing the benefits will not be a slam dunk. When analysts think about this deal, they tend to assume that Adobe and Omniture will be able to deliver significantly enhanced application measurability. The question for them is how big a win is that? But I don’t think the barriers to measuring applications are necessarily simple or exclusively technical.
I had the good fortune to meet Michael Scherotter of Microsoft at X Change and we got a chance to talk in-depth during what I think was the smallest Huddle of the entire Conference in that brutal late-Friday-afternoon close of conference time slot. I don’t blame anyone who split out to enjoy some pretty spectacular San Francisco weather and it ended up working out great for me. It was a really interesting wide-ranging conversation but of particular interest was Michael’s current focus on creating better and more integrated measurement for Silverlight – the development environment that may end up being Adobe’s most serious competition.
Application measurement has been on mind in our work this past year; partially because of the work we are doing in mobile. For the mobile space, applications have become a huge focus – and while mobile apps have some unique measurement challenges, they share many of the same challenges as fixed web apps. In addition, we tend to work on implementations for more complicated web sites, and in the past year we’ve tackled a range of implementations that were really applications not web sites. We even did a whole class at Think Tank on measuring applications, and though I didn’t teach that class, I wrote a goodly chunk of it.
So we at Semphonic have had lots of recent practice experience in measuring applications and how frustrating it can be. In today’s post, I’m just going to outline what I see as the core problems and then I’m going to follow-up with a more in-depth look at each.Building a framework for application measurement involves at least five (five!) significant problems.
Of these, the most obvious and maybe the most important is the mechanism for incorporating measurement. The current art of web analytics has been built around javascript tagging – with programmers doing page customizations and writing all sorts of little functions for click-handlers. This style doesn’t work for applications. Not only is javascript the wrong tool for the job, but this approach leads to far too much spaghetti code and far too much work for each measurement call to be appropriate in a development environment.
So the first task for measurement is to create an approach that lets developers integrate measurement quickly and easily. Programmers expect to be able to accomplish 3rd Party functionality like this with APIs and Function Libraries that require no more than a single call per invocation and encapsulate measurement variables in some logical structure.
Creation of a modular API or function-based approach to measurement is largely a technical challenge and really should be fairly straightforward. But the second major problem is more challenging. Application measurement has been shoe-horned into a measurement system designed for web sites based around page views. When we do application measurement, we have to spend a significant amount of time trying to come up with a translation between the way the application works and measurement concepts like page views and paths. It’s usually a clumsy fit. Omniture has done a decent job building a framework for video measurement – but I think that’s a much easier problem to solve. Useful analysis is going to require a real framework for application measurement and that doesn’t currently exist.
Problem three is one of the most neglected aspects of tagging in general – testing. We see lots of implementations where the level of testing on rollout is just pathetic. That’s a bad idea even when you’re tagging a traditional website. But you can’t get away with sloppy or non-existent testing when measuring applications – not least because your development groups won’t let you. Applications are built in formal stages with significant and carefully controlled testing.
There is no automated testing system for web analytics right now (though Observepoint moves in that direction) and it is much, much too hard to actually test. Having to integrate all sorts of new testing steps into an application process sucks down time and resources and makes getting measurement much more challenging than it should be.
Problem four is a combination of cost & performance. The traditional web analytics cost model is based on server calls. The cost of measuring web sites is generally considered fairly reasonable. But for many of our clients, trying to micro-track inside video has resulted in lots of server calls – more than the information is sometimes worth. For applications, the problems are likely to be similar. Will the server call model really hold up in application measurement and how much will it impact the style of measurement called for?
Lastly, we have the problem of analysis. Just as we don’t have a good framework for capturing and showing application usage, we don’t really have an analytic paradigm for using the data. Without such a paradigm, the whole exercise to integrate more measurability in applications could easily be an expensive waste. Indeed, it’s almost guaranteed to be an expensive waste in most early attempts. One thing I’ve certainly seen in our application measurement is that in addition to the usual concerns of web measurement, when we code application measurement there are almost always a whole set of measurement requirements and questions generated by the GUI designers. You might think the same would be true of websites, but it really isn’t. GUI application measurement is generally a new class of analysis in our field and in our experience so far it changes many aspects of what you collect and how you use the data.
I wouldn’t classify any of these concerns as trivial – and only the first (modularization) and third (testing) are primarily technical. Adobe or Microsoft really don’t have to deliver on every one of these challenges to be successful. Solving the technical problems would, at the least, open the door to easier resolution of the other challenges. But a really good application measurement framework would solve all of the first four problems and might even advance a point of view on good framework for analysis.The Really Big Topics at X Change
I’ve been doing a series of posts on thoughts from X Change Huddles and I started with two on behavioral integration of survey data – which was the very first Huddle I took part in. But this was by no means the hot topic at X Change. In fact, the Think Tank class on survey/behavioral integration was one of the least popular offerings we did (much to my chagrin). What were the really big topics? There were two – warehousing online analytics data and using a universal tag. Both came up over and over again. I was mildly surprised by how popular the first was – shocked by the amount of attention given the second. What’s interesting is that the two topics are more related that people might think – and that’s what I want to talk about today.
People have been talking about integrating online and offline data for years of course. But a couple of things this year felt different. Quite a few companies were actually doing real warehousing projects. Some of these were deep into production others felt more exploratory. Either way, people were clearly moving toward making this a reality. Given the broader economy, that’s impressive. Warehousing projects tend to be large investments – not easy to do in a seriously down economy.
The other thing that felt different this year was the number of companies that simultaneously feel confident of their basic web analytics infrastructure and reporting but dissatisfied with the maturity of their analysis. Achieving maturity in analysis has always been a problem – but in past years the big issue always felt like it was just achieving a quality infrastructure and reporting system. It would be an exaggeration to suggest that those issues have gone away but it felt to me like the balance has shifted. A lot more companies seem like they are getting quite a bit from their web analytics implementations and have reached the point where the existing tools just won’t give them any more.
Unlike data integration, the push for a “universal” tag is a lot newer. The idea is to implement a single point of data capture and then move the data to each of the analytics vendors who consume it. A single tag is appealing to many enterprises – after all, it isn’t unusual for a company to have five or six measurement tags on a single page. In addition to one or more full web analytics tags, ones for advertising, video, and various other niche measurement functions are all too common. A multiplicity of tags starts to be a performance burden and a nuisance from an IT perspective.
In some ways, the name “universal” tag may be a misnomer since not every universal tagging strategy even uses a tag. In some systems, wireline capture is used to grab the data which is then piped to the various information consumers. Alternatively, a single tag can capture the data to a central location which then moves just the necessary data to the consumers.
It’s a beautiful idea. The IT burden of tagging is dramatically lessened. Measurement across systems is unified with a single point of data capture (tag placement on existing pages is a common culprit in reconciliation projects). Performance on the page is no longer threatened.
Those are the immediate and obvious benefits.
But there are some less obvious but very real benefits that may be even more important.
X Change happened just before the Adobe deal, but many of those considering or using a universal tag approach named the ability to change analytic solutions as a key driver. For companies that may have migrated through two-or-three solutions in the last few years, the idea of being able to separate the capture layer from the reporting and analysis layer is pretty attractive. Let’s face it, one of the most difficult parts of switching vendors is re-tagging a site. And with the vendor landscape in web analytics looking rather less settled in some respects and the prospect of more acquisitions seemingly likely, the ability to swap software solutions without major infrastructure cost is undeniably appealing.
The last benefit I see is the one that may be the most significant – and it’s the fact that a universal tagging solution can enable warehousing with minimal additional fuss. After all, the whole point of a universal tag (however it’s implemented) is to create a single point of capture that can then move the data wherever necessary. Wherever necessary may well be your own data warehouse. A universal tag solution eliminates the need for you to move data to a web analytics vendor and then pay them to move it back to you. Instead, your warehouse could potentially take advantage of the same underlying data stream as all of the other measurement vendors.
That’s why I started this post with the observation that warehousing analytics data and universal tagging are more connected than people may think. Universal tagging is a potential enabler for data warehousing. That’s a good thing in-and-of-itself and I think the goal of warehousing and the savings around eliminating a feed from your vendor might become a real enabler of universal tagging.
And think about this - data warehousing in conjunction with a universal tag might make some intriguing types of analytics systems (like Google Analytics for basic reporting plus a warehouse for deep-dive analysis) possible that would otherwise not be.
So far, one feature that you just can’t get out of Google Analytics is a true data feed. And that’s a feature that Google may not even have on its radar. So if you wanted to warehouse analytics data, you couldn’t use GA for data capture. With a universal tag, a system like that suddenly becomes possible.
No matter how you slice it, I expect both warehousing analytics and universal tagging to grow significantly in popularity in 2010. We may not be quite at the tipping point yet, but I wouldn’t be surprised if it’s all that far away.
More thoughts from X Change on VOC Online Survey and Web Analytics Integration
[Before I dive into today’s topic, I wanted to highlight Jared Waxman’s comment on my last survey post. Jared made a really interesting point about a different way to think about sample size. I totally agree with his take. and it’s a different perspective than you usually get.]
My last post on online survey and behavioral integration made the point that the demands of behavioral analysis (which typically involve fairly small subsets of the total population) make large sample sizes an absolute requirement if you really want to take advantage of behavioral integration and not just say you have it. But the news about behavior integration isn’t all bad – even if you’re primary interest is on the survey side. Larger sample sizes are a burden. But one of the biggest concerns I heard during X Change was the validity of their online survey samples - and that's something that behavioral integration can help establish.
How do you know if your online survey sample really is representative?
From a pure survey perspective, you can take a methodological approach. A good methodology goes a long way toward getting you a good sample. But the online world is not as settled from a methodological perspective as the offline world and I heard lots of concern about the online surveys actually deployed at the enterprise level.
In the traditional survey world, key demographic variables can be used as a check on the validity of your sample. If you’re polling a presidential election and your gender, age or ethnicity break-downs don’t match the registered voter (or likely voter) population, you know you have issues.
But demographic variables in the online world provide no similar re-assurance. You have no independent measure of your actual demographics. Even if the online survey demographics match your broader audience profile, that’s no guarantee – it might even be suspicious. Of course, you may be able to match your online survey demographics with traditional research that includes a segmentation of online.
Even a good match here is no guarantee that you aren’t mis-sampling significant types of traffic. You may be missing out, for instance, on your natural search non-customer base and you’d never realize it with this type of check.
If you are doing behavioral integration, however, there is a simple and powerful way to validate your sample. Simply run a set of behavioral profiles against your survey population. Obvious variables to look at include pages viewed, acquisition sources, loyalty, and success event counts. With a large sample size, you can even check on geography.
In addition to the high-level behavioral variables, it's not a bad idea to check usage of key content types. It's possible, for instance, that your "job-seeking" population looks rather like your "shopping" population from a macro-perspective of sourcing, page views or time-on-site. But it would be distressing to find that your increased satisfaction scores are being driven by an increase of job-seekers coming to your site in a down-economy. Of course, surveys have an internal check (visit reason) that might catch this; but the behavioral measure is probably a more reliable indicator in this particular case.
If the behavioral profiles for your survey sample match that of your broader population, it’s an extremely powerful confirmation that your sample is valid. This is a great way to once and for all quash doubts in your organization about whether an online survey is really representative.
This works even if you are pulling a criteria-based sample. If you’ve made the decision that you’re only going to survey visitors with 5+ pages, you can compare the survey population to the total population of 5+ page viewers. As a bonus, you can also get a good sense of what you’re missing when you do that.
On the other hand, if you detect significant mismatches in the behavioral profiles, this can indicate problems in your sampling.
If you do find problems in your sample, you shouldn’t assume that you can just weight the population to fix the problem. Suppose that you find your survey sample consistently undercounts your natural search sourced population. It would be a mistake to simply weight what natural search respondents you have up to a representative number and assume you’ve fixed the problem.
You may only be sampling engaged natural searchers or natural searchers from brand terms. These populations may be fundamentally different than the broad natural search population.
You may need to weight by a combination of factors (e.g. pages viewed & source or pages viewed & source & visit number) or figure out how to actually adjust your sample.
Either way, using behavioral data as an independent confirmation of survey sample is a powerful tool. Not only can it help you figure out if you are getting a good sample, it’s a tremendous tool for PROVING to your organization that you are getting a good sample.
With organizations increasingly relying on the data collected from online opinion research, behavioral validation of the sample population is something you really should consider. After all, it’s really bad if your sample is wrong and people are relying on your numbers. It’s almost as bad if your sample is good and people don’t trust the numbers. And even if your sample is good and people rely on your numbers, an independent confirmation and the peace-of-mind it brings is still worth having.
Is it worth doing a behavioral integration just to validate your sample? If you take your research seriously (and you should), then I think it is. There are many other benefits to integration, but this is an oft-overlooked and quite substantial win that is available without all that much effort or cost. It’s not often I get to write a sentence like that!