SemAngel

O.D.ing on Analytics: Notes from the Berlin X Change

2013-06-16T18:23:28-07:00

As I write this, I am making the great swoop over the upper reaches of the North as I fly home to San Francisco from Berlin. X Change Berlin is over – X Change Los Angeles lies ahead. It was a great pleasure, as always, to be there and we’ve come a long way in a single year. This year’s X Change was, I think, better in nearly every respect. We had a larger group. We had more practitioners. We had more and better conversations. We had a better hotel. We even, though I cannot suppose this will continue to improve every year, had better weather.

For me, at least, X Change is an arduous journey. Enjoyable but deeply exhausting. The Web analytics equivalent of a crazy weekend in Vegas. It doesn’t help, of course, to deal with the inevitable pains of trans-Atlantic flying. I arrived in Berlin mid-afternoon after missing my connection in Amsterdam (late out of SF) and had the pleasure of seeing Matthias Bettag again. There’s no better greeting to another country than the face of a friend.

I was so tired I had a hard time staying awake till dinner and collapsed not long after to the expected first night’s troubled sleep. Monday, unfortunately, is my hardest day at X Change – Think Tank Training day.

I did my current favorite class – The Analyst’s Toolkit. It’s a compendium of analytics methodologies that we’ve developed over the last fifteen years: everything from Functionalism to Digital Segmentation to Site Topology. What I particularly like about the Toolkit class is the opportunity to show a broad range of analytic methods – I think it get’s folks thinking about which method is appropriate when and provides a broader grounding in the discipline than any other class I’ve done. I followed that with a very POPULAR course in attribution. We had sixteen attendees – which is a lot for a Think Tank. Paul Legutko and I have developed the class and it works its way from basic concepts of attribution (what is attribution, simple methods) to measurement infrastructure (stacking and warehousing) to more advanced concepts including a walk-through of building an actual, time-based full, stack attribution model along with concepts for extending the model. Going through the actual model building is hard – stretching my abilities as a teacher for sure.

Afterwards it was down to the InterContinental’s lounge for the reception (okay – I had time for a quick swim in the gorgeous pool). Quick swims seemed to be my theme this week, and at the reception I got to take a quick swim into the world of gaming measurement. We’ve only done a tiny bit of work in this fascinating industry but I got a great chance to talk with Alexandra Paun, an analyst for Wooga, and get an education in the state-of-the-art. Lots of interesting cohort measurement, really fast and agile development cycles – less segmentation than I think is likely optimal – the lack of which seems to be driven by the very short life-cycle of so many games. Fascinating stuff and definitely worth contemplating as a future Huddle topic. Most of us aren’t doing game measurement, but there are definitely learnings to be had there.

By Tuesday I was feeling at least reasonably normal, but I still skipped breakfast to get the extra time abed before the Keynote panel. We had four EU founders of analytics technology companies: Christian Sauer from Webtrekk, Simon Burton from Celebrus, John Woods from iJento, and Matthieu Lorens from AT Internet. Each of these guys has at least ten year’s experience in digital analytics and each has faced the inevitable frustrations, challenges and joys of entrepreneurship. It was a pleasure to hear these guys.

The most surprising moment was probably when Christian declared that being an analytics technology company would be a bad place to be in ten years. Which certainly got my attention – especially since the rest of us were quite optimistic as to the size and import of our current third wave (PCs, Internet, Analytics). His view was less dire than I at first supposed - more around the dangers of directly competing with Google Analytics – than about the future growth of analytics. He must be optimistic - Webtrekk has just entered the U.S. market with a presence in San Francisco. That leap across the Atlantic is huge and challenging, as I have good reason to know.

Then it was off to Huddles and their unique delights. There’s nothing quite like a Huddle – each is unique – even when the same leader hosts the same topic. You will never have the same discussion twice. There are so many I would love to do – so many I have to pass up. I try to balance topics, needs and sessions where I might be useful – and, inevitably, I get drawn into leading one or two due to sickness or late cancellation. I don’t mind - I enjoy it. I’m not as good as some at drawing people out, but I think I put more structure into the conversation than most. That first day, though, I was able to relax and let other’s do the leading.

Tuesday night is the big event. Matthias had found the Golf Club – perched riverside it gets its name, apparently, from a full golf simulator (you smack the ball for real and the machine figures out how far and where it would actually have gone). I managed to miss the bus (they came back for me), got chewed out by the bus driver, and arrived in the mood for a beer.

Not being much for golf, I mostly hung out by the river in the lovely, soft, summer evening air. Work, companies, old jobs, new analytics theories, tools – you will hear it all at an X Change dinner. It gets dark late in Berlin and the evening can be deceptive. Collectively, we didn’t head back till almost midnight. The young may still head to the bar but I need my rest.

By Wednesday it starts to feel like the homestretch. A nice late 9:30 start is certainly appreciated – that’s something we’ve learned over the years. Besides, you need time to enjoy a German breakfast properly. I survived filling in for Alex Schultz on Re-Targeting Metrics – a Huddle just a little bit outside my normal comfort zone. After the exotically named 4D Pie Chart Visualizations (a good bit of which we spent delving into the practical minutia of choosing scales, building motion charts, getting conditional formatting right and other core visualization choices), it was off to the close.

At the top of the Intercontinental reception afterward I finally got a chance to catch up with the “English” mob – some of my favorite folks in analytics. It’s hard to believe, but we were still talking analytics. Real-time, Asia, career-paths and, of course, X Change Los Angeles.

I hope I haven’t forgotten everything I heard. With so many measurement conversations, it can produce a kind of analytics “Hangover” – like the guys in the movie, I may have to re-piece my week together with the help of my phone. Well…I remember enough. I’ve got plenty to blog about including my next post. It comes from my very first Huddle – a Huddle that really got me thinking about the impact of Mobile and other channels on the way we build metric frameworks.

But for one day, at least, I had no desire to do or talk analytics. Thursday we took Matthias’ kids to the zoo and I got to déjà vu on chasing little kids around a playground, ogling black bears, watching hippos swim and eating bad zoo food. That was pretty sweet too.

To get your own taste of X Change - register now for the September X Change at the Ritz Carlton in Laguna Niguel (greater Los Angeles)...

X Change: Back with a Vengeance

2013-06-07T17:14:41-07:00

This weekend I fly to Berlin for our second X Change EU. We’re all sold out (as usual) – and I’m looking forward to a tremendous event. Special thanks this year to both Matthias Bettag and Michael Feiner who kept the momentum going while I was buried in E&Y transition stuff. I have to think about my Huddle schedule! It’s a good idea to register early since some Huddles fill up (whether in LA or Berlin) – and it’s a first come, first serve system. But there has to be some perk to running the event, right!

I’m thinking of joining Tom Betts’ Huddle on using “Behavioral Data to Aid Personalization”, Peter Pletsch on "From Web Analytics to Mobile Analytics and Back", Craig Sullivan on “Conversion Tools of the CRO Professional”, Alex Schultz on “Retargeting Optimization & Metrics”, and the beautifully titled “The perfect Data Visualization – Four-dimensional-Multi-Axis-dotted-line-Pie-Charts” with Karsten Courtin. What could be better than pie in 4 dimensions - how fattening is that? You gotta love it!

It’s the usual X Change fare – French Laundry fare crossed with all-you-can-eat buffet portions.

We’ve also added some of the traditional U.S. elements to the Berlin version including Think Tank (I’m doing my Analyst’s Toolkit class plus an Attribution class that Paul Legutko and I jointly developed). I’m also doing Tete-a-Tete’s – the one on one, opt-in sales meetings. Who’d have thought that I’d ever have welcomed a chance to do sales? But Tete-a-Tetes aren’t too bad. Not so much a cold call as an informal chat. And with fifteen minute slots, you don’t have those long awkward stretches to worry about.

Speaking of which, I’m finally in high-gear when it comes to X Change here in the United States. That aforementioned E&Y transition stuff put me WAY behind. The Huddle Leaders and topics are in place (stellar and fascinating), I’ve planned the last day (a bit different and something new), and I’m starting to gear up around the marketing. If you’re an X Change regular or have wanted to attend for years but never made it out, this should be your year. The Ritz is going to be a spectacular venue, I think our line-up is the best it’s ever been and, let’s face it, there’s never been a more exciting time in digital analytics than right now.

See you there (or here)!

Social Media, Simulation, and a Customer Intelligence System

2013-05-26T13:03:24-07:00

It’s conference season, so my presentations are starting to overlap in strange, circular patterns around my blogging. A month or so ago, I interrupted an extended critique of enterprise VoC and Customer Experience efforts to dive down into a discussion of Social Media and simulation based on my eMetrics presentation. That, in turn, got interrupted by two posts on big data that came out of a session I facilitated (rather like an X Change Huddle) on big data for regulated industries and a client internal webinar on building a digital big data system. This week I was out at VoC Fusion presenting on, you guessed it, my original blog topic. So I’m going to loop back and finish the eMetric’s discussion of simulation and Social Media, but not without a nod to creating a broader Customer Intelligence System (CIS) that incorporates social media data.

Simulation and Enterprise Reporting

The core of my eMetrics thesis was that enterprise reporting has been almost entirely focused on showing the current state of the digital marketing program. This focus on what’s happened is a crippling deficiency. It isn’t just that such reports aren’t as good as predictive models, it’s that the reports don’t help explain why anything happened. Since understanding is the lynchpin of action, with our relentless demand for actionable KPIs, we’ve eliminated virtually any chance of our reports actually generating action. Our dashboards don’t help us understand what will happen. They don’t help understand what happened. They only tell us what did happen. I’ve likened it to a weather report with nothing but the current temperature. Or, if you prefer, imagine a map like this:

Knowing where you are without knowing how you got there or how to go anywhere else just isn’t useful!

How do you do better? You build reports that embed models of the system. The model represents how the various aspects of the system fit together, captures which are most important, and describes the degree of interdependence between factors. When you build a model of a system, you’ve created real understanding. Represented in a report, you’ve shown a decision-maker how different factors fit together – meaning you’ve help identify the levers of change. Even better, a model is inherently predictive. By allowing a user to tune and change various aspects of the model, you’ve created a tool that helps run the business. That’s actionable.

While there are many ways to model systems, regression analysis is probably the most common technique. In regression analysis, you’re creating a model of how changes in a set of variables impact the target (dependent) variable. But regression isn’t suitable for every problem (though where it is suitable it’s a perfectly good technique for creating the type of report I’m discussing here). In creating reports of complex systems like digital marketing, you’ll often need to combine the results of multiple models built using quite different techniques. That’s where simulation comes in. Simulation techniques provide a way to combine different types of analysis and different problem sets and study how they interact. It’s ideal for a problem with multiple dependent variables and complex interactions between systems.

Simulation has another benefit. As I’ve talked about models, I’ve probably given the misleading impression that predictive models always generate understanding of how a system works. Unfortunately, that isn’t completely true. It’s true often enough – I don’t want to make it sound like there is a continual disconnect between models and systems. But there are times when the most predictive variables are not the ones that help you understand a system.

Going back to my weather forecasting example, imagine a primitive tribe that had been given a barometer. A clever medicine man could effectively predict the weather by tracking the needle on the barometer. Such a system will generate good predictions but no real understanding. For predicting weather, that really doesn’t matter. Weather is completely exogenous - there's no element of control. We don’t need to understand how it works, we just need to know if we'll need an umbrella for tomorrow's dance around the bonfire.

Business problems are different. We have control over aspects of the system and it’s important that we understand how the levers we control can impact the results. since your marketing spend and your competitor’s spend often have a very positive covariance, a regression model might work just as well or even better (against historical data) using your competitor’s marketing spend instead of your's as an independent variable. But you only have control over your marketing spend - not your competitor's. A well-built simulation model will probably demand the inclusion of both factors, which is surely correct.

Simulation is a natural technique for forcing an analyst to address the real-world.

The Customer Intelligence System (CIS)

Which brings me to my presentation this past week at VoC Fusion. In that presentation, I walked (well, maybe took a brisk trot) through the critique I’ve been developing of the way most enterprise’s are doing Voice of Customer research. Drawing on lessons from the evolution of behavioral analysis, I argued that enterprise VoC research is too fragmented, too siloed, too chaotic, too focused on site not customer, too taken up with meaningless or flatly misleading top-line metrics, too static, and too poorly disseminated. That’s a lot of “2s”! And while it might seem contradictory to complain that VoC is poorly done and poorly disseminated (the food is awful and the portions are too small!), the two are tightly bound. Part of getting the resources you need to do the job right is effectively disseminating what you have.

The solution? The integration of VoC efforts into a Customer Intelligence System that has intake from online surveys, offline surveys, social media, site and bricks-and-mortar feedback, and call-center data. The system would enforce standardized design, classification, and segmentation of the information and would be the center of a process in which VoC instruments and research are constantly tweaked and refined to answer emerging business questions. It would also serve as the foundation for an enterprise-wide dashboard capturing the state of the customer.

It’s an ambitious design, but it’s also the type of system that can be implemented for a fraction of the cost of a typical big data analytics mart and dramatically improves the enterprise’s understanding of the customer and the resulting ability to effectively shape the customer experience.

The Role of the Customer Intelligence System in Modeling

So how does a CIS fit into the reporting and modeling story? It doesn’t always. You might build many a forecast report and never need or to use the type of information contained in a CIS. There are a fair number of problems, however, where social media and other Voice of Customer data can fill what would otherwise be a pretty big hole in our models.

You can build a model out of any data. You can’t necessarily build a good one. At the core of most digital modeling problems are unresolved questions about demand, customer choice and attitudes, key segmentations, and competitive positioning. You can almost always build a model without accounting for these factors. I can, for example, build a model of Website traffic with nothing more than historical Website traffic data. It just won’t be a very good model.

Unless I understand how different types of visits and visitors generate different levels of repeat usage and how those visit-types interact with each other and drive the overall customer relationship, I haven’t really created a model that can drive understanding and help tune the system. In my eMetrics presentation, for example, I showed how social media data can be used to help model demand for a new product.

Here's another critical point. Taking your model down to the level of customer experience makes it much more likely that it will help identify the true levers of change. These types of variables have the dual benefit of being importantly predictive AND potentially controllable.

This doesn’t mean you build a CIS just to generate parameters for your model-based reports. That’s just one use of much broader system. But it shows how, as we drive deeper into customer analytics, the various components of the ecosystem begin to come together.

Model-based reporting (particularly simulation based reporting), will force you to go deeper into systems than you’ve ever had to before. The instant you put a forecasting tool into a decision-makers hands, you should be prepared for them to try the craziest stuff. What happens if I don’t spend anything on advertising? What happens if I kill my TV budget and spend 100% of my dollars on digital? Models based entirely on historical data will often miss the deeper connections that would help return plausible answers.

When you drive your models down to the customer level using data from a CIS, you're far more likely to be able to handle these types of ahistorical what-ifs. It’s more work. Sometimes, let’s be honest, it’s too much work for a single problem. But over time, the more you can integrate real customer data into your modeling, the better your analysis will be.

[Drop me a line if you'd like a copy of my VoC Fusion presentation!]

The Case for Building a Customer Intelligence System : "A Little Rebellion is a Good Thing"

2013-05-26T12:55:00-07:00

VoC Fusion is really the first conference at which I’ve presented a fully developed vision of what a Customer Intelligence System (CIS) designed to capture, understand, disseminate, and act on Voice of Customer data might look like. I believe that a well designed CIS is far more than just an Enterprise Feedback Management system. A CIS includes technology to store a wide-range of VoC data (including Social Media, Online Survey, Offline Survey, Online Feedback, Offline Comment Cards, and Call-Center), to process that data intelligently using advanced text analytics, to integrate the data with behavioral inputs, and to report out on that data using powerful data visualization tools like Tableau or Spotfire. It’s not just technology, either. An even bigger piece of a real Customer Intelligence System is the creation of a set of processes around this technology. Key processes include the standardization of intake, the creation of robust taxonomy and segmentation classifications and their standardization at the enterprise level, aggressive sampling to support behavioral integration and targeting precision, and the creation of virtuous cycles around survey design, analysis, and feedback.

It’s far from a trivial amount of work, but done thoroughly, it will transform the enterprise capability to understand customer drivers of choice and decision.

The Customer Intelligence System is not meant to be an evolutionary step in VoC. It’s revolution.

When Revolution is Necessary

Why revolution? Revolution is not to be taken lightly. Thomas Jefferson (from whom I’ve borrowed my sub-title) aside, revolution is risky and expensive. But sometimes it’s just plain necessary. In a set of pieces earlier this year, I laid out my case for why the current state of Voice of Customer in the enterprise is unacceptable. In today’s post, I’m going to summarize that case.

Each and every channel that captures key Voice of Customer data is seriously under-served in most enterprises today and there are gaping holes in the intersections between those channels as well as the distribution of information collected within those channels to the rest of the organization.

Existing Research Channels are a Mess

In Online Survey, most enterprise efforts are static, too site-focused, too long, under-sampled, and too concerned with top-line metrics like NPS (Net Promoter Score). Online surveys are an amazingly flexible and inexpensive way to understand your customers, so it’s a particularly sad that they are so poorly utilized.

In Social Media, the state of measurement is even worse - hardly worthy of the name. Social media metrics are a hodge-podge of the inconsequential and the mistaken. Social Media samples are mis-understood and poorly governed. The classifications of social media data are shallow and largely irrelevant. And the technologies used to generate social media reporting are deficient in their ability to effectively categorize the information.

It doesn’t get much better in Call-Center. Call-Center is the place that Voice of Customer data goes to die. No area of the enterprise is more siloed, more focused on their immediate operational concerns, and less integrated into the broader enterprise reporting and data streams. If you’re lucky enough to get a report of most-common Call-Center call-types, consider yourself in the vanguard of Call-Center reporting at the enterprise level.

Lack of Coordination Cross-Channel

The problems don’t end “in-channel”; indeed, they get worse (as is generally true), when you remove the silos and take a broader look at how the individual programs fit together to create an enterprise-wide view of customer.

At most enterprises, there is very little infrastructure (either technical or organization) for welding these inputs together into a coherent picture of customer attitudes. This begins at the most basic level in terms of setting research programs and standardizing core inputs.

At almost no enterprise is the research program across online survey, offline survey, social media, feedback mechanisms and call-center standardized and set in a consistent fashion. It’s not unusual even for obviously related items like online and offline surveys to be run by completely unrelated and uncoordinated teams. Often, when I ask the online survey teams for their offline survey data, I just get blank stares. With Social Media and Call-Center, it’s almost universally true that there is no coordination of research.

This has significant impact on the enterprise ability to really understand the customer. Every channel has limitations when it comes to research. By failing to standardize key questions and segmentations, you lose comparability across channels. If you ask questions differently or categorize answers differently, you can't compare segmentations across survey instrument. By failing to coordinate research across all the available channels, organizations leave gaping holes in their understanding of the customer and the customer journey.

Errors in Online Sampling Drive Constant Errors in High-Level VoC Metrics

In VoC, the analysis of the data nearly always requires careful stratification and segmentation. For years now, I’ve been pointing out how audience research in Social Media and Online Surveys IS NOT the same as traditional survey research. The sampling is different, less random, and more complex. As a result, top-line metrics like NPS and Brand Sentiment are extremely prone to error. So error-prone, in fact, that it's a serious mistake to use these numbers at the site-wide or social-channel level.

All those warnings don't seem to have done much good. I hear endless examples of organizations whose overweening VoC focus is on their NPS score. And at VoC Fusion, you're talking about the high-end of the market - companies with a true Enterprise Feedback Management (EFM) solution! Even if you’re doing the sampling right, NPS is, at best, a simple reading of your overall customer state relationship. Without deeper analysis of the real drivers of choice, it isn’t actionable and it produces no understanding of what drives the underlying customer relationship and experience.

If the best that a VoC program can achieve is to get senior decision-makers to follow their NPS scores, then we should invest the money spent on customer research into something genuinely useful – like better snacks in our meetings.

Technology for Text Classification is Poorly Adopted and Inadequate

Of course, one of the reasons for this heavy focus on simple top-line metrics like NPS is that the tools we use to work with text heavy data sources are terribly inadequate.

In Call-Center, the key data is often trapped in non-digital form. Many call-centers still use systems that make it challenging or impossible to extract even the basic operator actions or call information. This IS getting better. Current generation call-systems are much, much more open and call digitalization is an emerging and increasingly practical option.

In Social Media, we’re still struggling with technologies that do a very poor job of analyzing and classifying free form text data. Given that social media is 90% free-form text data, that’s a problem. If you’re using keyword-based systems for parsing and classifying Social Media data, it’s just not possible to do the job well. I can’t tell you how delighted (and surprised) I was to learn that E&Y has a strong document text analytics technology (via acquisition), and uses Crimson Hexagon for Social Media analysis. Very few of our clients are in anything like such good shape.

Dashboarding and Reporting is almost Non-Existent

As bad as all these problems are, no part of the overall use of Voice-of-Customer research is as broken as the reporting piece. There simply are not tools or efforts to combine this customer attitudes data across research channel in a consistent way and to distribute the resulting picture of customer attitudes out to the broader organization.

EFM tools are unable to do this. Most survey tools have very poor dashboarding and are completely siloed. There are some Social Media tools with strong dashboarding capabilities, but those tools are generally unsuited to real research and are locked into their proprietary, social-only data structures.

Even within these limitations, the state of enterprise customer reporting is remarkably poor. Very little survey data ever sees the broader light of day beyond the research teams that own them. And the enterprise that tracks anything about customer attitudes at the executive level except a few high-level metrics like NPS or Brand Sentiment is vanishingly rare.

Enterprise VoC programs just aren’t very good.

A Call to Arms

I’ll admit to being a perfectionist when it comes to measurement. Heaven knows I’ve complained often enough about the state of enterprise measurement when it comes to digital behavioral data. There is much, much work to be done before we get that right. Yet what we have achieved in the behavioral space is considerably better than the corresponding state of practice around VoC. Our technologies are better. Our efforts are more integrated. Our data is more standardized. And our efforts to socialize our data far more mature. If the state of customer behavior analysis is far from ideal, neither is it completely broken.

No so in Voice of Customer. The state of enterprise Voice of Customer is so bad, so broken in almost every important respect, that fundamental change is necessary. Change in technology. Change in ownership. Change in funding. Change in process. Change in focus.Enterprise's need to fundamentally re-think their whole approach to Voice-of Customer research.

The value of Voice of Customer programs at the typical enterprise is dangerously close to zero. Yet what data is more important, more valuable or as easily understood and acted upon as Voice of Customer? The opportunity is immense. The potential competitive advantage real. The current state abysmal.

It’s time for a change.

Aggregation and Detail in the Big Data World

2013-05-12T12:14:36-07:00

In last week's post, I took another crack at defining what makes "big data" real and not just more of the same with an extra helping of hype. Creating a sound definition of big data may not have a huge amount of practical significance - but it isn't without importance. Knowing what you're dealing with and why/whether it's really different from what you've done before has a host of implications for questions around resourcing, process and technology.

But finding a good definition of big data isn’t what I thought most compelling or important in the discussions that triggered these posts. It was the discussion around the nature and role of detail-level data in the big data world. If the theme of last week's post was that big data skeptics are somewhat right to question the hype but very much wrong to question the reality of big data, today's post addresses the most common piece of bad advice you'll get from big data experts.

The Many Advantages of Detail Data

If there’s one thing that virtually every big data vendor and consultancy will tell you, it’s that the key to big data systems is leaving your data at the detail level. Indeed, it’s precisely the ability to do this that makes big data technologies unique. They process enormous amounts of detail data (whether structured or unstructured) fast enough to allow for the data to be kept in its native form. What do you gain from this?

Independence from structure – removing layers of modeling and indexing
Ability to re-define the data model for every analytic problem – critical since most anslysis projects require unique aggregations
The ability to join multiple data sources at the detail level as necessary

These are all legitimately important things – and are all pretty much true. That being said, there’s no bigger mistake you can make in a big-data system than to believe that your data must always live at the lowest level of detail.

Taking a Rule One Step Too Far

The problem, as I see it, is that just as the big data vendors have largely failed to understand what makes big data problems unique, they’ve often failed to grasp what’s involved in the solution to those problems. They understand the technology, but not the analytics.

Just as with my last post on what makes big data truly different, it’s the stream nature of the data that drives my thesis – and though I’m going to use digital data as my example, what I’m going to argue applies equally well to sensor data, meter data, and a host of other important big data use-cases.

The problem starts with the lack of meaning at the event level. In digital, the analysis of page views isn’t interesting. The page is not a significant entity in marketing. Rather, the analysis is all about sequences of pages views – those sequences being a visit or a related part of the visit – in which the visitor is trying to accomplish something. If I’m building a model of the customer journey, I don’t want to capture every page. I want to capture what the sequence of page views was all about and how successful it was.

One of our main practice focus areas here at E&Y is how to use segmentation techniques to identify visit intent – what a sequence of pages tells us about the customer’s intent and interests. Over the past few years (as Semphonic), we’ve developed an entire analytics methodology around this. But whether you use those methods or not, the key point is that almost any analysis you do is going to have to create some level of meaning around the sequence of touches that constitute a visit. If you take the time and trouble to build a complex decision-tree or cluster analysis of visits, that’s an incredibly valuable foundation for nearly every subsequent analysis.

But if you listen to your big data vendor and insist on having nothing but the lowest level of detail data on your box, you’ll have to re-create that full segmentation EVERY single time you want to use it. That’s preposterous.

Think about this. It's the key point in this post. If you are always constrained to leave all of your analytics data at it's lowest level, you're force to re-create EVERY analytics step EVERY single time you want to re-use it. This might be reasonable when it comes to fairly simple techniques like sessionization, but it's madness when it comes to complex steps like segmentation. What's more, leaving the data in it's native detail state puts dramatic limitations on the number of analysts who can productively use the data. Big data environments are difficult enough without adding silly rules to make them harder.

Creating new and permanent levels of data that include not cubes but data aggregated into levels of meaning above the detail level (like a single row per visit based on segmentation) doesn’t violate any aspect of the big data paradigm - it’s essential to it.

This doesn’t mean you don’t need the detail. You do. Not every analytics problem will be captured within or will take advantage of a visit or task-based segmentation. You still have to have the ability to start over, and you'll use that ability all the time. But a large number of subsequent analytics tasks (including customer journey models) will benefit from having that visit-level segmentation-based aggregation and will be nearly impossible without it.

It’s just a case of people taking something true (don’t create complex data models, cubes, or fixed structures on your big data system) and taking it to a level where it no longer make sense – never have anything but the lowest level of detail on your big data system.

Summing Up

When you have visit-level, cluster coded data, you still have something that is, for all practical purposes, flat. It's no different than the call-center, call-level data your big data consultant will load onto your system without blinking an eye. But that call-center data isn't the lowest level of detail possible (unless it comes from call digitalization). It's data aggregated by the call-center system. If you'll load aggregations onto your system, why shouldn't you create them on your system too?

Whether the aggregation happens on your big data box or elsewhere is completely irrelevant. This visit-level segmentation coded digital data is just the kind of data your big data systems will chew up and digest wonderfully.

It's particularly useful as an integration layer between various types of customer touchpoints.

When you need to join call-center, Web, mobile, and bricks-and-mortar touchpoints, you simply can't do it at the detailed stream level. Data at that level is too disjoint - completely different for every type of stream.By using segmentation techniques to aggregate streams up to visit intents and success measures, you create a level of data perfect for customer journey modeling.

It's not detail level data, but don't let that worry you. That it happens to be easily used, meaningful, and valuable should not be taken as three strikes against it!

If you think your big data strategy needs a re-think, drop me a line. And if you're in Las Vegas this week for VoC Fusion stop by my presentation on Thursday to chat!

The Four V's, Big Data, and Big Data Skeptics: Half Right and Wholly Wrong

2013-05-05T10:16:33-07:00

This past week I did an internal webinar for a client on big data systems. It was a good session with lots of back-and-forth and questions – which not only makes for a more interesting and relaxed presentation, it’s more likely to stimulate my thinking. I hear more than enough of myself talking!

So even though this is a digression (from my posts on Forecasting and Dashboards) inside a digression (my larger series on re-engineering Voice of Customer at the enterprise level), I wanted to recap and expand on a couple of the central themes of that presentation.

First, a quick table-set. I’ve taken a crack in the past at defining what “big data” is all about. It’s always hard when a concept takes off to keep control over it – the Hype Cycle (Gartner's lovely concept) can take anything – from Presidential candidates to epistemology to piano-playing cats and so over-expose them as to make the underlying reality nearly impossible to discern.

Naturally, this creates a kind of backlash. Plenty of web analytics pundits are more than willing to describe big data as just hype. Not only is there a certain cachet in running counter to a popular trend, there’s a certain self-interest here too. Web analytics companies nearly all come from Web analytics tool backgrounds – said tools being, in most respects, the antithesis of big data tools. I imagine that it’s always safest to assume that anything you don’t understand must be unimportant!

I’m not a fan of the industry standard definition of big data (probably best exemplified by the four Vs: Volume, Variety, Velocity, Veracity) in part because I think it’s vulnerable to the criticism of skeptics that we've always had these exact same factors.

The four Vs do describe most big data situations, they just don’t get to the heart of what big data is all about. Back in the early ‘90s when I was doing credit-card work, we had volume most of today’s big data companies would still consider massive. We had plenty of velocity too, and veracity was pretty darn important when clearing card transactions. We didn’t have variety, but it’s implausible to argue that variety is essential to every big data application. There are plenty of big data applications that are single source. If I’m trying to mine CNN’s digital data stream, I don’t need variety to be in the big data universe.

So were all of us in credit-card working on big data in the early ‘90s? Some might say yes, but I don’t think so.

Instead, I’ve proposed a simpler, more basic, and more fundamental definition of what uniquely defines big data. Big data happens when you drive your data capture and analysis down from the traditional levels of analysis (like customer or transaction) to a level where the meaning of each event can only be interpreted in relationship to the stream of events. Digital is a paradigm case for this. Web site page events are not, in and of themselves, meaningful. The meaningful level of aggregation is somewhere in the sequence of events and that's what you have to interpret.

It’s not too different in utilities. When you move from the once a period reading of a meter per customer to constant collection, you change the nature of the analysis and data capture problem. No single meter reading is, in and of itself, important. It’s in the flow and pattern of the readings that meaning emerges. This is a different type of analysis.

It should also be clear from this why the Four V’s look like a reasonable definition of big data to those in the field. When you drive your unit of analysis down a level, you increase by one or more orders of magnitude the volume of your data capture and the velocity of your data. You place additional demands on data collection that can result in poor data quality. And while variety isn’t necessarily wrapped up in the concept, you have created a whole new set of challenges around joining data that lives at the stream level – making multiple sources (variety) far more difficult to handle.

But the beauty of the definition I’ve provided is that it makes it clear why my ‘90s credit card work – despite hitting the V’s pretty well – wasn’t necessarily big data. No amount of the four V's make for big data if you're just scaling up the same exact types of data and analysis as you've always done. It also explains much of why today’s generation of big data technologies are built the way they are and why they provide unique advantages that traditional transactional systems don’t. Those traditional systems sure-enough handled lots of volume - just not in the ways we need it handled now.

It also explains another aspect of big data that is particularly important and represents one of the biggest risks if you’re building a big data system. The nature of the analysis and the methods necessary to join, process, and understand the data all change at the stream level.

Traditional analysis techniques, from joining methods to sql queries to aggregatations to traditional statistical techniques like correlation, regression and clustering all work differently - if they work at all – when applied to this type of detail, stream data.

I’ve seen this cast as a debate between machine learning and traditional analysis; it isn’t.

Machine learning may (though I think it’s debatable) be particularly useful in big data situations because of the symptoms (the four Vs) that spring from detailed stream-level analysis. As far as I can tell, there is nothing about detailed stream-level analysis itself that makes machine-learning particularly suitable. The really important point isn't about machine learning - it's that your standard analytics toolbox is mostly out the window.

So while defining big data by the four V's may miss the mark, it's far, far more misleading to suggest that big-data is just "more of the same" - perhaps with a bit of an emphasis on the "more". If proponents of the more of the same view are claiming that we still need to decide how to structure data, how to join data, how to query data, and how to analyze data then their claim is merely empty. Of course we do. But if they mean to claim that we should use the same methods to join, structure, query and analyze the data as we always have in traditional transactional or BI systems, then they are flat-out wrong.

It's nearly always a safe bet that opposing the hype cycle will make you at least half-right. But if the big data skeptics are half-right about what's wrong with the hype, they are wholly wrong about the alternative.

[I should mention that I'm going to be speaking on a Big Data panel at the DAA Symposium in Washington DC on June 4th. The Symposiums are the probably the single best thing the DAA has created - I've attended a fair number and been consistently impressed. If you're in the area, do come out!]

The Role of Simulation in Building Powerful Enterprise Dashboards and Reporting Tools

2013-04-28T11:18:23-07:00

The vast majority of enterprise reporting and dashboarding is confined to “showing the current state”. We spend a huge amount of time showing people what happened – virtually none showing what’s likely to happen or what’s driving those numbers. We’re like weathermen who only report the temperature – never the forecast or the factors explaining that forecast. It just isn’t that useful. By embedding models into reporting, we have the opportunity to fundamentally transform them: from static snapshots that serve as little more than “Warning” signs to real business tools that help users understand what levers they have to drive performance and the implications of adjusting those levers. Embedding models inside our dashboards also creates a powerful discipline – connecting analysis and reporting – wherein every time our forecast is wrong we’re driven to go back and figure out why, continually refining our understanding of what drives the business.

It sounds great (I hope), but leaves open the question – how do you do it? How do you build models of complex digital marketing or operational systems?

There are actually quite a few different approaches – and it’s not as hard as it might seem – particularly to create an initial forecast that can be your entry-way into that virtuous cycle of continuous forecasting, analyzing, and tuning.

One technique that we’ve been experimenting with is simulation.

Why Simulation?

We’ve built models for many aspects of digital marketing: attribution, mix, internal optimization, use-cases, conversion funnels, onboarding and more.

Historically, none of these models have been simulation models. We’ve used a variety of techniques for these models – from multivariate regression to factor analysis to cluster analysis (at least to drive underlying segmentations).
So why add simulation to the mix?
Well, here are some examples from a simulation design program that we’ve created in a couple different problem spaces.

Here’s a high-level piece of one of our early, exploratory simulation models for a Product Launch:

And an even smaller piece of a broader simulation of a digital marketing system:

These aren't just diagrams. The simulation software they are built in allows you to define the stocks, flows, and parameters that make up the system and then use the simulation engine to see how changes will impact the whole system.

What do you gain by using simulation around these problems as opposed to our traditional models? The beauty of simulation is that it allows you to combine two or more very different models into a single system. For example, both our Product Launch model and our Digital Marketing System model require elements of campaign optimization – particularly mix modeling. Those elements are built using traditional techniques. But there's more to a Product Launch than optimizing your media spend and there is more to a digital marketing system than campaigns. In both cases, representing the Site System is also critical.

Now here’s the really interesting point – we use completely different techniques (segmentation and use-cases) to analyze the Site System. Those techniques are NOTHING like the techniques we use for campaign optimization. What’s more, there’s no way (at least as far as we’ve ever been able to figure out) to combine the two analytic techniques into a single method.

If you want to create a comprehensive dashboard of the digital system, however, you can’t just create models of campaigns and site and the leave them as independent entities. If you do, then you’ll miss the deep connections between them. As you adjust your marketing mix between online and offline or between display and PPC, you don’t just change the amount of traffic to the site; you change the mix of traffic to the site and the distribution of visit types. This means that campaigns and site systems need to be related to create an accurate view of the larger system.

Simulation is the only way I know of to accomplish that.

Challenges

It isn’t all hunky-dory though. First, simulation is a completely new discipline for us. We’re working to build out this capability, but I won’t pretend it’s easy. Building simulations isn’t much like building regression models (though there are some similarities), and the results of some of my initial simulations were downright puzzling.

Maybe that’s a good thing, though. Simulations create a real discipline in that, if you don’t have a reasonable working model of the world, the chances are high that your simulation won’t produce results anything like the real-world. Building a simulation forces you to tackle reality head-on and make sure you’re actually capturing most of the important elements of the system.

There’s a second problem with simulation that I’m less sanguine about. The images from above are directly from simulation software. They have a gee-whiz quality to them that I must admit I like. Particularly becuase they help capture something I think is vitally important in reporting - showing the interconnections between factors. But they most certainly are NOT the stuff of an executive dashboard. When you’ve created and tested your simulation, you need to instantiate it in a real tool. There’s no obvious tool for doing that - so grafting your simulation into a dashboard is very much a technical exercise.

Summing Up

Simulation combines traditional analysis (to drive the parameters of key systems) and the inter-relationships of complex independent systems, to create models that transform dashboards into working tools. They provide a way to forecast the future, identify the drivers of performance, track how actual performance deviated from expected, and give decision-makers the ability to try what-if scenarios with much higher real-world fidelity. It's cool stuff. They change your whole approach to reporting and dashboarding and replace static views with powerful tools for running the business.

Afterword

I’m speaking at the AIM Conference on Monday in Los Angeles – on how to measure beauty (or the Science of Aesthetics as my PPT is titled). I won’t pretend to have an answer, but it's certainly an interesting topic. I’m also very excited to be presenting out at VoC Fusion in a few weeks. It looks like a terrific conference. I’ve been building that presentation and it’s a really nice overview of my recent posts and work on creating a comprehensive Customer Intelligence System as well an early distillation of our integration with E&Y’s existing Customer Experience practice. It represents the first fruits of what I hope will soon be a very rich garden. Not only am I much pleased with the presentation, it’s great to be presenting outside the traditional analytics community. Really looking forward to it!