Blog – Causata

Causata at London Big Data week

2012-05-02T08:37:00.003-07:00

Last week was Big Data Week in London, with a packed week of excellent meetups on Big Data, Data Science, Data Visualisation, and the opportunities afforded by Big Data. We at Causata made the most of the week, attending lots of events and meeting some interesting people in the thriving London tech community.

I spoke at the Data Science meetup and talked about what it takes to determine cause and effect from data, and how I believe that the key is to perform experiments. One of the great strengths of Causata's real-time decisioning is that it allows statistically rigorous, controlled experimentation, allowing us to move beyond correlation to understand causation. Here are the slides I showed.
—Jason

The importance of getting data into production quickly

2012-04-17T22:46:00.000-07:00

Flowing Data is a great website for data visualizations. I love the one showing the sequence of Walmart store openings over time or the perfect choice of bike for each San Francisco neighborhood.

However, as someone who not only enjoys looking at data but is also focused on actually doing something with it, I felt today's article really hit home.

For a few years the Netflix prize has brought the top data scientists on the planet into an annual open competition to set a new bar on best movie predictions. It's the modern day equivalent of the KDD Cup, which was where Causata's COO Paul Phillips cut his teeth. The article highlights that the algorithm from the latest winning team was deemed not practical enough to put into production. I'm actually not that surprised -- though to be fair I doubt the competition had any requirements for that.

Nevertheless it does underscore a real gap I see in how people look at big data and making predictions with machine learning, especially when it comes to marketing or customer interactions. It's just not appreciated enough that the whole point of going through the effort to build a statistical model is to extract business value from it and that doing so quickly is essential. When I hear SAS analysts talk about how long it typically takes to put a model into production I'm always amazed. The months of recoding SAS code into SQL, the many compromises along the way, and the heavy validation effort in a database or data warehouse raise questions about whether it's the right approach.

This is a topic where I'm 100% sure we're way ahead at Causata. If you've built a statistical model in a tool like SAS or R it can be imported into Causata in seconds and any individual can then be scored in the next instant. Causata's real-time scoring also incorporates the latest data, including customer interactions in the last second. So if a new website visitor arrives via a high-value search term, a score will scream that they are a hot prospect. There are dozens of important use cases related to this capability. It's a huge deal. The philosophy for us is a bit like the agile software development process applied to data. You don't want a long period of time to pass before releasing a statistical model into production. Release early, release often.

Check out the full Netflix blog article that the Flowing Data piece refers to. The folks at Netflix explain how their business is evolving and this has changed the nature of the predictions and personalization they need to perform. This is just yet another reason why you need to get the data into production rapidly.

My prediction... in the next 12 months this agility topic is going to be top of mind for a lot more people. We see this growing awareness among the top marketing and analytics customers we're working with. How top of mind is it for you?

Keystone partnership and eMetrics SF

2012-03-04T23:41:00.001-08:00

We announced a new partnership this past week with Keystone Solutions. They've got a great team with a similar digital marketing heritage to us. We're really excited about working more closely together. The full press release is here.

Stay tuned for many more partnership announcements to come.

Many of the Causata team will be at eMetrics this week in San Francisco. Paul Phillips will be speaking on Tuesday morning. Don't miss it!

Banking on data

2011-10-18T12:34:00.000-07:00

Causata’s CEO Paul Phillips (left) moderated a panel at the BAI retail banking conference in Chicago last week.

Paul was joined (left to right) by:

Michael Wexler, Director of Digital Insights and Marketing Effectiveness at Citibank

Mike Olson, CEO at Cloudera

and Andrew Rosen, CMO at Bank of the West

The panel topic was “Winning and Losing with Customer Data in an Accelerating Digital World”. Michael and Andrew shared some illuminating stories about how important they see customer data and just how big an opportunity there is to impact their business and the customer experience. Paul and Mike gave their Silicon Valley thought leader take on where the technology is today and how companies should best take advantage of it. There was a clear consensus that the companies that embrace data will be the ones that win.

Determining the value of your data

2011-10-10T06:11:00.000-07:00

John Lovett has a thoughtful post at Clickz.com summarizing many of the struggles enterprises are having with Big Data. One problem is enterprises often store as much data as possible, for as long as possible, fearing they’ll be discarding valuable information if they don’t. Enterprises struggle to understand which data-points are valuable and which ones aren’t. Lovett writes:

“Understanding what data matters to your business requires empathizing with business stakeholders, examining marketing programs, and getting to the mission-critical values of the organization. In my experience, I've found that simply asking business stakeholders what metrics or KPIs are most important to them is a futile endeavor.”

At Causata, we believe that the relative value of data should be determined by how it can be used to drive your business forward and help you achieve your goals. We begin by organizing all of the data around a customer. You should be able to see every interaction a customer has had with your company. Those interactions should be stored in time order, so you can see how events in the past, predict future behavior.

The next step is to understand your business goals. Most often these are centered on increasing profit either by selling more products and services or increasing customer satisfaction. Take reducing attrition. You’ll need to understand the behaviors that lead customers to terminate their relationship with your company. You may look at things like: How many products did a customer browse online in the past week? How many times has she called to register a complaint over the past month? When was the last time she visited a store location? Leverage as much of your data as possible, especially web and mobile data rich with customer intent. Now conduct an analysis to see how each of these behaviors is correlated with customers who end the relationship with your firm. In the process you’ve transformed your data into customer intelligence. This intelligence can be used to power campaigns to reach out to those customers most likely to attrite.

This is a tremendously valuable view of the data because it gives you the power to anticipate dissatisfied customers and reach out to them before they leave. In doing so, you’ll be able to significantly increase your business performance.

This can be extended to any business goal. In the end you’re left with thousands of data-points that express customer intent. This, in turn, can guide an assessment of the relative value of various data-points. More importantly, it can be used to help you really understand how customer behavior relates to your business performance.

A look at Causata

2011-08-11T17:18:00.000-07:00

There's a good post about Causata by James Taylor on his blog Everything Decision Management.

Gareth

Management doesn’t value marketing

2011-06-20T11:06:00.000-07:00

According to a study cited in this article over at Marketing Week, 73% of CEOs believe that marketers “lack business credibility because they fail to quantify the success of their campaigns”. It goes on further to state that: “marketers focus too much on the ‘arty and fluffy’ side of marketing and not enough on its business science”.

Putting a hard number on the value of a marketing campaign is an ongoing struggle for many marketers. It requires a complete view of the customer, something that marketers may not have access to. Imagine a marketing campaign for a bank that leverages banner advertising, call center and direct mail with the goal of increasing product holdings among existing customers. Management rightfully wants to understand the business value created through this campaign.

To begin to answer this question, you’ll need a complete view of the customer, both before and after the after the life of the campaign. This complete view includes all customer interactions along with all other customer attributes. What banners was the customer exposed to? How many did she click on? Was she targeted for with direct mail? How did she respond? What are her total account holdings before and after the campaign? And so on.

Without this view it’s almost impossible to understand the cause of changes in product holdings because you’ll only be analyzing part of the story. This leads to approximations that can be divorced from business reality. Marketers are often forced into approximations because some number is considered better than no number at all. But this can lead to a big problem: When marketers report numbers that fail to align with financial statements, distrust of marketing grows.

Developing a complete view of the customer is a necessary condition to truly understand the success or failure of marketing campaigns. Marketers are going to continue to have trouble understanding the value they’re creating without it. And without understanding the true value they’re creating, business credibility may continue to elude them.

A CMO's Most Important Task

2011-05-23T10:13:00.000-07:00

The CMO Council’s report Renovate to Innovate: Building Performance-Driven Marketing Organizations (1) suggests that the most important task for CMOs is to drive their organizations to execute strategic, cross-functional campaigns:

“Most importantly, chief marketing executives have to revitalize marketing group cultures and mindsets. Narrowly focused, risk-averse managers in isolated silos of tactical execution (research, PR, advertising, events, marketing services and Web) must be integrated into cohesive, cross-functional campaign teams.”

Prior to integration of this sort, organizations must first understand the lifetime value of each customer regardless of how the customer chooses to interact with the organization. Only then can cross-functional teams can be organized around a common goal: maximizing lifetime customer value.

In the past I worked with a major retailer who failed to understand the drivers of lifetime customer value. A tension quickly emerged between the brand team, who viewed value creation on a long-term horizon, and the e-commerce team, who wanted to maximize in-session purchases. The question became: What metric should we optimize for?

Without an understanding of lifetime customer value, there is a significant risk in failure of cross-functional team alignment as each team clings to the metrics its used to. More importantly, there’s a real risk of creating teams that optimize the wrong metrics.

Understanding lifetime customer value requires organizations to:

Connect the data: Connect data from different channels organized around the customer.
Understand the data: Understand the key drivers of lifetime customer value. It’s just as crucial to understand how this changes over time.
Act on the data: Leverage all the data in any channel, ideally in real-time.

A common understanding of lifetime customer value will provide a solid foundation for these cross-functional marketing teams. Enterprises risk misalignment and suboptimal performance without it.
--
1. http://www.cmocouncil.org/resources/forms/download-report/index.php?id=204

Podcast: "Real-time customer intelligence with Paul Phillips"

2011-05-09T01:03:00.000-07:00

Causata CEO Paul Phillips was recently interviewed for Beyond Web Analytics by Adam Greco and Rudi Shumpert:

Join Adam and Rudi as they talk with Paul Phillips of Causata and real-time customer intelligence. Paul provides the team with insight on what exactly Causata is and the business problems that they are helping their clients solve. The conversation explores the real-time customer intelligence, pulling multi-channel data into a single source, and dives into what this means for companies of all sizes.

The full podcast is here.

New Causata whitepapers page

2011-02-25T06:50:00.000-08:00

We've gathered together all the Causata whitepapers on a single page for easy access. In addition to the whitepaper entitled Web data integration: Build or Buy? two other technical whitepapers are available.

Causata Distributed Data

At Causata’s core is a data structure that captures every interaction between your business and your customers.

Causata Identity Association

Causata has an identity association layer that disambiguates customer identity information to maximize the use of data.

Causata and Intuit: working together

2011-02-25T06:33:00.001-08:00

We're very happy to be able to talk publicly about Causata's relationship with Intuit. Intuit and Causata have been working together since 2010 on integration of diverse data sources using the Causata platform.

More details.

The five challenges to web data integration

2011-02-25T06:28:00.000-08:00

Data captured from the web brings with it a tough new set of analytical and low latency demands if its full operational value is to be realized. These challenges are:

1. Web data is high volume
2. Use cases are diverse and complex
3. Web data is low information density
4. Associating web observations is inherently ambiguous
5. Web “in-market” predictors depreciate very rapidly

The Causata whitepaper "Web data integration: Build or Buy?" explores these challenges in depth.

Big Data at Strataconf

2011-02-03T09:25:00.000-08:00

I and several of the Causata team are at O'Reilly Strataconf this week, a conference all about big data and the exciting possibilities it offers.

In a panel discussion in the opening session yesterday, Mike Olson of Cloudera made a very important point: big data is all very well, but things really get interesting when you combine diverse datasets.

This is an emerging theme in the conference, I'm seeing it everywhere.

In a tutorial on tools and techniques for the data analyst, Drew Conway and Hilary Mason took bit.ly data of web users who had clicked on links to Strataconf, and combined this with location data to plot a map. Interestingly for me, the Bay Area was only the second largest circle on the map - the largest was London! (Causata is in both).

Pete Skomoroch of LinkedIn combined the Strataconf attendee list with LinkedIn data to produce a visualization of the skills, companies, and job titles of the conference attendees.

Anthony Goldbloom of Kaggle announced the $3 million Heritage Health Prize, for using data mining to predict hospital admissions. There followed an interesting discussion on the whether it was fair to include external data sets, as is increasingly done to win data mining competitions. The consensus was that it's fine if it brings better predictions, provided it doesn't deter entrants to the competition who don't have access to this data. For me this just highlights the need for tools and datasets to do this well. I saw an interesting demo of using Google Refine to reconcile data against publicly available datasets, and there has been much discussion about the emergence of data markets.

All this is music to our ears at Causata: we derive powerful actionable insights precisely by combining customer data from multiple channels to give a comprehensive view of every individual customer.

Jason

Video: Paul Phillips, Causata's CEO, on creating business value from customer data

2011-01-06T05:44:00.000-08:00

In this short video, Paul Phillips, Causata's CEO, talks about how to create business value from customer data. With many companies exploring how to use web data to make better customer decisions, Paul explains the importance of connecting, learning and acting on data.

Learn more about Causata Technology

Integrating web data: the first step

2010-12-16T06:35:00.000-08:00

A hot topic at the moment is connecting web behavioral data to “offline” data –everything else that you know about your customers stored in legacy and backend systems. Clients tells us every day that using data from the web to inform and improve offline decisions is now a priority.

Many organizations have developed their customer analytics in silos with separate data infrastructure in different parts of the business. It’s common that the dotcom group in a company is so detached from the traditional offline part of the business that no data is shared at all. Not only are these companies not harnessing the power of the web data outside of the dotcom group, but they don’t have a good idea of which customers, are doing what online.

This can be the case even if they’ve got a well-managed CRM system and a sophisticated web analytics team. Web analytics systems are designed to show aggregated information across web site visitors, rather than store information about individual customers and weren’t designed to be connected to other systems.

At Causata we believe that connecting web data is essential.

With all the predictive power of web data there are a lot of cool things you can do because web data gives you customer intent. Customers are literally telling you “I like this TV”, “I’m interested in accessories”, “I’m in the market for a loan”, etc. You should to be listening to this information and acting on it quickly or you’ll lose out on opportunities to help customers and grow your business. We’ve seen again and again that online behavior can provide much more predictive power than traditional customer segmentation when it comes to understanding key events such as product purchases and attrition.

The conversations we have with clients generally focus on how they can understand why their customers behave the way they do and which customer actions can be influenced. There are huge opportunities for gains from linking data. There is, however, a basic step that nearly everyone can and should take right away.

If you’re a company that takes web data seriously then you probably already have access to some useful data. While many Fortune 1000 companies are experimenting with Google Analytics, which is free, about 40% of the Fortune 1000 care enough about web data to purchase a paid web analytics tool such as Coremetrics, Omniture, or Webtrends.

If you’re using a paid web analytics tool you’ll probably have already marked up your web pages to capture key interaction information for reporting purposes. Most of these tools will also let you access months of historical web data in batch form. It’s possible to combine this with your offline data.

The first step for integrating web data is to connect this historical batch data from your web analytics tool to your offline data. The challenge is to find a common key that will identify customers so you can join the two data sets.

Ideally, you will already have a Customer ID or Transaction ID of some sort in your web markup that you can use as a key to link the web data to the offline data. Otherwise, you’ll want to consider a few small changes to the website – typically a single field added to a handful of web pages is enough to get started. The benefit is huge because once a user identifies themselves you can understand their browsing history with your offine data. This data matching process can be improved further the more keys you add: things like a Cart ID on a checkout funnel or an email address when someone clicks through to your site from an email campaign (among others) can be used to merge offline and online data.

Once you have the online data connected it can be very enlightening. Suddenly it’s possible to see how people behave online, how online behavior differs according to different customer segments, and how web site activity relates to other offline activities in a store, a call center or anywhere else.

Of course, connecting the batch history from your web analytics tool to offline data isn’t the be-all-and-end-all – with the web you really want to be working in real-time – but it’s a great first step that everybody should be taking… now.

Gareth

Learn more about Causata Technology

More prediction, less reporting. Freeing up an analyst’s time so they can make a greater impact

2010-12-16T04:24:00.000-08:00

Why do so many organizations struggle to wring value from their data? For most the problem is not data quantity – it’s relevance and connectedness. Much of the data scattered across enterprises is practically worthless until it’s processed and connected in one place. Indeed, expecting valuable insights to spring from your data is like expecting a pile of coal to turn into diamonds. Raw ingredients have little value without the right conditions.

Irrelevant, unmerged data is so common that data scientists, analysts, and modelers spend much of their time trying to squeeze it into more valuable forms. Among the people I’ve asked it’s not a stretch to say they spend over 90% of their time on this. They’ll also tell you they’d prefer to be spending their time building predictive models because they know that’s where they can create most business value.

This picture gets worse when you start to tackle web data. Web data is notoriously sparse. Real-data lacks the pretty normal distributions people see in textbooks and this is only amplified on the web. Consider the example below showing page views: it may seem odd but most customers have none because not everyone is using the web yet, and the most active online customers with two or more page views are a small fraction of the overall population. Are these active online customers making more purchases across all channels? You can’t answer this question without connecting your data at the customer level across channels.

All the work to collect the web data in the first place, structure it into a dataset that can be used in a statistical tool like SAS or R, etc. can take a lot of time. The world would be a much better place if some of this effort were done automatically for a data analyst. Causata was designed with this in mind.

Causata automates the collection and connection of real-time web data so that it is structured around customers and can feed predictive models. Rich markup that characterizes customers is placed on web pages. This web data is combined with offline data and stored where it can be accessed easily using a unique identifier for each customer. Causata provides a large set of predefined variables suitable for customer analysis and modeling. More variables can be added as needed. These variables are constructed at query time from raw events so there is enormous flexibility for an analyst in terms of adding or changing variables at any time.

Causata displays the data graphically allowing an analyst to easily understand the distributions of the variables across any set of customers and also which variables have the most predictive power for a particular outcome. From there a dataset suitable for modeling can be exported to SAS/R. This means datasets for modeling can be created in minutes rather than weeks.

A huge portion of the end-to-end effort of creating value from customer data has been automated in Causata. If you’re an analyst, suddenly you can spend much more of your time on what you enjoy. You can actually explore the statistical models you want, put them into production and give your results a big boost.

Companies need to shift the analytical emphasis from reporting to operationalizing the power of predictive models by embracing opportunities to allow data analysts to spend more time on higher value tasks. There are so many places in a business that stand to benefit by being a lot smarter with data - marketing, personalizing an online experience and service are just a few examples.

Gareth

Learn more about Causata Technology

Simpson's Paradox meets marketing

2010-12-01T06:49:00.000-08:00

Part of Causata's name is drawn from the word 'causal'. In delving into masses of customer data Causata's goal is to surface causal effects. To do that Causata keeps incredibly granular data about events (any customer interaction) along a timeline. By examining events across time and controlling which people are exposed to particular interactions (such as who sees certain content on a web site), Causata teases out causal effects.

One of the surprising things when dealing with the data is how counterintuitive some results can be. A great example of this is Simpson's Paradox (it's not strictly a paradox, just very unexpected).

Imagine a situation where two pieces of advertising (called Version A and Version B) have been prepared for use in print and on the web. The advertisements are placed and the percentage of people responding to the ads is measured. From this data a simple table can be drawn up showing the number of responses, the number of impressions and the response rate for each of the versions and media:

	Version A	Version B
Web	1800/130000 (1.38%)	500/40000 (1.25%)
Print	750/40000 (1.88%)	2200/130000 (1.69%)

It's clear that Version A outperforms Version B in both print and on the web. From that it would be obvious to conclude that Version A 'works better' than Version B. But there's a problem.

When the results are combined a different picture emerges:

	Version A	Version B
Web + Print	2550/17000 (1.50%)	2700/170000 (1.59%)

Here it's clear that Version B produces a better response rate. That is Simpson's Paradox in action.

In this case this change occurred for two reasons: there's a large difference in the number of impressions each advertisement received (perhaps because of different sized print runs for a catalog or a difference in targeting on the web) and there's a confounding variable (a variable that needs to be taken into account when interpreting the results).

Here the confounding variable is the medium (web or print). Notice how for both versions of the advertisement print is more successful. For whatever reason people seeing the ad in print respond better to it than those who see it on the web. The choice of medium and the disparity in the number of impressions together mean that the wrong conclusion can be drawn if data is mixed.

Simpson's Paradox occurs because of an incorrect interpretation of correlation and causality. When looking at the combined data above it's tempting to say that Version B works better than Version A. There's a leap from the numbers to the conclusion, and because the confounding variable is ignored in the combined numbers its effect is also ignored.

Of course, the opposite is true. Version A is better than Version B. Once the true causal relationship is examined (the effect on response rate of versions A and B for web and separately for print) the true answer is revealed.

If you want to dig deep into this read Simpson's Paradox: An Anatomy.

At Causata we are mindful of Simpson's Paradox and its effect on conclusions drawn from data and are careful that our system is not itself fooled by this simple but tricky problem.

John.

Focusing on value 'to the customer'

2010-10-13T14:18:00.000-07:00

One of the most important things I learnt as Skype went from tiny company to household name was that our success depended on focusing on the user. At every company meeting we'd hear the mantra about it being the user that matters. Niklas Zennstrom, who founded the company, would tell us that if we ever found ourselves making a choice between what was best for the business and what was best for the user to choose the user every single time. It’s something that as a consumer I wish every company embraced.

This advice would be well taken by every corporation in America. Anyone who's struggled with their cell phone service can appreciate how important putting the user first really is. In fact, the number of times in the last year that I’ve dealt with companies where the person on the other end made the wrong choice shows there is a lot of room for improvement.

At Causata we’re engaged with many of the largest companies in the world. We speak to a lot of forward-looking people at retailers and financial companies and it is heartening to hear positive reactions to our approach to end user focus and customer value.

We explain, as a data driven company would, that you need to be aware of the value of every customer, specifically the potential value of customers going forward, but we’re very clear that we don’t think it’s enough to be thinking about the value to the business, you need to be thinking about the value to the customer.

A lot of companies blast out emails to millions of people. A couple of weeks later a small percentage of them respond and the company pats itself on the back. We’d say: Not so fast! What about the more than 90% of people whose time was wasted and whose inbox was cluttered with a message they didn't want to receive at that moment? Most consumers would say that their time is precious. Was the value to the consumer considered against the expected value of that email to the company? It's disappointing to say but I'd be surprised if it were.

At Causata, we perform predictions and calculate value at an individual customer level and know that you can predict with high accuracy who is likely to be interested in an email and who is likely to unsubscribe. We make available a host of other user-focused information too. So, if a company wants to take a user focused approach to sending that email, they absolutely can.

The world would be a better place if, as they conduct business - better yet at every customer interaction, companies didn’t just consider the value to them but the value to the customer.

Gareth

What Causata means by real-time

2010-10-06T09:10:00.000-07:00

Throughout Causata's web site one term is repeated constantly: real-time. But 'real-time' means different things to different people. For example, anyone who has worked with a real-time operating system will instantly think of hard constraints on execution time for critical routines.

At Causata we are actually using real-time in two senses: first to mean that the system responds within a small window of time when queried for a decision and second to mean that the system takes into account information shortly after it has arrived.

Causata is designed to provide decisions and customer profiles based on information in its distributed data-store to improve existing touchpoint systems. These profiles and decisions are designed to enhance interactions that a customer is having on the web, the phone, through a mobile device, via email, or at a teller. That means that Causata needs to be able to make a decision or serve a profile rapidly when the customer is interacting with the business. On the web that implies decisions and profiles in milliseconds.

To fully use customer interaction data Causata has to be up to date. The value of customer data depreciates quickly. For example, on the web the customer's trail of clicks indicates what's on their mind at that very moment. If a system can't accept and act upon the trail of clicks while the visitor is still on the site a great deal of information is wasted. Causata is designed to receive and incorporate the most up to date information as it happens.

And it's not just the web that benefits from real-time. With real-time profiles and decisions based on the absolute latest data any touchpoint can use information from any other. For example, emails being sent for marketing purposes can be personalized using information derived from Causata gathered from the web, call center, transactions and more.

John.

The challenge of identity association

2010-10-04T07:19:00.001-07:00

Because Causata wants to bring together all the information about a customer, a necessary first step is identifying which data is associated with which customer. And, unfortunately, customer identification is quite a challenging problem.

Imagine a customer researching digital cameras online. The customer visits the website of an electronics store and views a number of cameras; Causata records the web session with only an anonymous cookie tracking the customer.

A week later, the customer is walking by that same electronics store and decides to walk in. Having already researched cameras, the customer heads straight for a specific digital camera.

The customer purchases the camera with a credit card and presents a loyalty card. Causata records the transaction (down to the line item level) and the associated identity information as a retail session. The loyalty card has a name and address associated with it; it was captured when the customer signed up. Causata also has that information in its data store.

The next day, at a different location of the same electronics store, the customer buys a memory card for the camera and presents the same credit card (but has forgotten their loyalty card). Causata again creates a retail session and records the purchase with the associated credit card.

A week later, the customer logs in to the electronics store website to see how many loyalty card points they have. Causata records a web session capturing every click made plus the loyalty card information.

Causata notices that the last time this machine was used—when the customer was researching cameras—the same anonymous cookie was used. It can now associate the original research with the customer who eventually bought the camera.

Finally, the customer drops in to the same electronics store near their office and buys a lens cloth using cash and presents a loyalty card. Causata records a retail session with the loyalty card information.

All of these interactions build a complete picture of the customer: a customer who researched online, continued on to buy in store, made follow-up purchases, and tracked their loyalty points. Clearly, this is a loyal customer.

But without identity association it’s hard to see that complete picture.

The image below shows the individual sessions and their relationships.

After the customer has researched the camera online and bought it in store, the system records two separate, unlinked sessions: the anonymous web surfing session and the purchase. When the customer buys the memory card with the same credit card used to buy the camera, those purchases are linked via the credit card.

The picture becomes clearer when the customer logs in to the website to check their loyalty card points. The anonymous research web session is then linked to the purchase of the camera, and four separate sessions are joined together. Finally, the lens cloth is purchased, and this is added to the knowledge about the customer.

The connections between the interactions in the diagram above can also have probabilities associated with them. For example, it’s highly likely that the camera purchase, lens cloth purchase, and online loyalty card point tracking are sessions performed by the same person. They are, after all, linked by an individual loyalty card.

But the connection between the initial research session and the rest of the data is more tenuous. It’s possible that it’s the same person, but it’s also possible that more than one person used the same computer, thereby blurring the identity associated with the anonymous cookie.

Much more about the technology behind Causata can be found here.

John.

The Connected Customer

2010-09-30T07:44:00.001-07:00

When I talk about a connected customer I'm actually referring to three things

1. Customers of a business are connected to the business in multiple ways through different channels

2. The same customers are also constantly connected to the business because of the proliferation of mobile devices

3. Businesses need a connected view of each customer that takes into account all the ways customers interact with them

Causata's business is a consequence of these three. Causata talks about being in the business of providing real-time customer intelligence. What this means is gathering together all the clues a customer leaves when interacting with a business, analyzing them to discover customer desires, and providing the intelligence to systems that can act upon it.

Customer clues come in many forms both inbound and outbound. Inbound customer clues include every click the customer makes on a business' web site, the transactions they perform with the business, and calls they make to a call center. Outbound customer 'clues' are things like direct mail pieces that were sent to the customer, emails they've received, offers they've been shown on the web.

When drawn together this total view of the customer and their interaction history provides a rich set of data to analyze. But drawing it together can be very difficult. Causata's software is designed to make gathering customer clues easy: we integrate with a wide range of systems through standard APIs.

Of course, it's not just API integration but also data integration. That's why Causata's distributed datastore is schema-free and based on an indexed key/value store that we developed for the specific purpose of serving up real-time customer intelligence. This is designed to make bringing together disparate sources of customer data easy. And, happily, it has worked well in the real-world.

One significant wrinkle in developing such a system is that the identity of a customer is both fluid and vague. A customer might authenticate with a web site and provide strong identity information, might use a loyalty card in a store providing a different sort of identity, or simply be observed using the same credit card online and off. Equally a non-authenticated user of a business' web site might be one of many people as computers are frequently shared. To intelligently merge customer identity information Causata build 'identity association' into the platform.

Once you've got robust data about every customer interaction and can identity the customer it's possible to perform statistical analysis or machine learning based on very rich information. We've built both a user interface and machine learning facilities into Causata to provide both human and machine access to real-time customer intelligence.

Humans can sit at the Causata Customer Insight tool and analyze their customer base quickly and easily. Equally the intelligence coming out of Causata can be used to drive other tools. For example, Causata can feed into a web content targeting system, or an email targeting platform, or a system producing checkout coupons.

Ultimately our goal is to produce rich information that other systems can use. To do that we provide both access to raw information inside Causata and to the output of our machine learning algorithms.

A lot more detailed information about the technology can be found here.

John.

The Arrow of Time

2010-09-30T07:43:00.000-07:00

In building a system to glean real-time intelligence from customer interactions we had to make a number of decisions about data storage. One of these was how much data to store. If a specific use for customer data is known (such as mining it for particular values or relationships, or building a statistical model based on certain characteristics) then the data can be summarized before storage.

For example, instead of keeping a record of every transaction a customer makes, it might be good enough to store just the number of transactions and the average value. Or store the same thing on a monthly basis. This would greatly reduce the amount of data needing to be stored. Of course, the problem with that is that you have to know up front what question you are going to ask of your data.

At Causata we don't know what questions are going to be asked (we supply a flexible querying mechanism to ask any question) and we also wanted to build in future-proofness. If you summarize data you are inherently throwing away information. And that means throwing away information, means throwing away the chance to answer unseen questions.

So, at Causata we decided that the only real solution was to store every single piece of information about customer interactions. We call the smallest piece of information an event and store everything: every click on the company's web site (with all the available metadata), every transaction, every time an email is sent to the customer, etc.

Naturally, that means that we have to manage a huge amount of data which resulted in our parallel, distributed architecture using key/value pairs. And there's a massive benefit to doing this: we don't lose the arrow of time.

When data is summarized the relationships between events that can only been seen from their order is lost. In the Causata system we keep everything a customer does (and the company does to the customer such as sending a direct mail piece) in time order. That means we can reconstruct the complete customer history.

And it unlocks something even more valuable: we can examine cause and effect. We can, for example, look at everyone who received a particular marketing email and discover what they did next. By comparing with those who did not receive the email we can uncover the real effect of the message.

John.

Careers: Senior Interaction/UX Designer

2010-08-19T08:02:00.000-07:00

Causata, Inc. is recruiting a London-based Senior Interaction/UX Designer to join our venture-backed startup. This challenging role involves creating a web-based user interface for a product that is used to visualize massive amounts of complex data. You'll be working with the latest technologies and on a ground-breaking product.

We know that a great user experience is at the heart of any successful product. As our Senior Interaction Designer we want you to be a driving force in gathering insight into our users' needs, behaviors and intentions. You'll be translating this into innovative interface designs for state-of-the-art software that enables our users to explore and interact with huge amounts of data. You will be responsible for the UX and the UI working closely with (and managing) a visual designer, engineers and the product team throughout all stages of the product cycle.

Responsibilities:
• Taking part in requirements gathering and user research.
• Defining the user and interaction model.
• Creating a detailed user interface design for all our products.
• Developing high level and/or detailed storyboards, mockups and prototypes to effectively communicate interaction and design ideas.
• Championing user needs and the overall user experience.

Requirements:
• Solid academic background, Masters degree or equivalent experience, specializing in design, human-computer interaction, ergonomics, computer science or cognitive psychology.
• At least five years experience in application design, online interaction design or information architecture.
• Outstanding conceptual design skills and familiarity with high-level deliverables (scenarios, personas, flow diagrams, etc.) as well as low-level interaction design best practice.
• Skilled in at least one method of UI prototyping (Axure, Flash Catalyst, Flex, Fireworks, Illustrator, JavaScript frameworks etc.)
• Contribute to the creation of our product roadmap.
• Rationalize design decisions to stakeholders and peers.
• Create detailed specifications as needed.
• Enthusiasm for IxD, UX, and IA with a broad knowledge of current practice.
• Excellent leadership, communication and teamwork skills.
• Experience or interest in data visualisation and statistical analysis tools would be highly beneficial.

Visit Careers to apply.

The Evolving Enterprise

2010-08-18T02:21:00.000-07:00

"We have only two sources of competitive advantage:
1. The ability to learn more about our customers faster than the competition
2. The ability to turn that learning into action faster than the competition"

Quotation is attributed to Jack Welch, CEO General Electric

If Jack Welch’s assertions are true (and I believe they are) then
there is a major disconnect between the way enterprises currently
think about data management and how they should be thinking about it.
If rapid learning from customer data and acting upon that learning are
the only real sources of competitive advantage then why aren’t
enterprises worrying much more about organizing data to efficiently do
just that? Why is the focus of data management today on basic tasks
such as storage, retrieval and querying for reporting? This can’t be
right.

I expect this situation to change quickly. Over the last few years
there has been a rapidly accelerating effort on developing powerful
new analytical architectures, and these technologies are creating a
growing appreciation that enterprise data has been an enormous
unrealized asset.

If monetary value really can be extracted from new analytical
technologies in a way that very quickly generates returns of many
times cost, then enterprises will be quick to implement these new
technologies. What enterprises need right now more than anything else
is hard evidence that this can actually be done.

Rather tantalizingly, what we will see in this accelerating market are
widening gaps between the technically possible and the commercially
proven, and between the commercially proven and published case studies
in the public domain.

Here at Causata we like Jack Welch's words quoted above. It would be
hard to more succinctly express the essence of what Causata's software
encapsulates as a product than Welch’s points 1 and 2.

Paul.

Big Answers

2010-03-12T03:58:00.000-08:00

In a recent Economist story two striking facts extracted from Big Data were reported:

Best Buy, a retailer, found that 7% of its customers accounted for 43% of its sales.

and

In Britain the Royal Shakespeare Company (RSC) sifted through seven years of sales data for a marketing campaign that increased regular visitors by 70%.

And it's not hard to find other stories indicating that enormous gains in revenue can be extracted from the data that companies already have. For example, the web site NapaStyle.com reports a 10% lift in revenue per visitor after two months of mining customer clicks to understand behavior. World Kitchen reported an average order value increase of 20% using similar technology.

Anecdotal evidence says that Amazon.com is getting an additional 10% sales revenue by mining the books that its customers buy to recommend other titles they might like. And Netflix gets 60% of its rentals from people clicking through on similarly mined data guiding its recommendations.

Startup FlightCaster is taking a Big Data approach mining airline records, weather and FAA real-time traffic information to provide accurate predictions of flight delays well ahead of the airlines themselves. Their application provides flight delay information 6 hours before the airlines tell their own customers.

Fox Audience Network, which handles advertising across all Fox properties (including MySpace), is using a large cluster of machines crunching Big Data to obtain an order of magnitude increase in advertising margins.

And Europe's largest ad targeting network, nugg.ad is using Big Data to drive everything they do.

Users of Big Data shouldn't be surprised by Big Answers: Big Data represents a massive untapped resource that all companies already have. It's just a question of integrating the right software to obtain Big Answers.

John.