Although the fundamentals of analytics have existed for decades even before computers, the embracing of analytics is in an embryonic and elusive stage. Like the football runner, his knee has not yet touched the ground – the ball is still good! Analytics has not yet been tackled, so I will continue to write about it.

The Root Cause for Analytics – Globalization of Markets and People

The increasing number of articles and books on analytics list many reasons for organizations to apply analytics for insights and decision making. Some reasons are:

• The margin for decision error is getting slimmer
• Gut feel and intuition are insufficient for the correct or best decision
• Increasing complexity requires intense analysis
• The compute power from technology has dramatically increased
• The availability of data is exponentially growing

I believe the root cause for the escalating need for analytics is deeper than those reasons. My hypothesis is that the root cause is the globalization of markets and people.

Why Analytics?

Think back to just a few years ago. Since then, globalization of markets and people has intensified to a higher degree, especially with the escalation of social networking. Examples include Skype, fast wireless Internet connectivity, affordable smart phones, Facebook, 24/7 financial markets, nano-stock trading (recall the “flash crash”), and cloud computing. Basically, what has occurred is the digitization of communications.

In the days of wooden sailing ships, merchants and proprietors of local economies, presumably based on products and materials, were protected. But today, there is not only substantial free trade across country borders and efficient supply chains but also relatively more services, some of which are not physically bound, like outsourced call centers.

As commerce and communications of our planet Earth becomes increasingly bound together more tightly and vibrantly, broad things happen:

• Everyone’s personal values, preferences, and behavior matter more than ever because they impact more people than ever.
• Our global financial institution and investment systems more quickly pursue financial return values. Money chases more money at lightning speed.

Cause and effect relationships and linkages are at the heart of the growing need for organizations to require and leverage analytics. The countries and economies of our world have shifted from being disconnected to connected to inter-connected to interdependent. As this has occurred analytics provides insights for better and faster decision making.

There are few circuit breakers in a hardwired world like this that can shield an organization from the core drivers, such as consumer demand, actions by competitors, or external events like natural disasters, pandemics, or fluctuations in foreign currency exchange rates or commodity prices.

So, am I piling on?

Is there too much being written about analytics? Am I “piling on”? I don’t think so.

In fact, my speculation is that organizations are just at the dawn of awareness when it comes to how critical it will be for them to gain competency in applying analytics. We are only scratching the surface of the full potential of analytics and the benefits that applying them will bring.

Businesses are capturing a wide range of data in their modern BI systems in an effort to detect patterns and find correlations that are relatable to business performance.  With the proliferation of, and hype about, new data types, it can be difficult to quickly make a decision about which data relate relate to business performance.

Before looking externally, optimize your own sphere of control. Utilize the data you own and that you can directly impact before expanding into untested data types that are more difficult to access, analyze and combine with existing enterprise data. As an example, imagine the complexity businesses will face when accessing existing data types like inventory levels or headcount, vs. social media or external job market data, which are out of their sphere of influence.

It is tempting to look outside of the organization for the trendier and more hyped data types, while ignoring the fundamentals. While those external data sets can indeed be valuable, for performance sake it’s better to first focus on what you have, while still keeping an eye on technological developments externally.

For example, many organizations currently lack a firm handle on analytics fundamentals like sales forecasts, operations, and individual, team or company performance metrics. Many even lack the ability to track current headcount and attrition. Since business performance is heavily reliant on the people doing the work, it makes sense to measure the individual, team performance and the related factors that drive performance.  It is rare to find these measures on the web or on Twitter.

Innovative organizations are now measuring what seems to be an obvious factor – the people themselves. Who are they? How do they work? What innate characteristics tie to top or low performance? Do the characteristics of their people correlate to top business performance?  It’s simple to gather this data about your internal assets (the people) – you just ask since they are in your sphere of control.  And, data about the people themselves is reusable far into the future.

A strategic focus on local success factors is easier, faster, more structured and more easily obtained – and it can easily combine with existing data sources.  Correlating internal performance outcomes with talent characteristics may seemingly have less sizzle than a trendier Twitter sentiment analysis, but arguably more steak. And by steak, I mean control, predictive capacity, and direct information about performance.  Again, I’m not dismissing looking outside the organization and at unstructured data, but first things first!

As consumers have become aware of some of the data that is being collected about them, there has been backlash. Recent flare-ups with web companies like Facebook are good examples. The extent to which the tracking of behavior on the internet occurs – with Facebook, Google, and other public sites capturing data about who you are, what you are doing, where you are going, and what you want – is not known to most people. Even though many privacy policies technically declare intentions to collect and use data, the dozens of pages of “legalese” terms used aren’t read or understood by most people. There is often not a fully transparent, opt-in process with plain language declaring corporate intentions. In addition, some companies have pushed the limits of what their privacy policies technically allow. This is a bad practice that has resulted in trouble.

As much as I get excited as an analytics professional to dig into all the new big data sources available, I similarly become hesitant as a customer and citizen. As I predicted on the International Institute for Analytics’ 2012 predictions call, I believe that privacy concerns will be a major influence on how big data itself, and the use of it, evolves. There will need to be an extremely high level of trust between organizations who want our data and those of us who provide it. That trust must be earned and maintained. All it will take will be a few cases of violated trust, intentional or not, to derail the relationship and set us all back.

I have wondered what the “big moment” will be that causes everyone to realize how much about them is exposed and leads to a major popular revolt. Honestly, I thought the big blow up in December around Carrier IQ would be that moment. For those of you who missed the news, Carrier IQ (http://www.carrieriq.com/) is a company that provides software to mobile device manufacturers. The software collects usage information aimed at helping telecommunications companies and mobile device manufacturers identify hardware or network issues.

Someone posted a YouTube video showing the software capturing much more data than anyone would expect or want. The phone was even capturing key presses such as when you entered a password on a secure website. Naturally, this caused a huge uproar. (You can view this series of articles from CNNMoney for more detail: Part 1, Part 2, Part 3, and Part 4.)

In the end, it was determined that manufacturer HTC had accidentally turned on a debug mode in phones sent to consumers that was only supposed to be turned on during testing. The phones were capturing the information locally, but not sending it back to the carriers. However, even having such information stored on your phone is a huge risk if someone steals your phone and knows where to look for your passwords.

Ultimately, I came away believing that none of the companies involved had any dishonest intention. However, due to some errors in process and security procedures, a situation arose with potentially harmful impacts for users, including identity theft or the emptying of accounts by crooks able to access the data.

This is one example of how the world of big data enables analytics and actions that are very much like Big Brother. Without immense care and caution, organizations can wander into a quagmire like the carriers, software providers, and device manufacturers did with Carrier IQ. Those involved can only hope that nobody traces crimes perpetrated against themselves to the loss of their phone and the data that the software was storing on it.

Trust will be required for big data to realize its potential. If trust exists between consumers and corporations, for example, an organization or industry group can come up with its own guidelines that restrict how big data is stored and used. If that does not happen, governments will step in – and their regulations will likely end up being more restrictive and expensive, and could carry unintended consequences, as they often do. One way or another, the Big Brother potential for big data must be addressed soon.

While I was wrong that the Carrier IQ incident would be “the moment,” I do believe it is coming. Perhaps it will be the first leak of the electronic medical records of a prominent politician, or shoppers discovering that a loyalty card is being tracked with embedded RFID chips to monitor their in store activities without their consent (to my knowledge no retailers do this, but it is possible with today’s technology). We have been fortunate thus far that most coverage of big data analytics in the news has been favorable as the world marvels at the pure potential (see this week’s Wall Street Journal article.) We must anticipate possible negative perceptions and actions, and give the respect and care required to big data. This is the only way to avoid the perception, or reality, of big data becoming too much of a Big Brother for people to accept. If that happens, it may never reach its potential.

I am going to use the generic term MapReduce in this column. Hadoop is an open source implementation of the MapReduce framework, but there are others such as Teradata’s Aster Data offering as well. The important part for our purposes here is what a MapReduce framework can do, not which specific implementation you choose to utilize.

Scenario 1: A MapReduce platform as a “live” archive

This is a theme I heard repeatedly, and it was explicitly mentioned by speakers from show sponsor Cloudera. Consider a huge mass of historical data that rarely needs to be accessed. Traditionally, such data would be archived to tape or some other media and sent to storage (if it wasn’t simply deleted). While in theory the data could be accessed if needed, it is difficult and expensive to recover the data. In practice, people rarely leverage the archives.

With disk space being so cheap, the data can now be sent to the inexpensive, commodity hardware within a MapReduce platform. The data is still accessible at any time and is a “live” archive. Perhaps few users will use the data over time, but when they need to, they can. MapReduce is an inexpensive way to enable such an archive. It wouldn’t make sense to keep this archive live in a more expensive, formal data warehousing environment.

Scenario 2: A MapReduce platform as an ETL and filtering platform

One of the biggest challenges with big data sources such as web logs, sensor data, or even masses of email messages, is the process of extracting the key pieces of valuable information from the noise. Loading raw web logs into a database system to then throw away 90% or more of the data during processing isn’t the best way to go. Loading large, raw files into a MapReduce environment for initial processing makes terrific sense.

A MapReduce platform can be used to read in the raw data, apply appropriate filters and logic against it, and output a more structured, usable set of data. That reduced set of data can then be further analyzed in the MapReduce environment or migrated into a traditional analysis environment. The key is that only the important pieces of data remain, which makes it much more manageable. Typically, only a small percentage of a raw big data feed is required for a given business problem. MapReduce is a great tool to extract those pieces.

Scenario 3: A MapReduce platform as an exploration engine

Another recurrent theme at the show was the concept of a MapReduce platform being used for discovery and exploratory analysis. This is another solid application for MapReduce. Once raw data has been read and processed, further analysis can be done against the data within the MapReduce environment. As always, many paths of analysis may be tried before a successful one is found. Once the data is in a MapReduce setting, utilizing tools to analyze it where it already sits makes sense.

This scenario leads to a major decision point, however. Once a set of data is found to have high value via analysis in MapReduce, an important next step is to combine the new data with existing data. This is so that each data source can be made even more valuable by being combined with the others. Once you have distilled the data down to what is important, it should be loaded into the corporate systems that users have wide access to. It doesn’t make sense to pull all of the data out of a data warehouse, for example, just to match it with one new source of big data. It makes more sense to load the one new source of data alongside all the other pre-existing data within a data warehouse.

That last point is one where those loyal to MapReduce may differ with me. Many discussions at Hadoop World suggested that it does make sense to pull all corporate data into MapReduce. I predict that in the long run, however, things will go as I suggest above. Keeping data movement to a minimum is essential, as is making it available to as wide an audience as possible. For these reasons, MapReduce environments will augment, rather than replace, traditional environments.

The story I will be telling here is a true story that happened in my neighborhood a few weeks ago. While I don’t have any personal stake in the school, teachers, or students in this case, the story has eaten at me ever since I heard about it. Anyone who understands analytics will likely be equally appalled.

The local schools in my area are the best in the state and there are a lot of very smart, motivated kids that attend the schools. One of our neighbors has a very good student who is a senior in high school. We’ll call her “Sue”. Sue has always had good grades, she has never gotten into trouble, and she is taking a variety of challenging classes so that she can get into a good college next year.

Sue is taking an Advanced Placement course through the English department. This summer, her class was asked to write a paper as an extra assignment before school started. So far, so good. A new teacher in the department brought up the point that the school wasn’t using any plagiarism checking software and that they really should start using it. Again, so far, so good. Then things went horribly wrong…

The results of the plagiarism analysis showed that every single student in the class had cheated. All failed the assignment and initially the school planned to note the offense on each student’s official transcript. Goodbye, good schools! One might think that the school would question whether there were issues with either the software’s analysis or how the results were interpreted. After all, 100% of a group of “A” students with no history of trouble was flagged. The school did not question the results. The parents did, however. After some digging, the findings are quite troubling.

First, the software by default looks for any phrases of three words or more that match between two submitted papers. Each “offense” of a “copied” three-word phrase is tagged. Get tagged too many times, and you’re identified as a cheater. Let’s think about this criteria applied blindly without further thought. Assume students are writing about Tolstoy’s War and Peace. Two students start a sentence with “Tolstoy said that…” or “The meaning of…” or “The book refers to…”. They are now guilty of plagiarism. The software assumes nobody could have such phrases in common without copying from one another. Such tags are useful as a starting point. But, to be applied correctly, someone needs to review the papers and validate if any of the phrases really appear to be copied or just innocent matches like the examples. Nobody did that.

It gets worse. Part of the assignment was to provide definitions for a number of terms. Students were told that using dictionaries was allowed. Sue pasted definitions from a dictionary into her paper. It ends up that another student she barely knows had many of the same definitions. That, the parents were told, was the smoking gun. Clearly, the two girls had copied from each other. What wasn’t considered is that there are perhaps a handful of well-known dictionaries and dozens of kids in the class. Isn’t it reasonable that a couple of them may have used the same dictionary? Not according the English department. The only way the girls could both have a definition from Dictionary.com is if they copied from one another.

Sue’s parents managed to get the offense removed from her transcript, but not the 0% grade for the assignment. The teachers in the department are now unwilling to provide college recommendations for any of the students since they are all now viewed as cheaters. Some of these kids could quite literally have their futures stolen due to the misuse of analytics by a well-meaning school.

The point of this story is that analytics can’t be blindly followed. Results must be put in context. Unexpected results should be further examined. The English department’s personnel don’t understand what the plagiarism software is doing or why. They just assume that if the software says “potential cheater”, then it must be true. In this case, a horrible injustice has been done to some smart, honest kids.

As analytics become more pervasive and more and more tools make complex analytics “easy” and “push button”, we’ll continue to see examples like this. Those of us in the field have a responsibility to educate others of the limits and appropriate uses of analytics. In this case, the right course of action was to look at each student’s paper and interview them to get their explanation of the flagged passages. Also, the settings on the software should have been validated. To any fair-minded person willing to think logically, Sue did not cheat. To an English department using analytics software they don’t understand, she did. After all, the software found some red flags. What else do they need to know? The kids in the class may not get into a top college after all their years of work for they now have the brand of “Cheater” unfairly stamped on their chests.

Just like any other tool, analytics can be very powerful and helpful when used in the correct way. Analytics can also do major damage when used by those who don’t understand it and don’t use it correctly. In business, we’ve all seen cases where statistics or figures are shown to an executive without full context or needed caveats. After seeing the data, the executive takes some drastic actions that in some cases are not productive or required. It is imperative to ensure that analysis results are used correctly. Let’s review a few practices that should be standard when producing and using analytics. If these principles had been applied, this story would not have happened.

• Make sure that someone in the process fully understands the analysis being done, its strengths, and its weaknesses. Not everyone has to understand the gory details, but someone must.
• Make sure that the various settings and options utilized in a process have been chosen with good reason. Don’t just assume the default settings are best for your situation.
• When unexpected results are found, investigate further and ask critical questions before jumping to conclusions. No algorithm or software package is omniscient. “Because the software said so” is not an acceptable answer.
• When provided with additional facts or data that contradict an analysis, give them consideration. The goal should be the right answer, not your initial answer.
• Be careful to enable people to execute only the types of analysis they are prepared to execute correctly. It is easy for people to be in over their head and not even realize it. The results, as we’ve seen, can be devastating.

Last week I spoke for IIA at Predictive Analytics World. Whereas I often speak to executives who aren’t yet persuaded of the virtues of analytics, at this gathering—which also included attendees of the Marketing Optimization Summit and Text Analytics World—that wasn’t the problem. I was preaching to the converted, who already undertake a wide variety of analytical initiatives in their organization. The problem becomes not how to get going with analytics, but how to “industrialize” them.

As analytics grow more popular and central to business strategies, there is also a need to produce analytical miracles repeatedly, reliably, and quickly. The old days in which an analyst could take his or her time to produce custom analytical solutions are almost gone; instead, groups of analysts need to have “analytical service lines” in place. Some organizations might refer to them as “analytical solutions.”

These are necessary not only because the customers of analytical groups within organizations need quick and reliable delivery of analytics, but also they need to be familiar with the possibilities for analytical work. A “menu” of services that can be provided with speed and reliable outcomes can be very useful to the consumers of analytics. Incidentally, external analytical consultants should also provide a similar menu of analytical service lines.

One of the best examples I have found of such service lines or solutions is at HP Global Analytics—the analytical shared services group of the giant computer company. This group, largely based in India, has created a set of repeatable and scalable analytical capabilities serving many different functions across HP. For marketing, for example, they offer market intelligence, customer targeting, marketing spend allocation, and pricing analytics. In sales they offer sales force allocations, pursuit and conversion optimization, compensation optimization, and sales performance reporting. The group also has offerings in customer service, supply chain management, and HR—19 solutions in total.

So what comprises an analytical service line? It should have the following attributes:

• You’ve done it before, ideally several times—and it’s achieved a good outcome;
• You have at least some sense of a process that is followed to produce the desired result;
• You either have data readily available for the analysis, or know where to find it;
• You know what the likely decision outcomes are for the analysis;
• You know who the likely customers are for this service within your organization or client;
• You have created some degree of marketing materials to describe this service and its benefits.

As with all analytical capabilities, you can’t assume that your customer will understand analytical jargon on the service line card. Just as a restaurant markets the items on its menu by creating appealing descriptions and training service personnel to describe them, the analytical offerings on the menu have to be marketed and sold too.

Creating a set of analytical service lines, and executing on them effectively, will go a long way toward scaling up analytics in your organization and delivering them efficiently. Some analysts may yearn for the one-off, ad hoc analytical approach, but most will probably appreciate the increased influence they are having on the business.

Who knew that the view of analysts would change so much? I certainly didn’t! For years, like so many other valuable roles in business, analysts were relegated to the business social strata known as “Geeks”. Had I suggested that analytics people were “cool” or “sexy” when I first came out of school, everyone would have laughed. Today, there are regular references to analysts being “cool” and “sexy”…and they are not jokes! What a stunning turn of events this is. It is a rare opportunity to be in a profession that has experienced such a massive change in image, influence, and appeal. While a number of influences were at play here, I believe these changes were primarily driven by the realization of the value of data as an asset and the power the analysis of data has in driving sustainable business success. Analytics professionals are right in the middle of this action. I only wish I could claim I had planned it this way.

We should all embrace the new acceptance and importance that has been put upon our profession. It used to be that analysts would provide input to somebody who would provide input to somebody who would be in the room with a decision maker. Today, analysts are often included directly in the decision making process and can rise to the level of a true decision maker themselves. In many ways, analysts have “arrived” and become an accepted, respected, valued part of business today. We even have Chief Analytics Officers being named and entire organizations have been created, such as the International Institute for Analytics (IIA), that are focused solely on making analytics more a part of everyday life.

At the same time, I think we all need to be cautious about buying too much into the hype in the press. It may feel good to be called cool and sexy, but don’t drink the Kool-Aid too much. Another, newer “shiny object” will come along and excitement may shift, but analysts’ value to the organizations we serve must not. Analytics professionals need to stay focused on providing solid analytics and doing their job. They can’t get sucked into the role of a rock star.

Real rock stars who succeed long term first and foremost make good music and put on a good show. Rock stars who focus on being stars instead of making good music and putting on a good show typically fade away and are forgotten. There have been some “one hit wonders” who were a hot “A-list” star for a short time. Don’t be the Vanilla Ice of analytics in your organization!

Certainly, much of the progress the analytics profession has made is going to stick with us. But, don’t expect analysts to be the newest, coolest member of the “real team” at your organization forever. Someone newer and shinier is going to come along. Eventually, you’re going to have to get back to producing solid analysis and direction for your organization as the way to earn your keep. Enjoy the limelight today for what it is. Just don’t change who you are and what you do.

There was a neat “Blogarama” focused on “The Emerging Role of the Analyst” on October 6th. It was sponsored by the SmartData Collective. While I wasn’t able to post this entry on the official date, I would still like to invite readers to check out other opinions on the topic.

The Analytics Research Council for IT provides CIOs, CTOs and Chief Data Officers with a leadership view on the competencies required to enact breakthrough enterprise analytics, in addition to covering topics on leading practices in IT architecture and governance that support analytics decisioning.

Due to the intimate format and specificity of research content, Research Councils are structured to serve 2-4 executives per enterprise. Positions are offered by invitation only and all proceedings are governed by strict adherence to IIA’s confidentiality standards.

The ITARC is designed for senior IT executives (CIOs, CTOs, Chief Data Officers) and team members who actively manage the data and technology that power their organizations’ business analytics initiatives. Organizations are invited to join the council for a 12-month subscription. For more information, please contact Katherine Busey  at kbusey@iianalytics.com.

Looking back more than a few years, there weren’t many conferences focused on analytics, especially not from a business perspective. There were conferences that focused on tools, such as those hosted by SAS and SPSS. There were a few conferences focused on analytics from more of a technical and academic angle like KDD. There were hardly any shows focused on the business application of analytics. One of the earliest in that space was the National Center For Database Marketing (NCDM) show.

In the past year, I have been made aware of at least a dozen analytics oriented shows. Most of the new ones are going for a business spin. Some are general shows covering anything that can be called analytics. Some are focused on one topic such as web analytics. Some are focused on one industry, some are hosted by vendors, and some are regional while others are national. Analytics shows are popping up left and right and, overall, this is a terrific thing for the field of analytics. For the most part, they are incredible opportunities for us to collaborate and exchange ideas and experiences with one another.

My concern, however, is over-saturation of the market. How can this many analytics shows be viable in the long run? There is a risk that many of the shows will have trouble attracting enough attendees and compelling content to survive in the long run. Over time, it is almost certain that a few winners will emerge and many will disappear. In the meantime, I hope that people won’t attend a disappointing show and then write off the idea of analytics conferences in general. The analytics community needs to attract and engage as many people as possible. A compelling, value packed conference is one way to do that.

I think it is important for those of us in the analytics community to provide candid feedback to the organizers of the events we attend. Better yet, be an active attendee and provide input before the event. Guide the organizers down the path of what you want to see in a show, and which speakers have valuable insight. The organizers who listen will survive and morph into something with tremendous value. It is in all of our interest, and the interest of the broader analytics community, to make sure that when an analytics conference takes place, it is valuable and worthwhile.

Below, I’ve put a list of some shows coming up through the end of 2011. This list is not exhaustive, as it just shows the ones I’ve heard of through the end of the year, but you can see why I wonder if analytics conferences have gone wild!

• September 12 – 13: Predictive Analytics World Government
• September 18-20: Computerworld BI & Analytics Perspectives
• October 16 – 21: Predictive Analytics World New York
• October 23 – 27: Business Analytics Forum
• October 24 – 25: SAS Analytics 2011
• October 25 – 27: SAS Premier Business Leadership Series
• November 3 – 4: Predictive Analytics Summit
• November 10 – 11: Marketing Analytics
• November 15 – 16: Insight Live 2011
• November 30 – December 1: Predictive Analytics World London
• December 12 – 14: National Center For Database Marketing (NCDM)

I’m curious if you, our readers, have heard of others? What, where, and when are they? Were they valuable? Who can and should attend? Did any speakers blow you away?