In today’s discussion we will share some of our thinking and research to build models that measure what drives humans to perform.  Future articles will review the real challenges of actually defining or quantitatively measuring their performance.

Does Education or a Well-Aligned Degree Drive Performance?

One of the top five proposed factors for a performance model is the individual’s education – both the level of their degree, as well as their concentration of study.

We know from past experience that degree “types” and expected job performance don’t always align.  Consider, for example, the phenomenon that many excellent computer programmers have formal degrees in music.  In this case, there may be a pattern with this unrelated degree, or better, we might search even deeper for greater insight.

Data seems to show that the music degree, in this case, is a likely proxy measure of a creative streak, an out of the box thinker, someone who is intrinsically driven to create something elegant, balanced, and imaginative, someone with a native capacity to quickly “visualize patterns” in music or programming, or any variety of other role types.

Even for industry experts, it is easy to confuse knowledge with performance.  But of course they are not the same.  Technique and knowledge are important; you have to know what you’re doing.  But this is not the same as getting things done in an organization.

Does Experience Drive Performance?

Years of experience, or years in this role, can be predictive but have issues as factors that drive performance.  There can often be “survivor bias” when creating a model to measure human factors that drive performance.  Survivor bias can lead to good performers staying in a role, while “less suited” individuals would not stay in the role.

However, the “experience factor” leaves us no way to identify people who are up-and-coming.  Is there a way to know if a fresh graduate is well suited for a role?

Do Job Title and Organizational Size Drive Performance?

Job title and organization size tell us little to estimate their performance, but tell us much about the context of their performance.  Some may perform well in large groups but very poorly in small ones.  Depending on the domain, there may be interesting proxy measures for performance capacity – such as which tools they use and prefer.

Do Intrinsic Human Factors Drive Performance?

Another approach is to look at innate employee characteristics, such as the “types of tasks” an employee tends to be attracted to (their natural behaviors) or things they find deeply fulfilling (their personal drivers).  These are quantifiable by several methods and tend to remain stable through a worker’s career making them interesting to study over the long-term.

Innate human factors show what someone is intrinsically driven to, but don’t reveal whether they have the training or intelligence to perform.  As such, these innate factors are very interesting ways to discover raw talent before it has manifested in a career path.

Business progress continues to press on looking for ever-more-direct links between an individual and their business performance. Smart models will include intrinsic human characteristics and in the end, continued research will tell us which factors are strongest.  We will keep you posted as these studies progress.

There’s been a lot of noise lately about the concept of in-memory analytics.  As a recent example, if you followed the press around some major conferences in late April, you’ll notice that a lot of messaging was present around in-memory analytics products.  Many people have been asking questions about how in-memory can fit into their mix and how it relates to other options like in-database analytics.  So, what’s up with in-memory analytics?

There are several different types of “in-memory” offerings in the market.  Let me start by clarifying that what I will focus on here is specifically a new analytic modeling approach where analytics such as logistic regression models or neural networks are run in a massively parallel in-memory environment.  Such approaches are very relevant to organizations that need to build models at a new level of scale and how such offerings might fit into your company’s mix is important to understand.

First, does an in-memory analytics platform replace or augment traditional in-database approaches?  The answer is that it is quite complementary.  In-database approaches put a large focus on the data preparation and scoring portions of the analytic process.  The value of in-database processing is the ability to handle terabytes or petabytes of data effectively.  Much of the processing may not be highly sophisticated, but it is critical.  Think of the generation of hundreds of customer metrics based on detailed transaction history prior to building a model.  The metrics are later computed again in order to apply the scoring algorithm generated from the model.

What in-database analytics has not as broadly addressed is the model building process.  In-memory analytics addresses this step in a compelling way.  There are some very resource-intensive modeling algorithms.  Logistic regression is one example.  Traditionally, sampling was used and a limited pool of metrics was included in order to avoid resource constraints.  This is no longer necessary.

The new in-memory architectures use a massively parallel platform to enable the multiple terabytes of system memory to be utilized (conceptually) as one big pool of memory.  This means that samples can be much larger, or even eliminated.  The number of variables tested can be expanded immensely.  And, the number of iterations that can be run can go up exponentially.  Of course, this power comes at a cost.  So, it will only make sense to use it in cases where the costs can be justified by the benefits.

In-memory approaches fit best in situations where there is a need for:

• High Volume & Speed:  It is necessary to run many, many models quickly
• High Width & Depth: It is desired to test hundreds or thousands of metrics across tens of millions customers (or other entities)
• High Complexity: It is critical to run processing-intensive algorithms on all this data and to allow for many iterations to occur

One area that has been an initial focal point for in-memory analytics is the risk management function within major financial institutions. These institutions have hundreds or thousands of models that address various aspects of risk.  As new information comes in, the models need to be updated.  With in-memory analytics, these organizations are now able to update their models much more frequently than in the past, even daily or hourly, which leads to less risk and fewer losses.

The current generation of in-memory technologies won’t be a fit for everyone.  However, the use of such tools leads to several major benefits when they do fit:

• More accurate models will be generated since many more iterations to tune the results can be completed in a timely fashion.
• More frequent model updates since models can be updated regularly to keep scoring routines fresh instead of using existing scoring routines
• A much higher volume of models can be generated in the same timeframe which allows models to be applied more broadly to new problems
• Models can be built even on large and complex data in seconds or minutes, which allows expansion into near real time modeling

The combination of in-database analytics with in-memory analytics can be game changing in the right setting.  Be sure to consider how it may be a fit for your organization!

To see a video version of this blog, visit my new YouTube channel.

 Originally published by the International Institute for Analytics

Fundamentally, data are measurements. Most of the big data out there were not generated with analysis in mind. Much of it, in fact, is generated to send you a bill. That said, there is certainly residual value in transactional data. Still more of the big data out there is text, also not necessarily produced with the intent of analyzing it. That has residual value, too.

Regardless of source or type of data in your collection of big data, are you measuring what matters? Are you measuring it well—in other words, are your data of sufficient quality for you to make good decisions? And how much better could they be if you had better data?

Of the data you have, what questions do you ask? Based on the answers you get, how confident are you in the decisions you make? What could make you more confident? More data? Better data? Different data altogether? Maybe you would answer that with more analytical skills to produce better answers faster or some combination of these things.

Even with big data, we still need to think about how we can most efficiently and effectively learn from it. Sampling is a key strategy to help you learn faster; so is experimentation, which often involves measuring things you aren’t currently measuring — yes, more data of potentially greater value (which would be a sample). Simulation is another way to learn more efficiently and effectively in many cases, generating still more data (also a sample).

Too often, we settle for what we have because it’s easy; we take what comes to us because we can. Those who learn from data fastest and apply that learning to their advantage are thinking more strategically about what they do with their data (big or small), the data they still need to collect or generate, and how to measure what matters better and faster. Where do you put yourself on the learning continuum — are you measuring what matters? Are you trying new things?

Originally published by International Institute for Analytics

His discussion got us thinking … about why people are the way they are, how (if) they change, how to measure them, and how who we are impacts our business performance.  Like others before him – Dr. Seung’s approach is to map areas of the brain by EEG or fMRI.  This approach has been around for a long while and is a rather old school approach (though still unsolved).  Though fascinating, neither the EEG nor the fMRI reveals how the brain works, or makes any strides to helping us make this information useful by businesses.

To explore how the brain works, these brilliant scientists have taken on the monumental task of mapping how the millions of miles of neurons in our brains connect.  These networks of neurons are unique to each person, and formed by both genetics and experience.  The thesis is that this wiring pattern is what makes each individual unique.

All of this may sound mysterious and perhaps may feel irrelevant to today’s business focused Data Scientists, but it actually could not be more relevant to those of us who attempt to measure people and their affect on business performance.

This study lends credibility to findings that innate personality traits tend to be stable throughout an individual’s career (and beyond) making innate traits an interesting dataset to quantify and compare with business performance.

At this point, since the brain’s neurons and how they connect are yet to be mapped, we can only measure the outcome of these patterns i.e. an individual’s innate behaviors, drivers, career paths, job performance.  But, this does point to a future where we will be able to understand why people are they way they are, and run our businesses based on real brain science.

This will take decades, of course and may yield both good and bad things.  But we think it is interesting to see the world’s most brilliant scientists chipping away at the roots of what we in the trenches measure everyday.

Book: “Connectome: How the Brain’s Wiring Makes Us Who We Are” Houghton Mifflin Harcourt, Feb 2012, H. Sebastien Seung

URL:               http://hebb.mit.edu/people/seung/

Project:          http://wireddifferently.org/

Originally published by International Institute for Analytics

So the question is…when do you sample and when do you not?  And does it even matter anymore in the world of big data?  As I’ll lay out here, in most cases today there is no point in wasting energy worrying about it.  As long as a few basic criteria are met, do whatever you prefer.

First, let’s take care of the cases where sampling just won’t work.  If you need to find the top 100 spending customers, you can’t do that with a sample.  You’ll have to look at every single customer to accurately identify the top 100.  However, such scenarios, while common, aren’t the most prevalent type of analytic requirement.  They do represent an easy victory for the “no sampling” crowd, however.  Similarly, even a model built on a sample will need to be applied to the universe to use it appropriately.  So, when it comes time to deploy, sampling isn’t an option.

Second, let’s remember that many analytic processes are going to deal with or remove outliers and extreme values in some way.  As opposed to the “top 100” question above, many of the top or bottom observations may be removed or adjusted so as not to have too much influence.  Even if such observations are available in a dataset, they won’t be used.

The point above is important.  When building a customer propensity model, for example, you want it to apply broadly to the “typical” customer.  Perhaps there really is a customer that spends 1,000 times the next highest customer.  Even if true, that customer is so extreme and atypical that you shouldn’t include them in your model.  The model is meant to differentiate the masses and a few extreme customers can compromise the power of the model for the purpose it was intended.  Any customer who is legitimately that extreme is worthy of special handling from an organization to begin with.  You don’t need a model to tell you that.

Last, let’s come back to a typical scenario.  You need an average.  Or you want to get parameter estimates from some sort of predictive model.  Statistically speaking, a sample of sufficient size that is correctly drawn to mimic the population is going to get you the same answer as if you used all of the data.  There is no difference between the results from a sample or the universe for most types of metrics and models.

There are those who will vehemently argue that if you don’t need to sample, then don’t.  I can see that view.  One hole in this view, however, is that a correct modeling process will involve some combination of development and validation data sets…and these are effectively samples anyway!  Others will argue that you should only use the amount of data needed and that using more than the minimal sample required is a waste of time and resources.  I can also see this view.  One hole in this view is that if the resources available can easily handle all the data in a timely manner, then not much is wasted.

Where I net out is that I really don’t care.  If someone doing a project for me wants to sample, I’m ok with that as long as the sample is sufficiently large and drawn correctly.  If someone wants to use the universe, I’m ok with that too as long as the extra resources required compared to a sample aren’t pragmatically meaningful.  I am confident I’ll get the same results, so I’ll stay out of the argument over sampling.

I realize that this position of indifference may concern virtually everyone since most people land on one side of the fence or the other.  I guess my point is simply that there are plenty of other, more “meaty” topics to spend time debating when developing an analytic process.  I don’t see the use in losing much sleep over whether or not to sample in today’s world.  If the systems and tools in use can handle it either way, then I’ll let you have it your way!

One last unrelated note…if you think that you or someone you know might be an analytic superhero, be sure to check out the Analytic Superheroes site! 

Originally published by International Institute for Analytics

I routinely see opportunities for analysis in my daily life. Perhaps I was born with the DNA to constantly collect data and evaluate for better outcomes. For example, which check-out line should I enter at the store? Look at the length of the line and the number of items in the shopping carts of those already in line.

Of course, that example is a simple application of applying analytical investigation. A more complex opportunity is to answer this question – what angle and speed of an airline jet’s take-off ascent and landing descent will optimize its fuel usage? KLM Airlines has developed a sophisticated model to answer this question. It is based on data from thousands of past flights. The results save KLM remarkably significant fuel costs and substantial total expenses since fuel is a substantial portion of an airline’s expense structure.

Analytics’ perils from insect repellent

There is a problem, however, that not everyone thinks or behaves like a mosquito. They do not always inherently sense opportunities – the opportunities to apply analytics.

Perhaps I stretch this mosquito analogy too far when I presume that many opportunities may have insect repellent applied to them. For example, let’s consider the high expectations of service at a five star hotel. Ever wait in a long line at your hotel check out during the morning rush with others checking out? It might not appear cost-justified to the hotel, but it may be a very valuable extra expense to add one or more front desk staff if you irritate an important and delayed social media influencer who will complain on Twitter or TripAdvisor.com. How would you know? It is an opportunity for an analyst’s experiment or survey. The insect repellant analogy implies that an analyst may not “sense” an opportunity.

In the book Thinking, Fast and Slow by Dan Kahneman, recipient of the Nobel Prize in Economic Sciences for his seminal work in psychology that challenged the rational model of judgment and decision making, Kahneman explains the two systems that drive the way we think. System 1 is fast, intuitive, and emotional; System 2 is slower, more deliberative, and more logical. System 1 is largely unconscious and it makes snap judgments based upon our memory of similar events and our emotions. System 2 is painfully slow, and is the process by which we consciously check facts and think carefully and rationally.

A problem Kahneman points out is that System 2 thinking (slow) is easily distracted and hard to engage and that System 1 thinking (fast) is wrong as often as it is right. System 1 thinking is easily swayed by our emotions. As an example, he describes an observation that people buy more cans of soup in a grocery store when there is a sign on the display that says “Limit 12 per customer.” People miss the opportunity to analyze.

Life and an organization as a game requiring analytics

People enjoy playing games. To win or score high, one has to be superior with one’s analytical thinking. The electronic game industry has exploded. Angry birds anyone? The gaming industry (think Las Vegas) is enormous. In a sense one’s own life is a game with aspects that are both serious (e.g., personal tragedies and losses) and fun (e.g., friendships). Subconsciously we are analytical about decisions in our lives. For example, what traffic route should I drive to get to my destination most quickly?

Organizations are playing a game too by attempting to out-smart and out-maneuver whatever competitor, stakeholder, or obstacle it needs to defeat, satisfy, or overcome respectively. To win games or score the highest points, one needs to be smart, quick, and agile. Like the mosquito, one heeds to detect opportunities and apply superior analytics.

Analytics Professionals Struggle to Hire Just Like Non-Analytics Professionals

What struck me about this discussion is that analytics professionals (with the world of analytics solutions at their disposal) are having the same conversation non-analytics professionals have every day. The dilemma? How to reliably predict top performers when hiring.

It wasn’t lost on me that this traditionally difficult challenge, might provide an outstanding opportunity for the analytics community to teach businesses how to hire effectively – using an analytics approach.

Questions to answer:

• How to quantify the evaluation of analytics candidates?
• How to move beyond using proxy measures during the hiring process?
• How to create rigorous benchmarks that reliably predict top performers?

Evaluating by Proxy

Measures used today to spot the elusive ideal analytics professional span a range of “proxy metrics.” Some scan resumes for computer science, math or machine learning on resumes, others give puzzles to applicants. Some try to intuit whether a candidate is intensely curious. Others look for a storyteller – someone who can tell a good story using real data.

Bill Franks has had good success hiring outstanding analytics professionals. Among other traits his experience shows that outstanding analytics professionals are creative, suggesting that you “ask if they are artistic, or musical or have some kind of other creative experience in their background”.

It is possible, simple even, to come up with a benchmark of analytics professionals. Analytics professionals solve these kinds of problems all the time. It is this community’s strength and could be a time to shine as an industry, in a whole new category with the potential to affect bottom line business results and dramatically change the hiring industry.

Challenge to the Analytics Community

We suggest a challenge for the Analytics community can to solve together. What if we could use our own analytics methods to quantify the human characteristics that lead to success in our own field? This “benchmark” could lead to better hiring; reduced attrition and more focused professional development. Armed with a methodology, results, and the charts and graphs to prove it, we could lead the charge to introduce and implement similar analytical processes into the hiring processes of organizations at large.

Let’s start with the example of fingerprint matching. If you watch shows like CSI, you see them match up fingerprints all the time. A fingerprint image is totally unstructured and also can be fairly large in size if the image is of high quality. So,
when police on TV or in real life go to match fingerprints, do they match up actual images to find a match? No. What they do is first identify a set of important points on each print. Then, a map or polygon is created from those points. It is the map or polygon created from the prints that is actually matched. More important is the fact that the map or polygon is fully structured and small in size, even though the original prints were not. While unstructured prints are an input to the process, the actual analysis to match them up doesn’t use the unstructured images, but rather structured information extracted from them.

An example everyone will appreciate is the analysis of text. Let’s consider the now popular approach of social media sentiment analysis. Are tweets, Facebook postings, and other social comments directly analyzed to determine their sentiment? Not really. The text is parsed into words or phrases. Then, those words and phrases are flagged as good or bad. In a simple example, perhaps a “good” word gets a “1”, a “bad” word gets a “-1”, and a “neutral” word gets a “0”. The sentiment of the posting is determined by the sum of the individual word or phrase scores. Therefore, the sentiment score itself is created from fully structured numeric data that was derived from the initially unstructured source text. Any further analysis on trends or patterns in sentiment is based fully on the structured, numeric summaries of the text, not the text itself.

This same logic applies across the board. If you’re going to build a propensity model to predict customer behavior, you’re going to have to transform your unstructured data into structured, numeric extracts. That’s what the vast majority of analytic algorithms require. An argument can be made that extracting structured information from an unstructured source is a form of analysis itself. However, my point is simply that the final analysis, which is what started the process of acquiring the unstructured data to begin with, does not use the unstructured data. It uses the structured information that has been extracted from it. This is an important nuance.

One reason it is important is that it gets to the heart of how to handle unstructured big data sources in the long run. Clearly, some new tools can be useful to aid in the initial processing of unstructured data. However, once the information extraction step is complete, you’re left with a set of data that is fully structured and, typically, much smaller than what you had when you started. This makes the information much easier to incorporate into analytic processes and standard tools than most people think. Through an appropriate information extraction process, a big data source can shrink to a much more manageable size and format. At that point, you can proceed with your analytics as usual. For this reason, the thought of using unstructured data really shouldn’t intimidate people as much as it often does.

Originally published by the International Institute for Analytics

Statistics is not a branch of mathematics. It is its own field of study, but is not often seen as such.  I repeat: statistics is not a branch of mathematics!

1951 is the earliest date associated with this quote from the February 2002 issue of Notices of the AMS, where John Tukey stated “Statistics is a science, not a branch of mathematics, but uses mathematical models as an essential tool.”  Several decades later David S. Moore, retired Professor of Statistics, Emeritus at Purdue University and author of several textbooks:  “The trouble with statistics is that it is not mathematics.”  Around the same time, noteworthy statistician Professor David Hand, Senior Research Investigator, Imperial College of London, wrote an article: Breaking Misconceptions—statistics and its Relationship to Mathematics, which further articulated why statistics is not a branch of mathematics.  While statistics makes use of mathematics, it is its own distinct field of study.

January’s featured blog of the American Statistical Association, The Big Mistake: Teaching stat as though it were math links to an article showing that the confusion continues.  I am all for increasing our numeracy, but being more quantitatively astute requires more than mathematics.  In addition, I am all for mathematicians and mathematics teachers recognizing the relevance and importance of statistics—and concur with the main points of “Arthur Benjamin’s formula for changing math education” TED talk:

- One important subject every high school graduate should know is statistics;

- Teaching statistics is more useful to most on a daily basis than calculus;

- If taught properly statistics can be a lot of fun!

Simon King, Upper School Mathematics Department Chair at Cary Academy, teaches advanced analytics and statistics.  He gave an amazing talk at JMP’s Discovery Summit last year and it’s heartening to see how much value the students and parents are getting from the exposure to statistical thinking.  Developing critical thinking skills for problem solving is really what statistics and analytics are all about.

In Jeremy Shapiro’s post last May, he included Conrad Wolfram’s TED talk Teaching kids real math.   His plea to make math fun and interactive so that teaching it would be more effective is relevant for teaching statistical concepts as well.  More than 50 percent of our brains are dedicated to supporting seeing.  Visually and interactively exploring the shape and structure of the data, seeing how variables are related, what patterns appear, etc. can only help in the pursuit of understanding more abstract concepts, not to mention the productivity gains of making faster sense of text and numbers visually.

The importance of distinguishing statistics from mathematics is to appreciate that we live in a world where both statistical and mathematical thinking are needed. Mathematics is largely considered a deterministic way of thinking whereas statistics is characterized as probabilistic/stochastic.  The subtitle of David Salsburg’s The Lady Tasting Tea:  How Statistics Revolutionized Science in the Twentieth Century underscores this important distinction and the shift in view from philosophical determinism to one of embracing statistical approximation.  Change and uncertainty are ubiquitous.  Making better decisions in the face of uncertainty is largely what statistics—and to a great extent analytics—is all about.

Statistics has been called the science of science, the language of science, the logic of measurement, the science of information gathering, the science of learning from data, and many other things—all of these things are core to problem-solving / analytics.  Let us not call statistics a branch of mathematics—doing so limits the perceived value and marginalizes the many contributions statistics and statisticians have made and continue to make in solving real-world problems.  Change brings opportunity.  If statistics can be more fully recognized for the unique and important discipline that it is, we will all benefit.

This new privacy policy is one of many that will deeply touch privacy issues – see the Forbes article on Target. We can expect this kind of outcry to continue if not addressed proactively by the industry. And this is not only the question of storing day by day, probably forever, our personal data. The real challenge is to clarify who should be an owner of this data. Starting on March 1, 2012, the public really no longer gets to use Google services for free, they are paying by giving access to their personal data. Personal data will become the real currency of the digital economy.

That said, as one involved in this conversation from the perspective of potential for analytics, we look at ways to leverage this new data. It will be important for advertisers to adjust their marketing message to match the targeted user’s profile, as the collection of information about clients will enable them to profile clients very precisely. If Google users use the search engine, YouTube, blogs, Google Docs etc, it will be possible to determine their interests, views, opinions, career profile, and so on. Their social groups and the nature of those personal contacts, business contacts, interests etc, can be determined when they make use of discussion groups, e-mail, or calendars. Consequently, it is highly probable that marketing analysts will be able to determine age, sex, education, occupation, place of residence, and income. User activities can be monitored and their data will be integrated in the data warehouse. The history of behavior is retained and a user’s profile can be discovered. It’s worth remembering that the cross- and up-sell proposals for a given target group can make use of data-mining methods.

With these new possibilities arising from their policy change, at this time it appears that Google will be a perfect example of a company that’s competing on Big Data and analytics in the near future.