How To Gain A Competitive Advantage With Big Data

The chief objective for both small and large companies is growth.  Business growth creates more jobs, more revenue, and keeps the corporation healthy and self-sustaining.  The process to achieving growth year over year can be somewhat of a conundrum though.  Some industries have complex data streams that pour in among unrelated departments but the marriage of each piece of data can be crucial in the creation of a feasible growth trajectory.  Getting a handle on all your data can help you to interpret sales patterns, customer behavior, product lifecycle, and more.  Once you have a better understanding of your data you can apply the appropriate b2b pricing strategy to create steady growth.

Discovering Pearls

Service and parts organizations may have transactions with thousands of vendors and customers in a given month.  These transactions produce data that can be buried within the organization’s infrastructure, and as the company moves forward each day it may be difficult to stop and organize the data in a meaningful way.  There are big data solutions that are available to help companies unlock these pearls so proper analysis and planning for future growth can occur.  Finding an application that marries data housed within departments like accounting and operations can improve sales and create more revenue.

What Data Is Meaningful?

For service companies there is usually lots of data available for analysis.  It can be housed within the same department or completely separate from one department to the next.  The data that is most meaningful displays how sales prices were arrived at and how the sales and marketing teams achieved those sales.

• Operations Management data. Data from operations management teams may provide insight regarding products—how much it costs to purchase raw materials, the volume at which raw materials are consumed, the shelf life of all raw materials and finished goods.
• Sales and Marketing data. The sales and marketing team may track sales patterns and the buying behavior of its customers.  Access to that data can provide the company with valuable information to create a stronger sales strategy and product price optimization.  Sales and marketing personnel can also provide insight about the customers they sell products and services to.
• Accounting and Finance data. The accounting and finance departments use data to invoice customers and collect revenue.  In order to do so, however, they also need the date a sale was closed, the price of the goods or services being sold, the quantity sold, and any rebate or discounts being offered.

Making Meaning from the Data

Data mining provides corporations with a multitude of insights that may be otherwise buried, inaccessible, and meaningless.  Marrying information that can be shared within multiple departments allows for the corporation to increase its ability to reach more customers and potential for more sales.  As sales and revenue increases the business can grow, create more jobs, and stays self-sustaining and solvent.  Large corporations in particular, ones that have multiple departments working on related tasks and without the ability to share data in meaningful ways, can gain a competitive advantage with big data interpretation.

For more information, you can visit pros.com

Recently, I re-launched the debate on the Swiss Association for Analytics. However, I think it is worth a dedicated blog post. In order to answer this big question, we need to analyze the different phases of data mining and estimate which one can be automated. For this purpose, I have chosen the CRISP-DM methodology (I guess any other data mining process would lead to similar conclusions).

Business understanding

In this critical step, we transform a business problem into a data mining one. We need to understand what should be solved and why. Answers will lead to the following steps. It is clear that this step cannot be automated for a new project. The data miner has to interact with experts to define the data mining problem to solve.

Data understanding

This step consist in understanding the data, the way they have been collected, their particularities, etc. Again, the data miner works in collaboration with field experts to derive knowledge useful for preparing the data (next step). This is a manual task that cannot be automated.

Data preparation

In this step, we transform raw data into meaningful information to mine. An example is outlier detection (and removal). Some companies argue that their tools can automate this step. This is true to a certain extent, but there are limitations. Here is a simple example: what is the threshold for the variable “age” to be an outlier? 100, 110, 150 years old? This is problem dependent. The same issue happens for missing values. Detecting them is often straightforward, but deciding on the action to take needs manual intervention.

Another important aspect of data preparation is feature selection and extraction. While selection can be automated, extraction (through aggregation) needs understanding of the data. Finally, any data mining tool can automate the target variable detection. However, the final choice is left to the data miner, who knows the business problem to solve.

Modeling

This step is where we apply modeling algorithms to processed data. Among others, it involves selecting a data mining algorithm and tuning its parameters. This is certainly the task that can be the most easily automated. Some vendors claim that their tools can automate the model building process. The concept of testing several algorithms with different sets of parameters (tuning) can be automated to a certain extent. However, it supposes that there are enough data, that the choice of the algorithm is not business dependent (which is usually not the case) and that the evaluation criterion is known (see below).

Cross Industry Standard Process for Data Mining (CRISP-DM)

Evaluation

In order to validate our data mining results, we need evaluation criteria. Although applying a criterion can be automated and different modeling algorithm can be compared, the choice of the criterion may be business dependent. In the case of forecasting, for example, different evaluation criteria exist such as Root Mean Square Error (RMSE), Mean Absolute Error (MAE) and Mean Absolute Scaled Error (MASE). If we compare different forecasting algorithms on the same time series, we can use RMSE. If the goal is to compare different time series, MASE is more appropriate. This is business dependent and thus difficult to automate.

Deployment

In this phase, the goal is to transform our proof of concept or prototype into an industrialized solution. This step involves transforming our “one shot” project into a solution that can work with as few manual interventions as possible. Although standards such as Predictive Model Markup Language (PMML) are appearing, this step stills requires manual intervention. Questions such as where and how to integrate our data mining process within an overall solution/tool need to be explored.

As a conclusion, we have seen that most data mining steps from the CRISP-DM methodology cannot be automated and need manual intervention. Data preparation and modeling, to a certain extent, could be automated. However, as data mining professionals know, most of the effort in a data mining project concerns business and data understanding. Here is an excellent metaphor from Berry and Linoff (re-explained by David S. Coppock):

“The camera can relieve the photographer from having to set the shutter speed, aperture and other settings every time a picture is taken. This makes the process easier for expert photographers and makes better photography accessible to people who are not experts. But this is still automating only a small part of the process of producing a photograph. Choosing the subject, perspective and lighting, getting to the right place at the right time, printing and mounting, and many other aspects are all important in producing a good photograph.”

What about you? Do you think we can automate data mining?

For your information, here are other posts related to this topic:

Automation Will Change Data Science Beyond Recognition
Data Scientists Will Not Be Replaced by Automation
Data Scientist Scarcity: Automation Is the Answer
Data Mining Automation

Thinking Statistically

Here was the program of the first event:

• Can analysis of customer’s past purchase be used to predict the future ? - Vincent Schickel, Founder & CEO, Prediggo (20 min)
• Analytics @ Nestlé: The Case of Statistical Forecasting - Marcel Baumgartner, Demand & Supply Planning Expert, Nestlé (45 min)
• Break (10 min)
• Sponsor presentation – SAS (5 min)
• Statistical methods for decision making - Jacques Zuber, HEIG-VD and Statoo Consulting (45 min)
• Networking apero

Pictures of the event are available here. To join the association, go to www.swiss-analytics.ch. If you are interested in giving a talk or sponsor a future event, please contact us at info@swiss-analytics.ch

The book starts with an introduction to data mining and R. Rattle is a graphical interface for data mining using R. Although Rattle is introduced and screenshots appear in the text, it’s not needed to use the tool to benefits from the book. To be noted that all pictures are in colors. In addition to standard data mining algorithms (decision tree, association rules, SVM, etc.), Graham discusses topics such as data preparation, model evaluation and deployment (PMML).

One really good thing about Graham’s book is that each choice is explained. From data separation to algorithm tuning, everything is justified. It’s appreciated to read comprehensive theory before digging into examples and codes. The book is an excellent step by step tutorial with all codes needed for your projects. Each chapter is concluded with a summary of the R commands used.

If you are using R or plan to use R for data mining, you should definitely have this book with you. You may already know or have the book by Luis Torgo, Data Mining with R. These books are complementary. Graham’s book is a strong starting point for learning data mining using R. The book of Luis is an excellent continuation, full of case studies and explanations on how to avoid data mining pitfalls.

Data Mining with Rattle and R: The Art of Excavating Data for Knowledge Discovery (Use R!)

Know the value of your data

Data is all around us, and we are getting overwhelmed with it. Thanks to the internet, we have created in the past 15 years as much information as we did in over 100 years. The question is what to do with all this data? What value does it have?

Usually raw data at its source has little interest and contains too much noise to be useful. A couple of months ago, it was announced at CERN that there is over 99.9% probability that we have finally proven the existence of the Higgs particle. This discovery is set to be as big as the discovery of the Newton theory, but it did not come cheap. The construction of the new particle colider cost billions of Swiss francs, and produces Terra Byte of data each day. Data mining was crucial in analyzing this huge amount of data and isolate useful information for scientists to work with.

Data can also be worth more money than most people think. Take for example Amazon, the pioneer and leader in the eCommerce Industry. Amazon makes billions of dollars, but did you know that a few years ago, over 30% of its revenue came from its famous recommender system “Customers Also Bought”, or CAB. CAB is an unsupervised learning algorithm known to scientists as Item Collaborative Filtering. CAB works by analyzing the content of user’s shopping basket and compute the similarity between items. Then when a user looks at a specific product, CAB will predict what a user may like by proposing similar items. CAB can be seen as a personal shopping assistant.

Data has huge value but unfortunately it rarely comes as fruits hanging from a tree. This is where Data mining comes in and helps us transform this data into gold.

To know more about Vincent and his company, visit Prediggo

Taleb warns reader against people or situations that may seem impressive, but are due to luck (randomness). This concept is well summarized by the following quote: “Mild success can be explainable by skills and labor. Wild success is attributable to variance.” To demonstrate his ideas, Taleb uses the concept of “black swans” (rare events) and shows that we can’t predict them. One problem is that detecting risks and avoiding them is mediated by the emotional part of the brain, not the rational part (the one which could understand probabilities).

In his book, Taleb has a lot to say about journalists and traders (he is an option trader) for example, and how they are fooled by randomness (but he admits to be himself fooled by randomness: “I am just like every single character whom I ridiculed in this book.“). By the way, here is a good quote for traders: “There is no point searching for patterns that are available to everyone with a brokerage account; once detected, they would be self-canceling“. He also explains that we are not independence-friendly: “When viewing two events A and B, it is hard not to assume that A causes B, B causes A, or both cause each other“.

To conclude, Fooled by Randomness is an excellent reading full of examples where randomness is a trap. Any decision maker and anybody involved with probabilities should read this book.

Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets

We are proud to announce the 1st Swiss Association for Analytics Event!

When: Thursday, March 14th at 6pm
What: Analytics event and networking organized by the Swiss Association for Analytics
Where: Hotel Continental Lausanne, Place de la Gare 2, 1003 Lausanne (in front of the train station). Parking Simplon at 3 minutes.

Program of the event

• Can analysis of customer’s past purchase be used to predict the future ? – Vincent Schickel, Founder & CEO, Prediggo (20 min)
• Analytics @ Nestlé: The Case of Statistical Forecasting – Marcel Baumgartner, Demand & Supply Planning Expert, Nestlé (45 min)
• Break (10 min)
• Sponsor presentation by SAS (5 min)
• Competing on Analytics in Sports or What Is All the Fuss About Moneyball – Marcel Bieler, Senior Industry Consultant, SAS EMEA Region Risk (45 min)
• Networking apero

For more information, and to subscribe to this free event: event.swiss-analytics.ch.

We are looking forward to meeting you there!

This event is sponsored by SAS

Data Mining Research: Could you introduce yourself? What is your journey in the field of data mining?

Wil van der Aalst: Before working on process mining, I worked on Petri nets, the modeling and analysis of workflow processes, workflow patterns, and process-aware information systems. I also started the main scientific conference on Business Process Management and I’m the founder and leader of the IEEE Task Force on Process Mining. I got into the topic of process mining because I got bored with purely model-based research. For example, it is unsatisfactory to work on the verification and performance analysis of process models knowing that these models have nothing to do with reality.  Process mining provides an important bridge between data mining and business process modeling and analysis. Process mining research at TU/e (Eindhoven University of Technology) started in 1999. At that time there was little event data available and the initial process mining techniques were extremely naïve and hence unusable. Over the last decade event data has become readily available and process mining techniques have matured. Moreover, process mining algorithms have been implemented in various academic and commercial systems. Today, there is an active group of researchers working on process mining and it has become one of the “hot topics” in BPM research. Moreover, there is a huge interest from industry in process mining and more and more software vendors started adding process mining functionality to their tools.

DMR: What is process mining?

WvdA: The idea of process mining is to discover, monitor and improve real processes (i.e., not assumed processes) by extracting knowledge from event logs readily available in today’s (information) systems. Process mining includes (automated) process discovery (i.e., extracting process models from an event log), conformance checking (i.e., monitoring deviations by comparing model and log), social network/organizational mining, automated construction of simulation models, model extension, model repair, case prediction, and history-based recommendations (cf. www.processmining.org).

The most appealing form of process mining is process discovery. A discovery technique takes an event log and produces a model without using any a-priori information. For many organizations it is surprising to see that existing techniques are indeed able to discover real processes merely based on example behaviors recorded in event logs. Another form of process mining is conformance checking. Here, an existing process model is compared with an event log of the same process. Conformance checking can be used to check if reality, as recorded in the log, conforms to the model and vice versa. A third type of process mining is model enhancement. Here, the idea is to extend or improve an existing process model using information about the actual process recorded in some event log. Whereas conformance checking measures the alignment between model and reality, this third type of process mining aims at changing or extending the a-priori model. For instance, by using timestamps in the event log one can extend the model to show bottlenecks, service levels, throughput times, and frequencies.

DRM: What are the benefits for companies to apply process mining?

WvdA: Although event data are omnipresent, organizations lack a good understanding of their actual processes. Management decisions tend to be based on PowerPoint diagrams, local politics, or management dashboards rather than a careful analysis of event data. The knowledge hidden in event logs cannot be turned into actionable information. Advances in data mining made it possible to find valuable patterns in large datasets and to support complex decisions based on such data. However, classical data mining problems such as classification, clustering, regression, association rule learning, and sequence/episode mining are not process-centric. Therefore, Business Process Management (BPM) approaches tend to resort to hand-made models. Process mining research aims to bridge the gap between data mining and BPM. Metaphorically, process mining can be seen as taking X-rays to diagnose/predict problems and recommend treatment.

DMR: What are the future works in this field from a research point of view?

WvdA: The growing maturity of process mining is illustrated by the Process Mining Manifesto recently released by the IEEE Task Force on Process Mining. This manifesto is supported by 53 organizations and 77 process mining experts contributed to it. The active contributions from end-users, tool vendors, consultants, analysts, and researchers illustrate the significance of process mining as a bridge between data mining and business process modeling. The manifesto lists six guiding principles and eleven challenges. As an example, consider Challenge C4 in the manifesto: “Dealing with Concept Drift.” The term concept drift refers to a situation in which the process is changing while we’re analyzing it. For instance, in the beginning of the event log, two activities might be concurrent, whereas later in the log, they become sequential. Processes might change because of periodic or seasonal changes (for example, “in December, there is more demand” or “on Friday afternoon, fewer employees are available”) or changing conditions (“the market is getting more competitive”). Such changes impact processes, and detecting and analyzing them is vital. However, most process-mining techniques analyze processes as if they’re in steady state. This is just one of many open problems in the process mining field.

Given these challenges and the interest of industry, I hope that more and more data miners will start working on this exiting topic. In short: “It’s the Process Stupid!”, so start mining processes rather than data.

About Wil van der Aalst

Prof.dr.ir. Wil van der Aalst is a full professor of Information Systems at the Technische Universiteit Eindhoven (TU/e). Currently he is also an adjunct professor at Queensland University of Technology (QUT) working within the BPM group there. His research interests include workflow management, process mining, Petri nets, business process management, process modeling, and process analysis. More information: www.vdaalst.com

• Time Series and Forecasting (chapter 22): different forecasting models are described in details in this book chapter. Evaluation criteria are also presented. Although several equations are given, business problems such as planning are mentioned.
• Support Vector Regression for link load prediction: SVR are detailed as well as specific studies on sensitivity to input data, SVR parameters and training size. Other topics such as complexity and forecast horizon are analyzed.
• Forecasting with computational intelligence – an evaluation of Support Vector Regression and Artificial Neural Networks for time series predictions: a technical paper detailing what are times series and how SVR and NN (Neural Networks) can be used for forecasting. Use of training, validation and test set is advised.
• Error measures for generalizing about forecasting methods – empirical comparisons: analysis of the different evaluation criteria. Clearly separate two common goals that are to i) select the best forecasting model and to ii) calibrate a given forecasting model.
• Support Vector Machines and learning about time: good introductory article with several time series principles explained. Different ways to approach forecasting are presented.
• Forecasting – principles and practice: this online textbook is my favorite resource to learn forecasting. Focus is made on concepts, key ideas and business challenges regarding time series prediction. If you should read only one resource about principles of forecasting, it should be this one.
• Another look at forecast – accuracy metrics for intermittent demand: concise article full of advices to choose the right evaluation criteria for forecasting. If you want to read only one reference about evaluation criteria, I strongly advise this one.

Feel free to comment and add you own references!