Outlier Detection: A Survey

The first one, Outlier Detection: A Survey, is written by Chandola, Banerjee and Kumar. They define outlier detection as the problem of “[...] finding patterns in data that do not conform to expected normal behavior“. After an introduction to what outliers are, authors present current challenges in this field. In my experience, non-availability of labeled data is a major one.

The authors proposes three types of supervisions. In supervised outlier detection we make the assumption that labeled data are available. Semi-supervised outlier detection assumes that only one class of labeled data is available. Techniques which models normal instances as the only class are more popular (since normal instances are easier to obtain). The third approach, unsupervised outlier detection, is the most widely used one. The paper continues by describing three types of outliers. Authors then describes several applications of outliers detection in areas such as intrusion detection, fraud detection, industrial damage detection, image processing, etc.

Techniques used for outlier detection are then described. It is surprising to read that most data mining techniques can be applied to the task of outlier detection. For example: neural networks, SVM, rule-based, clustering, nearest neighbors, regression, etc. The articles continues with several other techniques. Authors also describe ways to evaluate results of outlier detection with false positive, false negative and ROC curve. To be noted the 19 pages (!) of references to other articles in the field. One of their main conclusions is that “[...] outlier detection is not a well-formulated problem“. It is your job, as a data miner, to formulate it correctly.

Link to Outlier Detection: A Survey

Meta Brown, General Manager of Analytics at LinguaSys, has kindly accepted to answer a few questions for Data Mining Research. I would like to thank Meta for her time and her advices in Analytics.

Data Mining Research: Who are you and what is your story?

Meta Brown: I’m a practical, plain-talking data analyst and engineer.  I use data to tell true stories, and teach others to do the same. So many of my clients do worthwhile things I could not do myself – improve medical care, provide valuable social services, create and sell useful products – that it is my pleasure to help each of them find ways to work a little better (or a lot) through analytics.

DMR: What services do you offer as a consultant in analytics?

MB: Guidance for organizations initiating or expanding analytics programs. Analytics only improves your bottom line if you begin with a realistic plan for making that happen. My role is to help you create and act on that plan, beginning with identification of goals and working backwards to define the steps you must take to meet those goals. This is an empowerment process that leaves your organization stronger, and more independent.

My availability for consulting is limited now, as my primary role is General Manager of Analytics at LinguaSys, a language technology firm. My work there is all about the coolest in text analytics.

DMR: What is the most important thing you have learned as a consultant in analytics?

MB: Consulting is valuable to those who are open to changing what they do. Others are wasting their money. Consultants have expertise, the ability to offer valuable information and advice, but only clients have the power to act on that input. Advice that is merely good, but executed brilliantly, yields far more value than brilliant advice without action.

DMR: What advice would you give to a business analytics manager?

MB: Set goals based on the primary concerns of decision makers, and make certain that most of your goals are things you’re sure you can do well and on time. Break up big goals into bite-size goals, and document your success as you go. This ensures survival, and leaves a little room for risk.

You can find more information about Meta Brown on her website: www.metabrown.com

After a short introduction, the book is divided into chapters with the following business analytics topics: importance, delivery, value, communication, execution plan and measurement framework. The final chapter brings everything together. Stubbs clearly shows the importance of business analytics (BA) and how it is useful for competitive advantage. Stubbs not only explains why, but how to find these advantages. The book goes one step further from Analytics at Work, by providing methodologies and procedures to achieve what is advised by the author. In addition to the process of making value from analytics, Stubbs define the analytics team members, one by one. To be noted, the excellent and comprehensive glossary at the end of the book.

The presence of checklists for BA managers are very appreciated. Stubbs emphasizes on the importance of communication when dealing with BA. Fundamental aspects of communications are explained with focus on BA. The funny thing with Stubbs’ book is that I can find myself in several of his examples. The Value of Business Analytics is definitely THE book for managing analytics projects. After reading his book, it’s clear for me that Stubbs has a lot of experience in the field. I will finish with a quote that summarizes the book: “Value is created only when action is taken, not when insight is generated.”

The Value of Business Analytics: Identifying the Path to Profitability

How India became the global hub for analytics?

The economic reforms at the start of the 90s led to liberalization of government policies and encouraged multi-nationals such as GE, American Express and British airways to enter India. Initially these companies sought to move their back-end operations such as transaction processing and customer service. These areas were traditionally considered low-value and resource intensive and these companies realised huge cost savings due to the low cost of labor in India.

As offshoring matured, so did the kind of work that was offshored. GE and American Express were amongst the first companies to identify an abundance of analytic talent in India. They started to move some of the high-end work such as data analysis and predictive modelling to India. The results were very successful and with the availability of PhDs and other highly skilled resources in India at about one-tenth the cost in the US, this strategy was a clear win for the businesses.

Soon, financial services companies such as Citibank and HSBC, online companies like Google and Amazon, and retailers such as Target, Tesco and Supervalu also set up their analytics teams in India.

Indian IT companies had already made huge strides in the global arena and they also realized that offering analytics services to their clients was the ideal way to move up the value chain. IT giants like Infosys, Wipro, Cognizant and TCS also jumped into the fray.

Last 10 years

All this happened in 90s and the first half of the 2000s and by this time, there were several people in India who had spent significant time in this field. Entrepreneurs with strong analytics experience set up companies such as MuSigma, Fractal analytics, Manthan systems and Marketelligent – niche analytics companies serving clients across the globe.

Finally, the latest entrants in the field of analytics in India are Indian businesses themselves. Financial services companies such as ICICI and HDFC led the way for Indian businesses. Now analytics usage has spread to retail, telecom, e-commerce, insurance, healthcare and even sports.

There are no official figures available around the size of the analytics industry in India. As per crude estimates, the analytics industry in India employs anywhere between 80000 to 300000 people, depending on how you define analytics. And it is growing at a rapid pace.

Analytics salaries in India have grown by 400% in the last 10 years and even the recession had little impact on this growth.

Future prospects

The future of analytics industry in India looks very promising. The growth is fuelled not only by global companies but also by the domestic market which is increasingly turning to analytics to stay competitive. The author firmly believes that businesses not only in India but around the world will continue to rely increasingly on analytics and it is truly a career for the future.

Author bio

Gaurav Vohra is an alumnus of IIM Bangalore with over 10 years of experience in the field of analytics. Gaurav has been in the analytics industry from its initial days and his career has spanned companies like Capital One and Information resources Inc., recognized as thought-leaders in the analytics space. Gaurav is now the co-founder of Jigsaw academy, a training institute that aims to meet the growing demand for talent in the field of analytics by providing industry-relevant training to develop business-ready professionals. You can visit Gaurav’s blog at http://blog.jigsawacademy.in

“Imagine you are a regional sales head for a major retailer in U.S. and you want to know what drives sales in your top performing stores. Your research team comes back with a revealing insight – the most significant predictor in their model is the average number of cars present in stores’ parking lots.”

The five following mistakes are detailed and explained by the authors:

1. Sophistication in statistics compensates for lack of data and/or business understanding.
2. Extracting meaning out of randomness
3. Correlation versus causation – modeling will help uncover causal relationships
4. Extrapolating the models way beyond the permissible limits
5. Imputing missing values with mean or median is the best way of treating missing values

This article is an excellent reminder for practitioners and I strongly advise it.

Read the full article: 5 Common Mistakes People Make in the Name of Statistical Analysis

The biggest added value of the book, and what makes it a unique resource in the field, are the tutorials. Several applications are detailed: aviation safety, unsatisfied customers, credit scoring and process control among others. Whatever your field of application is, you will find useful examples in the book. Even if the case studies aren’t in your field of interest, reading them takes you to an amazing journey in the world of analytics.

To be noted the interesting chapters about text mining and fraud detection. In the last part of the book, five chapters provide expert advices to both beginners and experienced data miners. The chapter “Top 10 Data Mining Mistakes” is remarkable. To conclude, this book is about applying data mining, rather than programming it. It gives excellent examples of enterprise applications in analytics.

Handbook of Statistical Analysis and Data Mining Applications

I recently read three interesting articles on this topic. The first one, Selling Information Governance to Business Leaders, by Sunil Soares, gives four tips explaining the value of analytics to business leaders. The first tip, focusing on business outcome, is extremely important. You may have a perfect data mining application, predicting your target with 99% of accuracy. If you can’t transform this 99% accuracy into a usable and industrialized solution for the company, the project will not bring any ROI for the company.

The second paper, Selling a Data Mining Project to Management is written by Casey Klimasauskas. It proposes an approach for selling a new data mining project to management. Part of the strategy deals with developing relationships with the stakeholders from the very beginning of the project. This minimizes the risk of future objections and increase the supporters of the project.

Finally, What the C-suite should know about analytics, is an interesting article written by Kishore S. Swaminathan. Found in the sascom magazine (but originally from Accenture), the article lists five areas to focus on in order to bring analytics to the management. The advices given in this article are very relevant. I particularly liked the 10 characteristics of an analytic leader. For more information, I provided the links to these three articles below.

Selling Information Governance to Business Leaders

Selling a Data Mining Project to Management

What the C-suite should know about analytics

The second paper, Data Science Revealed: A Data-Driven Glimpse into the Burgeoning New Field (EMC Data Science Community), proposes a benchmark of the data science community (understand data mining and business intelligence practitioners). The study tries to differentiate data scientists and business intelligence people. For that, about 500 practitioners have been interviewed. I find their concluding paragraph very relevant:

“Once companies have brought in the right talent, they need to create an environment conducive to effective data science. That means building high-performing, cross-functional teams that include a variety of roles, including programmers, statisticians, and graphic designers, and aligning them to directly support interested business decision makers. They should also loosen restrictions on data in the enterprise, allowing employees to more freely run data-driven experiments. Finally, data scientists should be given free access to run experiments on data, without bureaucratic obstacles, so that they can rapidly translate their own intellectual curiosity into business results.”

Data Science Revealed: A Data-Driven Glimpse into the Burgeoning New Field

Intelligence for Everyone: Transforming Business Analytics Across the Enterprise

Data Mining, Refining, and Storage

As companies are able to more effectively store and retrieve information via online storage services, the relevance of data mining as the crucial first step towards intelligent business decisions will become an important tool that was previously unavailable for small and medium-sized businesses and may usher in a new era of success for those who can afford to make the change.

Online Storage: Addition by Subtraction

The effect on large and small companies of using services that provide online storage should be evaluated by looking at what the process takes away rather than what it necessarily adds.  More and more companies are finding themselves practically bombarded with the exponential growth of data that is available about their customers, provided in no small part by the increasing number of internet-connected sensors that people own—smart phones, tablets, laptops, to name just a few.

Just consider this snippet taken from a white paper produced by Globalknowledge, a leader in IT and business skills training:

“According to International Data Corporation (IDC), ‘The proliferation of devices, compliance, improved systems performance, online commerce and increased replication to secondary or backup sites is contributing to an annual doubling of the amount of information transmitted over the Internet.’”

Indeed, a veritable ocean of information has been created, and its unexplored depths contain the potential for game-changing discoveries.  Unfortunately for medium-sized businesses, or even larger businesses that find themselves underprepared, the challenges of collecting their relevant data efficiently and storing it securely are enough to keep them too busy to participate in this feeding frenzy of information.  That’s where the developing industry of online storage comes in.

Online storage as a service is a young industry, and as with any young industry there are still some very real customer concerns that need to be addressed before it can be widely accepted as a solution.  Specifically, those who wish to provide online storage to customers at a large scale will need to bridge the security gaps that are still associated with it; however, as those issues are resolved with time, purveyors of online storage services that position themselves to accommodate this massive influx of data from small to medium-sized businesses may find that they have customers knocking down their doors.

The introduction of online storage as a service to small and medium-sized businesses that are having difficulty managing their data has a more profound effect on their bottom lines than it might initially seem.  That’s because rather than spending too much time trying to manage data, companies can shift their primary focus toward understanding it, which is a monumental task unto itself.

If the process of local storage and management was a difficult burden, then the process of knowledge discovery required to understand it is a seemingly insurmountable one—how can any person possibly hope to make sense of petabytes of information?  The answer to this question, and the key first step toward vital business intelligence for curious companies, is data mining.  As an automated process, data mining can help companies identify patterns in their data that can be analyzed in detail and used in predictive or product-oriented capacities.

In a dark room, data mining is like emergency lights that come on and provide much needed guidance and direction.  Without the benefit of online storage to relieve the mounting pressure from data overload, though, small and medium-sized businesses are in danger of being too busy to see the light.

Eric Greenwood is a technophile whose interests span the range of data management, online storage, business intelligence and much more – Read more of his work at the blog Online Storage!

If you come from the business side (e.g. C-level), the book may be interesting but the explanations about predictive analytics are quite light and you won’t see all benefits of these techniques in the company. I know that the main focus of the book is not about teaching analytics. It seems also not to be about filling the gap between analytics and action. I’m thus a bit confused about the real objective of the book. It is also explaining concepts at a very high level of abstraction. It is thus not directly usable in practice.

The book is divided in three parts. In the first part, James explains what are DMS and why they are useful for the company. The second part focuses on building these DMS. The third part is about the enablers (people, processes and technology), i.e. the aspects that will allow such DMS to be a successful initiative. Personally, I found the book very interesting starting from chapter 6 (Design and Implement Decision Services). The topic of fraud detection and prevention is very well studied throughout the book.

A very strange choice has been made to repeat in full text the expression Decision Management Systems hundreds of times. It thus makes the reading sometimes a bit tiring. The simple use of the abbreviation DMS would have solved this issue. To conclude, I found the book interesting and well written. However, keep in mind that it is written with a very high level of abstraction. You will thus have a clear understanding of the domain, but no practical advices.

Decision Management Systems: A Practical Guide to Using Business Rules and Predictive Analytics