High Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data

Machine Learning vs. Traditional Statistics: Different philosophies, Different Approaches

2016-10-26T16:52:00.001-07:00

"Machine Learning (ML)" and "Traditional Statistics(TS)" have different philosophies in their approaches. With "Data Science" in the forefront getting lots of attention and interest, I like to dedicate this blog to discuss the differentiation between the two. I often see discussions and arguments between statisticians and data miners/machine learning practitioners on the definition of "data science" and its coverage and the required skill sets. All is needed, is just paying attention to the evolution of these fields.

There is no doubt that when we talk about "Analytics," both data mining/machine learning and traditional statisticians have been a player. However, there is a significant difference in approach, applications, and philosophies of the two camps that is often overlooked.

What is ML?

ML is a branch of Artificial Intelligence (AI). AI focuses on understanding intelligence and how to replicate it in machines (systems or agents). ML aims at automatic discovery of regularities in data through the use of computer algorithms and generalizing those into new but similar data. Its main focus is the study and design of systems that can “learn from data” and its focus is inductive learning (learning by examples). ML is not the same as “data mining” or “predictive analytics” that are practices but a core part of both.

ML Roots started in 1950’s and many startups formed in late 80’s, early 90’s with applications such as real-time fraud detection, character recognition, and recommendation systems to be commercially successful (first generation ML systems). ML is also closely related to “Pattern Recognition (PR). While ML grew out of computer science, Pattern Recognition has engineering roots. The two however are facets of the same field where focus in both is learning from data. Today ML resurgence is the driver of the next big wave of innovation.

ML Application Variety

Data mining and predictive analytics

Fraud detection, ad placement, credit scoring, recommenders, drug design, stock trading, customer relationship & experience, …

Text processing & analysis

Web search, spam filtering, sentiment analysis, …

Graph mining

Other:

Speech recognition, human genome, bioinformatics, optical character recognition (OCR), face recognition, self-driving cars, scene analysis, …

ML Community/Practitioners

• Typically computer science and/or engineering background
•      More programming savvy
•      Not confined with a single tool
•      Open-source friendly
•      Rapid prototyping of the ideas/solutions desired

ML vs. Traditional Statistics

Historically, ML techniques and approach heavily relies on computing power. On the other hand, TS techniques were mostly developed where computing power was not an option. As a result, TS heavily relies on small samples and heavy assumptions about data and its distributions,

ML in general tends to make less pre-assumptions about the problem and is liberal in its approaches and techniques to find a solution, many times using heuristics. The preferred learning method in machine learning and data mining is inductive learning. At its extreme, in inductive learning the data is plentiful or abundant, and often not much prior knowledge exists or is needed about the problem and data distributions for learning to succeed. The other side of the learning spectrum is called analytical learning, (deductive learning), where data is often scarce or it is preferred (or customary) to work with small samples of it. There is also good prior knowledge about the problem and data. In real world, one often operates between these two extremes. On the other hand, traditional statistics is conservative in its approaches and techniques and often makes tight assumptions about the problem, especially data distributions.

The following table shows some of the differences in approach and philosophy between the two fields:

Machine Learning (ML)	Traditional statistics (TS)
Goal: “learning” from data of all sorts	Goal: Analyzing and summarizing data
No rigid pre-assumptions about the problem and data distributions in general	Tight assumptions about the problem and data distributions
More liberal in the techniques and approaches	Conservative in techniques and approaches
Generalization is pursued empirically through training, validation and test datasets	Generalization is pursued using statistical tests on the training dataset
Not shy of using heuristics in approaches in search of a “good solution”	Using tight initial assumptions about data and the problem, typically in search of an optimal solution under those assumptions
Redundancy in features (variables) is okay, and often helpful. Preferable to use algorithms designed to handle large number of features	Often requires independent features. Preferable to use less number of input features
Does not promote data reduction prior to learning. Promotes a culture of abundance: “the more data, the better”	Promotes data reduction as much as possible before modeling (sampling, less inputs, …)
Has faced with solving more complex problems in learning, reasoning, perception, knowledge presentation, …	Mainly focused on traditional data analysis

Learning can be achieved by manually writing a program covering all possible data patterns. This is exhaustive work and is generally impossible to accomplish for real-world problems. In addition, this program will never be as good or as thorough as a learning algorithm. Learning algorithms learn by examples (like humans do) automatically, and they generalize based on what they learn (inductive learning). Generalization is a key aspect of evaluating the performance of a learner. At the highest level, the most popular learning algorithms can be categorized into supervised and unsupervised types and each into high-level useful categories (also called data mining functions):

Supervised learning includes:

· Classification: Predicting to which discrete class an entity belongs (binary classification is used the most)—e.g., whether a customer will be high-risk.

· Regression: Predicting continuous values of an entity’s characteristic—e.g., how much an individual will spend next month on his or her credit card, given all other available information.

· Forecasting: Estimation of macro (aggregated) variables such as total monthly sales of a particular product.

· Attribute Importance: Identifying the variables (attributes) that are the most important in predicting different classification or regression outcomes.

Unsupervised learning includes:

· Clustering: Finding natural groupings in the data.

· Association models: Analyzing “market baskets” (e.g., novel combinations of the products that are often bought together in shopping carts).

Statistical Learning Theory

Historically, statisticians have been skeptics of machine learning and resistant to accepting it. This has been because of the liberal approach of ML and less emphasize on theoretical proofs. The good news is that "Statistical Learning Theory" has bridged the gap and has provided an umbrella theory where both sides can collaborate and operate. Basic statistical concepts is a cornerstone of many engineering and science fields, very much like math is. But sticking to traditional statistics thinking and practices would have prevented progress. These are two different things and ML has proved that in practice. For those interested to understand a bit about Statistical Learning Theory and its relation to ML, see the following lecture by Yaser S. Abu-Mostafa at Cal Tech.

---------------------------------------------------------------------------

I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

10 Required Non-technical Skills for a Data Scientist

2016-10-26T14:36:00.008-07:00

"Data Science(DS)" is nothing new but the term itself and the recent level of interest in it. As a practice it has commercially (not academically) existed for more than 25 years, mainly under "Data Mining (DM)" and "predictive analytics(PA)," since early 1990's. DM and PA got a lot of traction originally in financial, Telco, and retail industries that had a lot of granular historical data. Like anything that gets sudden attention and interest, DS has been misused and abused in a variety of ways. Given the fast surge in market demand in the last several years, many claim to be or want to be data scientists. True data scientists and DS managers who had to deal with screening DS resumes, can testify to the level of present noise (false positives) in that application process.

"Data Science" tries to be an umbrella field that covers more of what data mining and predictive analytics practices have covered. That is justified since with the growth of data of all kinds in recent decade and what is expected in the coming years, we need a lot more of the people with relevant DS skill sets. The challenge however has been the definition of that "skillset." What makes a good data scientist?

In my previous post "What is BuDAI?," I explained that a successful DS project requires the involvement of the data science team through the whole cycle. The core part of a data science project deliverable is the insight and decision coming out of analytics. The analytics could be trivial (generally aggregated view of data and only looking at a handful of variables together) where in that case there would be no need for DS. That would be in the realm of a data or business analyst. DS comes into picture usually where:

More sophisticated analytics approaches are required,
More complex transformations are required to prepare the data,
Granular or atomic analysis of entities of interests is required,
Analytics could be straightforward but big data is involved requiring attention to optimization of analytics,
...

Within BuDAI process, the S team has to interact with business, data engineers, data architects, project managers, and product managers to name a few. Aside from some relevant technical skills/knowledge[1] in math, stats, machine learning, programming, databases, and systems (the breadth and depth will depend on the level of seniority of the Data Scientist), through the years I have found the following ten traits to be as important as technical skills for junior hires and absolutely essential for senior data scientists.

Problem solving ability
Business acumen
Ability to question the work of self and others,
Passion for data (the more data, the better)
Attention to details and ability to validate own work in multiple ways
Statistical thinking (a thinker who knows when to reason deterministically and when not)
Passion for exploration and discovery (quick learner from fails)
Ability to devise optimal ways to experiment new or creativity (finding novel useful insight is cumbersome. One can never find a sure way to find it)
Presentation ability (written and oral)
Ability to simplify complex concepts for explaining to others.

----------------------
[1] This is the subject of another blog and given the today's coverage of data science, the required technical abilities vary greatly.

---------------------------------------------------------------------------
I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

What is DS-BuDAI?

2016-10-19T11:47:00.003-07:00

Data science[1] (covering data mining and related practices) is a multidisciplinary field that requires knowledge of a number of different skills, practices, and technologies, including but not limited to machine learning, pattern recognition, mathematics, programming, algorithms, statistics, and databases. In the context of big data, more skills and knowledge is required, such as knowledge of distributed computing techniques/algorithms and architectures. By nature, data science is a creative process that is a combination of both science, engineering, and art. Hence its success has been more dependent on the quality and the experience of the team that has been carrying it out. Thus in the past, for some time, data mining projects were not repeatable with the same level of success across different enterprises. However, with the maturity of the practice, that has changed.

Since the late 1990s there have been a variety of efforts to create standard methodologies and process models for data mining, such as CRISP-DM (Wirth and Hipp 2000). In this methodology there is an important focus on business, data, and deployment aspects, as well as the modeling, which used to be the main focus. Today, data science practices are more mature and well tested. Even though different methodologies may use different names for each step of the process, in general, I can logically divide any data science exercise into four phases (See Figure below):

Business Problem Understanding/Use,

Data Understanding/Use and Preparation,

Analytics and Assessment,

Implementation (Deployment and Monitoring).

In the context of big data, these logical phases stay the same; however, some low-level details of data preparation, analysis, and implementation may be impacted.

We all love acronyms and I have been using DS-BuDAI to refer to this process to communicate with business sponsors and users. The lowercase 'u' represents "Understanding/Use" to overemphasize their importance during Business and Data focused phases. It bridges the two. Analytics and Implementation are simply realizations of the data science deliverable.

The "Understanding" part needs no explanation specially in the context of business problem and data that are specifically going to be addressed and leveraged in the effort. "Use" however needs a bit of explanation given some recent experiences.

A DS project must start with a full understanding of the business challenge and how it could be solved leveraging data sources available or to be obtained. However, there could be cases that after everything is done and the value proven, the business users are not still willing to use the new insights for actions. This lack of responsiveness has a lot to do with the culture of the organization, how decisions have been historically made in the past, and the marginal improvement the new actions will bring. These however could be overcome with education and training and full support of senior management for change.

In some cases though, actionable insights are perceived by business users as "this is what we already knew" and "it is good that the data analysis confirms that." Basically saying that there is no novel new findings but a confirmation of what is known. There is truth to this perception sometimes but at times it is simply resisting change or accepting changes in practice.

In the context of data, "Use" also is essential. Collection, storage, preparation, and management of big data is still expensive no matter how much the storage costs have dropped in recent years with advent of open source systems and price drops in storage/processing systems. Data could easily be abused or misused. Sometimes too much data is used, and sometimes data is not used at the right level of details or aggregation.

The lowercase "u" in DS-BuDAI is to overemphasize understanding and use during business and data focused phases.

Originally published on10/19/16, 11:47 AM Pacific Standard Time

[1] "Data Science" is nothing new except the term itself and the level of recent interest for it.

The Age of Data Innocence is Over

2016-10-05T09:29:00.000-07:00

When I try to explain Data Science and Analytics to business people or those interested in these fields, I use the following example to describe the four pillars of: Data, Platform/Tools, Algorithms, and know-how.

To me, "data" is like a collection of bones (say of an animal) scattered around, some clean and some hidden in dirt. If these pieces are collected and put together correctly, it will mimic or resemble the real skeleton of that animal. Some of the bone pieces are perfect. Some are broken and have to be glued together, and some will be missing (hopefully not a lot). But if these bone pieces are organized together with care, it would give us a good view of the animal they once represented.

Then the infrastructure (data platforms and tools) are like the muscles that go around these bones, after they have been lined up correctly. The muscles make the skeleton move around and do interesting things. They can make the boring data come to life and express itself in many interesting ways.

The algorithms are then the brain. A very small mass of the whole thing through which, the muscles are controlled to make the skeleton body do more interesting things and in more novel ways. At the end, we have the lifeless and somewhat boring bone pieces moving around in harmony, doing interesting things. That is what Data Science and Analytics try to do.

Data used to be static and often boring if just collected and not used, but innocent. It used to be lying around in many places and sometimes at massive scale. It was up to the art and science of the data scientists and engineers to come to life. If they did their job right, it could come to life in many useful ways. It could do no harm of its own. It could not lie or cheat on its own.

The recent Volkswagen and Wells Fargo scandals have been a turning point, in which it signaled the end of data innocence. In the past, lies could be made by using data in a biased way and selectively. The data itself was innocent. The scandal shows that the data can be easily manipulated at the origin, right were the data is created.

The bone pieces in my example above could now be practically fake and made look real. In reality though, they collectively portray a skeleton of fiction or imagination, no matter how great and noteworthy are the platforms, tools, and people (know-how) who assemble and use it.

The good news is that for many applications to be useful, there is no incentive for those involved to fake the data. However one can envision many example applications in which there is an incentive to manipulate the data at the origin and fool everybody down the chain. That brings us back to the question of ethics and integrity in Data Science and Analytics and adds yet another important step to the long list of Data Validity (one of the 6Vs of Data discussed in my book) .

-----------------------------------------------------------------------------------------------------------------------
I discuss some related topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Analytics Divide and Big Data

2016-09-30T21:00:00.000-07:00

For more information and orders, visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

In a research report on analytics by MIT and IBM (Kiron, et al. 2011), three progressive levels of analytical sophistication for organizations are defined as aspirational, experienced, and transformed (See Table 1). In transformed organizations, analytics is used at all levels, day to day, strategically. It is considered an integral part of everything, including the culture. These organizations have high proficiency in both data management and analytics in terms of usage, skills, and tools. Both data management and analytics are enterprise-driven and are ingrained in the enterprise culture. Transformed organizations have robust data and analytics foundations and management competencies, making it possible to capture, combine, and analyze information from disparate sources, and to disseminate it across the organization so that individuals at all levels can consume it. In these organizations, one finds that processes, practices, and behaviors are aligned with the fundamental belief that business decisions at all levels should be based on data analysis.

On the other hand, aspirational organizations do not use any sophisticated analytics beyond spreadsheets, and do not have an integrated view of their enterprise data. They lack proficiencies on both the data management and analytics fronts. Their culture relies more on decision-making based on guts and intuition rather than data analysis.

The experienced organizations are somewhere in between, with initiatives that move them closer to transformed organizations. This study shows that experienced and transformed organizations continue to expand their analytics and information management capabilities to add more business value and differentiate themselves, while aspirational organizations keep falling behind. This growing gap or divide has major implications for businesses. For the following discussion, my focus is only on experienced and transformed organizations, since these strongly believe in the value of analytics and either practice it in full or have a goal to get there. I consider them standard organizations in the sense that, in today’s competitive world, they are more of the norm than exception.[1] These organizations have also the culture, the appetite, and the desire to deal with their big data challenge, if there ever is one.

Organization Category	Information Management Proficiency	Analytics Proficiency	Data Culture
Aspirational	Low	Low	Line of business driven
Experienced	Medium	Medium	Moving toward enterprise driven
Transformed	High	High	Enterprise driven

Table 1: Three progressive levels of analytical sophistication in enterprises (Kiron, et al. 2011).

One main differentiator between analytics in the traditional sense and big data analytics is that in the latter, the collected big data may or may not be useful for the specific business purpose intended. From the perspective of analysis, this falls into the category of you don’t know what you don’t know. However, if any insights are extracted, they could be enormously invaluable. Due to the maturity of traditional data management and analysis technologies, data that is stored in these environments is already known to be of high value. This data has been prepared to answer known business questions. The high value justifies their storage and management in enterprise data warehouses or data marts. With new big data, there are plenty of opportunities to ask new business questions never asked before, and the economic situation is favorable when investigating these questions.

Table 2 enumerates a few possible scenarios in today’s standard analytics environments (experienced and transformed organizations) when they are faced with big data. These environments already excel in dealing with traditional and proven analytics methods and technologies where storage, management, and analysis of the data follow standard processes and practices. Scenario 1 depicts the status quo in these environments—where, in the absence of any big data, it is business as usual.

Data Scenario	Big Data?	Storage	Analysis	Business Value
1	No	Standard	Standard	Known
2	Yes	Possible	Nonstandard	Somewhat known
3	Yes	Possible	Not possible	Not known
4	Yes	Not possible	—	Not known

Table 2: Big data scenarios in standard analytics environments.

However, in terms of their existing capabilities, they face different scenarios to deal with their big data challenges. The reader should keep in mind that the size of big data has to be interpreted in the context of time and place of each enterprise, given its sector and its place on the analytics evolution curve. In Scenario 2, the enterprise is capable of storing its big data, and can also analyze it using existing nonstandard[2] big data analytics techniques. As a result, the enterprise has some understanding of the hidden value in its big data, and can decide how much of it needs to be stored and for how long. In Scenario 3, the organization can cope with storing its big data, but does not yet have the capability to analyze it in any efficient way for assessing its value. The reason for this could be technological, methodological, skill set related, or budgetary. In Scenario 4, the enterprise at its current state is not capable of storing the big data (hence not able to analyze it either) for similar reasons to Scenario 3. Today, Scenarios 3 and 4[3] are still dominant for classic enterprises. Those operating under Scenario 2 are a small minority but are ahead of the curve compared to their peers. The curiosity of finding the potential value in big data is why big data has become a part of these organizations’ overall data strategies. Going forward, any enterprise data strategy that ignores big data should be considered incomplete.

----------------------------------------------------------------------------

Kiron, David, Rebecca Shockley, Nina Kruschwitz, Glenn Finch, and Micheal Haydock. 2011. Analytics: The Widening Divide. MIT Sloan Management Review; IBM Institute for Business Value.

[1] More than a decade ago, one could say that the reverse phenomenon was true, meaning that aspirational organizations were more of the norm.

[2] Big data analysis techniques are still in their infancy, and I consider them nonstandard in comparison with traditional data analytics tools and techniques (including data warehousing, BI, and data mining) that have matured, especially in the last two decades.

[3] “Without big data, you are blind and deaf in the middle of a freeway.”—Geoffrey Moore.
---------------------------------------------------------------------------

I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

What Constitutes a Big Data Scenario (Part 2): Human vs. Machine Generated Data

2016-09-06T21:25:00.000-07:00

For more information and orders, visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

It is projected that growth in global data generated per year is 40%, versus a 5% growth in global IT spending (McKinsey Global Institute 2011). From a cost perspective, this divergence clearly shows why big data technology solutions must be cost-effective today in order to stay within existing IT budgets and forecasts. However, the real value of any technology is ultimately defined by its business ROI and not its cost. This is particularly true for classic organizations that are exploring these new big data technologies. From an economic and business standpoint, the potential value
creation from big data opportunities has already been proven by numerous web companies. In many cases this value creation has been disruptive.

More than 80% of data generated is reported to be unstructured data, which includes:

• Semistructured data: weblogs, machine logs, etc.
• Unstructured text data: blogs, e-mail, comments, etc.
• Binary data: photos, images, audio, video, etc.

It has been reported that the data collected by the US Library of Congress by April 2011 has been around 235 terabytes. Fifteen out of seventeen sectors in United States have more data stored per company than the US Library of Congress (McKinsey Global Institute 2011).

Structured data is stored and resides in predetermined fixed fields. Unstructured data cannot be stored in fixed fields. Freeform text (books, e-mails, articles, blogs, etc.) and untagged video, image,
and audio are examples of unstructured data. Semistructured data also does not conform to fixed fields, but includes tags that identify its data elements. XML, JSON, and HTML-tagged text are examples of semistructured data. Multistructured data refers to a combination of all these data varieties.

What constitutes human or machine generated data is loosely defined. I like to differentiate between the two data using the following definition (See Table below). As humans interact with each other and with other organizations, or organizations with each other, a massive amount of structured data is generated in the form of transactions such as call records, payment transactions, sale orders, etc. These data are collectively generated through business processes and had been captured and analyzed long before the Internet became mainstream. Conventional big data technologies were originally developed to handle such data. Human interactions also create semistructured data, such as weblog data, that are newer; and their detail processing often requires newer big data technologies to be cost-effective. Human-generated data can also be directly created by humans as “digital content,” which could be either unstructured or binary. In a nutshell, human-generated data can be defined as the digitization of human interactions.

On the other end, I define machine-generated data as data capture machine-to-machine (Internet of Things) interactions. Machine-generated data may be the result of observing human behavior
instead of capturing their choices. This data could also be in structured, semistructured, or binary form. Data from RFID tags, computer logs, network logs, security cameras, etc., are typical examples.

Data Generation Origin	Definition	Information Management Proficiency	Examples
Humans	Data representing the digitization of human interactions	Structured	Business process data e.g., payment transactions, sales order, call record, ERP, CRM
		Semistructured	Weblogs
		Unstructured	Content such as Web pages, E-mail, Blog, Wiki, Review, Comment
		Binary	Content such as Video, Audio, Photo
Machines	Data representing machine-to-machine interactions, or simply not human-generated (Internet of Things)	Structured	Some devices
		Semistructured	Computer logs, Device logs, Network logs, Sensor/Meter logs
		Binary	Video, Audio, Photo

I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

What Constitutes a Big Data Scenario?

2016-08-22T19:11:00.000-07:00

For more information and orders, visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Big data is a new marketing term that highlights the everincreasing and exponential growth of data in every aspect of our lives. The term big data originated from within the open-source community, where there was an effort to develop analytics processes that were faster and more scalable than traditional data warehousing, and could extract value from the vast amounts of unstructured and semistructured data produced daily by web users. Consequently, big data origins are tied to web data, though today big data is used in a larger context.

The most common definition of big data (attributed to Gartner) is datasets that grow so large that they become awkward to work with using available data management and analysis tools. Difficulties include capture, storage, search, sharing, analytics, and visualization of such data. The definition is broad and can be extended in two ways. First, one expects that what is considered big data today will vary over time, as the technology will advance. What is considered big data today may be the new normal in the next several years. Second, it indirectly implies that what is big data may also change with the place—i.e., industry sector (vertical), organization, or enterprise—depending on where they are on their analytics evolution curve.

Those who are proficient in the world of analytics understand that the use of analytics in some verticals could be much more advanced compared to the others. For example, the banking industry has been an early adapter of business intelligence and predictive analytics. When it comes to systematic use of these technologies in application areas such as risk, fraud, and marketing/sales, this sector has been way ahead compared to other verticals. Large-scale retail and telecommunications have followed suit, while verticals such as health care, health insurance, and the public sector have been laggards. Hence, the interpretation of what is considered big data can differ across these verticals.

Forrester Research defines big data as the frontier of a firm’s ability to store, process, and access all of the data it needs to operate, make decisions, reduce risks, and serve customers. The main phrases here are “the frontier” and “all the data.” In essence, this definition is similar to the first definition though expressed differently. The trend of working with ever larger and larger datasets continues because, first and foremost, for some time web companies of the scale of Google, Facebook, Amazon, LinkedIn, and Yahoo! have had to capture and manage all the data on their sites (and elsewhere) at the most granular level to provide the services they offer. They had to think out of the box and innovate to cope with these challenges. Second, for more traditional companies in sectors such as financial, telecommunication, retail, etc., there are new potential business benefits in analyzing big data that will allow them to unravel new business trends, detect novel customer behavior, create new products and services, identify more complex fraud patterns, and assess more complicated risk events with better accuracy.

Though big data is always a moving target, current limits are on the order of terabytes and petabytes of data. These are the sizes of a single dataset or combinations of datasets that have to be analyzed for a specific analysis purpose at a specific time. Scientists have regularly encountered this problem in data mining, meteorology, genomics, complex physics simulations, biological/environmental research, Internet searching, and finance. Datasets also grow in size because they are being gathered by machines such as information-sensing mobile devices, aerial sensory technologies, computer software (logs), cameras, microphones, radio-frequency identification (RFID) readers, near-field communication (NFC), and wireless sensor networks.

I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Big Data Mining and Big Data Analytics

2016-08-13T12:27:00.000-07:00

For more information and orders, visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

In three previous posts, I made an attempt to describe Data Mining, Data Science, and Machine Learning:

Data Mining, Data Science, and Machine Learning (1), (2), and (3).

As explained in these posts, "Data Science" is nothing but a new term. The use of analytics, data mining, and machine learning has been the cornerstone of many applications for more than two and a half decades, e.g., in finance, Telecom, and retail. However, the popularity of web products from the likes of Google, Linked-in, Amazon, and Facebook has helped analytics become a household name, and its rise under a new term (Data Science).

Since Data Mining has been around for a long time, people may wonder how it relates to "big data Analytics"?

Big data is a new marketing term that highlights the ever-increasing and exponential growth of data in every aspect of our lives. The term big data originated from within the open-source community, where there was an effort to develop analytics processes that were faster and more scalable than traditional data warehousing, and could extract value from the vast amounts of unstructured and semi-structured data produced daily by web users. Consequently, big data origins are tied to web data, though today big data is used in a larger context.

What is considered big data today may be the new normal in the next several years. Also, big data may also change with the place—i.e., industry sector (vertical), organization, or enterprise—depending on where they are on their analytics evolution curve. Big data is always a moving target. Big data analytics is merely a new term meant to focus our attention on the analysis of big data and address the technological and business challenges associated with it.

The Figure below shows how I compare high-performance data mining (HPDM for short) and big data analytics (BDA for short). Any type of analysis, whether it is basic or advanced is covered under BDA. While using traditional tools and techniques, performing even basic computations on big data such as finding averages, counts, and sums in a reasonable amount of time is a challenge. As one expects, performing advanced analytics such as iterative graph algorithms
(e.g., PageRank), gradient descent (e.g., logistic regression), and expectation maximization (e.g., K-Means) introduces even bigger challenges.

The main focus of data mining, however, is advanced analytics. From its inception, data mining had to cope with the challenges of optimizing learning algorithms—which are typically iterative in nature and must converge—on very large data. These tasks are both I/O and compute intensive. In the context of big data, HPDM faces the same challenges that BDA faces in implementing advanced analytics algorithms.

Figure below is a conceptual graph comparing the operational ranges of BDA and HPDM across the two dimensions of “analytics sophistication” and “Data Volume.” BDA operates across the whole analytics spectrum, from basic to advanced analytics, but on the big data side of data volume spectrum. HPDM operates across the whole data volume spectrum, with its main focus on advanced analytics. The transition of HPDM into big data is disruptive, meaning that it requires a fresh look at the existing technology infrastructure, skills, and processes. The transition of BDA to advanced analytics can also be considered somewhat disruptive in the sense that it requires new techniques and approaches to implement these often iterative algorithms. I refer to the intersection of the two as big data mining, which emphasizes both the “big data” requirement and “data mining” (or advanced analytics) nature of it. Both are subsets of what is referred to as high performance analytics (HPA).

Comparison of big data analytics to high-performance data mining

I discuss these topics in detail in my book. Visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Ten General Principles in Data Mining/Science (Focus on Model Development)

2016-07-30T12:00:00.000-07:00

For more information and orders, visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Through years, working with different clients and applications, I have found a set of data mining general principles that also hold through in the context of big data. These are all listed in my book. Here I enumerate them using the terminology I have used in the book:

1. Use of “all the data” is not equivalent to building the deepest Analytics Dataset (ADS) in terms of the number of rows.

2. Given a choice between more of the data or a fancier algorithm, choosing “more data” at the top of the data preparation funnel is always preferred.

3. For most problems, when using the same learning algorithm, smart variable creation on a proper sample of data (a sampled ADS) outperforms the use of the deepest ADS, with primary and/or rudimentary (not well-thought) attributes.

4. For most problems, smart variable creation combined with a simple algorithm outperforms a mediocre variable creation exercise combined with the fanciest algorithms; whatever the ADS row size.

5. For some classes of algorithms, a smart ADS combined with a smart presentation of its variables to the learning algorithm on sampled data outperforms a smart ADS with the largest number of rows without proper presentation of variables.

6. In many problems, one has to deal with transactional data requiring creation of time-based variables that provide the learner with a short and long term memory of the past behavior of the entities to model. Depending on the problem, such variables need to be computed and updated from the transaction history of each entity and for every transaction in realtime or at specific time intervals.

7. For a fixed model complexity, as the number of rows (observations) in ADS increases, the training and test errors converge.

8. For some problems, all population must be represented in an ADS (e.g., social net analysis, long tail problems, high cardinality recommenders, search). For all other problems, sampling
continues to be valid. For a subset of these problems, sampling is mandatory, e.g., highly unbalanced datasets, segmented modeling, micro-modeling, and campaign groups. For the remainder, it is optional but not a limiting factor anymore. Historically for these problems, sampling had to be done to speed up the processing or to reduce the storage cost.

9. For big data, it is desired to use the same platform and interface for data understanding, preparation, and model development with minimal data movement and least iterations through the data to get to the result.

10. In the transition from model development to deployment, automatic code generation for computation of variables and models is of high importance to ensure quality control. Automatic code generation is mandatory in applications that require a large number of models.

The applications have been very diverse:

- Character recognition (hand-print English or cursive print languages) involving state-of-the-art image processing and text/character segmentation capabilities and use of neural networks with multiple layers, hundreds of nodes, and tens of thousands of weights (As an example see Machine-printed Arabic OCR),

- Real-time payment card fraud detection (See here),

- Segmentation of buyers and sellers at a large web auction site using both supervised and unsupervised techniques,
- Building models on-the-fly based on each SKU (millions of them) based on store characteristics for one the largest retailers,
- And tens of other applications in customer experience, risk, marketing, and fraud. Whatever machine learning or pattern recognition technique was used, the above always held true.

For more information and orders, visit the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

End of the Free Lunch for Analytics and Data Mining Software

2016-07-16T09:47:00.000-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info )

There has been an interesting phenomenon that is known to many as “What Andy giveth, Bill taketh away.” It means that every time Andy Grove (CEO of Intel) brought a new chip to market, Bill Gates
would soak up the new chip’s power to upgrade his software. For thirty years, Moore’s Law has continually enabled software performance to improve, often incrementally and, once a while, drastically. Many types of applications have enjoyed regular and free performance gains during
this time, without the need for rewriting old code or doing anything special. Many other applications have been created or have evolved by providing new powerful features made possible only because of faster hardware. By the mid-2000s, this so-called “free lunch for software” came to an end.

During this long period, software engineers simply did not need to account for parallelism and concurrency in their programs or algorithms for speedup. They just relied on the 50% expected yearly speedups of the processors that was made possible by Moor’s Law. However, by the early 2000s, a new reality started to emerge—one that created a crisis. In the new software world, not all software will enjoy faster year-over-year performance gains, as has been the case in the past. Only concurrent and parallel software greatly benefits from hardware advancements going forward, while single-threaded software will stay behind. As mentioned earlier, writing parallel and concurrent software
requires the use of different programming models such as multithreading and distributed computing, which are more complex.

This is not the first time that software has faced a big challenge. The first software challenge happened in the ‘60s and ‘70s when developers were writing assembly language programs. As the computers grew in power, memory capacity, and variety, there was a need for portability and some level of abstraction from the hardware. This was solved through high-level procedural languages such as C and FORTRAN, which were very efficient.

In the ‘80s and ‘90s, another challenge emerged: composing and maintaining large software. These were projects that involved tens or hundreds of programmers and where millions of lines of code had to be managed. Commercial and government entities such as Microsoft— while developing MS Word—and the US Department of Defense (DoD) faced such challenges early on. Software performance was not the main issue, since Moore’s Law had proven that it could deal with confidently. Object-oriented programming using C++, and then Java and C#, made large software development easier. Hundreds if not thousands of people could work on a single application simultaneously.

The third challenge is what we face now. The end of scaling of singlethreaded performance has created a major shift to symmetric parallelism. Until recently, programmers have been used to not being concerned about hardware. They believed that, as usual, Moore’s Law would continue to take care of them. Achieving this requires software developers to write programs differently and more
intelligently to exploit parallelism, distributed computing, distributed data, and modern hardware architectures.

Though the end of free lunch for software applies to all software, it is more profound in the world of data mining, machine learning, and analytics. See the book "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ) for more information.

Somewhere, Something Incredible is Waiting to be Known [1]

2014-11-01T21:27:00.000-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info )

It is projected that growth in global data generated per year is 40%, versus a 5% growth in global IT spending (McKinsey Global Institute 2011). From a cost perspective, this divergence clearly shows why big data technology solutions must be cost-effective today in order to stay within existing IT budgets and forecasts.

However, the real value of any technology is ultimately defined by its business ROI and not its cost. This is particularly true for classic organizations that are exploring new big data technologies. From an economic and business standpoint, the potential value creation from big data opportunities has already been proven by numerous web companies. In many cases this value creation has been disruptive.

Borrowing from Carl Sagan quote above, the curiosity to find something incredible that in this case is also of potentially high monetary value, is the driving force behind the acceptance of big data analytics in big classic organizations. As the cost of these technologies reduces and they become easier to employ, the required investment bet to get the potential ROI will be justified more easily. These organizations are not quite there yet but surely they are on the path to get there.

------------------------------------
[1] Quoted from Carl Sagan.

"The Pale Blue Dot" Effect and Big Data

2014-10-24T21:34:00.002-07:00

Many of you may have heard of the “Pale Blue Dot” which is a photograph of planet Earth taken in 1990 by the Voyager One spacecraft when it was leaving the solar system.[1] The picture was taken from a distance of about 3.7 billion miles from the earth. In the photograph, the earth with all its magnificence (i.e., life) only appears as a fraction of a pixel against the vastness of space, hence the name “Pale Blue Dot.”

This “Pale Blue Dot” effect may well represent insights that can potentially be extracted for some big data explorations. In such contexts, the insight itself may seem very small given the vastness of data collected, processed, and analyzed. However its value could be unimaginable when discovered. For example, some answers to potential cures for diseases may be hidden in DNA sequencing data, but it is extremely difficult and expensive to analyze this data and correlate it with known diseases, given the vastness of the data. However, if an insight is found and is leveraged for a cure, it will have huge value for society as a whole.

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info )

[1] Subsequently, the title of the photograph was used by Sagan as the main title of his 1994 book, Pale Blue Dot (Sagan, 1994).

My book titled "High-performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" is published

2014-10-10T22:00:00.002-07:00

My book titled "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" is published.

Order at CreateSpace

Order at Amazon

Here is the Book Site.

Description:
The use of machine learning and data mining to create value from corporate or public data is nothing new. It is not the first time that these technologies are in the spotlight. Many remember the late '80s and the early '90s when machine learning techniques-in particular neural networks-had become very popular. Data mining was at a rise. There were talks everywhere about advanced analysis of data for decision making. Even the popular android character in "Star Trek: The Next Generation" had been named appropriately as "Data." Data mining science has been the cornerstone of many data products and applications for more than two decades, e.g., in finance and retail. Credit scores have been in use for decades to assess credit worthiness of people when applying for credit or loan. Sophisticated real-time fraud scores based on individual's transaction spending patterns have been used since early '90s to protect credit card holders from a variety of fraud schemes. However, the popularity of web products from the likes of Google, Linked-in, Amazon, and Facebook has helped analytics become a household name. While a decade ago, the masses did not know how their detailed data were being used by corporations for decision making, today they are fully aware of that fact. Many people, especially the millennial generation, voluntarily provide detailed information about themselves. Today people know that any mouse click they generate, any comment they write, any transaction they perform, and any location they go to, may be captured and analyzed for some business purpose.

Every new technology comes with lots of hype and many new buzzwords. Often, fact and fiction get mixed-up making it impossible for outsiders to assess the technology's true relevance. I wrote this book to provide an objective view of analytics trends today. I have written it in complete independence, and solely as a personal passion. As a result, the views expressed in this book are those of the author and do not necessarily represent the views of, and should not be attributed to, any vendor or employer.

Due to the exponential growth of data, today there is an ever increasing need to process and analyze big data. High-performance computing architectures have been devised to address the need for handling big data, not only from a transaction processing standpoint but also from a tactical and strategic analytics viewpoint. The success of big data analytics in large web companies has created a rush toward understanding the impact of new big data technologies in classic analytics environments that already employ a multitude of legacy analytics technologies. There is a wide variety of readings about big data, high-performance computing for analytics, massively parallel processing (MPP) databases, Hadoop and its ecosystem, algorithms for big data, in-memory databases, implementation of machine learning algorithms for big data platforms, and big data analytics. However, none of these readings provides an overview of these topics in a single document. The objective of this book is to provide a historical and comprehensive view of the recent trend toward high-performance computing technologies, especially as it relates to big data analytics and high-performance data mining. The book also emphasizes the impact of big data on requiring a rethinking of every aspect of the analytics life cycle, from data management, to data mining and analysis, to deployment.

As a result of interactions with different stakeholders in classic organizations, I realized there was a need for a more holistic view of big data analytics' impact across classic organizations, and also the impact of high-performance computing techniques on legacy data mining. Whether you are an executive, manager, data scientist, analyst, sales or IT staff, the holistic and broad overview provided in the book will help in grasping the important topics in big data analytics and its potential impact in your organizations.

Data Mining, Data Science, and Machine Learning (3)

2014-09-19T15:28:00.003-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info )

Experiences on building Data Science Teams

By education, I was trained in Intelligent Systems focusing on Machine Learning and Pattern Recognition. Consequently, for some of us who have spent our entire career in developing and promoting data mining science, machine learning/pattern recognition, and analytics, "data science" is nothing but a new term. As mentioned, in essence it is the same as data mining science, with a few twists. There is no doubt that the popularity of the term has been directly related to the attention big data analytics has received in recent years, especially due to the success of big web companies. Popularity of these companies has finally brought the importance of analytics to the attention of masses.

Prior to this recent wave, these technologies were already an integral part of many things in our lives, without us noticing it. However, they were never publicized to the scale they are today. The most notable commercial applications of machine learning have been around since late 80's and early 90's; real-time payment card fraud detection, hand-print recognition (US mail, checks, ...), and recommender systems. A startup I worked for called (HNC or HNC Software, acquired by FICO) was the pioneer in the first two. The core of HNC's business from early 90's was to get data from some corporations on their customers and develop intelligent solutions leveraging machine learning and advanced analytics on that data. That was the core value the company would provide and the reason for its valuation. Due to the big data craze and the recent publicity analytics has finally received, today many companies try to do that on any data they can find: public or private. And with the explosion of human and machine-generated data, there is a lot more that can be done.

Data science is "the practice of extracting insights from data of any size or variety, using a multitude of disciplines and technologies for the purpose of creating new data products and services or improving the existing ones."

As a part of four startups, I have spent my whole career in developing and promoting novel applications of machine learning and pattern recognition for real-world problems such as hand-print recognition, control of non-linear systems, real-time behavioral fraud detection, and many more. At some of these companies, I had to hire and build data science teams with the right skill sets. I also worked with a couple of universities locally in the late 90's when there were not huge demands for these skill sets and not much of publicity. I am currently advising a university of establishing an undergraduate Data Science program based on the experience of all these years working in the field.

Here are the skill sets that make a perfect data scientist. Keep in mind, that not many people can be found to possess these all, given the multidisciplinary nature involved. That is why it is an important managerial task to hire a data science team that collectively addresses the immediate needs of the business:

(1) Passion, love, and patience for data (often imperfect data) including all it takes (if necessary) for identification of all sources of data, collection, and validation for the prototype system,
(2) Deep knowledge of machine learning (or pattern recognition), and statistical modeling. These will provide solid ground for quantitative analysis. Real-world experience using these is sometimes a must.
(3) Good computational skills and knowledge of main programming principles - programming experience in at least a couple of languages (one third and one fourth-generation language),
(4) Solid foundation in mathematics including linear algebra, numerical analysis, and probability theory (Bayes),
(5) Business acumen - Focus on data and analytics applications that provide high impact to the business (creating data products and services),
(6) Inquisitive (ability to ask questions, challenging assumptions, validating thoughts/ideas, ...) and pragmatic (there are no perfect solutions. Good is often best given the time and resources),
(7) Some working knowledge of databases or newer data stores. A solid background on fundamentals of computing is a plus, including high-level architectures.
(8) Ability to communicate findings to business, peers, and customers (not for everybody),
(9) Ability to formulate a business problem into a data mining problem (four phases discussed earlier) and execute.
(10) Problem solver + statistical thinker.

It is important to know that tools, packages, and platforms change and as long as the person has the core traits and skills, it should be possible to adapt to new tools and platforms if necessary. Though this is not true for all.

Here are some real-world observations. Obviously I could never hire anybody with these traits directly from universities. But the successful hires from the universities were those with computer science background and focus on machine learning or some similar fields. They were the easiest to train and develop. Applied statisticians (or applied physicists or other analytics disciplines) needed to be developed in two areas: one was in the programming/computation dimension and the other on the acceptance of machine learning techniques and approaches (sometimes at odds with what they have learned traditionally in their fields). Usually statisticians had a narrow set of skills in computation and preferred to use a single tool like SAS or the like. Software engineers (with scientific programming interest) were not a good choice for a data science position since there was a lot more that had to be done to develop and train them.

At the end of the day, our data science team was a combination of the following:
(1) A sub-group of people with more of the skill sets above combined,
(2) A sub-group with statistics focus and bias (but exposure to machine learning techniques which are now somewhat a part of the statistical tools sets),
(3) A sub-team with data collection/manipulation/validation skills that could grow to a data scientist,
(4) Software engineers focused on implementation especially productionalizing the analytics process discovered. They would also help with the prototyping aspects if it required integration of tools and systems.
(5) Project managers to coordinate efforts.

The business acumen and customer facing skill sets take training and development. Some people just preferred the technology and the back office. These were in their nature.

Almost guaranteed, the data aspects of the data scientist job (collection, validation, manipulation, ...) are always learned and practiced in a commercial setting. They are never taught in any university program and every time it surprises people who come from academia.

Big Data Confusion

2014-07-22T12:10:00.001-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

I was away for FIFA World Cup for a couple of weeks and am back now. The first thing I noticed last week was that my Twitter account had been hacked. But now all is sorted out and this blog will be tweeted.

In the process of writing my book, I did some research on the definition of "big data" and "big data technologies." It suffices to say that I found a lot of conflicting definitions, and as a practitioner in the world of analytics, I thought how others who do not have the deep experience in analytics may feel about all this! At the end, I found the following definition of big data the most appropriate.

The most common definition of “big data” is datasets that grow so large that they become awkward to capture, store, search, share, analyze, and visualize them using available data management and analysis tools. First, what is considered big data today may be the new normal in the next several years. Second, it indirectly implies that what is big data may also change with the place—i.e., industry sector (vertical), organization, or enterprise—depending on where they are on their analytics evolution curve.

In the last few years, the term “big data” has been tied closely to “Hadoop” by the media. This is unfortunate. The reader understands that an organization may choose, or may have already chosen, to solve its big data problems using big technologies other than Hadoop. At the same time, one can also use Hadoop to solve a more conventional data problem, and not necessarily a big data problem. Hadoop is only a piece in the whole big data puzzle, but it is an influential one. This 2013 survey (Gartner Survey: Big Data Adoption in 2013 Shows Substance Behind the Hype) like a few other surveys conducted by other research organizations reflects what organizations think of big data.

In my upcoming book, I have tried to address fact vs. fiction (hype) about big data analytics. Big data analytics can be approached from many different angles as it relates to business use cases, analytic processes (methodologies and algorithms), platform architectures, etc. As far as platforms are concerned, when we talk about big data, we are talking about one of these options:

- Hadoop, MapReduce, and YARN,
- Massively parallel databases (Relational and NoSQL),
- Real-time event stream processing,
- In-memory distributed analytics,
- Big data analytics appliance.

Again, the Gartner Survey illustrates that aside from the hype, what organizations consider big data technologies (cloud is at the top), what they use them for (customer experience at the top), and what type of data they use them for (transactions at the top). I have not seen a single unified understanding of big data in these surveys and there are many of them. That shows how lose the definitions and understandings about big data are.

Data Mining, Data Science, and Machine Learning (2)

2014-06-12T10:27:00.002-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

A question that often comes along is: "What is the difference between Machine Learning and Data Mining Science (now Data Science)?" Newcomers often confuse these, not mentioning the businesses who try to use these terms to make their products more appealing to their customers and investors. I make an attempt to describe the difference between the two here. For more detail, you can read my new book.

Data Mining Science (the practice is referred to as Data Mining) covers all aspects of creating value out of data from capture, storage, quality checks, analysis, deployment, to infra-structure. It requires some knowledge of a variety of disciplines including mathematics, machine learning, statistics, programming, and databases. On the other hand, machine learning (ML) is a set of techniques and algorithms that can be applied to the data (any type of data from numeric, text, binary, ...) when placed into the right form; meaning a set of exemplars (or patterns or observations or records) each with a set of features that captures the problem of interest. ML (very close to pattern recognition) is a branch of artificial intelligence in computer science that in general focuses on the general problem of "learning from data" or "learning by examples."

Machine learning and pattern recognition are very close in principles and established fields taught in top universities for a long time, Both try to address the problem of learning with the difference that ML has its roots in computer science while pattern recognition has its roots in engineering. Statistical Learning Theory is a sub-field of ML that focuses on the formalization of the problem of learning, basically focusing more on the theoretical aspects. Statistical learning theory explains why machine learning techniques in general work, something the practitioners had experienced through empirical results. Traditional statisticians have always been skeptics regarding machine learning applications even though nowadays these techniques are a part of many statistical toolsets.

The fundamental assumption in machine learning that distinguishes it from traditional statistics is the principle that it makes little or no pre-assumptions about the problem including data distributions. Traditional statistics's roots go back to when there were no computers around, and analyzing any data centered around data reduction and simplification (sampling, variable reduction, linearity assumptions, ...). These often required making restrictive assumption about the problem and the data distributions.

Historically, ML techniques had to deal with solving much more challenging problems and as such, they do not make pre-assumptions about the problem and instead, uses the power of the computer to search and optimize for "a good" solution, often using heuristics. Machine learning is not necessarily in search of the best optimized solution, but "a good solution" and many times it uses heuristics in its approaches. In other words, traditional statistics forces severe assumptions on data to get its best solution, but the solution is only best assuming the correctness of those assumptions. However, dealing with many real-world machine and human-generated data, those pre-assumptions are rarely correct. Hence the resulting solutions can be mediocre at best. ML generally does not seek or claim to find the best possible solution. Often given the complexity of the problems, the best solution may not be achievable or worth the time and resources to find it even if it exists.

Another fundamental pillar of machine learning is the concept of "generalization" and the trade-off between accuracy and robustness. ML solves this using empirical approach of using training/validation/test approach while traditional statistic uses statistical tests on the training data(coupled with tight initial assumptions) to address this.

As a practice, data mining (and data science) has four phases of equal importance:

(1) Business problem understanding

(2) Data understanding and preparation

(3) Model development and assessment

(4) Deployment and monitoring.

Business Understanding: Data mining starts from the full understanding of the business problem where business domain knowledge and data mining knowledge both have to be leveraged. The final result of this process is to set an ROI expectation and a formulation of the business problem into a data mining problem.

Data Understanding and Preparation: Understanding the data does not need any explanation. By definition, it requires proper storage, low-level quality checks, and access to "all relevant data" for the desired business problem. The data preparation covers all aspects of data manipulations that will be required for "model development." Sometimes the data preparation is minimal but very often it requires sophisticated and innovative ways of converting "raw relevant data" to so-called an "analytic data set" (or ADS) that well represents the business problem of interest and can be fed to into the algorithms of interest. An ADS is a set of exemplars each with a set of features that capture the problem of interest.

Model Development and Assessment: The carefully designed and processed ADS is the input to this stage. This stage is where machine learning is applied to process structured, unstructured, or multi-structured data. The results of modeling are assessed against the business goals, and a good solution is selected.

Model Deployment and Monitoring: Building the most sophisticated models are only useful when those models can be operationalized and continuously monitored for performance degradation.

In earlier days of data mining, many projects failed because they were solely focused on the modeling and learning algorithm cuteness (academic orientation) rather than focusing on the business value. Many others failed in practice because of lack of attention to data storage, quality, and preparation for modeling. Of those that passed these hurdles, many failed because they could not be deployed in operations.

Today, many of these early failures have been addressed and good practitioners evaluate the whole life cycle at the first phase before attempting for a solution. The tools used in the process can change and often are from a variety.

With the explosion of data and the popularity of analytics and ML in general, all players in the market are using the term "Data Science" and "Data Scientist." The data platform vendors (MPP, NoSQL, Hadoop vendors) use the terms to emphasize the database/data store, and basic analytics aspects. Outside the context of big data, I do not consider basic analytics even close to what a data scientist has to do. Startups and big web companies may emphasize more on the programming requirements aspects of a data scientist.

In my opinion, Data Science is mainly a sexier and more appropriate name for "Data Mining Science." "Mining" does not portray the right image because it is generally associated with dangerous and hard manual labor. Also Data Science as a practice is more focused on creating new products (data products) and tends to be much higher-level in the organization's leadership.

In the third piece of this topic, I try to enumerate what skills are required for a data scientist, what skills one needs in a data science team, and what needs to be taught in a data science curriculum.

For more detail and in the context of big data impact on data mining, you can read my new book.

In future blogs, I discuss "my ten commandments" for data mining, i.e., the general principles to be aware.

Analytics Maturity

2014-06-09T16:00:00.003-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Like many new technology terms in the last two decades, "analytics" has been used and abused in different contexts. Depending on the background, data professionals have a different view of what analytics is. If one comes from IT, database, reporting, or business analysis backgrounds, he/she will consider any manipulation of data that generates reports, aggregated results, or data slicing and dicing as analytics. If one comes from the data mining, machine learning, or statistical modeling experiences, he/she will only consider analytics as where sophisticated algorithms are applied to the data.

The good news is that this battle has already been fought and settled. Today, there is a clear distinction between these two interpretations of Analytics. The former is called basic analytics (low end or spreadsheet-kind of analysis) often referred to as looking at the behind mirror. The latter is referred to as advanced analytics (high end) where the goal is to use sophisticated techniques on the past data to make predictions in the future. With big data, even basic analytics like simple aggregations and tabulation of the data for reporting becomes a challenging task if response time is at all of concern. Those who focus on basic analytics tasks are like journalists while those who focus on advanced analytics may be called innovators.Both types of analytics are essential to the well- being of a business.

The fundamental principle is that an organization cannot transition into advanced analytics era if they have not already mastered the basic analytics applications. In other words, basic analytics is the requirement before entering into advanced analytics, and both are dependent on solid data management infrastructures. This so-called analytics maturity is discussed at length in my book in different contexts (including big data context) and assessing it is necessary prior to any effort to augment a firm's analytics capabilities.

Data Mining, Data Science, and Machine Learning (1)

2014-06-06T10:18:00.001-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

People often ask me about data science and its relationship with data mining science (often just referred to as data mining) and machine learning. In my book, I provide a viewpoint on this topic. For those like me who have spent their entire career in developing and promoting data mining, machine learning, and data analytics, “data science” is nothing but a new term.

The use of machine learning and data mining to create value from corporate or public data is nothing new. It is not the first time that these technologies are in the spotlight. Many remember the late ‘80s and the early ‘90s when machine learning techniques—in particular neural networks—had become very popular. Data mining was at a rise. There were talks everywhere about advanced analysis of data for decision making. Even the popular android character in “Star Trek: The Next Generation” had been named appropriately as “Data.”

Data mining science has been the cornerstone of many applications for more than two decades, e.g., in finance and retail. However, the popularity of web products from the likes of Google, Linked-in, Amazon, and Facebook has helped analytics become a household name. While a decade ago, the masses did not know how their detailed data were being used by corporations for decision making, today they are fully aware of that fact. Many people, especially the millennial generation, voluntarily provide detailed information about themselves. Today people know that any mouse click they generate, any comment they write, any transaction they perform, and any location they go to, may be captured and analyzed for some business purpose.

All these have contributed to finally bring analytics to the forefront of many conversations even among regular people. A decade ago, we could not comfortably tell a customer how we anonymously analyze their detail transactions in real-time to protect them even from fraud (See Chapter 9 of this book).

Big Data Analytics Confusing Landscape

2014-06-06T07:42:00.001-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info ).

Any new technology brings its own new buzzwords and lots of promises. It becomes real confusing for outsiders to assess its relevance and to separate noise from reality. In my book, I provide an objective and holistic view of big data technologies and the impact of new big data technologies in classic organizations. Leveraging my 20 years experience in advanced analytics and machine learning, I tried to cover many topics that are relevant to anybody who is interested to get into this field.

Who is my audience for my new book?

2014-06-06T07:25:00.000-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data."

I wrote this book for a variety of audiences (See here for details). Most importantly, there are many people in the technology, science, and business disciplines that are curious to learn about big data analytics in a broad sense, combined with some historical perspective. They may intend to enter the big data market and play a role. For this group, the book provides an overview of many relevant topics on the subject.

Hadoop Summit 2014 Take Aways

2014-06-05T12:46:00.002-07:00

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data."

Notable Trends from Hadoop Summit 2014 #HadoopSummit

- FYI: Hortonworks and Yahoo! are hosts of this event. This event has been held in San Jose Convention Center for the last four years (I think) and attracted 3200 people this year.

- As a sign of maturity of Hadoop, one can see the list of Diamond and Platinum sponsors that include many heavyweights such as AT&T, Microsoft, SAP, Teradata, Cisco, IBM, Informatica, Oracle, SAS, and VMware.

- With YARN finally released at the end of 2013, there was a high expectation to hear about Hadoop maturity stories/experiences in classic environments. There were stories and experiments that were shared showing that Hadoop with YARN is moving along its maturity curve, and that is good news for the classic organizations.

- Still lots of tools in Hadoop ecosystem are low level but the trend continues by different vendors to make the tools for integration and use more efficient and higher level. For classic organizations, this will be the way to get Hadoop use more widespread beyond storage and ETL.

- With Yahoo! being a host, there were lots of presentations from them regarding various aspects of Hadoop 2.0 usage at Yahoo!

- One thing I have found is that there is no "single big data technology" that solves most of the business problems out there especially for classic organizations that have lots of investment in the legacy big data analytics technologies. For now, for most business problems, more than one big data technology will be required.(Good to read my book on the subject when it is out).Many participants from classic organizations in banking, telco, retail, and others emphasized this as their experience playing with Hadoop. Remember the Hadoop newcomers used to call it the "holy grail", but those days are over. What we will see is more of convergence and integration rather than pure replacement.

- Queries using MR are slow. There are now many solutions on Hadoop to address "low latency" queries for big data such as Impala (Cloudera), Drill (MAP R), Shark (Spark), Presto (Facebook), HAWQ (Pivotal), and Apache Tez. All these are making things more confusing, and there would be no answer on what is the right way for sometimes to come. There were a few presentations focusing on Tez and how it improves performance for Pig and Hive.

- Spark-on-YARN enables Spark application run on Hadoop with no need to create a new cluster. There were some early results shared by Yahoo!. Also, there is SparkR from UC Berkeley that promises interactive analysis of large data in parallel from the R shell.

- There were new use cases from travel industry, electronic consumer peripherals, to selling cars showing Hadoop traction in many more industries.

- Hadoop security is coming along and still more work needs to be done. Check out Apache Sentry, Apache Knox, and Project Rhino.

- About cost of ownership of Hadoop cluster, what I have found is that one should always start with a public cloud implementation for discovery and exploration, and even at earlier stages of operations. There would be a point where depending on the application-specific factors, an on-premise implementation would be more cost effective and will also provide better ROI. For those applications that data security is of high importance, on-premise from the start will be the way to go for now.

Finished a book...

2014-05-27T16:37:00.002-07:00

I now have finished a book titled "High-Performanceon Data Mining and Big Data Analytics."

It is due to come out in July 2014 and will be available through Amazon.com.

See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info )

An Overview of High Performance Data Mining and Big Data Analytics

2012-04-04T14:59:00.000-07:00

<Update May 2014>
I have written a book on this title due to come out in July 2014.
See the book site for "High-Performance Data Mining and Big Data Analytics: The Story of Insight from Big Data" (http://bigdataminingbook.info )
-----------------------------------------------------------------------------------------------------
<April 04, 2012>
With the exponential growth of data comes an ever-increasing need to process and analyze the so-called Big Data. High performance computing architectures have been devised to address the needs for handling Big Data processing not only from a transaction processing viewpoint but also from an analytics perspective. The main goal of this paper is to provide the reader with a historical and comprehensive view on the recent trend toward high performance computing architectures especially as it relates to Analytics and Data Mining.

There are a variety of readings separately on Big Data (and its characteristics), High Performance Computing for Analytics, Massively Parallel Processing (MPP) databases, algorithms for Big Data, In-memory Databases, implementation of machine learning algorithms for Big Data platforms, the Analytics environments of the future, etc. However none gives a historical and comprehensive view of all these separate topics in a single document. It is the author’s first attempt to bring about as many of these topics together as possible and to portray an ideal analytic environment that is better suited to the challenges of today’s analytics demands.

(Use Fullscreen button down the page for easier viewing of the Document)

201201_High_Performance_Data_Mining_published_20120501.pdf by Khosrow Hassibi