The article, “Mass. firms see riches, jobs in charting oceans of data”, discussed reports on Big Data by the Mass Technology Leadership Council (MTLC) and McKinsey Global Institute regarding the role Massachusetts plays as a hot bed for both Big Data software firms and the people with the skills to level Big Data in their businesses. 

Big Data’s promised business benefits range from capturing more of a consumer’s wallet to discovering the cure for cancer.  Software firms certainly see the potential of Big Data. They’re investing in better ways to capture, integrate, manage and analyze Big Data. Both established high tech firms and venture capitalists are making significant investments in it.

The Big-Data Skills Gap

The gap we are going encounter in this industry is the shortage of both the skilled people to develop the Big Data analytics models, often referred to as data scientists, and the data-savvy business people to take advantage of them in their business. The McKinsey report states “By 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.”

Our industry has faced a shortage of skilled people in business intelligence, analytics and data integration that has kept business from effectively using the data they already have. With the onslaught of Big Data and the advanced skills it requires, we’re destined to fall even further behind.

Data Scientist Definition

If we are ever to fill that gap we need to broaden our definition of what a data scientist is. This title and the job descriptions that are posted for these jobs are too concentrated on programming and IT skills. In order to create Big Data analytical and predictive models the person does need to have programming skills, but more importantly, that person needs to understand statistical modeling. To develop the model, the person also needs to understand their business and industry. It also helps if they understand how to apply customer behavior and economic models. 

Although Big Data means “a lot of data,” the reality is, in many cases, that data is incomplete and inconsistent.  Developing analytical models means dealing with dirty data and gaps.  Many “numbers” people have trouble dealing with these conditions.

Data scientists need to have expertise in statistics, economics and business besides the basic programming skills. Although the ranks of the software engineers and IT is an excellent place to start, our industry needs to broaden its appeal and reach to people with a mathematics, actuarial, statistical, economics or business backgrounds if the data scientist gap is ever to be filled.

McKinsey mentions the shortage of the data-savvy business managers who will be the users of the Bid Data analytical models developed by the data scientists. Fortunately, many business schools have started offering analytical curriculum at both the undergraduate and graduate level. For business people with deep business expertise, it is time to brush up on statistics.

• Business people are averse to change; they don’t want to invest time in learning a new tool
• Business people don’t understand that BI tools such as dashboards are more powerful than spreadsheets; they’re foolish not to use them
• Spreadsheets are filled with errors
• Spreadsheets are from hell

 IDC estimated that the worldwide spend on business analytics in 2011 was $90 billion. Studies have found that many firms have more than one BI tool in use, and often more than six BI tools. Yet a recent study found that enterprises have been “stuck” at about a 25% adoption rate of BI tools by business people for a few years.

So why have adoption rates flatlined in enterprises that have had these tools for a while? Are the pundits correct in saying that business people are averse to change, lazy or just ignorant of how wonderful BI tools are?

The answers are very different if you put yourself in the business person’s position.

First, business people are likely working many hours and multi-tasking. Demands on their time are ever increasing regardless of the state of the economy. Without any spare time, they need a compelling return on investment before learning a new tool. Spreadsheets have been the bedrock of reporting and analysis for quite a while, so if it ain’t broke don’t fix it.

With spreadsheets business people can slice and dice data, pivot it, calculate new metrics and display in compelling visuals. What they need more than a new slick presentation tool is comprehensive, clean, current and consistent data.

Second, moving to BI tools mean that business people have to depend on their IT staff to create the dashboards, cubes, visualizations and reports. Since many business people feel that there is a significant IT bottleneck resulting in a long waits before they get their reports, they keep turning to spreadsheets. After all, neither their competition nor their boss is going to wait for IT to generate the report they needed yesterday.

There are certainly many compelling reasons to convince a business person to use a BI tool, but the usual slick demo and “it will practically do your job for you” pitch is not going to change a business person’s mind.

Next: What are the real problems and how do we entice business people to use BI tools?

EMC's Chuck Hollis says, of big data analytics:

"The race is now on to acquire -- and maximize the productivity of -- the key talent behind this wave: data scientists and their supporting data science teams.

At EMC, we've been working hard to understand who these people are, what makes them different, how they work -- and what they think is important."

Take a look at Understanding The New Rock Star: The EMC Data Science Survey

IDC recently stated “In 2011 the business analytics technology market, which includes software, hardware, and services for data warehousing, query, reporting, analysis, advanced analytics, content analysis, analytics spatial information management, and analytic applications, surpassed $90 billion in worldwide revenue.” Business intelligence (BI) has ranked in the top tier of IT application priorities for over a decade.  Vendors have made significant improvements to BI, data integration and data warehousing products during that time. Experts have established best practices in these areas.

Yet despite the attention, technological capabilities and investments there are dirty secrets in our industry that analysts and pundits do not like to admit:

• Spreadsheets are the BI tool of choice for business people. Many enterprises use several BI tools, yet the only pervasive BI tool is a spreadsheet.
• Manually coded extracts are the predominant data integration tool to get data into those spreadsheets (and often into the databases) used for reporting. IT often used a data integration tool to get data into a data warehouse, but to then get data into data marts, OLAP cubes or other databases or tables used for reporting IT uses SQL scripting. Business people use the capability that Microsoft Excel offers to gather data and often supplement this by leveraging Microsoft Access.

Many are in denial about these dirty secrets, but in order to change something one must first recognize there are problems. When IT and analysts do confront these dirty secrets they often lash out at spreadsheets and SQL, but the reality is the use of these are symptoms of problems, not the problems themselves.

The result of this situation is that managing the information that business people need takes longer and costs more. In addition, there is a perpetual and increasing backlog to obtain comprehensive, consistent and current business information.

Subsequent posts will discuss the causes of these dirty secrets and what both IT and business need to do differently.

This debate is often an ideological battle, where people cite Inmon or Kimble to justify their position. At this level, the debate is about theory rather than the business, data or analytical needs of enterprise business people. But before people build a data warehouse, they must understand those needs, as well as the industry best practices that will help fulfill them.

The biggest reason why IT groups have this debate is because their view of dimensional data modeling is too simplistic. IT developers generally view dimensional models as fact and dimension tables placed in either a star or snowflake schema.  IT understands how to implement the basic concepts such as surrogate keys and slowly changing dimensions (SCD), but they hardly, if ever, use much of the advanced (also known as hybrid) design constructs.

They see the advanced concepts, such as rapidly changing, casual, hot swappable, heterogeneous or junk dimensions; how to implement hierarchies; bridge and outrigger tables; and when to use the various categories of fact tables, as esoteric.

(Admit it, it was tough just reading this sentence without thinking it was time to check your Facebook page!)

So why is it so tough to grasp these advanced concepts? A big part of the problem is that they are generally explained in an academic context. They’re not being connected to the real-world use cases where they should be used. Thus, they become geek-speak and are ignored.

Complicated as they may sound, the advanced dimensional design approaches have each been formulated based on real-world business and data requirements that occur across all enterprises. Rather than esoteric, these concepts are based on a pragmatic approach to implementing successful data warehouse and BI solutions.

Until IT understands the depth and practicality of advanced dimensional modeling, the decision whether to implement a normalized versus a (simplistic) dimensional model is a false debate. IT either builds an overly complex 3NF data warehouse that quickly gets overwhelming, or they build an overly simplified dimensional model that needs to be continually overhauled to support the inevitable expanding and changing business requirements. In either case, the business is underserved when it comes to getting the information they need, and the costs of BI keeps rising without the expected business ROI.

If companies understand and implement advanced dimensional models, then they can leverage the best practices that have been developed through years of real-world experience.