This is one potential trouble spot with big data that I don’t think most people recognize or consider. Many big data initiatives today are that special combination of new data being applied to a new problem. This makes it critical to validate your assumptions and the influence they have on the results of the analysis. If your results aren’t stable across the range of reasonable assumptions, then you have a problem.

An Example

I recall many years ago when I was first building predictive models that included TV advertising data. The data was at an extremely high level to begin with and on top of that we had to make many assumptions about the data as we prepared it for our models. For example, what decay rate would we use for the advertising impressions? How would we reconcile any differences in projected impressions from different sources?

The guidance I was given at the time, which is what I think most people usually follow even today, is that if a model’s parameter estimates come out significant and the model explains a good bit of the variance, then you have found a model that is good and you can use it. However, I stumbled upon a huge problem with that approach.

One day I had what would have been considered a good model under the above rules. However, for some reason, I decided to see what would happen if I changed my assumptions about the decay rate and a few other points and re-ran the analysis. I was astonished to see that I still had significant parameters that in total explained a lot of the variance. However, the new parameter estimates were different from my original ones by more than the margin of error!

In effect, my assumptions did more to determine my results than did the model itself. The team and I did more extensive testing to finalize assumptions we all agreed were the best possible for the advertising data. However, I am still uncomfortable today with the idea that assumptions can in many cases do more to determine your answer than the analysis that uses those assumptions.

Be Sure To Test Your Assumptions

I recommend that you make a point to test the impact that your assumptions have on your results even if a new analysis looks great at first. If you find that minor changes in your assumptions have a substantive impact on your results, then you should go through a much more detailed process of validating your assumptions. This is especially true if changing assumptions leads to results that will actually point to different decisions. With big data, this extra work may be necessary frequently because you are often breaking new ground where assumptions haven’t stood the test of time and application.

Of course, there is always the possibility that your assumptions are wrong. You may also not be able to prove what the best assumptions are.  For an example of this, look at the impacts on a retirement portfolio caused by changes in the average compound interest rate earned over time. There is no way to know what the actual rate of return will be, but you are wise to use one more towards the lower end of what you think is reasonable to be safe. By understanding how the rate of return assumption impacts the ending value of your savings, you are able to choose assumptions that best fit your mindset, risk tolerance, and needs.

Following the approach I have outlined here won’t remove all your risk. But, it will certainly ensure that you better understand the risks you are exposed to. In a situation where one set of reasonable assumptions produces a result that says “go” and another says “no go”, I suggest that you make everyone aware of the issue and then have a candid discussion about the implications of choosing one set of assumptions over the other as you determine the best way to proceed. This leads to a more informed decision, which is what you should always strive for with any analysis.

To see a video version of this blog, visit my YouTube channel.

Originally published by the International Institute for Analytics

When we named this year’s IIA Summit the Chief Analytics Officer (CAO) Summit, we did so because it aligns with our belief that analytics is the future of business and the last competitive differentiator in the market. Therefore, the idea that in the not too far off future companies will include a CAO type role in the C-suite does not seem far-fetched. Especially given the gravitas of the job and the role analytics plays in the success or failure of business today.

With just 2 weeks until the2013 CAO Summit in Chicago (June 18-19), we are more excited than ever about the group of analytics leaders and experts we are convening to share and discuss best practices from the frontlines of business. Recent additions to the speaker line-up include Dan Yoo, the Senior Director of Business Operations and Business Analytics from LinkedIn, Dan Wagner—the CAO mastermind who helped win POTUS the White House—, and Michael Cousins, the VP of Research and Analytics at Cigna.

Regardless of your specific title, if you’re charged with applying analytics to help drive your business, we’d like you to join us and be part of the conversation. There are just a few spots left, so register today for IIA’s CAO Summit in Chicago!

It’s obviously possible, of course, to fold Big Data and analytics into the responsibilities of existing C-level executives—a CIO, a CFO, a COO, or even a CEO. A recent Deloitte Analytics survey found that respondents were most likely to say that analytics reported to the CEO. On the one hand, it’s great that the topic has the attention of the CEO. On the other, will it really get enough attention with all the other things that CEOs have to worry about?

There are many other examples of organizations that have successfully folded analytics into other C-level jobs. At Procter & Gamble Co. , CIO (and head of Global Business Services) Filippo Passerini has done a great job of bringing analytics to the executive suite. At Caesars Entertainment Corp. , the analytics group reports to CFO Donald Colvin, and the company has probably produced more positive analytical results than any other. At software giant Intuit Inc. , Nora Denzel has a unique job that incorporates Big Data, social design, and marketing. Since she took it, Intuit has delivered several new data and analytics-based products or features to its customers. In other words, combining analytics with other functions can work quite well.

Despite these successes, I still think it makes sense to have a dedicated role for analytics. There are increasing numbers and various versions of these. One variation is the chief data officer (CDO) role, which is pretty common in large banks. In principle I think it is a fine idea to combine the responsibility for data management and governance with the application of data, i.e., analytics. In practice, however, most of the CDO incumbents seem to spend the great majority of their time on data management, and not much on analytics. Most of them don’t have strong analytics backgrounds either.

There are some exceptions, of course. John Carter was the CDO at Equifax Inc. , and while there he led efforts to build the company’s analytical capabilities—while still wrestling with many data issues as well. And Carter has a Ph.D. in Statistics. Now, however, Carter has a different job at Charles Schwab Corp. He is senior vice president of analytics, insight, and loyalty. I get the impression he has an easier time addressing analytics in his new role.

There is another new analytics-intensive role at eBay Inc. Zoher Karu, who led analytics efforts at Sears Holdings Corp. , will be the new “vice president of customer optimization and data.” I might prefer the term “customer analytics,” but optimization is of course an analytical concept.

Then there are the C-level titles that are purely focused on analytics. FICO, the University of Pittsburgh Medical Center, and the Obama 2012 campaign are three organizations that have named chief analytics officers. The insurance giant AIG created a new role called chief science officer for analytics veteran Murli Buluswar. If you are really serious about analytics and you want to employ them in a variety of functions and units around your organization, I recommend this sort of job and title.

However, I don’t want to quibble. I will be happy if any organization creates a senior management role with analytics in it, regardless of the specific title. The perfect analytics title must not be the enemy of the good title.

Originally published by WSJ’s CIO Journal.

In this research, however, we studied the Big Data activities of large, well-established businesses—companies like United Parcel Service Inc. , Wells Fargo & Co., United Healthcare Products Inc., and other giants. All are doing something with Big Data, but large volume, high velocity, and unstructured data in those organizations must be integrated with everything else that’s going on in the company. Overall, we found the expected co-existence; in not a single one of these large organizations was Big Data being managed separately from other types of data and analytics—it’s all blended together.

Big Data may be new to much of the world, but many data-focused executives in large firms view it as something they have been wrestling with for years. Some managers appreciate the innovative nature of Big Data, but more find it “business as usual” or part of a continuing evolution toward more data. However, they are still impressed by the lack of structure of the data they are now able to manage, and the opportunity/cost ratio of Big Data technologies.

There are also continuing—if less dramatic—advances from the usage of more structured data from sensors and operational data-gathering devices. Companies like General Electric Co. , UPS, and Schneider National Inc. are increasingly putting sensors into things that move or spin, and capturing the resulting data to better optimize their businesses. Even small benefits provide a large payoff when adopted on a large scale.

Like many new information technologies, Big Data can bring about dramatic cost reductions, substantial improvements in the time required to perform a computing task, or new product and service offerings. Like traditional analytics, it can also support internal business decisions. Most of the companies we interviewed had a specific benefit in mind. Each benefit choice has implications for the leadership of the Big Data initiative and the way that benefits are managed.

Big Data introduces some highly specialized features that set it apart from legacy systems. In the report we describe a “big data stack” that is optimized around the large, unstructured and semi-structured nature of Big Data. It starts with storage at the bottom and tops out with user consumption activities—in short, it’s not just about Hadoop. We also describe how Big Data technology architectures interface with more traditional business intelligence and reporting architectures. It’s clear that there are many alternatives now to the traditional data warehouse, as I described in my last post.

As with technology architectures, organizational structures and skills for Big Data in big companies are evolving and integrating with existing structures, rather than being established anew. There is no separate “Big Data Department;” instead, existing analytics or technology groups are adding big data functions and data science skills to their missions. The companies have several different approaches to finding scarce data scientists, and some are also worried about establishing data-savvy leadership. However, there doesn’t seem to be the data scientist hiring frenzy in these companies that you find in Silicon Valley.

In this guest column I have previously described a new paradigm for managing analytics, “Analytics 3.0.” It’s the combination of traditional analytics and Big Data—certainly what we found in these big firms—and it’s necessary now that the data-driven economy applies not only to online business, but to virtually any type of firm in any industry. Some of the other attributes of Analytics 3.0 we found in this study include the combination of multiple data types, new approaches to data integration, much faster processing of data with new technologies, and the integration of analytics with operational and decision processes.

Even though it hasn’t been long since the advent of Big Data—a decade or so for online firms, and less for big companies in other industries—these attributes add up to a new era. Some aspects of this new world will no doubt continue to emerge, but organizations need to begin transitioning now to the new integration of big and small data. The new model means change in skills, leadership, organizational structures, technologies, and architectures. Together they comprise the most sweeping change in what we do to get value from data since the 1980s.

Originally published by WSJ’s CIO Journal.

“With the trend towards Big Data and Data-driven Machine Learning methods

• Statistics will become less important
• Statistics importance will not change
• Statistics will become more important, as the foundation of Data Science
• Not sure”

“Statistics will become more important” was the clear winner. While this poll is not a scientific survey, it’s still interesting to see what people are willing to take the time to express their views on. It’s unfortunate that the word “statistics” has different contexts, and has been viewed so negatively for so long by so many. This is perhaps why we’ve seen other terms come in (and out) of fashion: data mining, data science, predictive analytics, etc., to rebrand what are essentially statistical concepts. It’s ironic that many people put statistics in a box as though it’s forever one thing — only deals with small data, is about hypothesis testing, etc. — when statistics is fundamentally about dealing with change.

In earlier posts, I’ve written about statistics as its own discipline; how it is core to data analysis and value creation, and how statistical literacy is growing in importance (let me take this opportunity to remind you that we are celebrating the first-ever International Year of Statistics!). Jeff Leek of Simply Statistics has a nice YouTube video on The Landscape of Data Analysis showing that statistics is foundational to data analysis. While other disciplines also contribute, they don’t contribute as directly.

The first time I heard statistics described as “the language of science” was many years ago in a conversation with David Salsburg, author of The Lady Tasting Tea and first statistician Pfizer ever hired. To expand on that, to be more scientific in any decisions you make — in science, in industry, in government — you will need statistics! And statistics needs you! Robert Tibshirani, eminent statistician at Stanford University, is quoted in The New York Times Bits blog last year: “Statistics is unusual.  … It’s a service field to other disciplines. It doesn’t rely on its own work. It needs others.”

It wasn’t too long ago that universities required you to meet foreign language requirements, especially for graduate degrees in the sciences. It now appears that a new language — statistics — is working its way in to the curricula for degrees in science as well as business. Recently, a proposal was made to establish a statistics curriculum within the chemistry departments of US colleges and universities. In addition to the evolving curricula in business schools to include more statistics, data mining, predictive analytics (and offering new degrees in these areas), even the hard sciences are incorporating more statistics to better prepare their graduates for jobs in industry.

Many of our customers confirm that they spend a few years investing in their new hires to instill in them best statistical practices because their academic training has not adequately prepared them to do the work that is needed. One of our longtime partners, Predictum, has been offering courses like Data Analysis and Statistics for Scientists and Engineers since 1997.

Statistics as a word may have some baggage (many unfortunately did not have the best introduction to this powerful subject and think of it as “sadistics”), but “statistical thinking” is another term that casts everything in a more strategic light. Statistical thinking is being scientific about problem-solving, speaking the language of science in any given context, because what is science? Science — good science — is the efficient and effective way of understanding the natural and social world to be more informed and make better use of that information.

Good science and speaking its language with some level of proficiency, is required to derive value from the growing volume and complexity of data we continue to amass. May we all learn, at some level, the language of science so we can make more informed decisions, best utilize scarce resources and compel better actions.

An Exemplar – whether professional analyst, IT professional of any sort, or analytically oriented business person – does three basic things:

•  Exhibits the hallmark behaviors of an analytical culture, including delving into detail, searching for truth, testing and learning, and purposefully blending evidence and experience in decision making.
•  Uses analytics – models, applications, informal analyses – regularly in work and decisions. When presented with a new situation or problem, the Exemplar’s first instinct is always to find some data and analyze.
•  Improves analytics by developing data, sharing analyses, and participating in projects to build and refine analytical models and applications.

Professional analysts do these things for a living. As Exemplars, they take some extra steps: making their analytical methods transparent to business colleagues, being visibly analytical both in validating opportunities for analytical applications and in measuring the performance of those applications, and sharing their output and expertise across the analyst community and its projects.

Managers generally and executives in particular are rightfully expected to lead by example. So there are additional opportunities and responsibilities for them to be analytical Exemplars for their management colleagues and the organization at large.

If you’re a manager:

•  Recognize and reward analytical behaviors and analytics-based business results. Set expectations and objectives about analytics use, and incorporate them into the performance management system.
• Institutionalize devil’s advocacy in your team. You can become more fact-based through persistent and effective devil’s advocacy – someone on the team who scrutinizes data use and rigor of discussion and injects alternative viewpoints. If team members tend to be reticent, stir things up (and make it a game) by designating a devil’s advocate at each meeting.
• Enable analyst rotation. As discussed under the role of Personal Trainer, an organization builds its analytical muscles with the help of analytics professionals “in the field.” If you manage analysts, establish a rotation program. If you’re a business manager, request and welcome the itinerant rotating analyst.

If you’re a senior executive:

• Sponsor analytics projects and enable them to succeed. If analytics are new to your part of the business, or the project involves important decision points, you should be much more hands-on than usual as the project sponsor.
• Invest in building analytical capability. The enterprise at large and each of its major business units and functions should have a strategy for using analytics and an investment plan for building analytical capability – people, process and technology.
• Employ analytics prominently in the performance management of the enterprise. Use analytics in setting strategic direction. Use analytical scorecards and dashboards to monitor and understand performance and make decisions. Regularly review performance drivers, strategic assumptions, and the business model itself. And do these things where the organization can observe, not behind closed doors.

The real litmus test of analytical executives is not just using their scorecards but developing them – playing active roles in defining, refining, and field testing their scorecards, and looking for ways to make them more analytical. Many scorecards I see are high on visual display and low on analytics; they roll up important business information in traditional reporting fashion. An analytical scorecard is connective and predictive. It shows how business metrics correlate and drive one another, and it anticipates future performance and the levers to pull to improve it. An analytical scorecard is also in motion, incorporating new metrics and analyses as they prove valuable. The executive Exemplar owns and manages the analytical scorecard.

I hope you’ve enjoyed this tour of analyst roles – Data Demon, Opportunity Finder, Personal Trainer, and Cultural Arbiter, Community Organizer – and a role for everyone (but especially analysts, managers, and executives) – Exemplar of analytics in action.

Part of the answer can, I believe, can be found in the gym.  In the face of the same fitness equipment, the same workouts and educated, advisory support from the same certified trainers some exercisers thrive and others fail.

In predicting analytics success, a key differentiator is not tools or even natural ability, but attitude. Gym goers who thrive – losing weight, gaining strength, sculpting their minds as well as their bodies – have a particular mindset. While this prestigious group has a diverse set of motivations and goals they exhibit similar characteristics. They:

• Value Innovation: As your fitness level improves so must your program. Maintaining the same routine may be comfortable but it doesn’t improve your fitness. The fitness junkie is inquisitive and open-minded, constantly looking for new ways to challenge themselves and keep things interesting.
• Challenge the Status Quo: The fitness savant views goals as stepping stones, not an end point. Run 2 miles? Check. Now…can I go faster? Longer? Join a soccer team to exercise my new skill?
• Are Self-Motivated: While he keeps an eye on others in his league, fitness junkies primarily compete with themselves. No resting on your laurels here!
• Court Failure: OK. Clearly nobody likes to fail. But progress requires risk. And exercising to failure is a tried and true method for increasing strength. After all, how do you know your limits if you never reach them?
• Monitor Performance: Fitness junkies track progress (journals, heart rate monitors, Fit Bits) in the interest of continuous improvement rather than for punitive purposes.
• Solicit & Share Knowledge: The fitness junky with the enviable abs and prodigious stamina views both the trainer and fellow exercisers as valuable sources of knowledge. They are eager to share experiences about what works, and doesn’t, with others in the same boat.
• Manage Change: The fitness guru doesn’t practice good behavior only in the gym. They take their healthy habits on the road, modifying their daily regimens, eating habits and other behaviors (e.g. steps or elevator).

When applied in the context of analytic practices these attitudes also separate organizations that behave intelligently from those that are merely informed. Innovative, adaptive enterprises arm themselves with both the analytic toolkit and the behaviors required to wield the resultant insight. Without the right culture, motivation and chutzpah analytics are interesting but, it could be argued, of limited value.

Kimberly Nevala is a director on the SAS Best Practices team. She specializes in strategies for BI, data governance, and master data management programs, and conducts client workshops and industry presentations in these areas.

With professional analysts and data scientists in short supply, organizations have got to excel at engaging and retaining the analysts they’ve got. Not just to keep the expertise available, but for purposes of institutional knowledge and model management as well.

What does it take for top-notch analysts to thrive in your organization? In addition to the universals – competitive pay and fair recognition – these four conditions are essential:

• Challenging work – top analysts move on when the supply of challenging business problems runs dry. Maintain a queue of initiatives and, of course, don’t let analysts get sidetracked into too much by way of simple report preparation.
• Opportunity to explore – the “2012 Analytics Professionals Study” confirms that the dominant characteristics of analysts include curiosity and creativity. Give your analysts tools and time to explore alternatives and discover and refine solutions.
• Results in action – analysts are pragmatic, and they seek the satisfaction and recognition of seeing their models and applications making a difference in the business. Close the loop and measure the business impact of analytics initiatives.
• Community of colleagues – analysts’ curiosity and creativity are also satisfied by the chance to work with and learn from other highly capable analysts. The Community Organizer helps make that happen.

Many enterprises begin their journeys to analytical maturity with small “pockets” of analysts and analytical activity scattered across the business. Other enterprises have strong centers of analytics in one or two functions – often supply chain, marketing, or finance – and isolated pockets elsewhere. In both cases, you can drive learning and engagement by connecting your analyst groups into a community. It may begin as a loose network or association, then later, as the enterprise’s commitment to analytics grows, evolve into a more formal center of excellence with specific development and deployment objectives and responsibilities.

Community members share knowledge, experience and best practices. They advise and assist on one another’s problems and projects. If you have an established analytics center in one function, the community helps spread its expertise across the enterprise even without benefit of a formal analyst rotation program. A community enables its members to grow and thrive, individually and as teams or discipline groups. It improves learning, development, deployment, engagement and retention. Forming a community builds both capability and momentum for analytics.

A Community Organizer takes the initiative and plays the catalyst. Seeking out the pockets of analytics and performing the introductions. Organizing informal gatherings and eventually a series of “Analytics Days” bringing together analysts and business partners from across the enterprise. Putting in place some infrastructure, email distribution lists or a community space on the internal social network. Serving as a community communications officer, a go-to person for information, publisher of news, and tracker of the community’s accomplishments.

Even if you’re not in the lead in forming and enabling the analyst community, you can help build it in other informal ways.

• Network outside the enterprise and recruit analysts to join it. Be on the lookout through social media and at industry conferences. Referrals yield strong candidates, and referral fees should reward you for finding new additions to the analyst talent supply.
• Serve as an information clearinghouse. When you come across useful articles or links or other resources, point them out and pass them along to colleagues likely to be interested, and post them to the community workspace. You’re contributing to the development of individuals and the “reference library” of the community.
• Blog your work in the community workspace. Whenever you solve a new problem, learn something useful, find a new tool, develop a new trick of the trade, or encounter an obstacle – jot it down. And encourage your colleagues to do likewise. When enough people are doing so, the community has a timely and “mineable” database of experience and best practices.

That last recommendation I wouldn’t have made a few years ago – seems like extra work. But for those of us even mildly addicted to Twitter or Facebook, jotting things down has become natural. So do it on the job for the benefit of your colleagues.

The role of Community Organizer centers on the fourth of the essential conditions listed earlier. However, a vibrant community also serves the other three. It gives analysts visibility and collective voice, making it easier for them to publicize their capabilities, state their needs and preferences, and focus their energies on the most important business opportunities.