Sure, big data technology can read every tweet ever tweeted. It can search and organize boundless volumes of books. It can even help keep track of sports statistics that were previously unrecordable.

But now it can make the world a happier place.

John C. Havens, founder of the H(app)athon Project, wants to harness emerging technologies like big data to bring about tangible and positive world change.

It all started when Havens wrote an article titled “The Value Of A Happiness Economy,” in which he argues that happiness is much more linked to helping others than it is to annual income. He cites statistics that show “after a person or family receives a salary of $75,000 per year, increasing the amount of money brought home doesn’t increase a feeling of well-being.”

Havens points out that the act of “paying it forward,” or performing a favor for someone with the expectation that they return it to someone else entirely, can do wonders for your mood. Seven out of ten people “were happy when they did something good for other people, but only one out of 10 people ever experienced generosity on a daily basis.”

He suggests that big data cen be used to tackle that discrepancy. Enter the H(app)athon App, which is currently in development. It will prompt you to assess your happiness regularly and provide you with suggestions on how to get happier.

Ultimately, he hopes to use all the data this app collects to create “a happiness economy” where people measure their success by how content they are, rather than how wealthy they are.

It’s not just about improving well-being around the world. Havens has political ambitions for his project too.

By analyzing user responses with big data technology, H(app)athon will create what it calls a “global mood ring” – a realtime analysis of happiness around the world. The aim here is to “help policy makers or anyone else see why the holistic measure of well-being could help guide their decisions with greater context than the GDP.”

Watch the H(app)athon App’s video below to learn more.

The H(app)athon Project Video Introduction from The H(app)athon Project on Vimeo.

Please follow SAI on Twitter and Facebook.

Are you ready for Big Data at the big screen?

It’s true. Hollywood has discovered Big Data’s talents to determine how to distribute and promote movies. Advertisements viewers see before a movie is screened, and how that movie will be distributed later through channels like Netflix are all determined by data analysis. And with huge troves of data available to mine, marketers are becoming increasingly sophisticated at targeting movies.

For instance, demographic and geographic information is added to Twitter, Facebook and early reviews to decide – often at the last moment – what advertisements to show movie-goers, said Bill Livek, CEO of Rentrak, a media measurement company.“Big Data empowers the studios to change their advertising. As the movie rolls across the United States, they use that information,” he said.

Livek was speaking at a panel on Big Data at the Tribeca Film Festival in New York, sponsored by SAP.

Richard Whittington, general manager of media industry solutions at SAP, moderated the discussion and stressed the economic importance of Big Data. “Big Data is new oil,” he said, drawing a parallel to the effects of oil on the global economy.

Panelists talked about the various ways in which data is determining business decisions. Christina Warren, a senior technology analyst at Mashable talked about data-driven journalism, which is currently getting more and more fashionable. It is based on Big Data.

“Oftentimes, pieces of content will go viral. Our goal is to cover the content before it goes mainstream,” explaining Mashable’s philosophy.  Mashable uses Youtube share counts, Twitter, Facebook and so on to monitor thousands of data sources.

Eugene Hernandez, director of digital strategy, Film Society of Lincoln Center, had a different take.  He said that relying only on popularity – as in creating content just based on the number of clicks – could miss some very good films.

“If we can find a film that has tremendous artistic value but has tough subject matter, an analyst may say, it may be tough to get an audience,” he said.  But yet that film might appeal to the membership of the Film Society of Lincoln Center.  “We can take something that is challenging and show it to people. There is tension between what the data tells you and the heart tells you.”

That said, Hernandez stressed that the Film Society is “constantly looking at the data” such as whether its members are buying tickets online in an effort to improve their operations. However, how the curators feel about a film is still the most important factor.

“Sentiment is everything. It starts with sentiment. The heart leads, and we like to figure out how data can help us connect with our audience.”

Other panelists seemed to agree. Jason Kassin, co-founder and CEO of Filmtrack, an organization that helps manage IP in the film industry, said creating a film “is still an organic, artistic process.” But with the advent of Big Data, “the ease of looking at this data will help make better decisions.”

A topic that was repeatedly touched upon is the changing nature of the movie industry. Livek said consumer behavior has changed radically when it comes to viewing movies and TV. “Eighty percent of the viewing of a TV show is happening more than three days after airing,” he said. Independent films are also growing disproportionately because of the new platforms that are available.

Stacy Spikes, CEO and co-founder of MoviePass, pointed out some of new insights that come from Big Data. “We can see that people who go to Bollywood movies travel in groups of four to six.”

The Chinese economy is also opening up to the American movie producer, and could potentially double the addressable market, according to SAP‘s Livek. Big Data will be used to strategize in these emerging markets.“We can take cost out of the equation and make the industry more productive,” he said.

This story originally appeared on SAP Business Trends.

It’s not enough for an organization to simply have consistent data. It also must agree on how to use and deliver that data – preferably at a company-wide level. Without this agreement, it’s difficult to link strategic objectives with tactical actions such as divisional planning, budgeting, and reporting.

To realize the benefit of your updated architecture, you need to choose an information framework to structure and prioritize your data:

• Start by understanding your company’s standard information needs on a daily, monthly, or quarterly basis. Match the time horizon of your analysis to the temporal nature of your data.

• Select a framework that allows you to focus on the key performance indicators that matter most to your business. There are many standard pre-defined data frameworks based on industry, business role, or user. The key is to be consistent across the organization so that the business can more easily examine performance and make decisions in a unified way.

• Ensure that the design of your decision-making processes (including analytics content, navigation, alerts, and decision flows) align and drive to the outcomes that are important to the company. Most companies value results more highly than elegant processes.

• Consider your framework both from a top-down and a bottom-up perspective. This can help ensure that you don’t inundate decision makers with excessive or unnecessary metrics.

A consistent information framework is critical to providing transparency across organizational silos and hierarchies.

Access Actionable Information

Everyone in your organization should have timely information to create actionable insights, not just a few select individuals or a particular department. The proliferation of mobile devices and ability to harness large volumes of data make this an imperative. Business users need easy-to-use operational information to manage their day-to-day responsibilities.

The following principles can help you deliver the necessary information in the most appropriate format:

• Provide role-appropriate access to your information. The information available at each level should be actionable at that level and connected to overall company goals in a transparent way. Ensure that your standard metrics for performance management are readily accessible. This type of relevance and easy access promotes user interest and engagement with the data you present.

• Make information available in a visually appealing and easy to use format. Dashboards, with the ability to drill into detail, are a highly effective way to present user-relevant and actionable insights. The visual layout and navigation can be customized to reflect priority metrics and decision flows specific for the user, business, or industry.

• Use real-time delivery of information to enable rapid response to issues and to optimize business performance. Alerts can inform users of anomalies that require immediate action. Trend analysis identifies opportunities for innovation, resource reallocation, or strategic redirection.

• Allow users to create their own analytics in a self-service manner and make those analyses available to appropriate parts of the organization. Enable your information workers to be creative and social with their analysis.

Providing information that supports the strategic goals of the company—in formats and with devices that allow business users to easily analyze this information and share their ideas—can have a critical impact on organizational performance.

Is your company succeeding at delivering the right information to the right people?

Next week, I’ll discuss How to Get from Data to Decision—Enabling Real Time and Predictive Analysis. Stay tuned!

It starts with the three Vs of big data. Most of the discussion is centered around the first V, Volume. Finding the best way to store terabytes, or even exabytes, of data is a thorny question.

Historical data is just the first building block of a big data strategy. Having a strategy to deal with the velocity and the variety of your data are key to becoming a real-time enterprise. Thats where Complex Event Processing (CEP) comes in.

A CEP Engine combines information from a variety of sources. It looks for patterns in these event streams and then responds in real-time. You could have a number of streams of information coming from employees, machines, or customers. Tying all of this incoming data together allows you to find complex patterns across different channels. Once a pattern is recognized the system can trigger a response.

How is CEP being used?

CEP is already a mature technology in a few specific markets, but it is poised to cross the chasm into a number of data driven industries. So far the widest adoption is in the financial industry. There’s a whole breed of Wall Street traders that use finely tuned complex event processing algorithms to trade at high frequency and high volume.

This requires scraping thousands of news articles a day and comparing them to even small movements in price to find micro events. Traders then create rules to buy or sell based on which micro events the CEP engine registers.

The next use case might surprise you: chances are you are driving a complex event processing system to work everyday. Cars nowadays are amazingly engineered systems with a lot of sensors. When the speed sensor shows the car suddenly slowing down, but the brake pedal hasn’t been pressed, and a tire sensor is suddenly showing low pressure, the car can trigger an emergency procedure to increase traction control, tighten the seat belts, and deploy an airbag.

All of this can happen faster than you can react and hit the brakes. The combination of signals from the tire, the brakes, and the speedometer defines a complex event. The safety processes are the logic that gets triggered when a complex event is registered.

Where will Complex Event Processing show up next?

There is a huge opportunity for experts who can work with CEP systems to understand what signals are most important. The opportunities for CEP are far ranging. Some of the most promising areas of innovation are the utilities, oil and gas, retail, and discrete manufacturing industries.

This is a third post of a series based on a SAP-sponsored breakfast meeting organized in Sydney earlier this year with speaker Donald Feinberg, Gartner VP and Distinguished Analyst explaining the “Nexus of Forces”: social, mobile, cloud and information.

In the first two posts, Donald covered why in-memory is disrupting everything, and why every organization will be running in-memory in 15 to 20 years time, and the business impacts of the new in-memory computing possibilities.

In this post, Donald explains why you should have an in-memory action plan — and how to create one.

These comments are based on my notes taken from the speech, formatted for legibility.

Why you need an in-memory action plan

You need to change the way you look at IT infrastructure, applications, and the infrastructure that’s running those applications. Truly, with some of these new technologies like in-memory technology, there are no barriers, things that you can’t do. Words like “no we can’t do it” start to go away.

I’m not going to tell you it’s going to be cheap, I’m not going to tell you there’s not going to be bumps in the road as you’re doing it, but things that you really thought were not possible are possible now. Period.

What do you do in your organization to start to adopt or use some of the in-memory technologies? You are going to spend money on this. Whether the TCO is less or not, you still have to build your skills, you still have to buy applications, you still have to buy the technology and infrastructure and things like that.

Build a business case first. Show the value of what you’re going to do. The return on investment may be long or may be short. We recommend short at first. Small projects with quick return on investment will get you more projects that are bigger and have a greater impact on the company. But you have to prove it first – that’s the key.

Assign a small team of people to look at this. Most companies don’t have a research and development organization in IT (the big ones do). But there’s no reason you can’t have one person looking at the things that are possible with the new technologies, looking at how they can make your current applications more efficient, or start to change how you use them.

So set up a CTO or department of the CTO that has somebody in there who’s just looking at the stuff that’s out five years or ten years from now, so that you will be ready to start to do projects with it when it matures to the level of risk that you’re willing to take.

Always do a POC, proof of concept. Do not just assume that because it looks good on paper it’s going to work for you. You need to test it with your data, with your applications, with your people.

Brainstorming. A lot of people don’t realize that your business unit people are much more IT-aware than they have ever been before. Brainstorm with them on what some of these things can happen, in the business, and how they can make use of it. Who has the budget today? IT? Or the business unit? So if you don’t do this, they’re going to do it anyway, and they’re going to implement the technology without IT. The big disadvantage is that the company doesn’t get the broad skill base that is necessary, and that technology is not shared across the business units. It’s much better to keep it in IT, not because you order it so, but because you are moving along in these new ways, with the business units and what they need.

If you believe what I’m telling you about in-memory technology, as being part of your future, it’s not too early to start to define a strategy for how in-memory is going to enter into your organization and be used.

You may decide that part of the strategy is “we’re going to wait two years to let it mature”. That’s fine, but start looking now at where it can fit and when within the organization, so that you’re prepared and ready to accept it when it comes along. If you’re an early adopter, start tomorrow. If you’re more risk-adverse, next year, the year after.

But at least understand the strategy for how this is going to fit in your organization, because as we believe, it IS coming, whether you want it or not, so you may as well start now to look at a strategy for where it’s going to fit in the future.

Questions and Answers

What would you reply to somebody who said “I’ve already got enough problems in my organization already”?

From a short-term standpoint, I can’t disagree with that.

But some of the new architectures and the in-memory technologies can maybe help you with some of the issues that you have today.

It depends on what the issues are. One of the issues a lot of people have is speed: my applications don’t run fast enough. So maybe there’s in-memory technology that can speed that up. Or maybe moving it to mobile will make it run faster.

Looking at the nexus of forces and looking at technology as a solution to some of your problems may actually help you short-term.

Cloud – maybe cloud can save you some time. I’ll give you a simple example: how are your development costs? Use the public cloud for that. Let your developers develop on an Amazon AWS.

Why is that good? Your people don’t have to set up the development environment. You make a phone call, and you have it. When the project’s done, you make another phone call, and not pay for it any more. You don’t have to go out and buy a server that then you’ve got to figure out what to do with after the development project is done. So there’s a place where cloud immediately can help you.

So some of this new technology is mature enough to solve some of your problems. And then, when you start putting your head together with the business units and start to have an impact on the competitiveness and the bottom line of the organization, that’s where you can really make a difference. Some of this technology may enable you, if you’re a retailer, to turn your inventory one more time a year. Is there any retailer that doesn’t want to do that? And not be out of something when somebody wants it?

If the business unit wants to be an early adopter, but the IT unit is risk-averse and conservative, how does the business user drive this change?

I’ve been around a while in this business. If there’s one thing I’ve heard over and over since I started in the 60s, it’s “IT has to communicate with the business”.  We’ve learned that lesson – that doesn’t work. Going out to dinner with your business liaison once a month and talking with them is nothing.

So one of the concepts that we came up with around twelve years ago, with respect to BI specifically, is the BI Competency Center. The reason that has worked is because it takes business people and IT people and puts them together, working together, not talking. So they make decisions together.

If I’m going to do a new project, all the business units decide what the priority project is. This is a concept that works. Some of your companies can’t afford to have full-time people in it, so you do it virtually: you have a meeting once a week. But they still manage projects, they still make buying decisions on products, they still set strategy for the company. The group should not be run by IT (which is hard to swallow sometimes) – but by the business unit. And most important: the CIO can not be the sponsor. It must be higher in the organization.

So if I’m going to have a “business technology competency center” where people from the business and the industry are going to get together to look at new technologies and where they may work, the sponsor has to be the CFO, the CEO, the Board, somebody like that. Then they will work together to do this.

Risk-adverse IT organizations are normal. You have a job to do to keep the lights on and you’re not going to do it if you take risks. It’s that simple – you’re not going to have a job if you take risks.

So how do you fit that with adopting new technology? Again, just like with the research and development with one person, you can take a couple of people from your organization as part of this “business innovation competency center”, sponsored by the CEO, so you can go hire some new people to do it if you need to, or move people over and backfill them.

They may take on a project with a business unit where you see tremendous value to the business, and you look at something that is, say, in beta. And you look at that technology to enable that business unit to be more competitive, more productive, more profitable, and it doesn’t affect the rest of your organization. You still can deliver the things you’re doing, because it’s “outside”. How do you get to that? You have to get senior management in the organization – not the IT organization – behind you. How do you get that? A small project, to demonstrate to them the value of this kind of thing.

Now one thing that comes to mind immediately: if you look at what’s happened in the past ten years with data warehousing – my area – every time there’s been a recession, database sales and data warehousing sales have dropped off. Except in the 2008/2009 worldwide recession, where every segment of IT was negative growth except DBMS, which was flat. In that environment, flat was positive.

Why? Because when the CIO came in to the CEO and said “I need more money to spend on my data warehouse” and the CEO says “are you nuts, with this economy?!”, you pointed to a flat screen on his wall that had key metrics of the business in “real-time” – for the first time, senior management, the CFO, COO, CEO, could physically see the value that information was bringing to their business.

If you can demonstrate physically to them the advantages of some new technology, then they’re going to buy into it and start to fund it. You can’t say something like “I want a new ERP package” – in an economy like 2009, that will get you fired for asking. But if you have some real strong value that you can demonstrate quickly or instantly to them, they’re going to spend money on it if they think it’s going to save money or help them. So that’s what you have to do. Lots of people say “only large companies can afford that” – but anybody can put it together with at couple of visionary people from the business units and one or two people from IT to put this together, and they can be virtual.

Pfizer is one of our BICC case studies. They have 150 people full time in the BI competency center: 75 employees and 75 consultants. Most people can’t afford to do that, and I’m not suggesting you do. But here are models in-between that make sense, that will fit in everybody’s budget.

Taken to its full extent, the technology also offers the potential to help companies uncover new growth opportunities before competitors, rethink business processes on the fly, and empower people to decide and act with real-time insights.

For example, companies may gain insights into more granular customer segments and new formats of data such as machine sensors and social data. At the same time, the technology is enabling businesses to perform predictive analytics that help shift monetization models from purely product-driven to service-driven models based on unique customer preferences.

To better understand current perceptions and expectations for the new technology in enterprise environments – as well as the interaction with other technology megatrends such as cloud computing and mobility – SAP has partnered with Knowledge@Wharton, the online management journal from the University of Pennsylvania’s business school, to launch a brief, executive-level survey of the key issues.   Topics covered include:

• Current pain points regarding access to real-time business information
• Opportunities for breakthrough innovation
• And outlook for the future

All participants will receive a full report on the findings, including industry comparisons, and will be entered into a drawing for a free 7” Kindle Fire HD. 

TAKE THE SURVEY

https://valuemanagement.sap.com/Sapbenchmarking_Portal.html#ID=316

Note to readers: SAP’s Performance Benchmarking program is a strategic service sponsored by its Value Engineering organization. Originally launched in 2004 together with ASUG as a forum to exchange metrics and best practices, the program today has grown into a global effort and one of the largest such programs in the industry—with more than 14,000 participants from more than 4,000 companies and studies available in 12 languages. Participants receive—free of charge—customized and confidential benchmarking comparisons against industry peers as well as aggregate analyses. To participate in the SAP benchmarking program, visit https://valuemanagement.sap.com.

There’s no doubt that big data is playing an increasingly big role in business, politics, healthcare, education, retail and numerous other industries. With the right tools and expertise, organizations can slice and dice data to reveal trends and other information that will inform decisions about future strategy and direction.

“[Big data] is the idea that we are finally able to do with a huge body of data things that were impossible to achieve when working with smaller amounts, to uncover to new insight and create new forms of economic value,” said Kenneth Cukier, co-author with Viktor Mayer-Schönberger of “Big Data: A Revolution That Will Transform How We Live, Work, and Think,” and data editor of The Economist. “So, for example, the reason we have self-driving cars or very good computer translation is not because of improvements in processors or algorithms, though they are useful, but because we have vastly more data from which the computer can calculate the probability that a traffic light is red and not green, or which a word in one language is a suitable substitute for a word in another.”

During the 2012 election, big data was used to great advantage—namely, by Barack Obama, in his re-election campaign.

A November 2012 article in Time magazine, “Inside the Secret World of the Data Crunchers Who Helped Obama Win, reported, “Data-driven decision making played a huge role in creating a second term for the 44th President and will be one of the more closely studied elements of the 2012 cycle. … In politics, the era of big data has arrived.”

Cukier said campaigns have always been run on information, but in 2012 big data helped optimize that information and activities around it.

“Obama’s data scientists made pioneering innovations on applying big data to politics,” said Cukier. ”They learned through testing what the optimal sum was when requesting a donation. Every piece of promotional literature online and offline is tested before it goes live at scale. They micro target down to subgroups of the population that were otherwise not picked up by campaigns in the past because it was hard to get granular information on those groups and what moved them to vote a certain way. As a result, campaigns try to tailor their activities down the seemingly ‘individual’ level — not  broad, lumpy categories like the ‘soccer moms’ used in Clinton’s campaigns in the 1990s.”

The role of big data will only increase in the 2016 presidential election, with practitioners using technology to hone in on data much more granularly than ever before. In addition, say experts, the use of big data will overlap with social media and other platforms.

“For the 2016 presidential election, we can expect big data to play a much more central role than ever before,” said Cukier. “[We’ll see] micro-targeting of individuals and narrow subgroups of the population, tailored messages over social media platforms, instant feedback loops to the campaign about what works and what doesn’t to move voters, and data collected from the private sector to build predictive models of what works best for fundraising, activating the base of supporters and the get-out-the-vote activities.”

Pay attention to some of the hotly contested issues and the preparation for 2014 Senate elections, and you will see big data at work and a preview of what’s to come in 2016.

“There will be data testing in the run-up to 2014,” said Steve Parkhurst, political consultant, GPH-Consulting. “There have already been debates this year on issues like the sequester, the Second Amendment and immigration, for example, and I’m sure the big data databases are filling up with usable data. The data drill-down is no doubt tracking where voters are looking and what those voters are saying, especially online.”

All of this is not to say that the process will be easy. Analysis of big data is a highly complex task, and at this point in time it requires a very specialized—and often expensive–skill set.

“Not just everyone can set up a big data shop,” said Parkhurst. “The people who are really good at it are going to charge a premium price.”

Luckily, technology providers are heeding the call for products and services that can tame the big data beast. Indeed, said Cukier, two of the biggest barriers to big data are narrow thinking and a dearth of leadership.

“The only real obstacle is one of mindset,” said Cukier. “We need to think creatively about what we can do with the data and how important it is. It takes both ingenuity and leadership.”

This is the second post based on a SAP-sponsored breakfast meeting organized in Sydney earlier this year, as part of a ANZ/APJ innovation analytics tour, with speaker Donald Feinberg, Gartner VP and Distinguished Analyst explaining the “Nexus of Forces”: social, mobile, cloud and information.

After covering why in-memory is disrupting everything, and why every organization will be running in-memory in 15 to 20 years time, Donald went on to look at the business impacts of the new in-memory computing possibilities.

These comments are based on my notes taken from the speech, formatted for legibility.

Business impact of in-memory computing

What is the impact of in-memory computing on your business? It’s about running the business, growing the business, and transforming the business, and you need to look at the business impact of this technology across all of these.

Run the business

One of the biggest advantages of memory that people forget is this: right now, you have lots of applications. And today, people typically have one application per server. Let’s say you have your corporate running on ten servers today, and it’s spread out across locations, because of storage access and the speed of the processors and the speed of the applications and the database access.

If I can consolidate that down to a single server, I‘m going to save a lot of money, right off the bat. Not only power, floor space, cooling, but replacement costs every three to four years for ten or twenty servers is more than one. It’s not necessarily a single server — it may be one or two — but it’s going to be much fewer.

The people required to maintain it are going to be fewer, your maintenance costs per year are going to be less, everything is less. So the speed of these in-memory technologies on just running your business – forget about transforming for a minute – is going to be a huge savings. Because if one applications runs a hundred times faster on a server, I can get more applications on that server.

When I said you you’re going to run your whole business in-memory in 10 or 15 years, I left off the fact that it’s going to be on a single server the size of what you think of as a desktop server, plugged into the wall with no special air conditioning needs. That’s the kind of miniaturization and speed that in-memory is bringing to the table, with huge savings.

I know many of you are saying “he’s not talking about high availability or disaster recovery”. All of that is coming — and it also is miniaturized. You’re not going to run your business on one of these, you’re going to run your business on two of them, sitting next to each other, duplicating everything it does, synchronously. That’s your high availability. Then you’ll put another one somewhere else, in somebody’s home, 250 or 800 kilometers away, and that’s your disaster recovery center. You hire a disaster recovery manager in Perth, and put the disaster recovery in his house — that’s the way it will be in the future.

Transform the business

The latency with in-memory is so low that you can do things synchronously that you wouldn’t have thought to do synchronously before. It’s not only a matter of how many things you can do, and how much you can fit into this box because of the speed, but it’s also because of what the latency is going to give you.

Why is that important? Think about where information and mobile and social come together, and you need to do messaging and things like that. Because of this lower latency, I can start to do things I couldn’t even consider before, because I couldn’t get it fast enough to even think about it.

How many of you may have applications that you thought about doing, but because things took so long on your system, it’s just not reasonable to do? I’m not talking about the demonstrable ones like if you’re in the manufacturing business, your MRP run takes four to six hours overnight, now you can run it in five seconds. So you can use the application differently.

And other things that you couldn’t do at all now become possible. As we start to do sentiment analysis, looking at social networks, and building it into a planning application that I’m running in seconds, that’s huge in the way you can change your business.

Think about if somebody says to you “I want to buy 10,000 cases” and you don’t even know if you can produce that. And then he says “I want it next week.”

How long does it take your company to commit to that, and to figure out a price, that may in fact be higher because I’m going to bounce other customers off the production line in order to get this done? If you can do that with a latency in seconds, it changes the way you do business.

That now is getting into “transform the business” because an application that you view as “a forecasting package that I run overnight” is not a just a forecasting package if I can run it in five seconds or two minutes. It becomes a sales tool, changing the way I’m doing business.

The example that I like to use is this: airlines want to sell you discount tickets. Most people don’t know that airlines re-price all the tickets on all their planes every night. So your company goes and buys a full-fare ticket because you need it refundable.

The next day, that flight may have two more discount tickets because they have a yield that they need for each plane, for each flight, so they can actually go through a whole calculation that tells them how many discount tickets they can have. Now, why is this valuable to them? Well, if you get on to, say, Qantas today and say “I want to go to Singapore and I want a discount ticket” and there are none on the day of the flight that you want, most of you wait until tomorrow to see if there are any, right?” Not true – most people don’t even know that happens. Instead, what you’re going to do is switch over to Singapore Airlines and if they have a ticket, you’re going to buy it and Qantas just lost the revenue.

But if Qantas could re-price every seat on every plane every time a ticket was sold, that business wouldn’t go away. If you had an application like that, which in-memory will allow you to do, and you went to the CEO of the airline and said “we have this application, do you want it?”, how much do you think they would be willing to pay? I’ll tell you — they won’t even ask how much it costs. That’s how much it transforms their business, and changes what they do. They’ll pay whatever you want.

In-memory computing technologies

So far, we’ve been talking just about in-memory DBMS. Here are some of the other ways the technology is used.

In-memory data grids have been around a long time. If any of you do web applications, you may be using some them. Memcached is the one that comes to mind – an open-source product – where your data’s in memory, in the application, and scales across multiple computers, multiple servers. That technology’s been around a long time and enables some of the biggest web applications that you’re all using, including Amazon, including eBay, and all the spinoffs of those.

High-performance messaging infrastructure. Think about what happens if you want to send a message out to four or five thousand of your customers at a time. It’s an SMS message or whatever, in-memory’s going to be able to do that much quicker.

Wouldn’t it be nice if you’re an airline, and you’re cancelling a flight, to get those messages out quickly? Or, in retail, if you’re going to have a special pricing discount, you’re going to send out to all the customers registered on your site, and you’re a big retailer with one hundred thousand or a million customers, think about how high-performance messaging is going to happen.

Complex event processing. That’s what fraud detection is all about, especially for cloned cell phones, for trading fraud, for credit-card fraud, for anything where some analysis is taking place on streaming data coming into a computer and in real-time. I make a decision on an event that’s happening, and then do something about it.

In-memory application servers. These are necessary if you’re going to do this consolidation onto a single or double box of all your applications. Your application servers have to be in-memory, and they can’t be based on disk drives, or they’re not going to run as fast as all the other technology that is enabled with the applications running in the application server.

All of these together make up “in-memory technologies”. The providers of this technology are going to merge together and all of this is going to become an in-memory megadata platform over the next three to five years. Data grids are going to go away and just become part of the in-memory database. These two will be the first to merge, and they’re merging already with in-memory analytic applications and application servers.

That’s the future, as they merge together, which will enable you to run your whole business in memory.

Drivers of in-memory computing

So what drives all this? Well, big data. Now remember “big data” is not just about volume. When we mention big data with respect to in-memory, people think we’re crazy, because big data is a lot of data, and people say “I’m not going to put a petabyte in memory: it’s too expensive!”

“Big Data” is volume (big size) and/or velocity(how fast the data’s coming in) and/or the variety of data(unstructured data). In-memory can support velocity today, that’s one the first use case of it, high-speed data coming in through event processing, smart metering, etc. And it can support unstructured data. As the price comes down, as compression gets better, it’ll also get start to get larger and larger on volume of data.

Real-time analytics. For years, Gartner has said there is no such thing as “real-time.” Today, you are running analytics on data that is coming from a transaction system. If I have to say it that way, there’s a latency there. Some ETL or data integration process has to move data from the transaction system to the data warehouse before you can do those analytics. The only way you can do real-time analytics is if it’s being done on the transaction data when it’s completed. So that is one of the drivers for this.

24×7 with no batch windows. If you batch window drops to less than zero, you’re going to have to run things very quickly. Batch is going away. That Materials Requirement Planning batch run that takes six hours? If it starts to run in 3-4 seconds, it’s really no longer batch.

So the whole concept of batch disappears with in-memory technology. Any time you see words like “awareness” then you’re talking about in-memory. In order to make any applications aware of things it means real time, and it means you need the speed and low latency of in-memory technology to do it.

Inhibitors of in-memory computing adoption

So what’s slowing us down?

A lot of these are perceptions. So the perception that it’s a complex architecture: it doesn’t have to be.

The perception that it’s unrealistic: today, this technology is emerging, and yes, it’s disruptive, but no, you can’t do everything with it. So the expectations have to be set right. There are of course no standards, there aren’t a lot of skills and there’s not a lot of best practices yet, because this is just emerging with those. That will happen over the next few years.

So yes, there are many drivers, but at the same time there are many inhibitors, a lot of which you can change by setting expectations and perceptions correctly. So you start to think about IT looking at all this data and saying “what do I do with it all?” and the bottom line is: if your assumptions are that you can’t do anything with it, you’re not going to do anything with it.

[Next post on “creating an in-memory action plan” coming soon. In the meantime, if you're interested in hearing Donald Feinberg talk about this, a web seminar is available (registration required)]

Share and Enjoy

• Facebook • Twitter • Delicious • LinkedIn • StumbleUpon • Add to favorites • Email • RSS

Gartner’s stats are impressive indeed. But if you really need confirmation that Big Data analytics is going mainstream, look what’s happening in baseball. According to a recent article in The New York Times, as more Major League Baseball teams use data to make team-building decisions, they are leaning on sportscasters to help audiences better understand what the data means.

Big Data and Baseball

Robert Ford, 33, who was hired by the Houston Astros in the off-season to call games with Steve Sparks, explained that the team “wanted a broadcaster who is at least comfortable with exploring the idea of discussing advanced statistics and what they mean.”

“We need (sportscasters) to tell the story of how we are making decisions and putting the organization together,” George Postolos, the Houston Astros’ president and chief executive, told the newspaper.

For example, why was Ben Zobrist added to the team’s lineup? As the pre-game notes to the media explained, Zobrist has a high WAR (wins above replacement) ranking, which is a measure of a player’s offensive and defensive contributions relative to others who play his position and could replace him.

Apparently, all of the Big Data details are not going over the heads of the fans. For millions of them – especially fantasy league enthusiasts – stats like WAR, VORP (value over replacement player), and BABIP (batting average on balls in play) are important numbers. Like real team management, fantasy team owners give these and other stats a lot of weight when selecting players.

Fan interest in player data is so strong that providing this information is now considered part of the fan experience at ballparks. For example, at Yankee Stadium kiosks provide fans with the opportunity to play fantasy baseball scenarios utilizing over 100 years of team data rendered with the help of SAP software.

If the guys up the radio booth are going to be talking Sabermetrics, –the analysis of baseball through statistics that measure in-game activity – might Gartner be understating how many businesses will invest in Big Data?

According to the report, one of the reasons why companies are taking a harder look at Big Data is in response to the increasing media coverage of it. “This makes IT and business leaders worry that they are behind competitors in launching their big data initiatives,” Frank Buytendijk, research vice president at Gartner.

If fantasy league owners fantasy team managers are crunching numbers to select a utility infielder, it’s a good bet that business leaders will take a closer look at Big Data too.

Consumers want and expect relevance, and companies know that big data analysis can make their marketing and sales be relevant.

Consumer Data is the New Currency

Consumers trade their personal data for discounts, relevant marketing, and will basically sell their souls to the devil for perks. Okay, maybe consumers wouldn’t sell their souls for perks and marketers collecting the data aren’t devils, but there is a gray area and level of comfort that consumers feel when sharing their data in exchange for benefits.

Companies need to recognize and respect this gray area – and draw a solid line that is not crossed when marketing.

Now, this isn’t something new. It’s definitely an issue that has been around for ages, but  since companies are able to collect data through monitoring behaviors and tracking online actions, consumers are not always privy to what data is being collected and when they are actually sending their data to companies for analysis.

Since these valuable online actions provide big data and help marketers better understand what consumers want, how they act, and how to motivate, this data is essential in running a successful business.

So, let’s find that solid line in collecting big data and use it.

Marketing for Relevance Without Being Creepy

Today’s consumers are smart enough to know that companies know plenty about their habits, experiences, and preferences – and they expect companies to use that data to market appropriately. As much as consumers don’t love giving up their data, they will if it means a better experience. So use the big data to your advantage, but follow these three rules:

1. Let consumers know what is being collected and when it’s happening. Although you don’t have to do this all the time, figure out a nice way to ask permission for consumer information.

This rule reminds me of the day in 2010 when I went shoe shopping at Zappos.com (love them) and clicked on a few pairs. I left the site without making a purchase and when I went onto another site, I saw the same shoes pop-up in the Zappos ad on the right side of my screen. Not cool. Creepy, so don’t do that.

While I love Zappos and the strategy of targeting me and showing the products I just viewed, I don’t love the cookies on my browser that I didn’t know were installed. Simply let your consumers know what is happening on the site and consider sending emails for items left in their shopping cart (Victoria’s Secret does this in a sensitive way), rather than “following” them around on the Internet dangling the carrot in front of their noses.

2. Don’t trick them. Consumers don’t love receiving emails that are relevant when they don’t know how they ended up in the inbox. Even though a consumer might be interested in the latest tech toys that will help them save more money and be more successful, if they aren’t sure how you received their email address and ended up in the “to” field, they are more likely to question the “why” of the information rather than receive the value of the content.

3. Be open about opt-in and set expectations. When a consumer opts-in, many things happen. The first is that the company has a more qualified lead because that consumer specifically said that they wanted in. The second is that the consumer gets what they want. And the third, is that a relationship begins between the consumer and the company.

To keep this relationship growing, be sure that expectations are set during the opt-in process. Let consumers know what is happening – you are collecting their personal information to send them targeted emails once a week with messages about sales and project ideas. If that is what a company presents and delivers, it is a two-way street to mutual happiness.