The ElastiCube Chronicles

Google Says: Americans More Patriotic Since 9/11

2011-09-11T05:34:00.000-07:00

America is a patriotic nation. Notwithstanding the definite upsurge in the period following 9/11/2001 ten years ago, overall the level of American patriotism tends to remain remarkably consistent. Polls support this statement, as reported in a Forbes article last year:

Patriotic attitudes are generally very stable. In a question Gallup asked in January 2001, 87% said they were "extremely" or "very" proud to be American. When Pew repeated the identical question last year, 86% gave that response. In 2001 and 2009, only 1% said they were "not at all proud." The 9/11 tragedy produced more overt displays of patriotism and heightened sentiment, but responses soon returned to the norm.

So how did 9/11 impact American patriotism? Back in 2005, an MSNBC article reported the results of a poll conducted by the Roper Reports unit of NOP World. NOP’s vice president of consumer trends was quoted as saying:

“We tracked patriotism, spirituality and religion, and giving to charities and volunteerism right after 9/11,” Silvers said. “All three popped up. Within about nine months, volunteering was down and so was religion, but what has stayed with us is patriotism, and it’s obviously fueled by a couple of things. The shift point was 9/11.”

As are most of the folks working at my BI software development company (SiSense), I could be called obsessed when it comes to applying business intelligence methodologies to anything. In this spirit, I automatically started thinking about how I might be able to uncover some trends showing if and how actual data might support the articles and surveys I saw about American patriotism and 9/11, as the ten-year anniversary of the 9/11 tragedies approached.

I decided to take a look at Google search data and see if and how online searches reflect the “patriotism effects” of 9/11 in America. After looking around and reading some more, I came up with two searches to check out: flag sales and volunteering.

What led me to check out flag sales? I saw a recent article that says that, “One response to the catastrophic events of Sept. 11, 2001, came quickly in a traditionally American way -- flag sales soared.” The article quotes the president of Gettysburg Flag Works in New York State as saying, “‘On 9/11, 12, 13, 14, we sold out everything that was red, white and blue. We had lines out the door,’ recalled Mike Cronin, president of Gettysburg Flag Works. “I was in flag shock for a couple of weeks.’”

Not surprisingly, a sharp increase in searches related to flag sales can be seen in Google search data in the month of September 2001. For the years following 2001, however, searches for flag sales were relatively flat.

Flags Sales 2001

Flags Sales 2007

Interestingly, though, this year’s tenth anniversary of the 9/11 attacks seems to be resonating more with Americans than on previous anniversaries: flag sale searches increased significantly over the past few weeks.

Flags Sales 2011

When I went looking for articles which might discuss the increase in community service and volunteering among Americans, I didn’t find much. However, a look at Google search data for “volunteer USA” shows an interesting consistency with what I saw for flag sales searches: significant spikes in searches after the original 9/11 and then again this year.

US Volunteers 2001

US Volunteers 2009

US Volunteers 2011

From the graphs above it is quite clear that the 10th anniversary of 9/11 has a great impact on patriotism, even more so than years before.

In any case, it’s hard to believe that ten years have passed since thousands of Americans died on 9/11. Hopefully, America is stronger as a country for it and its people are stronger as a nation. In case you feel like reading more about how patriotism has increased in the US since 9/11/01, I leave you with the following article, published this week in a Florida newspaper, which talks about American patriotism (and the difficulty in measuring it with surveys): Since 9/11: Patriotism - Ten years after the deadliest terrorist attack inside the United States, love of country remains as strong as ever, if not a little stronger.

By: Roi Hildesheimer | The ElastiCube Chronicles - Business Intelligence Blog

The In-Memory Technologies Behind Business Intelligence Software

2011-08-25T06:41:00.000-07:00

If you follow trends in the business intelligence (BI) space, you’ll notice that many analysts, independent bloggers and BI vendors talk about in-memory technology.

There are technical differences that separate one in-memory technology from another, some of which are listed on Boris Evelson’s blog.

Some of the items on Boris’ list are just as applicable to BI technologies that are not in-memory (‘Incremental updates’, for example), but there is one item that merits much deeper discussion. Boris calls this characteristic ‘Memory Swapping’ and describes it as, What the (BI) vendor’s approach is for handling models that are larger than what fits into a single memory space.

Understanding Memory Swapping

The fundamental idea of in-memory BI technology is the ability to perform real-time calculations without having to perform slow disk operations during the execution of a query. For more details on this, visit my article describing how in-memory technology works.

Obviously, in order to perform calculations on data completely in memory, all the relevant data must reside in memory, i.e., in the computer’s RAM. So the questions are: 1) how does the data get there? and 2) how long does it stay there?

These are probably the most important aspects of in-memory technology, as they have great implications on the BI solution as a whole.

Pure In-Memory Technology

Pure in-memory technologies are the class of in-memory technologies that load the entire data model into RAM before a single query can be executed by users. An example of a BI product which utilizes such a technology is QlikView.

QlikView’s technology is described as “associative technology.” That is a fancy way of saying that QlikView uses a simple tabular data model which is stored entirely in memory. For QlikView, much like any other pure in-memory technology, compression is very important. Compressing the data well makes it possible to hold more data inside a fixed amount of RAM.

Pure in-memory technologies which do not compress the data they store in memory are usually quite useless for BI. They either handle amounts of data too small to extract interesting information from, or they break too often.

With or without compression, the fact remains that pure in-memory BI solutions become useless when RAM runs out for the entire data model, even if you're only looking to work with limited portions of it at any one time.

Just-In-Time In-Memory Technology

Just-In-Time In-Memory (or JIT In-Memory) technology only loads the portion of the data into RAM required for a particular query, on demand. An example of a BI product which utilizes this type of technology is SiSense.

Note: The term JIT is borrowed from Just-In-Time compilation, which is a method to improve the runtime performance of computer programs.

JIT in-memory technology involves a smart caching engine that loads selected data into RAM and releases it according to usage patterns.

This approach has obvious advantages:

1. You have access to far more data than can fit in RAM at any one time
2. It is easier to have a shared cache for multiple users
3. It is easier to build solutions that are distributed across several machines

However, since JIT In-Memory loads data on demand, an obvious question arises: Won't the disk reads introduce unbearable performance issues?

The answer would be yes, if the data model used is tabular (as they are in RDBMSs such as SQL Server and Oracle, or pure in-memory technologies such as QlikView), but scalable JIT In-Memory solutions rely on a columnar database instead of a tabular database.

This fundamental ability of columnar databases to access only particular fields, or parts of fields, is what makes JIT In-Memory so powerful. In fact, the impact of columnar database technology on in-memory technology is so great, that many confuse the two.

The combination of JIT In-Memory technology and a columnar database structure delivers the performance of pure in-memory BI technology with the scalability of disk-based models, and is thus an ideal technological basis for large-scale and/or rapidly-growing BI data stores.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Putting The Costs of BI In Perspective

2011-08-09T04:16:00.000-07:00

Successful business intelligence (BI) solutions serve as many business users as possible. As more users use it, the more value the solution brings.

However, if you’ve had any experience with BI, you must have noticed that as the number of users grow – so does the complexity (and consequent cost) of the solution. This is a fundamental reality in the traditional business intelligence space, although many startups in the space are attempting to change it – each according to their own vision and understanding of the space.

But why is buying a BI solution for dozens or hundreds of users so much more complicated than buying a solution for a select group of power users?

Perspective #1: The Cost of Software Licenses

People often think that the answer to this question lies in software costs, but in fact software costs are usually the red herring in the process of business intelligence costing.

It is obvious that the more users your solution has the more software licenses are going to cost. Therefore, you might be tempted to choose a vendor that sells software for 30% less than another vendor – but basing a decision solely on this is a big mistake as license costs have little bearing on the total cost of a BI solution, and hardly any impact on ROI.

Some proof to this can be found in open source. Open source BI provides (by definition) free software, and there is no shortage of open source BI tools/platforms. However, none of them are doing as well as the established non-open source vendors, even though they have been around since the beginning of the century. They’re having trouble acquiring customers, at least compared to commercial vendors. It is very easy to assume that if software costs were significant inhibitors in the BI space, open source solutions would be much more prominent than they actually are.

Another hint at this can be found in the ‘commercial’ (non-open source) world, where BI vendors do charge for licenses but will usually provide significant discounts on purchasing of large volumes of licenses. BI vendors do it for reasons that go beyond the obvious attempt to motivate potential buyers to expand their purchase orders. They do it because they realize the total cost of the solution – to the customer – grows significantly as the number of users grows, regardless of license costs (preparation projects, IT personnel assignment, etc). They need to take this into account when they price their software.

Tip: Pay attention to software costs, but there are way more important things to consider. You should really leave the license cost comparison to last.

Perspective #2: The Cost of Hardware

Two things that have great impact on the hardware requirements of a BI solution are the amounts of data being queried directly by business users, and the number of business users doing the querying concurrently. Depending on which technology you use, each user can add between 10%-50% to the configuration of hardware resources required (disk, RAM and CPU).

(For you technology geeks out there, there is an interesting discussion about this topic on Curt Monash’s blog. Check out the comments section, as it will also give you a good idea on what hardware configurations can be used, when different technologies are utilized)

The tipping point, however, is when your requirements grow beyond what can be fitted inside a single commodity hardware box (read: cheap off-the-shelf computer). If this limit is hit, you basically have three options, none of which are practical for most companies:

1. Buy a high-end proprietary server
2. Clustering / sharding
3. Build a data warehouse / pre-processed OLAP cubes

Unfortunately, BI technologies that were designed prior to the 21st century (RDBMS, OLAP, In-Memory Databases) don’t leave much room for innovation on this particular aspect. They were designed for hardware that was different than what exists today. So while there will always be a limit on what can be achieved with a single hardware box, with traditional BI technologies the threshold is too low to be feasible for most modern companies that both have large volumes of data and seek extensive usage at reasonable and consistent response times.

The good news is that this is not the case with new technologies that are designed specifically to utilize the modern chipsets that are available on any commodity 64-bit machine, and therefore get more (orders of magnitude more) juice out of a single 64-bit commodity box. Running dozens or hundreds of users on a single box is more than possible these days, even when data is in the 100s of GBs size range.

Tip: If you do not wish to spend loads of money on high-end or proprietary servers, and your internal IT department has better things to than to manage a cluster for BI, you should really give preference to technologies that would allow you to set up your BI solution on a single commodity box.

Perspective #3: The Cost of Starting Too Big… or Too Small

After talking to business managers, executives and other stakeholders, you’ve determined that this BI solution you’re considering has the potential of serving 100 users. How would you then go about calculating your project costs? This is where things get tricky, and where most BI buyers fail to protect their wallets. Making the wrong decision here is far more significant than any decision you make on software licenses or even hardware.

Even if the development stage of your BI project goes without a hitch, getting a hundred users to use any kind of software, in any company, is a challenge that is not at all easier than any technical challenge you will encounter during the various stages of the project. You could easily find yourself spending tons of money on the development and deployment of a complicated 100 user solution, only to find that only 15 of them are actually using it.

So instead of your total cost per user being reduced due to the ‘volume-pricing’ model, you actually paid much more – because each one of these 15 users absorbs the cost of the 85 others who find it utterly useless, too difficult to use or completely misaligned with their business objectives. You'd be surprised how often this happens.

The obvious way of dealing with this common problem is to start off small (10-20 users), and expand as usage of the system grows (assuming it will). But when it comes to traditional business intelligence solutions, there’s a catch - deploying a solution for 10-20 users and deploying a solution for 100 users are utterly different tasks and require significant changes in solution architecture.

Following this path will save you some cost on the software licenses you did not purchase straight off. However, if demand for the solution grows inside the business, you will have to re-design your solution – which would probably end up costing more than it would have initially.

Tip: The correct way of dealing with this challenge is to seek a solution that scales without having to re-architect the solution as usage grows. Buying more software and upgrading hardware when the time comes is relatively easy and inexpensive, while rebuilding the entire solution from scratch every year or two costs way more.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

SiSense Welcomes Crispin Read as Executive Chairman

2011-07-20T07:38:00.000-07:00

I am excited to share with you all that Opus Capital's Crispin Read, former General Manager of ERP solutions at Microsoft, has joined SiSense as executive chairman. We are all looking forward to working with him!

PALO ALTO, Calif.--(BUSINESSWIRE)-- SiSense, a provider of leading-edge business intelligence (BI) solutions, today announced that Crispin Read has joined the board of directors of SiSense as Executive Chairman, effective immediately.

“We are delighted to welcome Crispin as Executive Chairman,” says Elad Israeli, Co-Founder and Chief Executive Officer of SiSense. “Crispin’s experience and track record at Business Objects, Cartesis and Microsoft, give him the perfect background to support SiSense as we continue to execute our ambitious global growth strategy.”

With over 20 years of experience in business intelligence and business applications marketing, Crispin Read currently serves as Vice President of Business Intelligence Investments at Opus Capital. Read joined Opus Capital from Microsoft Corp., where he served as General Manager of Microsoft’s ERP business, leading product management and marketing for Microsoft Corp.’s portfolio of enterprise resource planning (ERP) applications. In this role he was responsible for ERP strategy and execution worldwide. Prior to Microsoft, he was Chief Marketing Officer at Cartesis SA, a venture-backed financial performance management company that was acquired by Business Objects in 2007. From 1995 to 2004, Crispin served in various roles at Business Objects, including Vice President of Product Management and Vice President of Product Marketing. During this period Business Objects grew revenues from $30M to $930M.

“SiSense has absolutely amazing technologies that vastly reduce the cost and complexity of BI deployments. As a result, countless companies and departments will now be able to enjoy previously unobtainable benefits of business intelligence”, says Crispin Read. “I’m excited to join the talented and visionary SiSense team on this journey.”

About SiSense
SiSense is leading the way to a new era of business intelligence, where fast and flexible BI is available to any business user analyzing any type or amount of data. Based on advanced technologies developed during the course of five years, SiSense delivers an end-to-end BI solution that allows organizations to mash-up a variety of data sources containing mountains of data into server-based interactive dashboards and reports in less than a day, thus securing immediate ROI. SiSense customers enjoy very high performance do-it-yourself BI, without IT projects, data warehouse systems, OLAP cubes or programming. For more information and a free trial download of the software, visit www.sisense.com.

The ElastiCube Chronicles - Business Intelligence Blog

Columnar RDBMS, Gourmet Fast Food and Santa Claus

2011-06-18T08:51:00.000-07:00

Boris Evelson of Forrester recently published a blog post titled It's The Dawning Of The Age Of BI DBMS (Database Management System). I took note that in this post he classified Vertica, ParAccel and Sybase IQ in a category he named ‘Columnar RDBMS (Relational DBMS)’, and that started off a friendly email exchange as to what the heck that really means.

I said: “RDBMS is tabular, by definition.”

...and Boris said: “To me if I can access something via SQL, it’s relational.”

Who’s right is a matter of perspective, I suppose. But technically, defining RDBMS by the existence of SQL access is incorrect. According to Wikipedia, the short definition of an RDBMS is a DBMS (Database Management System) in which data is stored in tables and the relationships among the data are also stored in tables. The data can be accessed or reassembled in many different ways without having to change the table forms.

Note that the word 'table' appears three times in the short definition of the term. SQL does not appear anywhere in the definition.

What's worse, columnar databases do not store data in tables - again, by definition. So, come to think of it, how can such a thing as a Columnar RDBMS even exist? (get the title yet?)

I suppose that the only thing Columnar RDBMS could technically describe is a database system that stores data in a column-oriented manner, yet still relies on the fundamental mechanisms of an RDBMS (tables, SQL, indexing, etc). In practicality, this means that each field is stored in its own table, with an additional field for correlation. But that is a side-ways implementation technique that is mainly practical for somewhat extending the lifetime of existing software assets that are reaching their scalability limits, and hardly deserves its own DBMS category.

I know we wouldn't describe ElastiCube as being an RDBMS (even though it supports SQL access) and I'm pretty sure Vertica wouldn't describe their technology as an RDBMS either.

The similarities between ElastiCube, Vertica and RDBMS are sufficiently described within the 4 letters D-B-M-S. The letter R is what differentiates between them.

SiSense refers to ElastiCube as a Columnar DBMS or Column-oriented DBMS and I think this describes Vertica equally well. These two databases are not similar in the way they work internally, but neither are SQL Server and Oracle - which are still both RDBMS.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

BI-as-a-Service - Some Questions Worth Asking

2011-06-16T00:06:00.000-07:00

For a couple of years now, there has been a substantial amount of hype in the business intelligence (BI) space regarding “cloud BI,” or business intelligence systems hosted by Internet “cloud computing” service providers. This “cloud BI”, which is actually SaaS (software-as-a-service) BI, has been riding the wave of cloud computing in general, with the lower startup costs, faster deployment and easier scalability that cloud-based software implementations promise business customers. Several new companies have emerged and are promoting a new golden age of BI which they say will be faster, easier and cheaper than conventional business intelligence.

While this sounds fantastic at first glance, it might be a good idea to look beyond the hype to determine if deploying a BI solution in the cloud offers the same types of advantages of other SaaS solutions, such as CRM, accounting and email.

To this end, consider the following questions:

Does SaaS BI reduce dependence on IT staff?
Does SaaS BI compromise your data security?
Is all your business data already in the cloud?
How much will hardware cost you in the cloud?
Is BI backbone technology becoming more 'cloud-aware'?

1. Does SaaS BI reduce dependence on IT staff?

One of the main claims of SaaS software solutions in general is the dramatic reduction in dependence on expensive (and overworked) information technology (IT) professionals. In many organizations, the limited availability of IT staff is a major bottleneck when considering implementing a new software system that will be used across the organization. The cloud offers an appealing solution: since there is no hardware to install, no software to upgrade, no storage to backup and no new security mechanisms to implement, adopting a cloud solution (mostly) circumvents the need for extensive IT services (whether from in-house personnel or consultants).

In the realm of conventional BI systems, dependence on IT is a common and frustrating bottleneck for business users. The IT department (or consulting firm) is required for every piece of the BI puzzle, from the data warehousing to creating OLAP cubes to creating and customizing individual reports. In most companies, business users quickly discover that getting what they want from their company’s BI system, including incorporating new data sources, adding reports, customizing dashboards and extending the system to more users/departments, requires IT resources which are often unavailable when needed. The result is a frustrating and compromised system which fails to deliver on its full strategic potential.

So, does moving to the cloud solve this central problem of conventional BI?

The truth is that hosted BI solutions are really just outsourced IT departments which happen to come along with a bag full of their own home-grown or third-party software systems. All the stages of the familiar BI system deployment – from requirements specification consulting through data warehouse creation through report customization – still require the involvement of IT staff.Since these IT professionals are experts in their software and environments, they will likely be more efficient than hiring or retraining your company’s own staff (although they will not likely be more effective than any other dedicated outsourced technical BI team). However, think carefully about how much you want to be dependent on outsourced IT services for the lifeblood of your company’s strategic decision-making platform.

If your BI needs are modest and don’t change often, then, in theory, you will probably end up saving money as compared with hiring your own IT staff for BI. However, as the system grows and extends (as it always does), you will be at the mercy of the schedules and price rates of a third-party IT team which you are locked in to. If you thought internal IT can be a bottleneck, imagine how difficult it will be to get good and timely service from an external IT department located far away and busy with numerous other customers as well.

2. Does SaaS BI compromise your data security?

There are two unrelated issues to think about here. Read the rest of the article...

The Collapse of Traditional Business Intelligence

2011-05-31T04:45:00.000-07:00

The traditional approach to business intelligence has gone bankrupt. In its place, a new wave of companies provides alternative solutions based on innovative technologies and new business models.

Changes are Happening in BI
During the past few years, dramatic changes have been occurring in the world of business intelligence (BI). These changes all go towards one goal: removing the barriers - firmly set by the traditional BI vendors - which prevent wider usage of these decision-making systems within organizations.

These barriers include great complexity, high cost, excessive dependence on external system integrators and general dissatisfaction among business users with the tools foisted upon them by the traditional BI software vendors.

Naturally, these changes are leading relatively young and innovative companies to the field. Whether by utilizing newer technologies such as columnar databases or via a software-as-a-service (SaaS) business model, their goal is to change the rules of the game in favor of the BI customer.

The Importance of Data Analysis for Organizations
Business intelligence is a concept which has already been around for more than 20 years, and most organizations understand well the advantages of making decisions based on real data as opposed to relying on intuition and guesses. This has led to tremendous demand for these types of solutions, and the phenomenal consequent growth of the BI industry.

In today’s dynamic world, BI solutions are more necessary than ever to organizations interested in making their operations more efficient, primarily in controlling expenses and maximizing revenues. Furthermore, the Internet’s growth as a means of marketing and distribution is increasing competition in almost every field. Without BI solutions, more and more organizations will be finding themselves left far behind.

The New Self-Service BI Concept
Most of the traditional BI solutions (SAP, IBM, Microsoft, Oracle, etc.) are designed for implementation by subcontractors – experienced professionals generally known as “system integrators” – who specialize in customizing and deploying these solutions (in fact, most BI companies are actually integration companies).

As a result, these solutions require technical expertise which does not exist in the average organization. Additionally, the cost of the actual software in this situation is small relative to the service contracts required with the system integrator. And if that’s not enough, the organization’s complete dependence on the contractor can continue indefinitely, a situation that severely limits the adaptability and use of the solution for the frequently-changing needs of almost every organization.

Organizations considering implementing a traditional BI solution face these serious obstacles. This is the main driver for the most prominent industry trend of “self-service” BI, based on newer technologies and more business-friendly pricing models. These newer BI solutions provide at least the same degree of functionality and power as the traditional products, but are designed to be implemented, customized and managed by the people already found in most organizations.

Self-Service BI Products and Tools
The various types of “self-service" BI solutions can be categorized as follows:

1. Software-as-a-Service (SaaS) and/or cloud-based BI – This approach enables the implementation of certain, usually simple, business intelligence solutions without heavily involving of the organization’s IT department. The basic idea is to fully outsource all the IT services involved in implementing and maintaining a BI solution. These solutions eliminate, at least on paper, the need for expert technical staff, the need to buy and maintain dedicated computer hardware and the need to manage software updates. On the other hand, this approach does not remove the dependency on an outside service provider – it actually increases this type of dependence.

2. BI tools for analysts – This type of BI focuses on tools which make the organization’s data analysts more efficient in their work. Analysts spend a tremendous amount of time gathering and organizing data, usually in Excel, and then preparing graphs and reports. BI tools in this category generally provide facilities to more easily gather, organize and present data, including special analytical and graphical features. However, since these tools do not contain centralized data repositories and reporting facilities, they are not ideal for the multi-user environments which characterize most organizations interested in a BI solution.

3. Data warehouse/OLAP-replacement BI solutions – Solutions designed to serve many users and/or to process complex (or very large amounts of) business data represent the biggest technical challenge. Meeting this challenge in the traditional way – by using a data warehouse and OLAP cubes – is what positioned BI into the exclusive domain of the wealthy and the brave. The tremendous complexity, lack of flexibility and very high cost of this approach is what gave rise to alternative technologies which could deliver the same results (many users and much data) faster, easier and more cheaply. SiSense, for example, developed its own BI technology (called ElastiCube) which exploits columnar database and advanced memory-management technologies to deliver enterprise-scale BI without the complexity, rigidity and high cost of traditional BI solutions. Solutions of this kind represent the basis of an entirely new approach to the BI challenge.

Conclusion
In recent years, great strides have been made to enable the widespread deployment of enterprise business intelligence solutions. Whereas in the past, BI was the exclusive province of large and wealthy organizations, today it is also readily accessible to small and medium-sized companies. Now, even startups can (and should!) take advantage of the substantial business benefits provided by such solutions.

You, too, are invited to join the BI revolution!

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Are BI Appliances Simply 30 Year Old Databases?

2011-04-28T06:03:00.000-07:00

In a thought-provoking blog post published by WIT, a business intelligence consulting company in the U.S., the author writes of latest acquisitions relating to Business Intelligence appliances.

BI Appliances

It got me thinking. I’ve been seeing and hearing the term ‘BI appliance’ a lot recently, and whenever I do - I find myself struggling to understand what it means.

One characteristic that seems to be commonly identified with BI appliances is that they are a combination of software and hardware that form specific functions that have to do with analytics (i.e business intelligence). WIT’s article lists a few examples, including HANA (SAP), HP Business Decision Appliance (Microsoft), Netezza (acquired by IBM) and Greenplum (acquired by EMC).

But is proprietary hardware really required for a so-called BI appliance? No, it’s not. And indeed, I have noticed numerous references to Vertica (acquired by HP) and ElastiCube (by SiSense) as BI appliances. Interestingly enough, both are software-only solutions (i.e. software appliances).

It makes sense, as it shouldn’t matter if your ‘appliance’ runs on proprietary hardware or commodity hardware, if it essentially does that same thing.

The BI Appliance Wars

In a recent interview and in response to quips made by Netezza’s CEO regarding HP’s latest acquisition, Vertica CEO Chris Lynch had this to say about Netezza:

“Their tag line is ‘The power to question everything'. So the first question is: why do they need proprietary hardware? The second question is: why are they using a database engine that’s based on technology from 1982?”

He is obviously angry, but I agree with the premise of his argument. If you’re in the analytics business and you require proprietary hardware – there’s something seriously wrong with your database software technology. Commodity hardware is so powerful today with 64-bit computing and multi-core CPUs, that it’s hard to imagine what type of BI solution would require proprietary hardware. That is, if your technology was engineered in the 21st century.

The established vendors are not oblivious to this, but rewriting their entire codebase is not something they are willing to do. So some are partnering and/or merging with hardware companies as an alternative. But at some point, scraping this codebase will be unavoidable, or customers will flee due to availability of much better and cheaper alternatives.

BI Appliance or BI Tool?

As if to toss a little more confusion into the mix, the WIT author asks:

"Though I wonder - with memory becoming cheaper and cheaper and with 64 bit platform, why do you have to have a special appliance? Why not use an in-memory tool with tons of RAM ?"

The question itself indicates a misunderstanding of why appliances exist in the first place, and there are a several answers to this question. Here are a few:

RAM is cheaper, but it's not cheap. Disk was and always will be cheaper than RAM.
The price of a computer jumps significantly beyond 64GB. A PC with 64GB of RAM costs significantly less than a server machine with 65GB of RAM, even though there is supposedly just a single GB of memory difference.
In-memory databases assume that the main bottleneck is I/O. However, when dealing with large amounts of data, this is no longer true. At such volumes, bottlenecks are between RAM and CPU.

For more information about this, please read In-Memory BI is Not the Future, It's the Past.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

BI vs. Big Data - Watch a Columnar Database in Action! (Video)

2011-04-14T13:52:00.000-07:00

I was recently fortunate to speak at one of the database technology conferences held in Israel. Big parts of this conference revolved around ‘Big Data’ and I was asked to give the Business Intelligence perspective on this fascinating subject.

As part of my presentation, I attempted to show the impact of columnar database technology on the basic premise of business intelligence - the ability to have business users perform ad-hoc analytics and reporting tasks over as much data as possible.

In order to do that, I represented a business user building a report over a very large operational database containing 13 tables, the largest of which hold 100 million and 40 million rows. While databases of this size were once rare - now, any company who has a properly tracked website quickly accumulates even more data than that.

To demonstrate, I was using a front-end analytics tool (SiSense Prism) to create reports that query the database directly - a feat not advisable with a relational database. So instead of querying the source database, the data was replicated (but unmodified) into a columnar database that was designed specifically for ad-hoc analytics - ElastiCube.

The computer holding the ElastiCube was a $1200 off-the-shelf PC with 6GB of RAM, 100GB of disk space and a single quad-core CPU (64-bit). The Prism front end could be installed on any computer, as it does not process the queries or hold data - only requests query results.

For your convenience, here is the video of this demonstration.

Interesting Points
One thing you should understand from this video is how simple drag-and-drop operations of a business user (or multiple users) within his or her desktop tool turn into complex database operations that would choke any relational database (joining, grouping, aggregating), yet are handled by a columnar database without any difficulty.

Which brings me to my final, and perhaps most important point -

Whichever business intelligence front end tool you pick, dealing with issues like this (and their subsequent side effects) is in fact 90% of the life-time cost of a BI solution, and often why the solution stops being used all together. This is because BI solutions which rely on relational back-end technology must be designed to assume there had been significant trimming, de-normalizing and pre-aggregation of the data conducted prior to being delivered to business users. This process never ends, and only becomes more and more difficult to maintain over time.

Columnar databases change this reality entirely, and combining them with 64-bit and multi-core computing makes for a dramatic evolution in BI development.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Chatting with Albert Einstein about Business Intelligence

2011-04-06T05:10:00.000-07:00

Albert Einstein is arguably one of the most intelligent people who have ever lived. Have you ever wondered what it would be like to chat with him about business intelligence? I have.

Mr. Einstein, thank you for taking the time to talk to me today. I work for a business intelligence vendor called SiSense and I would love to hear your thoughts on a subject you know more about than anyone else – intelligence.

"Whoever undertakes to set himself up as a judge of truth and knowledge is shipwrecked by the laughter of the gods.”

Modest as always! However, instead of talking about personal intellect, I would love to get your perspective on what we call ‘business intelligence’.

“One may say the eternal mystery of the world is its comprehensibility.”

That is exactly why I’m talking to you. While great technological and functional strides have been made in the business intelligence space, no one has really been able to figure it out yet. Maybe you can help us out?

“Technological progress is like an axe in the hands of a pathological criminal. It has become appallingly obvious that our technology has exceeded our humanity”

Mmm, right. But business intelligence technology is not exactly the atom bomb.

“Concern for man and his fate must always form the chief interest of all technical endeavors. Never forget this in the midst of your diagrams and equations.”

Tough crowd. Okay, let me try to clarify. Business intelligence describes the ability of a business to make informed decisions based on facts inherent in its data. Experience shows that the challenge in achieving this can be easily tracked down to the technology being used behind this ability. Innovators in the space place a lot of focus on this, but I think they find it challenging to explain to the average person why it is so important.

“Most of the fundamental ideas of science are essentially simple, and may, as a rule, be expressed in a language comprehensible to everyone.”

So what you’re saying is that the technology doesn’t matter, if you cannot explain its benefits in simple terms. Fair enough. So what if I told you that modern business intelligence technology simplifies the ability of decision makers to gain knowledge and insight into their business operations and through that make better informed decisions?

“The true sign of intelligence is not knowledge but imagination.”

But you cannot disagree that following business metrics is a good idea.

“Not everything that can be counted counts, and not everything that counts can be counted.”

I guess that’s true. So, are you saying we should give up on the whole idea of business intelligence being realistic?

“Reality is merely an illusion, albeit a persistent one… Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.”

But that is exactly what business intelligence means - never stop questioning. However, without getting the occasional answer, this could prove to be quite a tedious challenge, right?

“The devil has put a penalty on all things we enjoy in life. Either we suffer in health or we suffer in soul or we get fat.”

I guess I could lose a few pounds, but what part would technology play in all this?

“We can't solve problems by using the same kind of thinking we used when we created them.”

That is why new technology is constantly being thrown into the business intelligence mix. Modern chipsets like CPU and RAM, Columnar databases, NoSQL databases, etc. These technologies did not exist before, and they're all being used to make business intelligence more approachable.

“Everything should be as simple as it is, but not simpler.”

Having things simple is important, no doubt, but there is also the issue of business intelligence costs going through the roof.

“Sometimes one pays most for the things one gets for nothing.”

We indeed encounter people dropping loads of cash on something that could be easily done in Excel. Isn’t that insane?

“Insanity is doing the same thing over and over again and expecting different results.”

Interestingly, it indeed seems like many business intelligence vendors just re-wrap their old products based on technology invented before you were born. You end up paying for the same thing again and again, without really getting any value back. How can new business intelligence vendors succeed where the others have failed?

“You have to learn the rules of the game. And then you have to play better than anyone else.”

This was a fascinating chat. I’d like to thank you for your time.

“The only reason for time is so that everything doesn't happen at once."

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

* Einstein quotes are the courtesy of BrainyQuotes.

There is still hope for Business Intelligence

2011-03-16T13:02:00.000-07:00

A few months ago, I wrote an article about the relationships between business intelligence platform vendors and their distribution networks, and where these relationships are headed.

The key message in that article was that Value-Added Resellers (VARs) must understand that self-service business intelligence is where the market is headed – and adjust their business models accordingly. A company selling BI solutions based on another company’s BI platform will need to provide real added value to the customer in order to stay in business. In the not-too-distant future, this value will almost certainly come from industry-specific professional knowledge and experience (as opposed to purely technical expertise). More and more potential customers will no longer accept lengthy projects and, with the new software and technologies now emerging, it is no longer justifiable.

Winds of Change
The Israeli BI market ('the Pond', as it is referred to by the locals) is a good example of how the BI market looks like in most parts of the world. It’s packed with BI companies that are in fact service and/or placement shops and are forced, by their own out-dated business models, to sell to a just handful of high end companies (by Israeli standards). That, and the lack of decent tools, has just made it impossible for the market to evolve. It just amazes me that in the 21st century, a few patched up software components are used to sell ridiculously long projects that end up in nothing, taking yet another bite out of the BI industry's credibility.

Yotam Aharonson, the CEO of New AspeQt - a known and respected BI integrator in Israel – recently published an article in local Israel media which finally made me feel like things are starting to change. In my excitement, I had it translated to English and posted below (the original Hebrew version can be found here).

I believe it stands for itself as an explanation and an argument.

Self-Service BI: A Revolutionary Idea that has Become Reality
By Yotam Aharonson, New AspeQt

The King has No Clothes
In a world accustomed to thinking in terms of long and expensive projects, complex licensing models and long-term temporary employee placement, new voices are now being heard. System integrator New AspeQt has set a mission to itself - to bring into the Israeli market an insight that has already spread to quite a few markets around the world: there is a different way!

New AspeQt believes that the only way to deploy “business intelligence” in an organization, such that it will be possible to adapt to ever-changing information and business needs, is by enabling the organization to create and manage its own reporting and analysis facilities, while reducing dependence on external consultants. The combination of a system integration company that understands the value of this new approach, together with a BI platform that enables simple and flexible implementation, is a revolution in itself! (New AspeQt has found that the Prism BI application from SiSense is most suitable for this challenge.) For the first time, control is transferred into the hands of the customer from the very beginning of the process and remains there throughout.

Trapped in the Clutches of BI System Integrators
It would seem that for a BI system integrator to espouse self-service BI would be to shoot itself in the foot, or at least to harm its basic interests, namely, to sell work hours, the bread and butter of any system integrator. New AspeQt has a different perspective: While most integrators have, in fact, become employee placement companies which provide skilled BI workers to work at their customers’ sites, New AspeQt sees itself as a young and dynamic company which serves as a skilled in-and-out “commando”: New AspeQt transfers the required knowledge to the client’s employees, along with professional guidance, to quickly bring the client to full BI independence.

Many organizations evaluating the implementation of a BI solution mistakenly think that BI is a finite project. This fundamental misconception can cost a company a lot of money and, more importantly, create a “Gordian knot” of permanent dependency on the system integrator. An organization’s information and data needs in the business world are constantly growing and changing. Given this reality, dependence on an external provider will, at best, lead to ever-growing and accumulating expenditures. At worst, in an attempt to reduce these expenses, the organization may simply avoid further development and essentially lose their “business intelligence” edge.

If it Works for Yahoo and Cisco, Why Not for Me?
Many organizations in Israel and around the world have already chosen the do-it-yourself BI solution from Israeli software vendor SiSense. For a significant number of these customers, the person implementing the system was not even in the company’s IT department, but rather among the “BI consumers” themselves, such as marketing, operations or finance. All of these customers tried the software and its self-service method for a month with no up-front expense. They realized that the software’s promise is not just a marketing slogan, but a genuine reality. When the information consumer builds his own solution himself, there’s no chance that the solution will not fit his needs!

By: Yotam Aharonson, CEO, NewAspeQt | The ElastiCube Chronicles - Business Intelligence Blog

Just Another Fool with a Tool

2011-02-16T00:57:00.000-08:00

Once, when Business Intelligence (BI) wasn’t a multi-billion dollar industry, it was an idyllic concept which revolved around enabling an analytic approach to business decision making. That’s what made me fall in love with BI over 10 years ago and what, in my view, gives the BI space its misty glow.

At the heart of the Business Intelligence concept lies the seemingly trivial ability to get answers to important questions. The CEO may ask the CFO how quarterly sales look compared to the previous year, and the online marketer may ask himself to which landing page they should direct new traffic for maximum conversions. To each of them, their question is equally important and has great impact on their course of action.

The general ability to get questions like these answered effectively is what measures the “intelligence” of a business. An intelligent business operates based on informed insight, while a regular (“dumb”) business operates based on guesses and hunches which could easily be wrong.

Sic Transit Gloria Mundi – Thus Passes the Glory of the World
Being part of the Business Intelligence space used to mean something. The great strides made by BI innovators like Business Objects, Cognos and Microsoft have done a great job to promote the Business Intelligence concept and maintain its lucrative appeal. The fact that they couldn’t make it affordable beyond a thin tier of wealthy companies actually helped maintain that perception: not just any company can use business intelligence, and not just any company can sell business intelligence. Business Intelligence was for “serious” companies only.

Long since then, however, the good name of Business Intelligence has been tarnished. I don’t know if you’ve noticed, but the rate at which new “business intelligence vendors” keep popping up is ridiculously high.

Why the double quotes? Because you don’t need to know anything about Business Intelligence to launch an AdWords campaign or do some SEO work, and online marketers have been taking advantage of that fact to feed off the business intelligence prestige and throw a lot of attention towards their own goods. But these rarely have anything real to do with business intelligence. At the best of times, these are simplistic reporting tools, data visualization gizmos or expensive implementation projects masked by buzzwords like SaaS.

Is Your Business Really Intelligent?
The saddest thing is, companies genuinely wishing to introduce the modern and effective approach to business decision making inherent in BI get entangled in all this hype and end up paying a lot of money for “business intelligence,” when, in fact, all they have done was have someone develop a few reports or create a pretty dashboard for them.

Business Intelligence is not about what reports or dashboards you have, how detailed they are or how dazzling they look. It’s about being constantly hungry for new information across your entire business, and being able to retrieve it easily and quickly without constantly calculating how much it would cost.

Otherwise, you’re just another fool with a tool.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Thoughts about Business Intelligence and the Cloud

2010-12-10T11:48:00.000-08:00

Business intelligence in the cloud is a hot topic recently, as part of the hype surrounding the cloud in general. I am not a big fan of cloud BI and I have mentioned that several times. However the topic does merit discussion.

The advantages of the cloud over on-premises are pretty straight forward. However, as far as business intelligence implementations are concerned, the question to me was always whether the benefits outweigh the unique challenges the cloud introduces. If all business data was in the cloud, there was a definite case to make for implement business intelligence software in the cloud. But since most business data isn’t, the benefits of cloud BI are not as obvious.

The blogosphere and analyst community in the business intelligence space are not sparing any words on the subject. There are several startups in this space as well, such as GoodData, PivotLink and others. But is the business intelligence space really heading in the direction of the cloud? I believe the answer is no.

The main reason I do not believe that the BI space is headed towards the cloud (at least for now) is because business intelligence backbone technology doesn’t seem to be headed there. In fact, it seems to be going in the opposite direction.

If you take a careful look at the new technology promoted by the established business intelligence vendors like SAP, IBM and Microsoft, and even those promoted by slightly less established vendors (yet successful) such as QlikTech and Tableau – it is all technology that is either ‘desktop enabling’ technology or in-memory technology.

These technologies, in-memory in particular, aren’t very cloud friendly and weren’t designed with the cloud in mind at all. They are designed to extract more juice out of a single computer, but very hard to distribute across multiple machines as in the case in most cloud implementations. Also, to benefit significantly from these types of technologies, you need very powerful computers, a premise which goes against proper cloud architecture that dictates that computing operations should be parallelized across multiple cheaper machines.

On the other hand, the current cloud BI platform vendors are using the same traditional backbone technology the on-premises vendors do, and by that they suffer from the same drawbacks most BI vendors do such as complexity and long development cycles. And when these drawbacks come into play, whether the data is in the cloud or on-premises isn’t even the main issue.

Even if the ‘pure cloud’ BI platform vendors did develop better technology more suited for running BI in the cloud, it is still years away. So while you can use the cloud for some types of solutions (mainly around other cloud data sources) the fact of the matter is that the cloud BI hype is at least a few years too early.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Business Intelligence Vendor Websites – How to Read Between the Lines

2010-12-01T15:10:00.000-08:00

If you’ve begun looking for a business intelligence solution, you probably noticed very quickly that there are quite a few business intelligence vendors out there. Narrowing the overcrowded field of vendors in order to come up with your own business intelligence short list will usually start with a review of the vendors’ Websites. Armed with your short list, you can begin contacting vendors to receive proposals.

This article is going to help you discover some key differences between vendors based on what they share on their Websites – and what they don’t. Read between the lines while looking at BI vendor Websites to quickly figure out which business intelligence vendors are probably right for you, and which ones aren’t.

The Customers Page

Obviously, a business intelligence vendor Website without a page dedicated to featured customers is cause for concern.

Assuming the vendor does indeed have a Customers page, the first thing you should look for is whether the featured customers are big corporations or smaller companies. This is an important distinction because business intelligence solutions for big corporations often have very different functional (and other) requirements than business intelligence for smaller companies. You should focus on vendors which sell to companies like the one for which you’re seeking a solution. If you need BI for an SMB, a business intelligence vendor that only lists Fortune 500 corporations on their Customers page probably won’t fit your needs. Their solutions are probably too complicated and/or expensive. Similarly, if you need BI for a large corporation and the business intelligence vendor only lists SMB customers, the solution may not deliver the functionality, performance or scalability you need.

The second thing you should look for is whether you recognize any of the customers listed on this page. Having recognizable names says a lot about the credibility of the business intelligence vendor. Well-known companies with recognizable brand names do not trust their business operations to just anyone. A business intelligence vendor with recognizable names on its Customers page is less likely to disappoint you than a company listing only unknown names.

The Case Studies Page

Reviewing the case studies on a vendor’s Website is a good way to determine whether the solutions the vendor provides were implemented in business scenarios that resemble your own. Most vendors should include at least a few case studies on their site.

It’s important to note whether the solution has been implemented in companies within the same industry as yours. Different industries will share many common needs, but there will also be special needs and requirements unique to each one.

Another important thing to look for is whether the solution is being used in the same way you’d like to be using your business intelligence solution. There are several types of business intelligence solutions available – some are used more for reporting and some more for analytics. Some are maintained primarily by IT and some are designed for more hands-on use by business professionals. Case studies help you determine whether the solution can meet your goals and whether it can be used by the people you intend to use it.

The Partners Page

The Partners page is a great place in which to gather information about the vendor. You may think that the Partners page is mainly used for finding representatives in your own region, but there is a lot more you can tell about a vendor from this page.

One thing you should look at is whether the vendor has a long list of service integrators listed as partners. This is important because it says a lot about how difficult the solution may be to implement.

While software vendors prefer to sell software licenses, service integrators typically make most of their revenue from projects they execute, charging by the hour or by the day. When a business intelligence vendor has a long list of service integrators as partners, it usually means that the solution they sell is a challenge to implement and/or requires extensive on-going maintenance and consulting. Otherwise, the service integrators wouldn’t see much business value from working with that vendor.

The second thing you should look for on the Partners page is whether the vendor has multiple software/technology vendors listed as partners. Typical business intelligence applications require several tools and technologies to be fully implemented, and when a vendor lists technology/software partners, it usually means they only provide a portion of the business intelligence stack themselves.

The Free Trial Download Page

Even though free trial downloads are commonly found in the software industry, they are not commonly found in BI vendor Websites.

There are two major reasons for this. The first is that 99% of the business intelligence vendors out there are not software providers, but rather service providers. As opposed to selling products, they partner with existing software/technology vendors to use their products as building blocks for implementing custom solutions.

The second reason for vendors not having a free trial available for download is their lack of confidence in your ability to quickly and easily use their software (due to it being too complex and/or because it requires professional services to get started).

Therefore, when a vendor does in fact have a free trial version available for download, it typically means both that this vendor is willing to sell directly to you (thus removing the mark-ups taken by third-party service providers and resellers) and that this vendor believes you would actually be able to use their software yourself. It also stands to reason in this case that you’re looking at dramatically shorter implementation times, without extensive third-party service provider fees.

The Services Page

Vendors in the business intelligence space basically have two ways of making money: through selling software or through selling services (e.g., consulting, implementation, maintenance).

It is very difficult for a vendor to successfully focus on both, as they require completely different staff and operations. Building commercial software in the BI space requires significantly deeper pockets than simply providing services which utilize existing (third-party) software. This is why the business intelligence space has very few vendors which provide end-to-end “productized” software and significantly more service providers who will custom build a system.

It’s important to understand that the total cost of owning a BI solution is rarely the cost of software licensing, but rather all the work that needs to be done in order to customize this software for your own needs. Therefore, if a business intelligence vendor presents itself as a software vendor, while having a wide range of professional services for sale, it should make you wonder what it is about their software that requires so many services around it. As previously mentioned, true software product vendors prefer to focus on selling licenses, not on selling services. It is very difficult to do both equally well and, from a vendor’s business model perspective, it’s wrong to even try. So business intelligence software vendors try to make their software as easy to implement in as self-service a manner as possible. Otherwise, they would have to spend more than they’d want on training, support and customization services.

The Pricing Page

Pricing pages are not something you often find on business intelligence vendor Websites. Those which do have pricing pages usually have some empty statements such as “contact us” instead of actual pricing figures.

Usually, business intelligence vendors do not publish pricing information because most of the actual cost of a system deployment will go towards the implementation and customization projects. These projects are very hard to price before doing extensive on-site analysis and estimating project scope. Therefore, the pricing would be meaningless.

For a vendor that actually sells software products, a typical reason for not posting pricing information is to hide the mark-ups their service integrator partners get on the deal. In many cases, the service integrator is the actual one doing the selling to the end customer and they enjoy margins which tend to be significant. Publicly posting the list price would prevent third-party resellers such as these from adding significant markup to the sale.

However, there are rare cases when a business intelligence vendor does post pricing on their Website. These are typically companies with unique business models and offerings, and by that fact alone they are worth your close attention.

The Online Community Page

Having an online community (via blogs, forums, etc.) is a very big benefit for customers in the BI space, as it allows you to leverage the experience of other users to accomplish tasks and solve problems. Regardless of which vendor you choose, you will always have a question at some point and it’s great to be able to get it answered without having to rely on the vendor’s official support channels.

Online communities also give you a good indication of how popular the solution is and what types of things people are doing with it. Generally speaking, your experience with a business intelligence solution will be much more fluent if you have access to the collective knowledge of an active online community.

When a business intelligence vendor does not have an online community, you should immediately be concerned about how much support will cost you and how quickly you will be able to get responses to your inquiries.

Good luck on the search for your perfect business intelligence solution!

Provided as public service to the business intelligence customer community by SiSense, a BI vendor leading the way to a faster, easier, better era of business intelligence.

Microsoft’s Announcement of BI Road Map – a Public Relations Nightmare

2010-11-15T22:57:00.000-08:00

During Microsoft’s PASS Summit 2010, Microsoft announced their strategic plans for their Analysis Services business intelligence platform. But as intriguing as Microsoft’s announcement was from a technological perspective, it also had another less desired effect – it presented Microsoft’s long time business partners, namely BI consultants and service providers, with big questions in regards to where they fit in Microsoft’s grand plans for the BI space.

There are two main reasons for this:

1. While Microsoft heavily relies on its partner network for selling more SQL Server licenses, they have been heavily marketing PowerPivot which is positioned as self-service BI. Neither the partners, nor Microsoft, have yet to figure out how to position the two offerings in a way that makes sense to a typical customer.

2. Microsoft’s road map clearly shows a shift away from OLAP architecture and to a new paradigm they call the BI Semantic Model that doesn’t coincide with their partners’ existing and hard earned training and expertise in implementing/selling Microsoft BI solutions. Microsoft partners would need to re-align their entire business based on a product that isn’t even released yet, let alone implemented anywhere.

Following this announcements, high profile evangelists of Microsoft BI solutions have openly expressed their concern regarding this radical move. See examples here and here. Ever since, Microsoft has been working overtime on some major damage control, trying to explain itself and reassure its partner network that OLAP is not going anywhere and that these new plans are complementary to the existing Analysis Services offering (i.e. "bla bla").

Regardless of Microsoft’s post factum attempt to re-establish calm amongst its partner community from a PR perspective, that cat is already out of the bag. And honestly, Microsoft’s move was not unexpected. Check out the article published two months ago titled ‘Business Intelligence Vendors and their Partners – Rough Seas Ahead’ which specifically discusses what PowerPivot (and similar technologies) would mean for the BI implementation business.

Whether Microsoft’s ideas are practical or just pipe dreams remains to be seen. However, one thing is certain – considering the fact Microsoft rely so heavily on its dedicated partners for sales and marketing, I would have expected this to be handled with much more finesse. This announcement was a very poor display of handling public relations.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Microsoft’s BI Roadmap says NO to OLAP Cubes and MDX

2010-11-12T15:24:00.000-08:00

So Microsoft PASS Summit 2010 was kicked off on November 10th, and the burning topic was where Microsoft’s Analysis Services product is headed in light of Microsoft’s new PowerPivot offering. Chris Webb, probably one of Analysis Service’s biggest fans and experts, said it best:

“The last few days have been quite emotional for me. I’ve gone from being very angry, to just feeling sad, to being angry again; I’m grateful to the many members of the SSAS dev team who’ve let me rant and rave at them for hours on end and who have patiently explained their strategy – it’s certainly helped me deal with things. So what’s happened to make me feel like this? I’ll tell you: while it’s not true to say that Analysis Services cubes as we know them today and MDX are dead, they have a terminal illness. I’d give them two, maybe three more releases before they’re properly dead, based on the roadmap that was announced yesterday.”

The full post and consequent comments can be found here.

Readers of The ElastiCube Chronicles may recall a previous post titled ‘Is Microsoft Admitting that Analysis Services is not Fit for the Mid-Market?’ published back in August 2010, in response to the official release of PowerPivot. Well, I believe that question has been officially answered - Yes.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

The New Tableau 6.0 Data Engine – First Impressions

2010-11-12T14:03:00.000-08:00

Tableau 6.0 is out, and according to Tableau Software’s CEO one of its main features is a new data engine. Here’s an excerpt from one of the articles covering Tableau’s latest release:

"Our new Tableau Data Engine achieves instant query response on hundreds of millions of data rows, even on hardware as basic as a corporate laptop... No other platform allows companies to choose in-memory analytics on gigabytes of data …" Christian Chabot, CEO of Tableau Software, said in a statement.

These are bombastic claims indeed and the underlined segments of the CEO’s quote are particularly interesting. So with the help of my friend, colleague and brilliant database technologist Eldad Farkash, I decided to put these claims to a real life test.

Since this data engine was claimed to be utilizing in-memory technology, we set up a 64-bit computer with adequate amounts of RAM (hardly a corporate laptop) and used a real customer’s data set consisting of 560 million rows of raw internet traffic data. To make it easier, we imported just a single text field out of this entire data set.

Initial Findings:
1. Surprisingly, and unlike what Tableau’s CEO claims, Tableau’s new data engine is not really in-memory technology. In fact, their entire data set is stored on disk after it is imported and RAM is hardly utilized.

2. It took Tableau 6.0 approximately 5 hours to import this single text field, out of which 1.5 hours was pure import and the rest a process Tableau calls ‘Column Optimization’ which we believe is creating an index very similar to that of a regular relational database. For comparison, it took QlikView 50 minutes and ElastiCube 30 minutes to import the same field. That is an x7 difference. All products were using their default settings.

3. Once the import process completed, we asked Tableau to count how many distinct values existed in that field, a common query required for business intelligence purposes. That query took 30 minutes to return. For comparison, it took both QlikView and ElastiCube approximately 10 seconds to return. That’s an x180 difference. Again, both products were used with their default settings.

Initial Conclusions:
Tableau’s new data engine is a step up from their previous engine which was quite similar to that which Microsoft Access had been using in Office 2007. That is good news for individual analysts working with non-trivial amounts of data using earlier versions of Tableau, which were quite poor in this respect. This release, I imagine, also helps Tableau against SpotFire (Tibco), which until now was the only pure visualization player who could claim to have technology aimed for handling of larger data sets.

From a practical perspective, however, the handling of hundreds of millions of rows of data as well as the reference to in-memory analytics are more marketing fluff geared towards riding the in-memory hype than a true depiction of what this technology is or what it is capable of. Tableau’s data engine is not in the same league as in-memory technology, or pure columnar technologies like ElastiCube, when it comes to import times or query response times. In fact, it is slower by several orders of magnitude.

Choosing a BI Vendor - Making the Short List

2010-10-24T14:43:00.000-07:00

There is no shortage of business intelligence vendors out there. They all claim to be powerful, easy-to-use, flexible and affordable. So how do you pick the one that is right for you?

In order to be able to choose the right BI vendor from the abundance out there, the best way is to follow high-level, yet restrictive, criteria and only then compare them on a feature-by-feature basis. Here are a few tips that will help you do that, as well as avoid common mistakes typically made when choosing a BI solution. This is the 21st century, and BI solutions are completely different than what you may be used to. If you follow these tips, you’ll end up with a very short list of vendors, and then it’ll just be a matter of choosing the one you feel most comfortable with in terms of specific features, pricing, support, etc:

Find a Complete Solution, Not Just Pretty Visualization.

The visualization of data is important, of course, but the biggest mistake you can make is judge the BI vendor based on the pretty dashboard samples they show you on their website or during a demo. Every BI vendor can do that because visualization software components are a dime a dozen. The real challenge is customizing these dashboards to your own needs and having them show your own data. This part usually takes most vendors months, and costs you bundles. If the BI vendor cannot get your own data to show the way you like it within just a few days, you could probably find a better one.

Beware of the Data Warehouse.

A data warehouse is a centralized database filled with all the business’s data, and for years it’s been making a ton of money for BI vendors and bringing nothing but grief to customers. Today’s BI technology does not require a data warehouse, even when there are multiple data sources involved, large amounts of data or multiple users querying the data. There are very specific scenarios where a data warehouse is a good idea, but they are most likely not relevant to you. If the vendor requires a data warehouse to proceed with implementation, it is most likely you should keep looking.

Beware of the OLAP Cube.

OLAP, which stands for Online Analytical Processing, is 20 year old technology designed to improve query performance over medium to large datasets. OLAP is also very lengthy and costly to implement, and there is really no need for it anymore. Today’s BI technology can handle even huge amounts of data without OLAP, at fractions of the time or cost. If the BI vendor requires OLAP to assure you acceptable query performance, you should probably move on.

Refuse to Make Significant Upfront Investments.

Many BI vendors will promise you the world, but will demand significant upfront investment in preparation projects, hardware and software before you even get to run a single report on your actual data. Do not agree to this, and demand to have at least one solid report or dashboard running over your own data before you commit to anything significant in advance. If the vendor is not willing to do so, it’s probably because they would have to spend weeks on development before they can reach that point. That typically means this vendor is either using very old technology or is simply trying to pull one over you.

Be wary of Vendors whose Business is Prof. Services.

Vendors who sell real home-grown BI software products (in contrast to OEMing someone else's software) do not like engaging in long professional services projects because it hurts their margins. That is why they prefer to create software that is easy enough to be used directly by the customer or through a third party (which usually lives off these professional services contracts). If you choose a BI vendor who makes most of his business off professional services (as opposed to software sales), you can pretty much be sure that they will take their time building your solution. These types of BI vendors also live off on-going maintenance services, so what you initially pay for the solution is actually only the beginning. Whenever possible, try to choose a BI vendor that focuses on selling BI software to the end customer, not to the professional services community.

Make the Vendor Prove it To You.

The most important thing is to make the vendor prove what they claim prior to investing too much money upfront. This proof must be in the form of reports, dashboards or analytics in real life scenarios, running on real data, used by the actual end users and within a reasonable amount of time. If a vendor is not willing to accommodate this simple request, you really should find one that does. Many vendors provide free trial versions, as well as utilize technology that speeds up implementation tremendously. If the one you're in contact with now doesn't, they shouldn't make your short list.

Provided as public service to the business intelligence consumer community by SiSense, who is leading the way into a new era in business intelligence.

From OLAP Cubes to ElastiCubes – The Natural Evolution of BI

2010-10-18T16:30:00.000-07:00

OLAP (Online Analytical Processing) technology is the most prevalent technology used in corporate BI solutions today. And while it does what it’s supposed to do very well, it has a bad (and accurate) reputation for being very expensive and difficult to implement, as well as extremely challenging to maintain. This fact has prevented OLAP technology from gaining wide popularity outside of Fortune 500-scale companies, which are the only ones who have the budgets for company-wide, OLAP-based BI implementations.

Since the inception of BI and consequent entrance of OLAP technology into the space, the need for BI has been rapidly growing. Recognizing that OLAP-based solutions were (and still are) hard to introduce into a wider market, thought leaders and visionaries in the space have been since then trying to bring BI down to the masses through technological and conceptual innovation.

The most recently recognized innovation (even though it’s been around for quite a while) was in-memory technology, whose main advantage was cutting implementation time and simplifying the process as a whole (a definite step in the right direction). However, as described in my recent article, In-Memory BI is Not the Future, It's the Past, using in-memory technology for speedy BI implementation introduces significant compromises, especially in terms of scalability (both for data volumes and support for many concurrent users). Now, after in-memory technology has been on the market for some time, it is clear that it is not really a replacement for OLAP technology, but did in fact expand the BI market to a wider audience. In fact, it is probably more accurate to say that in-memory technology and OLAP technology complement each other, each with its own advantages and tradeoffs.

In that article I also briefly mentioned the new disk-based ElastiCube technology (invented by SiSense). ElastiCube technology basically eliminates the inherent IMDB tradeoffs by providing unlimited scalability using off-the-shelf hardware while delivering both implementation and query response times as fast (or faster) as pure in-memory-based solutions. This claim was the subject of many of the emails and inquires I received following the article’s publication. I was repeatedly asked how ElastiCube technology had achieved what OLAP technology had failed to do for so many years, and what role in-memory technology played in its conception.

Thus, in this article I will describe how ElastiCube technology came to be, what inspired it, what made it possible and how it has already become a game-changer in the BI space, both in large corporations and small startups.

A Brief History of BI and OLAP
OLAP technology started gaining popularity in the late 1990s, and that had a lot to do with Microsoft’s first release of their OLAP Services product (now Analysis Services), based on technology acquired from Panorama Software. At that point in time, computer hardware wasn’t nearly as powerful as it is today; given the circumstances at the time, OLAP was groundbreaking. It introduced a spectacular way for business users (typically analysts) to easily perform multidimensional analysis of large volumes of business data. When Microsoft’s Multidimensional Expressions language (MDX) came closer to becoming a standard, more and more client tools (e.g., Panorama NovaView, ProClarity) started popping up to provide even more power to these users.

While Microsoft was not the first BI vendor around, their OLAP Services product was unique and significantly helped increase overall awareness of the possibilities offered by BI. Microsoft started gaining market share fairly quickly, as more companies started investing in BI solutions.

But as the years passed by, it became very apparent that while the type of multidimensional BI empowered by OLAP technology was a valuable asset to any organization, it seemed to be used mainly by large corporations. OLAP is just too complex and requires too much time and money to be implemented and maintained, thus eliminating it as a viable option for the majority of the market.

See: Microsoft (SSAS), IBM (Cognos)

The Visualization Front-End Craze
As more companies began investing in BI solutions, many vendors recognized the great opportunity in bringing BI to the mass market of companies with less money to spend than Fortune 500 firms. This is where visualization front-end vendors started popping up like mushrooms after the rain, each of them promising advanced business analytics to the end user, with minimal or no IT projects involved. Their appeal was based on radically reducing the infamous total cost of ownership (TCO) of typical BI solutions. These products, many of which are still available today, are full of useful and advanced visualization features.

However, after years of selling these products, it became very clear that they are incapable of providing a true alternative to OLAP-based solutions. Since they fail to provide similar centralized data integration and management capabilities, they found themselves competing mainly with Excel, and were being used only for analysis and reporting of limited data sets by individuals or small workgroups.

In order to work around these limitations (and increase revenues), these tools were introduced connectivity to OLAP sources as well as to the tabular (e.g., spreadsheet) data they supported until then. By doing that, these products basically negated the purpose for which they were initially designed – to provide an alternative to the expensive OLAP-based BI solutions.

See: Tableau Software, Tibco SpotFire, Panorama Software

The In-Memory Opportunity
The proliferation of cheap and widely available 64-bit PCs during the past few years has somewhat changed the rules of the game. More RAM could be installed in a PC, a boon for those visualization front-end vendors struggling to get more market share. More RAM on a PC means that more data can be quickly queried. If crunching a million rows of data on a machine with only 2GB of RAM was a drag, users could now add more gigabytes of RAM to their PCs and instantly solve the problem. But still, without providing centralized data integration and management, this was not a true alternative to OLAP-based solutions that are still prominent in massive organization-wide (or even inter-departmental) implementations.

Strangely enough, out of all the in-memory technology vendors out there, only one realized that using in-memory technology to empower individual users wasn't enough and that the way to gain more significant market share was to provide an end-to-end solution, from ETL to centralized data sharing to a front-end development environment. This vendor is QlikTech and it is no wonder that the company is flying high above the rest of the non-OLAP BI players. QlikTech used in-memory technology to cover a much wider range of BI solutions than any single front-end visualization tool could ever do.

By providing data integration and centralized data access capabilities, QlikTech was able to provide solutions that, for other vendors (in-memory or otherwise), required at least a lengthy data warehouse project if not a full-blown OLAP implementation. By utilizing in-memory technology in conjunction with 64-bit computing, QlikTech solutions work even on substantial amounts of data (significantly more than their traditional disk-based competitors could).

However, QlikTech has not been able to make a case for replacing OLAP yet. I believe this is not only because of the scalability issues and hardware requirements involved when large amounts of data and/or users are involved, but it’s also because they do not inherently support dimensional modeling like OLAP does. Apart from making life simpler for IT when maintaining multiple applications, OLAP’s implementation of a dimensional model also gives end users, via supporting front end tools, a broader range of flexibility in creating their own BI applications.

Microsoft, the newest entry into the in-memory BI game, also started marketing its in-memory PowerPivot solution as an alternative to OLAP, basically admitting it gives up on its Analysis Services as a viable solution for the wider mid-market.

See: QlikTech (QlikView), Microsoft (PowerPivot)

The SaaS/Cloud BI Hype
The SaaS/Cloud hype hasn’t skipped over the BI space, though running BI in the cloud does not dramatically change anything in respect to implementation time and/or complexity of implementation. In fact, cloud BI vendors use the same technologies that are widely used on-premises. There are several startup companies in this space, competing for niche markets.

It’s still hard to tell what impact the cloud would have on the BI space as a whole as none of these companies has yet to prove there’s even a viable business for hosting BI in the cloud. One thing is certain, though: these companies cannot rely on in-memory technology to grow significantly. The costs of hardware and the amount of work required to support the number of customers they would need to thrive are prohibitive, to say the least. For more on the problem with cloud BI, see my earlier post, Would I Use Cloud Business Intelligence?

See: GoodData, YouCalc, Birst, PivotLink, Indicee

ElastiCube: Convergent Technologies for an Optimum Solution
ElastiCube technology was officially introduced to the market in late 2009, after more than five years of research and development conducted in complete secrecy. After being proved practical and effective in the real world (by being successfully implemented at over 100 companies, paying customers in numerous industries, from startups to multinational corporations), SiSense secured a $4 million investment to continue the development of the ElastiCube technology, and to expand awareness of the Prism Business Intelligence product which is based on the technology.

ElastiCube is the result of thoroughly analyzing the strengths and weaknesses of both OLAP and in-memory technologies, while taking into consideration the off-the-shelf hardware of today and tomorrow. The vision was to provide a true alternative to OLAP technology, without compromising on the speediness of the development cycle and query response times for which in-memory technologies are lauded. This would allow a single technology to be used in BI solutions of any scale, in any industry.

Here are the 10 main goals on which SiSense focused when designing the ElastiCube technology:

1. A data warehouse must not be assumed to exist for effectively querying multiple sources.

2. A star schema must not be assumed to exist for effective querying large amounts of data.

3. The solution must provide unlimited scalability, both in terms of number of rows and number of fields, within a finite and reasonable amount of RAM.

4. The solution must be able to operate using off-the-shelf hardware, even for extreme data/user scenarios.

5. The solution must provide high-speed, out-of-the-box query performance, without requiring pre-calculations.

6. There must be a separation between the application layer and the physical data layer via a virtual metadata layer.

7. There must be support for a dimensional model and multidimensional analysis.

8. The same application must be able to support a single user with a laptop to thousands of users via a central, server-based data repository.

9. Without running an SQL database, an SQL layer must be available to conform to industry standards.

10. The solution must offer the ability to incorporate additional/changed data (e.g., new rows, new fields) on the fly, without reprocessing the entire data model.

The presently available version of Prism, based on ElastiCube technology, delivers on every one of these requirements. Even though it would be a lot of fun for me, I unfortunately can’t delve into the nuts and bolts of how these goals are technologically achieved. What I can say is that ElastiCube utilizes columnar storage concepts as well as just-in-time in-memory query processing technology. If you want to read a little about it, you can take a look at SiSense’s ElastiCube technology page.

I can add that the feasibility of ElastiCube was greatly affected by the amazing CPU and disk technologies that now come with any run-of-the-mill personal computer.

ElastiCube is extremely powerful technology that enables speedy implementation of individual, workgroup and corporate-wide BI. As a solution that delivers the promise of OLAP-style BI without the cost, time and IT overhead of OLAP, it is no surprise that Prism is rapidly gaining popularity in the market. Businesses that use ElastiCube technology include household names such as, Target, Yahoo, Cisco, Samsung, Philips and Caterpillar. But a significant portion of business that use ElastiCube are significantly smaller, such as Wix and other startup companies - who otherwise could not afford BI at all.
See: SiSense (Prism)

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

In-memory BI is not the future. It’s the past.

2010-09-24T07:25:00.000-07:00

In recent times, one of the most popular subjects related to the field of Business Intelligence (BI) has been In-memory BI technology. The subject gained popularity largely due to the success of QlikTech, provider of the in-memory-based QlikView BI product. Following QlikTech’s lead, many other BI vendors have jumped on the in-memory “hype wagon,” including the software giant, Microsoft, which has been aggressively marketing PowerPivot, their own in-memory database engine.

The increasing hype surrounding in-memory BI has caused BI consultants, analysts and even vendors to spew out endless articles, blog posts and white papers on the subject, many of which have also gone the extra mile to describe in-memory technology as the future of business intelligence, the death blow to the data warehouse and the swan song of OLAP technology.

I find one of these in my inbox every couple of weeks.

Just so it is clear - the concept of in-memory business intelligence is not new. It has been around for many years. The only reason it became widely known recently is because it wasn’t feasible before 64-bit computing became commonly available. Before 64-bit processors, the maximum amount of RAM a computer could utilize was barely 4GB, which is hardly enough to accommodate even the simplest of multi-user BI solutions. Only when 64-bit systems became cheap enough did it became possible to consider in-memory technology as a practical option for BI.

The success of QlikTech and the relentless activities of Microsoft’s marketing machine have managed to confuse many in terms of what role in-memory technology plays in BI implementations. And that is why many of the articles out there, which are written by marketers or market analysts who are not proficient in the internal workings of database technology (and assume their readers aren’t either), are usually filled with inaccuracies and, in many cases, pure nonsense.

The purpose of this article is to put both in-memory and disk-based BI technologies in perspective, explain the differences between them and finally lay out, in simple terms, why disk-based BI technology isn’t on its way to extinction. Rather, disk-based BI technology is evolving into something that will significantly limit the use of in-memory technology in typical BI implementations.

But before we get to that, for the sake of those who are not very familiar with in-memory BI technology, here’s a brief introduction to the topic.

Disk and RAM

Generally speaking, your computer has two types of data storage mechanisms – disk (often called a hard disk) and RAM (random access memory). The important differences between them (for this discussion) are outlined in the following table:

Most modern computers have 15-100 times more available disk storage than they do RAM. My laptop, for example, has 8GB of RAM and 300GB of available disk space. However, reading data from disk is much slower than reading the same data from RAM. This is one of the reasons why 1GB of RAM costs approximately 320 times that of 1GB of disk space.

Another important distinction is what happens to the data when the computer is powered down: data stored on disk is unaffected (which is why your saved documents are still there the next time you turn on your computer), but data residing in RAM is instantly lost. So, while you don’t have to re-create your disk-stored Microsoft Word documents after a reboot, you do have to re-load the operating system, re-launch the word processor and reload your document. This is because applications and their internal data are partly, if not entirely, stored in RAM while they are running.

Disk-based Databases and In-memory Databases

Now that we have a general idea of what the basic differences between disk and RAM are, what are the differences between disk-based and in-memory databases? Well, all data is always kept on hard disks (so that they are saved even when the power goes down). When we talk about whether a database is disk-based or in-memory, we are talking about where the data resides while it is actively being queried by an application: with disk-based databases, the data is queried while stored on disk and with in-memory databases, the data being queried is first loaded into RAM.

Disk-based databases are engineered to efficiently query data residing on the hard drive. At a very basic level, these databases assume that the entire data cannot fit inside the relatively small amount of RAM available and therefore must have very efficient disk reads in order for queries to be returned within a reasonable time frame. The engineers of such databases have the benefit of unlimited storage, but must face the challenges of relying on relatively slow disk operations.

On the other hand, in-memory databases work under the opposite assumption that the data can, in fact, fit entirely inside the RAM. The engineers of in-memory databases benefit from utilizing the fastest storage system a computer has (RAM), but have much less of it at their disposal.

That is the fundamental trade-off in disk-based and in-memory technologies: faster reads and limited amounts of data versus slower reads and practically unlimited amounts of data. These are two critical considerations for business intelligence applications, as it is important both to have fast query response times and to have access to as much data as possible.

The Data Challenge

A business intelligence solution (almost) always has a single data store at its center. This data store is usually called a database, data warehouse, data mart or OLAP cube. This is where the data that can be queried by the BI application is stored.

The challenges in creating this data store using traditional disk-based technologies is what gave in-memory technology its 15 minutes (ok, maybe 30 minutes) of fame. Having the entire data model stored inside RAM allowed bypassing some of the challenges encountered by their disk-based counterparts, namely the issue of query response times or ‘slow queries.’

Disk-based BI

When saying ‘traditional disk-based’ technologies, we typically mean relational database management systems (RDBMS) such as SQL Server, Oracle, MySQL and many others. It’s true that having a BI solution perform well using these types of databases as their backbone is far more challenging than simply shoving the entire data model into RAM, where performance gains would be immediate due to the fact RAM is so much faster than disk.

It’s commonly thought that relational databases are too slow for BI queries over data in (or close to) its raw form due to the fact they are disk-based. The truth is, however, that it’s because of how they use the disk and how often they use it.

Relational databases were designed with transactional processing in mind. But having a database be able to support high-performance insertions and updates of transactions (i.e., rows in a table) as well as properly accommodating the types of queries typically executed in BI solutions (e.g., aggregating, grouping, joining) is impossible. These are two mutually-exclusive engineering goals, that is to say they require completely different architectures at the very core. You simply can’t use the same approach to ideally achieve both.

In addition, the standard query language used to extract transactions from relational databases (SQL) is syntactically designed for the efficient fetching of rows, while rare are the cases in BI where you would need to scan or retrieve an entire row of data. It is nearly impossible to formulate an efficient BI query using SQL syntax.

So while relational databases are great as the backbone of operational applications such as CRM, ERP or Web sites, where transactions are frequently and simultaneously inserted, they are a poor choice for supporting analytic applications which usually involve simultaneous retrieval of partial rows along with heavy calculations.

In-memory BI

In-memory databases approach the querying problem by loading the entire dataset into RAM. In so doing, they remove the need to access the disk to run queries, thus gaining an immediate and substantial performance advantage (simply because scanning data in RAM is orders of magnitude faster than reading it from disk). Some of these databases introduce additional optimizations which further improve performance. Most of them also employ compression techniques to represent even more data in the same amount of RAM.

Regardless of what fancy footwork is used with an in-memory database, storing the entire dataset in RAM has a serious implication: the amount of data you can query with in-memory technology is limited by the amount of free RAM available, and there will always be much less available RAM than available disk space.

The bottom line is that this limited memory space means that the quality and effectiveness of your BI application will be hindered: the more historical data to which you have access and/or the more fields you can query, the better analysis, insight and, well, intelligence you can get.

You could add more and more RAM, but then the hardware you require becomes exponentially more expensive. The fact that 64-bit computers are cheap and can theoretically support unlimited amounts of RAM does not mean they actually do in practice. A standard desktop-class (read: cheap) computer with standard hardware physically supports up to 12GB of RAM today. If you need more, you can move on to a different class of computer which costs about twice as much and will allow you up to 64GB. Beyond 64GB, you can no longer use what is categorized as a personal computer but will require a full-blown server which brings you into very expensive computing territory.

It is also important to understand that the amount of RAM you need is not only affected by the amount of data you have, but also by the number of people simultaneously querying it. Having 5-10 people using the same in-memory BI application could easily double the amount of RAM required for intermediate calculations that need to be performed to generate the query results. A key success factor in most BI solutions is having a large number of users, so you need to tread carefully when considering in-memory technology for real-world BI. Otherwise, your hardware costs may spiral beyond what you are willing or able to spend (today, or in the future as your needs increase).

There are other implications to having your data model stored in memory, such as having to re-load it from disk to RAM every time the computer reboots and not being able to use the computer for anything other than the particular data model you’re using because its RAM is all used up.

A Note about QlikView and PowerPivot In-memory Technologies

QlikTech is the most active in-memory BI player out there so their QlikView in-memory technology is worth addressing in its own right. It has been repeatedly described as “unique, patented associative technology” but, in fact, there is nothing “associative” about QlikView’s in-memory technology. QlikView uses a simple tabular data model, stored entirely in-memory, with basic token-based compression applied to it. In QlikView’s case, the word associative relates to the functionality of its user interface, not how the data model is physically stored. Associative databases are a completely different beast and have nothing in common with QlikView’s technology.

PowerPivot uses a similar concept, but is engineered somewhat differently due to the fact it’s meant to be used largely within Excel. In this respect, PowerPivot relies on a columnar approach to storage that is better suited for the types of calculations conducted in Excel 2010, as well as for compression. Quality of compression is a significant differentiator between in-memory technologies as better compression means that you can store more data in the same amount RAM (i.e., more data is available for users to query). In its current version, however, PowerPivot is still very limited in the amounts of data it supports and requires a ridiculous amount of RAM.

The Present and Future Technologies

The destiny of BI lies in technologies that leverage the respective benefits of both disk-based and in-memory technologies to deliver fast query responses and extensive multi-user access without monstrous hardware requirements. Obviously, these technologies cannot be based on relational databases, but they must also not be designed to assume a massive amount of RAM, which is a very scarce resource.

These types of technologies are not theoretical anymore and are already utilized by businesses worldwide. Some are designed to distribute different portions of complex queries across multiple cheaper computers (this is a good option for cloud-based BI systems) and some are designed to take advantage of 21st-century hardware (multi-core architectures, upgraded CPU cache sizes, etc.) to extract more juice from off-the-shelf computers.

A Final Note: ElastiCube Technology

The technology developed by the company I co-founded, SiSense, belongs to the latter category. That is, SiSense utilizes technology which combines the best of disk-based and in-memory solutions, essentially eliminating the downsides of each. SiSense’s BI product, Prism, enables a standard PC to deliver a much wider variety of BI solutions, even when very large amounts of data, large numbers of users and/or large numbers of data sources are involved, as is the case in typical BI projects.

When we began our research at SiSense, our technological assumption was that it is possible to achieve in-memory-class query response times, even for hundreds of users simultaneously accessing massive data sets, while keeping the data (mostly) stored on disk. The result of our hybrid disk-based/in-memory technology is a BI solution based on what we now call ElastiCube, after which this blog is named. You can read more about this technological approach, which we call Just-in-Time In-memory Processing, at our BI Software Evolved technology page.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Business Intelligence Vendors and their Partners – Rough Seas Ahead

2010-09-01T06:33:00.000-07:00

The traditional business intelligence ecosystem is built on the numerous strategic partnerships that exist between BI software vendors, which provide the technology, and value added resellers (VARs), which provide customized solutions based on that technology.

The Relationship between BI Software Vendors and their VARs

As in all partnerships, both sides need to have something significant to gain for their partnership to be successful. In the business intelligence industry, this has indeed been the case for a long time. The software vendors use their channel partners to distribute their software to a larger audience and these, in turn, have made a pretty penny from commissions, consulting and implementation fees.

There has always been a distinct difference, however, between the business goals software vendors set for themselves and those sought after by their VARs.

The software vendors, for their part, want to sell as many software licenses as they can to new customers, as well as to charge software maintenance fees from their existing clientele. This provides them a steady income stream from existing customers while new customers grow the business. Their partners, on the other hand, prefer long and complex implementation projects from which they generate significantly more revenue than they do from commissions on software license sales.

This symbiosis used to be great. Since most traditional BI companies are focused on high-end corporations with huge budgets, there was enough to go around. These customers have large numbers of employees who can benefit from BI (read: big money selling software licenses for the software vendors) and who have no problem spending hundreds of thousands (or millions) of dollars on implementation projects (read: significant income from project fees for the implementer).

Mutually Beneficial Relationships?

It so happens, however, that changing conditions over the past couple of years (and particularly during 2010) have brought the traditional business intelligence industry to a point where the mutual vendor-VAR benefits are not as obvious anymore. While these conditions have contributed to a deterioration in relationships between BI software vendors and their partners, the good news is that companies exploring business intelligence options stand to benefit substantially from the situation.

Let’s take a look at some of the conditions affecting the BI industry in recent years:

1. Tough Economic Times

Obviously, the economic crisis which began in 2008 affected everyone, vendors and customers alike. Business intelligence as a concept was actually positively affected by this crisis as it became painfully obvious how important it is to track a business’s operational and financial performance. On the other hand, available budgets shrank significantly and there was a smaller pie to share between BI software vendors and their partners. This fact has been causing friction between the two sides as each attempts to vigorously protect its own piece of the pie.

2. Too Many Partners

In an attempt to gain more market share, software vendors invested extra effort in recruiting more and more VARs for their partner networks. While this had a positive effect on software vendors’ revenues, it wasn’t as good for those in the partner network. Having more partners leads to more competition which, in turn, means more investment in marketing and sales (and lower profits). To make matters worse, in a further attempt to increase revenues, some software vendors actually began competing with their own partners on implementation deals.

3. QlikTech and their IPO

Ever since QlikTech began gaining popularity, their main sales pitch has been shorter implementation times and reduced ongoing costs (due to the supposedly fewer IT personnel required to maintain their BI solution). While this holds mighty appeal to BI customers, it flies in the face of the entire premise of BI resellers, which rely on project implementation and BI maintenance revenues. QlikTech addressed this issue by providing their VARs higher commissions on software license sales (as compared to those offered by Microsoft, Cognos or Business Objects, for example). Coupled with the implementation and maintenance work a QlikTech solution still requires, the higher commissions provide reasonable revenues for their partners.

Along with their impressive sales and growth numbers, QlikTech’s recent IPO revealed that they generated $157M in revenues during 2009 with total expenses of $150M. The resulting profit of $7M is not great.

Whether QlikTech’s intentions are to be acquired soon or to keep growing their business remains a mystery, but either way their partners should pay close attention. If they do seek a quick exit, their partners face an uncertain future. If they intend on growing their business and improving profitability, they will have to raise their prices and/or expand their partner network significantly and/or increase their direct involvement in both software sales and implementation. Existing partners will not be pleased with either of these alternatives.

As the successful pioneer of a newer, faster, easier approach to BI, the QlikTech example should be considered carefully by VARs as an indication of what the future may hold for the BI industry as whole.

4. The Self-Service BI Hype

The hottest thing in the BI industry today is the self-service BI concept. Regardless of whether it’s promoted by vendors providing personal analysis tools or cloud BI platforms, the basic idea behind it is the same: traditional BI is too expensive, takes too long to implement and is a big pain to maintain. Instead, the customer wants tools to enable self-reliance (as opposed to relying on external consultants/implementers who live off service fees). Whether these solutions actually deliver what they promise is beside the point (you can read my opinion about cloud BI here), but the buzz is out there and the market hears it, so it’s getting harder these days to justify long and expensive BI projects.

5. Microsoft PowerPivot

PowerPivot is Microsoft’s attempt to promote the self-service BI concept. By introducing PowerPivot, Microsoft is basically giving up on penetrating the mid-market with SQL Server Analysis Services and is trying instead to do it by introducing stronger BI capabilities in their Office product. While some believe that PowerPivot is just a lot of hot air, the fact remains that Microsoft is investing a lot of effort and money on marketing it. This places their existing partners – who rely on SQL Server sales – in a very problematic situation. These partners prefer SQL Server-based solutions, which provide more license commissions and more project hours, yet they need to fight Microsoft’s own marketing machine which is now essentially promoting self-service BI. Not an enviable situation to be in, to say the least.

What Does the Future Hold?

It’s great that so much emphasis is being placed on simplifying business intelligence and making it accessible to companies that do not have multimillion dollar budgets. Since established players and new startups alike are now beginning to focus on this type of approach, it is actually realistic to expect that self-service BI is on its way to gradually becoming a commodity. Customers will benefit greatly from this trend.

On the other hand, business intelligence VARs must understand that this is where the market is headed – and adjust their business models accordingly. A company selling BI solutions based on existing BI platforms will need to provide real added value to the customer in order to stay in business. In the not-too-distant future, this value will almost certainly come from industry-specific professional knowledge and experience (as opposed to purely technical expertise). More and more customers will no longer accept lengthy R&D projects to achieve BI and, with the new software and technologies now emerging, it is no longer justifiable.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Business Intelligence? Yes Minister!

2010-08-24T06:32:00.000-07:00

I recently watched an episode of the classic BBC series ‘Yes, Minister’.
In this episode, the minister asked his assistant, Bernard, to inquire about a new hospital where there are supposedly no patients and a ridiculous amount of administrative staff. Bernard conducts some research and returns to the minister with his results.

Here’s a short transcript:
Bernard- You asked me to find out about an alleged empty hospital in north London.

Minister- Oh, yes.

Bernard- Well, in fact there are *only* 342 administrative staff in the new St. Edwards hospital. The other 170 are porters, cleaners, laundry workers, gardeners, cooks and so forth.

Minister- And how many medical staff?

Bernard- Oh, none of those.

Minister- None??

Bernard- No.

Minister- Bernard, we are talking about St. Edwards’s hospital, aren’t we?

Bernard- Yes, it’s brand new. It was completed 15 months ago and is fully staffed. But unfortunately at that time there were government cutbacks and consequently there was no money left for medical services.

Minister- A brand new hospital with over 500 non-medical staff and no patients??

Bernard- Oh, there is one patient, sir.

Minister- One?

Bernard- Yes. The deputy chief administrator fell over a piece of scaffolding and broke his leg

Minister- Oh god. Thank heavens I wasn’t asked about this in The House. Why hasn’t it got out?

Bernard- Well actually I think it’s being contrived to keep looking like a building site and so far no one realized it’s operational.

Minister- I think I better go and have a look at this before the opposition does.

This episode (‘The Compassionate Society’, 1981) is hilarious. But how is it relevant to business intelligence?

Well, think that instead of a conversation between a minister and his assistant about a new hospital, this conversation is actually between a CEO and his CIO regarding an ongoing business intelligence project. You've got yourself a conversation which is still happening more often than not in BI projects, 30 years later.

A business intelligence solution without users is like a hospital without patients. It provides work for IT/implementers/consultants (=administrative staff) but it is of no use to the business users (=patients) for whom the BI solution (=hospital) is actually intended.

Spending too much upfront and going into long development cycles only to find out your ultimate customers - the business users - won't actually use it is a sure-fire way of making a bad investment, running out of budget and getting laughed at by the BBC. You don't want that, do you?

For tips on how to approach your next BI project, check out this post.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

Is Microsoft Admitting that Analysis Services is not Fit for the Mid-Market?

2010-08-12T03:51:00.000-07:00

I recently read an article posted on the SQL Server team's blog (Technet) written by Shimon Shlevich, a product manager at Panorama Software, focusing on Microsoft's recently-launched PowerPivot in-memory offering.

According to the author, Microsoft has two main goals with PowerPivot: to "introduce a new in-memory engine for data processing" and to "promote the self-service BI concept extending the usage of BI systems to a wider audience."

There are, of course, other reasons which the author did not mention, such as Microsoft trying to get a fighting chance against QlikView, which has been constantly beating Microsoft at mid-sized and departmental deals.

In addition, Microsoft is trying to motivate their customers to upgrade to Excel 2010, in which PowerPivot is provided for free in the form of an add-in. Microsoft is not a natural BI company and their cash cows are still Windows and Office, so it only makes sense. Will it work? Who knows. Will it change the BI space? Probably not.

To me, the most interesting thing about this post is the fact that PowerPivot is meant to promote the self-service BI concept, which in most people's minds is the complete and utter opposite of what Analysis Services delivers, namely a heavy, IT-centric business intelligence solution.

If this is true, Microsoft is basically admitting on their own blog that Analysis Services has failed to provide a viable solution for mid-sized companies and departments (where self-service BI is widely used) and that their new BI marketing strategy is based on Office, not SQL Server.

This fact is well known to people who are familiar with the trends and nuances of the BI space, but Microsoft saying this on their blog is, to me, a very big deal.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

The Google AdWords Broad Match Modifier

2010-08-05T12:16:00.000-07:00

A couple of months ago, Google added a modifier option for the Broad Match type. This option basically allows you to control which words must appear in exact or synonymous form within the search phrase.

For the record, I have always disliked the Broad matching option. Especially since Google introduced Expanded Broad matching where Google shows you ads for every word they deem close enough to the actual word you bid on, not just the exact word.

Ever since they did that, it's become very difficult to control Adwords campaigns because the search phrases you bid on become much less targeted and cost you many irrelevant click$.

The new Broad match modifier is a step in the right direction as it allows you to better control the phrases that bring up your ad.

Let's say you bid on the phrase: Banana Cake Recipe
Setting this phrase to Broad matching, will show your ad to surfers searching for Chocolate Cake Store and Growing Bananas in Brazil which may be irrelevant to what you're advertising.

If you use the new modifier like so: +Banana +Cake +Recipe
Your ad will show only to phrases that contain some variation of all three words. That means that the search phrase must contain all three words, or whichever word Google decide is synonymous to the each of the words you specified. They may, for example, bring your ad up to someone searching for Banana Muffin Recipe. You may like that or you may not.

You can also choose to apply this modifier to only a subset of the words in the phrase, but then you need to determine whether it's worth your while to split the phrase into two separate ones.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog

The Catch 22 of Traditional Business Intelligence

2010-08-05T03:24:00.000-07:00

So much has already been said about how much of a pain business intelligence is. The complexity, the constant IT bottlenecks, the crazy cost of hardware, software, consultants and whatnot. Gil Dibner of Gemini Venture Funds (formerly of Genesis Partners) described it very eloquently and in great detail in his blog post about the SiSense investment round.

Since business intelligence imposes so many limitations and challenges, every existing BI vendor picks its favorite ones and positions itself as the best at addressing those specific ones. Some focus on providing easy-to-use front-end tools for the business user, some on handling complex ETL scenarios and large data sets, others on open source software to remove software licensing costs and so on.

Business intelligence vendors have been constantly rolling out new functionality and technology through the years. But still, it seems like business intelligence has been standing still. No progress has been made in expanding it to the wider market that can't afford long and costly development/customization cycles. In fact, most of the BI vendors which do not sell enterprise-class solutions (e.g., SAP, IBM and Microsoft) haven't been able to grow much and remain focused on niche markets.

Well, my friends, it's time somebody told it to you straight.

Business intelligence can deliver on its promise, but the entire idea needs a complete overhaul. As long as vendors keep improving specific functions within the traditional BI paradigm, no progress will be made. The traditional business intelligence paradigm needs to be scrapped and replaced by something that is much easier, much faster, much cheaper and much more manageable.

What is the core of the traditional BI problems? The traditional paradigm contains an inherent flaw that prevents it from taking BI to the next level where ROI is indisputable and business users get another powerful tool added their arsenal - in companies of all (or most) sizes.

The Inherent Flaw in the Traditional BI Paradigm

If you search "why business intelligence projects fail" in Google you will find an abundance of white papers and articles (mostly written by BI vendors themselves) giving their two cents' worth. When BI vendors pick their top reasons, they usually pick issues dealt with by their offerings and not by the competitions'. Marketing 101. Fair enough.

But one top reason on which they all seem to agree for a BI project's failure is the lack of up-front planning. That is to say, in order for a business intelligence project to succeed, you must compile your requirements ahead of time, coordinate with all the relevant parties (IT, business departments and executives) and plan the project in accordance to those requirements. Otherwise, you are destined to fail.

In other words, they blame you - the consumer - for a failed BI project. Had you planned ahead, the project would have been a success and you wouldn't have flushed hundreds of thousands of dollars worth of software licenses, hardware and personnel time down the toilet.

Sadly, they have a point. Since traditional BI solutions aren't very sympathetic to unplanned changes from an architectural point of view, anything you don't think of in advance is difficult and expensive to introduce later. So you better think long and hard about what you need, because otherwise any requirements you miss could mean the difference between a successful project and a complete mess.

But herein lies the catch.

It doesn't matter who you are or how much experience you have, it is utterly impossible to know in advance what your requirements are when it comes to BI. BI is highly dynamic and requirements change all the time because the business changes all the time and because business users come up with new ideas all the time. A report you need now is not the report you need later, an analysis you do now may only be relevant for a short period of time and meaningless shortly thereafter.

Most importantly - if you are a department or company seeking a BI solution but without any BI development experience, you have no way of knowing how a particular requirement will affect the architecture of your solution. Thus, you could easily find yourself disregarding the immediate testing of some particular capability because it seems trivial to you, just to discover later that the entire solution comes tumbling down when you actually try to use it, and that without it the system is useless.

You cannot imagine how often this happens, especially when a solution calls for OLAP cubes built over a data warehouse (blah).

It's the traditional BI vendors who made up the rules for this game over 10 years ago. They are the ones who've been aggressively promoting a paradigm where everything needs to be thought of in advance - otherwise you are sure to fail. It makes sense because these vendors focus on enterprise-wide BI for fortune 500 companies where the complexity of a BI project is masked by the complexity of the corporation's own business processes. These organizations are used to things taking years to reach perfection because every other process they have pretty much takes the same amount of time.

But trying to implement the same concepts on slightly smaller corporations is the exact reason why most BI projects fail.

Don't get me wrong. It's always good to plan ahead. But know this - business intelligence requirements are impossible to predict and nearly impossible to measure until the end users use it on real data - in real-life scenarios - over time.

You cannot do this with traditional BI without investing a TON beforehand, and even then you have no guarantees. When you go for BI as advocated by the traditional platform players, you are basically throwing hundred dollar bills down a wishing well and hoping for the best.

Learn from the thousands and thousands of companies who have already learned this harsh lesson with blood and tears. Don't do it. There are ways to change the rules of the game while still getting the same class of business intelligence, without compromising on functionality, speed or capacity. But you cannot expect to find it by turning to the traditional BI players who have an over-sized BI developer eco-system for which they need to provide work. This can only be done by younger, innovative BI companies armed with new technologies, fresh ideas and sensible pricing models.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog