Data Warehousing

Above and Beyond Technical Debates - Columnar Databases

2009-07-07T14:33:28+00:00

As this is my first blog post as the new CTO and VP Product Management at Infobright, I would like to take a few minutes of your precious time to reflect upon my first 30 days with this great company. I started at Infobright on June 1st, 2009 after spending a great deal of time with our CEO Miriam Tuerk and members of the Infobright team. After several days, I came to the conclusion that Infobright is a terrific company with a team of professionals that are very talented, have a great work ethic, are of high integrity and most importantly of all, are very passionate about our products and our customers and partners. This, I decided was the type of team and technology that would be a great fit for me. After 30 days, I am finding Infobright a great company to be part of. I feel very privileged to have a chance to work with such a terrific group of smart professionals. What’s very cool is that when I tell people in the data management space that I work for Infobright, every single person that was familiar with Infobright told me it was a great company. That has been terrific to hear.

Having spent many years in the database management space either working with great teams to build tools that run on databases (Powersoft/PowerBuilder, Powersite, ObjectCycle, Lumapath, Centive, Ascential) or being part of a team behind databases from the likes of Sybase, iAnywhere (part of Sybase), IBM and EnterpriseDB, I’ve been fascinated by all the technical debates that exist in this terrific market. Some of my favorites are from industry pundits and respected analysts such as Curt Monash who is always driving some very critical debates in the database space or the insight emanating from academics, peer CTO’s or database architects. Just recently our Chief Scientist, Dominik Slezak and Victoria Eastwood our VP of Engineering came back from the ACM SIGMOD with all kinds of insight from the various sessions they attended that further add to the discussions and debates around the topic of columnar databases.

Above and beyond the technical debates that exist in the market around the topic of columnar stores, which again I enjoy immensely, we must do our best to look beyond these healthy and productive debates by continually focusing on solving real world customer pain and by working hard to deliver compelling business value and benefits to our respective customers and partners. Customer success means a great deal to us and that was an important attribute that I saw in Infobright. The customer passion was very visible when speaking to members of our team. Our team also recognizes that we have a strong opportunity with our solution in the market, but they also recognize that we have lots of work to do as would any company at our stage in the market. The good news, lots of great innovations are taking place here at Infobright and the contributors on our open source offering are starting to contribute, which is very exciting. We can’t thank them enough. I’m very excited about seeing our 3.2 release become available in alpha form and look forward to the input and feedback as we move towards our release candidates and production versions. Coming on board at the start of the next major release is a great opportunity and I’m very excited about what we have in store for our customers and partners as we move forward.

According to a 2008 Forrester Research report, the market for open source databases is project to climb from $850 million to $1.2 billion by 2010. No question about it, we are in for some exciting times in this industry and I look forward to being part of this great team at Infobright. Thanks for taking the time to read the post and look forward to posting more as we travel down our path at Infobright.

SIGMOD 2009 Update #3

2009-07-03T20:27:05+00:00

After Many Eyes (see Update #2) I got so excited that I couldn’t focus on anything. According to the plan, I attended Transactions, Security, and Caching but my thoughts were elsewhere. (Sorry.) The thoughts came back just before Surajit Chaudhuri’s talk in the Industrial Directions session. I treat it as a very good summary of both current and future directions for extending query optimizers by paying more attention to the usage information, which – if appropriately gathered – may provide the optimizer with more hints regarding the actual data distributions, the current state of the database, the contextual knowledge about the domain of application, et cetera.

The discussion after the talk was quite inspiring too. One of the questions – given that the talk referred broadly to the example of cardinality estimation – was whether cardinality is still something that we should be so interested in, in particular in the case of technologies based on massive sequential scans (including column scans in the columnar stores). I fully agree with the speaker’s answer that it may be important anyway in order to optimize the size of intermediate structures occurring for more complex queries. Nevertheless, let’s remember that the cardinality estimation is just an example of what the optimizer may rely on. In general, I can feel that practically all the currently existing query optimization approaches might benefit from extensions described in this talk, of course after appropriate translation.

Last but not least, let me briefly summarize the closing part – Award Presentations.

In the first talk, Masaru Kitsuregawa (Edgar Codd Innovations Award) referred to the history of hash-joins and, more generally, parallel computing in database applications. It was especially interesting to listen to the story of parallel development of Functional Disk System in Japan and Gamma System in USA. It’s also worth looking at the newest Prof. Kitsuregawa’s Information Grand Voyage project.

In the second talk, Jeffrey S. Vitter and Min Wang (Test of Time Award) spoke about wavelets. Let me refer to ICDE 2009 (Update #2), where the analogous award was granted to the wavelet applications as well. It looks like 1999 was a magic database year for wavelets, with two significant papers (SIGMOD 1999 and ICDE 1999) in the areas of data processing and data analysis.

In the third talk, Daniel Abadi (Jim Gray Dissertation Award) described his contribution to C-Store – the second famous (after Peter Boncz’s dissertation on MonetDB) academic project related to column stores. The main invention of C-Store – creation of multiple partial data copies (projections) sorted in multiple ways to provide optimal basis for query execution – has become the starting point of Vertica’s commercial product. In the meantime, Daniel keeps searching for new research directions and has just turned out to be a brilliant blogger. I really admire his ability to combine academic and industry interests, which is crucial for the scientific progress.

The forth talk was by Georgia Koutrika on CourseRank: A Social System for Course Planning developed at Stanford (Best Demonstrations Award). If you missed it, no problem – you can always attend the online lecture!

As a summary, the last three days passed very quickly. (It’s always like this when there’s something interesting going on.) I wish I attended the sessions on Monday, especially the Web of Concepts by Raghu Ramakrishnan. (I still remember his remote keynote earlier this year.) But let’s look into the future, to SIGMOD 2010 in Indianapolis!

Best greetings and keep in touch – Victoria has just posted her SIGMOD summary too,

Dominik

SIGMOD 2009 Update #2

2009-07-02T17:00:45+00:00

Sitting at keynote #2 by Many Eyes – a “living laboratory” web site where people may upload their own data, create interactive visualizations, and carry on conversations. I immediately thought about uploading all our forums in order to visualize the main trends of discussions. Indeed, Many Eyes provides interesting tools for the text visualization – something that is more difficult and often more important than handling numeric information. Also, whenever I attend the talks about data/knowledge visualization/representation, I automatically recall such topics as approximate querying (do we need exact answers to visualize information?), our discussions whether and how to visualize our knowledge grid, as well as general aspects of business intelligence, data mining and human-computer interaction.

But let me go back to yesterday’s sessions…

After the already-mentioned Data Services session, I attended Query Optimization. Among the others, I liked ParAccel’s talk about additions to the PostgreSQL query optimizer, although I fully agree (that was one of the questions after the talk) that the columnar store specifics should be taken into account at this level to a larger degree. Further, in the Systems Research session, I enjoyed the overview of today’s leading Storage Class Memory technologies (I should put it together with the questions after our talk and our experience with the Sun’s Open Storage), as well as DryadLINQ, which is Microsoft’s way to make distributed computing on large compute cluster simple enough for ordinary programmers.

In the meantime, we had a pleasure to attend the 40 Years of Relational Model Celebration with a number of excellent presentations about Edgar Codd and the history of the whole idea, from both academic and industry perspectives. It was also a pleasure to meet and discuss with Curt Monash. Finally, after the Tour of Newport (I was supposed to go to Harbor Cruise but I thought the tour would be more interesting – and I’m not disappointed!), it was a pleasure to talk with the Tableau people during the banquet (we should try to do something more together).

All together, I do like SIGMOD.

More to come,

Dominik

SIGMOD 2009 Update #1

2009-07-01T15:10:11+00:00

When attending ICDE 2009, I blogged every evening. This time I wanted to blog right after waking up. However, it doesn’t work so well. There is always something to be done in the morning – emails, forums, urgent things that you forgot to do yesterday… As a result, I’m already sitting at the Data Services session (I switched from the Database Optimization session in the very last moment) and I’m still gathering my thoughts…

Yesterday’s Keynote was excellent! Hasso Plattner – the co-founder of SAP and the head of the HPI Institute – combined general observations on the evolution of OLTP and OLAP systems with a specific case of SAP applications. Indeed, there is a growing interest in technologies that would be able to combine OLTP and OLAP features. You can find some relevant examples in Section 3. (I like especially the categorization of the types of updates.) Indeed, the proposed in-memory column store approach may turn out to be the way to go. (Certainly, there are some technical details to be further discussed, e.g., the SAP’s dictionary-based compression that was also presented in the Column Stores session. However, in general, isn’t it surprising to see the column store-based mechanisms as an efficient support for OLTP?!)

I also liked the discussion after the Keynote. Obviously, implementation of the idea of in-memory column stores in production would require a significant development effort. So, one of the questions was whether SAP could be potentially ready to make such an effort. Prof. Plattner’s answer was honest and straightforward: If you can see that your technology does not meet requirements any longer, if you can see a good alternative, you have no choice – you need to rewrite the code. Actually, SAP has already done a similar exercise in the past…

Besides the Keynote, Victoria and I enjoyed very inspiring presentations on MonetDB in the already-mentioned Column Stores session, the comparative “Shared-Nothing-MapReduce” analysis in the Large-Scale Data session (with a very hot discussion, as usual), and the Peta-Scale Yahoo! presentation in the Data Warehousing morning session. Of course we also enjoyed presenting Infobright!!! (See the forum for details and the presentation attached.)

Best greetings,

Dominik

Good Morning SIGMOD!

2009-06-30T11:06:56+00:00

30^th of June, 6:53 AM, Providence. Three months after ICDE 2009. A couple of hours left to the opening of SIGMOD 2009. Just like before, I was late for the workshops, as well as the first part of PODS. Just like before, I’m going to use the blog for the daily updates and the forum for more real-time communication.

I invite everyone to join discussion!

The first day starts with the keynote on Enterprise Applications - OLTP and OLAP - Share One Database Architecture. It refers, in particular, to in-memory column databases. Very interesting! Have a look at the paper related to this talk. – It’s available for everybody, not only the SIGMOD/PODS participants.

The next item for today is the Industry Session 1. The first talk (10:30-11:00) is about Infobright. Together with Victoria Eastwood, we’ll present the current state of technology and some research directions. I’ll summarize it in the next update. I’ll also attach the final version of presentation.

Best greetings,

Dominik

December Conferences In India - Experience Workshop

2009-06-09T18:35:23+00:00

I’m getting more and more excited about attending the December 16-20 conferences in Delhi. (PReMI 2009 and RSFDGrC 2009.) I’ve already written several times about our rough data contest that is a part of those events. This time let me mention about another initiative - the experience workshop that is aimed to attract research and industry to present social and business benefits of real-life applications related to the main conference topics.

The paper submission deadline for PReMI 2009 has been just moved to June 25. The other ones are on August 15. So, if you are in India or you are going to visit India in December, if you are interested in pattern recognition and machine intelligence, or data mining and soft computing, and - perhaps most of all - if you are interested in helping to build the bridge between theory and applications, please join us in Delhi!

Best greetings,

Dominik

Organizing data and more about rough data contest

2009-05-03T08:14:33+00:00

Infobright’s query performance depends on quality of the Knowledge Grid. Definition of quality of particular Knowledge Nodes should reflect their role in query optimization and execution. Quality may be related to average closeness of the minimum and maximum values in Data Pack Nodes, to the percentage of zeros in our Histograms, Character Maps and Pack-to-Pack Nodes et cetera.

Quality of the Knowledge Grid depends on organizing the incoming data rows into Data Packs, more precisely, the policy of putting particular rows into particular 2^16 row groups that are then transformed into collections of compressed Data Packs. In the current implementation, while loading data into Infobright, the initial row ordering is not changed. Surely, one can pre-order data prior to load but it may not necessarily improve all Knowledge Nodes that are useful in particular query workloads. Actually, pre-ordering will improve only a fraction of Knowledge Nodes and make the others worse. Moreover, it will significantly slow down the data load. Hence, we started implementing alternative algorithms for on-fly organization of rows that will improve quality of the whole Knowledge Grid, with no harm to the data load speed.

Let me skip the algorithmic details. There will be more about them at SIGMOD and/or VLDB this year. Instead, let me focus on the results. Consider the following example:

Some time ago we were writing about Rough Data Contest organized as a part of the academic conferences to be held this December in Delhi. I posted the rough data to be analyzed by the contest participants, with the rows corresponding to the 2^16 row groups taken from one of our benchmark datasets, and with the columns corresponding to the min/max statistics stored in Infobright’s Data Pack Nodes. However, I realized that the majority of information about dependencies between our benchmark dataset columns was completely lost. Therefore, I decided to create the contest data one more time, but now using the above-mentioned algorithm for more careful organization of rows during the data load.

The result is attached to the latest post on Rough Data Contest forum thread. You can see that the number of non-trivial min/max values is now much higher. It turns out that organizing data in purpose of improving Knowledge Nodes can bring us also other positive effects. In particular, it leads me back to our presentation at VLDB 2008. One of the questions from the audience was: Aren’t you afraid of losing correlations between columns when grouping data into Data Packs? So the answer should be: Yes, there is such a risk but we can attempt to cope it by more careful organization of rows during data load.

Best greetings,

Dominik

Infobright Query Videos

2009-04-16T21:07:18+00:00

Kevin Galligan, one of the ICE community members, has posted some rather unique videos showing how his application gets rapid access to information through the unique architure of the Infobright Knowledge Grid. The videos on his blog show 3 columns of a table, broken down into blocks, which are actually 65,536 items of data each (we call these datapacks.) While you are watching his videos, be aware that there are over 900 columns in the table - he’s only visually showing you 3 of them. Check it out…

ICE 3.1.1 is available now for Linux

2009-04-16T20:43:31+00:00

Today we are releasing v3.1.1 of ICE for Linux. Windows32 binaries will be available in the next few days. This version is a maintenance release primarily focused on community submitted issues and problems around the use of the Infobright high-speed loader. A full list of issues fixed in this release is detailed in the release notes.

I’d also like to mention that recently we rolled out a new contributed software section that contains some important utiities, as well as contributed documentation from some of our community members. On this page you’ll find instructions for deploying Infobright within the Amazon cloud, connectors for popular BI tools like Jaspersoft and Pentaho, loading utiilies and migration tools from SQL Server to Infobright. If you have software or documentation that you’d like to contribute to the Infobright project, please don’t hesitate to get in touch with me directly.

ICDE 2009: Conclusion

2009-04-03T09:49:42+00:00

Wonderful conference! In my posts, I concentrated on research observations. However, I should also mention great atmosphere, perfect organization and all those pleasant and interesting conversations with the ICDE 2009 participants.

I should also say a few words about Shanghai. Although I didn’t have too much time, I visited the Yu Gardens and I could admire beautiful views during welcome reception on Huangpu River. Shanghai is definitely a good destination!

As you noticed, we have put all the ICDE updates to the blog. As I was writing them in a hurry, I may still correct some mistakes and add more links. Let me also invite everyone once more to our ICDE 2009 forum for further discussions.

Looking forward to the next ICDE,

Best greetings,

Dominik