Community

CAOS Open Source Adoption Survey

2009-10-28T16:29:38+00:00

Given current economic conditions there is considerable interest in open source software and whether open source licensing can help users lower the cost of enterprise computing compared to traditional proprietary licensing.

The 451 Group would like you to contribute to its CAOS Open Source Adoption Survey covering end user attitudes to the potential financial benefits of open source. The survey can be found here.

In return for your contribution, you will receive a copy of the research results, which will be delivered in an update to our second Commercial Adoption of Open Source (CAOS) research report "Cost Conscious, A practical guide for understanding and calculating the financial benefits of open source for enterprise IT projects". You will also receive an invite to a special 451 Group Webinar at which we discuss these results.

The survey includes about 20 questions. We are not asking for details of specific projects, so no sensitive data will be revealed. We are asking about the benefits and risks of open source in general, and how you evaluated the cost of open source in any relevant projects. All responses are of course confidential. The survey will be available for the next two weeks, but please complete it at your earliest convenience.

Infobright v3.2.1 is available for download

2009-10-28T15:53:50+00:00

Both Infobright Community Edition (ICE) and Infobright Enterprise Edition (IEE) have been upgraded to v3.2.1.

ICE v3.2.1 gains these features:

Support for "named pipes" on Windows
Enhanced SQL capabilities (STDDEV, VARIANCE, STDDEV_SAMP, VAR_SAMP, STDDEV_POP, VAR_POP)

IEE v3.2.1 gains these features in addition to those listed above:

Multi-server high availability
DML performance improvements for INSERT operations
New platform support (64 bit SUSE Linux Enterprise 10, 32 bit Windows Server)

ICE is available freely for download on this site. IEE subscribers can download updates to IEE from the Infobright Support Portal.

Data Warehouse Database Conversions

2009-10-08T18:06:54+00:00

Over the years I've worked on many conversion projects, from one operating system to another, from one BI or ETL tool to another, but the conversions I've done the most that's a tough challenge for many is converting from one database to another.

The premise of SQL was to make an easy to use portable language to access data. So why is it perceived by many that converting from one database to another is a huge challenge? Because they underestimate the work involved by not knowing who uses the data and how and rushing it into production with improper QA.

There are many tools to help with conversions and the first place to look is with the database vendor you are migrating to. Many database companies have migration toolkits to make this task easier. Infobright has several free tools in it's download section of their website called ICE Breaker to make it easier to migrate from Microsoft SQL Server and Oracle for example.

These tools typically

1) generate the DDL (create table statements) so you quickly have a working schema

2) export the data into a flat files

3) import the flat files into the schema on the new database.

The premise of these tools is to get you up and running as fast as possible. But it's the next steps are what has helped make my conversions a success:

1) evaluate who in the company is using the data and how. You can do this by contacting all department managers and asking them, preferably in a face to face meeting. I know this sounds daunting, but without their support, success is unlikely. All it takes is one department telling you in the 11th hour that it's their crunch time and they haven't had time to convert their MS Access reports and your conversion is on hold for months. You will be surprised in all the clever ways people access a data warehouse with MS Office without IT's knowledge.

2) setup user training on how to switch their applications from the old database to the new. Setup training for smaller groups from throughout the company with multiple classes as most departments can't afford to lose too many of their staff at once.

3) coordinate with desktop support to have them include new ODBC drivers and any client software needed so that they can push this to every desktop that needs it well in advance.

4) update the ETL process to keep the new database updated. This can be as simple as pointing process to a new ETL tool connector, or as difficult as editing shell scripts. This is a good time to evaluate how your ETL process is working for you and make it better.

5) update the BI tools to point to the new database.

6) there's nothing that will reduce the trust in your ability to deliver if your conversion produces wrong results in queries and reports. This is why it's very important to setup a QA process early in the conversion to make sure reports balance and that data is correct. The biggest mistake I see is waiting until the end of the project to start this.

7) use the professional services group from the new database vendor to help kick start the project and then come back in the later part of the project to help fine tune everything. Nobody knows the database product or has access to support better than a consultant that works for the vendor. Spending weeks fumbling trying to learn things on your own just wastes time and adds to project costs.

Having a working database as quickly as possible, even with stale data to do the above tasks in parallel will allow the conversion to be successful in the shortest amount of time with the least impact to the business.

VLDB 2009 (update)

2009-09-16T10:27:58+00:00

The pdfs of VLDB 2009 papers are already available in DBLP. Let me immediately draw your attention to the section on Experiences and Lessons, wherein the last paper is about DBLP, by Michael Ley. It’s really great to read about something so popular among my academic colleagues. I hope you’ll take a look at it.

Another paper worth reading is, surely, the 10-year Award Keynote. Put it together with the tutorial on column-oriented database systems, as well as with some recent announcements about hybrid columnar stores. It confirms that we did the right thing to focus on columnar architectures a couple of years ago. It also shows that each columnar solution needs to have something beyond basic mechanisms (such as, e.g., columnar scans) to make it really unique and successful.

Best greetings,

Dominik

MySQL CheatSheets Helpful

2009-09-15T13:40:43+00:00

OK, so most of you know that I lived in the world of PostgreSQL for several years before moving into the world of Infobright and MySQL. With that in mind, I've been determined to continue to hone my skills around MySQL as it serves as the primary interface on top of our open source columnar data warehousing engine. Yes, I've hacked around with MySQL for many years and was of the first OEM'rs of MySQL when I was at Ascential. I even created a website on the side called www.MyCollegeFAQ.com that is based on MySQL. It is a free (no advertising) and open site that is used to help high school and college students get answers to their frequent asked questions about getting into college. Always had an affinity towards the MySQL database but other efforts led me to PostgreSQL. One tool that I've found helpful these days is the MySQL Cheat Sheet that is installed on my iPod Touch. It was developed by the folks at Concentric Sky. This handy reference tools is a great companion reference for all the various commands, data types, sql formats, etc. for MySQL. They say that if you study something for 10,000 hours, you become an expert. Let me say, I'm not quite 10,000 hours into MySQL, thus the Cheat Sheet. If you are interested in getting the Cheat Sheet, it is on the Apple App Store. This is not just a reference to a website tuned for Safari on the iPod Touch or iPhone, it is a simple and easy to use application that is a growing dictionary for MySQL reference information that is continually updated. It's 0.99 and well worth it. Enjoy.

Infobright On The Cloud

2009-09-11T13:00:19+00:00

Cloud computing to Infobright is somewhat analogous to supporting an operating platform and yes, we support the deployment of our market leading open source columnar database on the cloud and more specifically using an open cloud manifesto approach. The good news is that Amazon has continued to lead the way when it comes to supporting a businesses (whether small, medium or large) cloud initiative and Infobright is happy to support our open source columnar database on the Amazon cloud.

These days some data warehouse solutions in the market put proprietary and complex wrappers around their cloud offering. With their approach, you are not in control, they are in control and with a proprietary approach, they end up locking you in and who knows who is going to touch or view your highly sensitive data. At Infobright, we take the opposite approach. We believe strongly in openness and also believe that the customer should be in control of their cloud data warehouse efforts, especially with the types of critically important and sometimes highly sensitive data stored in the warehouse or mart. For us, it is a matter of guiding our customers on how to take advantage of cloud infrastructure and solutions and putting you in control with information to assist in your efforts. If cloud solutions such as monitoring, management, deployment technologies are truly open, then they should easily integrate with other open technologies such as Infobright. To assist you, we have developed a document containing instructions on how to get Infobright running on the Amazon cloud. With this approach, you are totally in control. What follows is a link to these instructions. If Cloud is a consideration, we believe you should consider an open and unrestricted approach, not one that involves a lock-in and proprietary approach.

http://www.infobright.org/downloads/contributions/Amazon_Cloud_EC2_Instructions.pdf

The Open Source World keeps growing! - Stay tuned for TunedIT

2009-09-08T19:13:26+00:00

Pretty soon there will be one year since we have released the first ICE. There will be surely a lot of noise around this first anniversary. In the meantime, let’s look around. Of course everyone knows MySQL and Pentaho (and some might hear about Weka - now also a part of Pentaho). Some may also know other open source columnar stores, such as MonetDB and LucidDB. However, it’s even more important to realize that there are newcomers to the Open Source World practically every day.

A couple of days ago we received the following message from one of our friends, who had helped us in making Infobright’s technology successful and then decided to go back to the Realm of Data Mining. Of course we wish him success also with this new undertaking and we hope to be able to do something cool together in the nearest future. Here is the message:

=======

I am pleased to announce that TunedIT system has been publicly released and is available at:

http://tunedit.org/

TunedIT facilitates evaluation and comparison of data mining and machine learning algorithms. Its website contains exhaustive data on performance of nearly 100 algorithms from Weka and Rseslib libraries, tested on tens of different datasets from UCI Repository. You can extend this information with your own test results, as well as upload new algorithms and datasets.

I invite you to visit and explore TunedIT and to contribute new algorithms, datasets and results. I hope TunedIT will prove useful in your research. If you find our system interesting, please help us improve it by sharing your comments.

=======

So I’ve just looked at TunedIT and I guess I’ll be a frequent visitor there. (Apart from Infobright, working on new data mining algorithms is a kind of my hobby; moreover, as I wrote before, there may be a place for data mining algorithms even inside Infobright engine.) Surely, there are such websites as that of the UCI Repository but - as the name suggests - it’s just a repository of data mining / machine learning data benchmarks, without such a convenient and sophisticated testbed framework that TunedIT provides.

So, maybe TunedIT should be rather compared to tpc in the world of databases?

And, by the way, there is also a mention about Rseslib - again, something for the fans of rough sets!

Best greetings,

Dominik

VLDB 2009

2009-08-21T23:06:12+00:00

I wish I could attend VLDB 2009 next week, the more so as it’s organized in such a nice place in France, so close to Poland. However, there were too many things happening around Infobright’s Release 3.2 to get prepared for the conference. Of course I’m not complaining, because it was a lot of fun to put so much innovation into the architecture of the new version. On the other hand, I will be looking forward to the new VLDB papers available in DBLP and I’ll try to catch up with the reading as soon as possible.

As for now, I’m looking at the conference program and I can see many familiar topics. I will miss the cloud keynote by Raghu Ramakrishnan, a kind of continuation of his ICDE 2009 talk. This is one of the examples of a growing importance of the cloud among the database researchers and practitioners.

Some of the VLDB 2009 papers have become popular even before the conference. For example, take a look at the last paper in Research Session 26. The paper can be downloaded from the homepage of the HadoopDB project. I like it, although I keep thinking whether shared-nothing is really the only way to go when attempting to leverage benefits of Hadoop and analogous frameworks. As the authors mention in Introduction, parallel databases are not necessarily shared-nothing. The authors mention Exadata as a (partial) counterexample (see the footnotes on page 1). Moreover, even in the realm of MapReduce we have such approaches as, for example, Phoenix. So, as usual, there is more analysis ahead…

Best greetings,

Dominik

Columnar Database: A Strong Choice For Marketers

2009-08-13T17:02:39+00:00

According to Chris Dunne, VP of Market Development for ChoicePoint Precision Marketing, part of LexisNexis, "Improved marketing efficiency, increased intelligence and lower overall costs are all possible with a columnar database structure. Database marketers should investigate the advantages this new approach could bring to their own marketing systems"

As the adoption and use of columnar databases grow in the world of marketing, companies like LexisNexis Choicepoint and others will likely experience significant costs savings as Chris further points out by stating "Another benefit of using columnar database solutions is the lower cost of ownership". Chris attributes this to columnar requiring "less throughput capacity than their traditional full-table file counterparts, hardware requirements are reduced. Expensive large-scale disks can be replaced with lower-cost alternatives while still improving performance."

The types of critical information needed by marketing range from understanding and discovering customer usage patterns, customer churn, campaign management, website navigation and customer interactions. All of these sources of information likely land in a data warehouse. The ideal data warehouse for marketing initiatives would be a columnar database because of the nature of the complex queries that marketing professionals need to answer rapidly on a day to day basis. Any other solution would likely increase costs, create lag and friction in getting results to the marketing team and be less likely to see significant returns on investments, especially when open source columnar databases are available at much lower costs than their commercial counterparts.

In fact, Chris points out that "advantage of columnar database is its ability to easily integrate into an existing marketing system simply by feeding data from a data warehouse or from outside sources".

Chris clearly understands the benefits of columnar database and to gain more of insight, I would encourage you to read an article he published in DMNews found here

Look for Focus When Choosing A Business and Solution For Your IT Project

2009-08-03T15:06:11+00:00

With technology and new innovations rapidly evolving in the open source market, there are times when a company or even the team behind the company tend to stray away from their core purpose and focus for delivering a solution to the market. In fact, a few startup entrepreneurs do this full time, constantly bringing new solution by inventing a new company while letting their investment in time, passion and energy from their early ventures diminish rapidly. Instead of staying laser focused, they become less interested over time and do little to help the original venture while the original venture attempts to keep leveraging the entrepreneurs name or background in the forefront of the company.

When looking at a solution that claims to have a well known professional behind the company, where the company keeps touting their name, be sure to understand what that person does on a day to day basis to truly help the company. Are they active in the strategy and planning phases or day to day operations of the company? Are they committed full time or only partially committed. Marketing tends to do whatever they can to leverage the visible entrepreneur behind the company, but before buying into this type of pitch, ensure you understand the full picture. Be suspect if they aren't giving you a straight answer.

Companies who do have a strong well known entrepreneur behind them will do whatever they can to use the person as a part of their brand imagine. This shouldn't be about brand image. It should really be about how the person helps the company with a path to success for customers and the business itself. This is the case for Steve Jobs. Clearly the industry understand how important Mr. Jobs is to Apple. Mr. Jobs keeps Apple focused and contributes every ounce of his energy into Apple while also focusing on delivering world class consumer products that consumers love.

When evaluating a startups solution and their business, look for a true partner that is focused on your success not just the name of an entrepreneur or industry pundit that is associated with the business.