Regardless of what back end you choose the mysql query engine is week. It only supports nested loop scan, no merge join, no hash join, nothing fancy. Further sub quires can’t even use indexes so they are near useless. The rule when using mysql is *don’t join*. So the case where mysql can be put to good use is where all your quires are over a single table, which is why it has done well for web apps I wold suspect.

was similar to those in a mid-2006 post on MySQL Performancing Blog, which said:

One of the reasons elevating this problem in MySQL is lack of advanced join methods at this point (the work is on a way) - MySQL can’t do hash join or sort merge join - it only can do nested loops method which requires a lot of index lookups which may be random.

What’s the current situation, mid-2008? Surely data warehousing storage engines such as Infobright’s aren’t so artificially limited in their join strategies.

At Google, our mission is organizing all of the world’s information. We use literally thousands of different data formats to represent networked messages between servers, index records in repositories, geospatial datasets, and more. Most of these formats are structured, not flat. This raises an important question: How do we encode it all?

That sounds like a lot. On the other hand, if “data format” is just a synonym for “table structure,” “file structure,” and/or “schema,” it sounds more plausible. Varda goes on to say

a simple lists-and-records model … solves the majority of problems

Come to think of it, that sounds very consistent with the idea that MapReduce solves a large fraction of Google’s data management issues.

The Globe now reports that a multimillion dollar deal the prior year with the Massachusetts Department of Education was equally dubious. Lowlights include:

A former state Department of Education administrator has told investigators that she was offered a private sector job by a sales representative for Cognos ULC, around the same time that the software company was attempting to win a lucrative education contract in 2006, according to officials briefed on the matter.

While pushing to land the smaller education department contract, Lally specifically emphasized that he could have money added to the state budget to fund the deal because of his friendship with DiMasi, according to Chew and a former Cognos employee. The contract provided for a data warehouse system, which would allow the department to collect, track, and share data about students, teachers, and finances across the state.

In April 2006, Lally’s prediction proved true. House lawmakers added a new $5.2 million line item for the contract through a House budget amendment. DiMasi told fellow legislators at the time that the amendment was a priority, according to an official with direct knowledge of the budget negotiations.

“The two finalists - we spent a long period of time putting them through their paces,” Bickerton said.

That period of time: two days, according to a timeline provided by the Department of Education.

• EnterpriseDB asserts that MyISAM is the only MySQL storage engine with decent performance.
• EnterpriseDB then bashes MyISAM for all sorts of well-deserved reasons, especially ACID-noncompliance.
• EnterpriseDB asserts that row-level triggers, lacking in MySQL but present in PostgreSQL, are the most important kind of trigger.
• EnterpriseDB claims PostgreSQL is superior in procedural language support to MySQL.
• EnterpriseDB claims PostgreSQL is superior in authentication support to MySQL.

Of course, the question of MySQL performance with a transactional storage engine — i.e., InnoDB — is somewhat controversial. E.g., there are many confident guides to InnoDB tuning. (Another post I found was this one, focused on Drupal. ) The bottom line seems to be that MySQL can be both transactional and decently-performing, but the price is sacrificing some of the load-and-go ease one gets from a default configuration. And apps written for MySQL/MyISAM will not necessarily perform well over MySQL/InnoDB.

When in the Course of an increasingly competitive global economy it becomes necessary for one data set to dissolve its connections to a constraining environment, the separate but inherently unequal station to which the Laws of Whose budget is larger prevails.

Related links:

• Cartoons from DATAllegro
• April Fool press release from Netezza

Related links:

• Humor from Netezza
• Another gerbil-based solution

He seems to believe this is an important distinction, but I’m not convinced yet. Rather, I think the essence of (complex) event/stream processing lies in its performance for certain kinds of data processing problems, some of which require super-low-latency and some of which don’t.

Maybe there’s some kind of a partition of the problem space, in which if you want event IDs you’re better off with an Apama-like rules-engine paradigm, while if you don’t need them you do better with Coral8/Streambase-style SQL. But off the top of my head, I don’t see it.

Am I missing something?

• The IRS has a data warehouse running on Sybase IQ, with 500 named users, called the CDW (Compliance Data Warehouse). (Computerworld)
• By some metric, it’s a 150 TB warehouse. (Computerworld)
• By some metric, they add 15-20 TB/year, with a 4 hour load time. (Computerworld)
• As of 2006, there were 20-25 TB of “input data”, with a “70% compression rate”. (Sybase)

I can’t entirely reconcile those numbers, but in any case the database sounds plenty big.

Computerworld also said:

the research division also uses Microsoft Corp.’s SQL Server to store all of the metadata for the data warehouse and the rest of the agency. Managing and cleaning all of that metadata — 10,000 labels for 150 databases — is a huge task in itself,