• Business is strong. Multiple vendors claim to be going gangbusters, with the happy sounds coming out of Vertica and Infobright being echoed by several competitors. Hearsay suggests some other companies in related businesses are doing well too. Depending on who you talk to, the business pickup dates back to Q4, give or take a quarter.
• Oracle Exadata has become a formidable competitor, on the strength of Exadata 2. Exadata 2’s positioning and perception among Oracle users seem to be pretty much in line with what Oracle portrayed to me.
• Teradata is portrayed as a weak competitor. Competitors don’t worry about Teradata nearly as much as they do about Oracle. That said, I suspect a bit of wishful thinking; Teradata is clearly still getting a lot of business the other vendors would dearly love to have.
• HP Neoview is reeling. (Almost) nobody sees Neoview competitively. The Walmart Neoview installation is said to have stayed small at best. JP Morgan is said to have completely thrown Neoview out (and a bunch of HP engineers with it).
• (Almost) nobody mentions competing against DB2 either. This continues to baffle me.

• Vertica had a “killer” Q4 and is doing very well in Q1 again.
• Vertica burned hardly any cash last year; i.e., it was close to cash-flow neutral in 2009.
• Vertica is hiring aggressively, e.g., in sales.
• Vertica is well down the path with several CEO candidates who Jerry regards as outstanding. He is hopeful there will be a new CEO in April. (But I bet that would be late April, given what Jerry mentioned about his own travel plans.)
• Absent a full-time CEO, Jerry and Andy Palmer are spending a lot more time with Vertica.
• One Vertica customer is approaching a petabyte of user data. The last time Vertica had checked, that customer had been more in the ¼ petabyte range.
• Other multi-hundred terabyte Vertica databases were mentioned, including one where Vertica claims to have beaten Teradata and perhaps other competitors in a head-to-head competition (it sounds like that one’s too recent to be deployed yet).
• Vertica sees Aster and Greenplum competitively more often than it sees ParAccel.
• Vertica sees Sybase IQ competitively a lot in financial services (in new-name accounts for Sybase as well as where some kind of Sybase DBMS is an incumbent), and more occasionally in other sectors.

NDA parts of the conversation also gave me the impression that Vertica is moving forward just as eagerly as it’s peers. I.e., I didn’t uncover any reason to think that Ralph’s departure is a sign of trouble, of the company being shopped, etc.

On the other hand, I didn’t uncover any other clear reason for Ralph’s departure either. The party line is that Ralph left for “personal reasons”. It was his decision to leave. He did a great job while at Vertica. Basically, Vertica is saying that, even though all was going swimmingly, Ralph just up and quit, leaving some very valuable unvested stock options on the table at what had been his first CEO gig ever.

Contacting Ralph didn’t add any immediate insight. He responded quickly via a Twitter DM, but was in the airport for a trip to his home country of South Africa, and deferred discussion until after his return.

Incidentally, Infobright and ParAccel have both also had recent CEO turnover. Stated reasons in each case were of the “Right person to lead the next stage of the company’s growth” variety.

Highlights on the market share/sector side include:

• Infobright’s customer base grew 500% over the past year, to 120 paying customers.
• This included end users (60%), as well as ISVs and SaaS providers (40%) who embed Infobright’s DBMS in their application. It is great to see both segments taking advantage of our technology.
• During the same period, Infobright’s open source software was downloaded 35,000 times.
• The end user applications were heavily clustered around web and online analytics tracking, with a focus on understanding customer behavior on the web.
• Infobright also continues to see the growth of application-specific data marts.
• There is also continued interest and growth in using Infobright technology to analyze IT logs and telecom CDR (Call Detail Record) data, to identify fraud or security issues, to understand and improve network performance, and other purposes.

Product highlights include:

• Infobright be much more transparent in 2010 about its plans.
• Infobright will start posting and commenting on future releases and themes in March of this year. (However, they haven’t run much of that by me yet, and we’re past the middle of March.)
• Infobright expects to drop 3-4 interim releases for every major release, with at least two major releases in 2010.
• Some of Infobright’s major improvements this year will be:
  • Continued SMP performance improvements “without the need for complex hardware configurations or administrative effort”.
  • Extending the “hit rate” of the Knowledge Grid, which is central to Infobright’s performance story.
  • Better international support with UTF-8 extensions.

• XtremeData still hasn’t sold any dbX stuff (they’ve had a side business in generic FPGA-based boards paying the bills for years). Well, there may have been some paid POCs (proofs of concept) or something, but real sales haven’t come through yet.
• XtremeData does have three prospects who have said “Yes”, and expects one order to come through this month.
• XtremeData continues to believe it shines when:
  • Data models are complex
  • In particular, there are complex joins
  • In particular, two large tables have to be joined with each other, under circumstances where no product can avoid doing vast data redistribution
• XtremeData insists that all the nice things Bill Inmon – including in webinars — has said about it has not been for pay or other similar business compensation. That’s quite unusual.
• XtremeData is coming out with a new product, codenamed the Personal Data Warehouse (PDW), which:
  • Is ready to go into beta test
  • Should be launched in a month and a half or so
  • Will have a different name when it is launched

Naming aside,

• The XtremeData PDW consists of XtremeData software running on a Cray CX1 box.
• Thus, the XtremeData PDW will plug into a 20 amp wall power socket. It consumes 1600 watts.
• The XtremeData PDW also inherits the Cray CX1’s noise cancellation feature.
• Bottom line on the form factor: The XtremeData PDW is meant to be stuck in the corner of a business analyst’s office, not a computer room.
• The XtremeData PDW will have 16 1 TB disks (going up in size later), for 5 TB of uncompressed user data.
• Pricing isn’t finalized for the XtremeData PDW, but it will be around XtremeData’s usual figure — $20K/TB of uncompressed user data.
• XtremeData hasn’t “released” compression yet, but it’s “ready to go.”
• The XtremeData PDW will not include FPGAs, unlike other XtremeData dbX appliances. It will just run the XtremeData dbX software on 8 Nehalem chips.
• XtremeData calls this a “3-node” machine. I didn’t bother asking why it wasn’t 4-node. (Perhaps there’s a head node of some kind that properly isn’t counted.)

Some comparative notes:

• A Netezza Skimmer has similar size and price to the XtremeData PDW, seems to draw less power, has less uncompressed user data capacity (but already has compression), is also in essence a three-node system (I think), and of course has a lot of software connectivity. If XtremeData can match Netezza’s compression, the XtremeData PDW will have a 2X or so price/TB advantage over Netezza Skimmer – but Netezza’s compression is of course a moving target. I don’t know how happy Skimmer is outside a computer room.
• Kickfire manages similar amounts of data on a smaller box (5 rack units vs. 7), drawing less power (600 watts vs.1600), also with a lot of BI and ETL tool connectivity.

… or something like that. I don’t intend to write seriously about NorthScale until I have a better idea of what MemBase is.

In the mean time,

• VentureBeat put up a solid post on NorthScale’s company history and so on
• Om Malik bought into the NorthScale memcached pitch
• TechCrunch has a low-quality post about NorthScale (although it wasn’t as error-riddled as the same author’s post about nStein, which Seth Grimes properly blasted)

NoSQL = HVSP (High Volume Simple Processing) without joins or explicit transactions

Within that realm, Dwight offered a two-part taxonomy of NoSQL systems, according to their data model and replication/sharding strategy. I’d be happier, however, with at least three parts to the taxonomy:

• How data looks logically on a single node
• How data is stored physically on a single node
• How data is distributed, replicated, and reconciled across multiple nodes, and whether applications have to be aware of how the data is partitioned among nodes/shards.

After talking with Dwight, and also with Cassandra project chair Jonathan Ellis, I feel I’m doing decently in understanding the first of those three areas. But there’s a long way yet to go on the other two.

In Dwight’s opinion, as I understand it, NoSQL data models come in four general kinds.

• Key-value stores, more or less pure. I.e., they store keys+BLOBs (Binary Large OBjects), except that the “Large” part of “BLOB” may not come into play.
• Table-oriented, more or less. The major examples here are Google’s BigTable, and Cassandra.
• Document-oriented, where a “document” is more like XML than free text. MongoDB and CouchDB are the big examples here.
• Graph-oriented. To date, this is the smallest area of the four. I’m reserving judgment as to whether I agree it’s properly included in HVSP and NoSQL.

As Dwight sees it, JSON (JavaScript Object Notation) is the emerging markup standard for the document-oriented data models, and to some extent the BLOB part of key-value models as well. Reasons seem to include:

• JSON is something web developers are likely to know anyway.
• JSON, unlike XML, is schema-less. In the NoSQL world, that’s perceived as a good thing.
• Perhaps for both these reasons, JSON is perceived as easier to use than XML.

Except as noted, I’m not aware of anything that solidly contradicts the above.

Dwight went on to say that there are two main NoSQL replication/sharding models, in line with the seminal papers to which I previously linked:

• Based on or resembling Dynamo. The core idea here is accepting eventual consistency among nodes as being good enough, even if that means you sometimes read dirty data. The benefit is that you never are blocked from writing. By way of contrast, systems that enforce true inter-node consistency (think of a two-phase commit) can shut you down from writing if consistency guarantees aren’t being confirmed in a timely manner. Thus, in a Dynamo-like scheme you write data to multiple nodes, via consistent hashing; then when the time comes you read one or more nodes, and hope that what you’re getting back is a correct result.
• Based on or resembling BigTable. In this model you’re trying to keep the nodes fully consistent in the usual way, e.g. by synchronous replication. Indeed, what’s being kept consistent is both data itself, and metadata about the data’s location. Details surely vary a lot from implementation to implementation.

I’m fuzzier on this stuff than on the data models, because to date nobody has ever explained to me how an actual live system (MongoDB, Cassandra, whatever) implements its replication strategy. Also, while I think that in both these models applications are allowed to be ignorant of the replication/sharding strategy, I’m not as sure of that as I’d like to be.

If we stop here, we already have something useful. MongoDB has a document data model, and is in the BigTable-like replication camp, at least at first. Cassandra has a table-like data model, and is on the Dynamo-like eventual consistency side. But to say those are the only differences that matter would be like saying that all shared-disk RDBMS (e.g., Oracle and Sybase IQ) are essentially alike. That, of course, would be nonsense.

So a third dimension needed in this taxonomy is how the systems actually bang data on and off of disk (or silicon, as the case may be). I don’t yet have an overview of that. I know something of how Cassandra does it, and will write about same in a future post, but that’s about it. So please stay tuned.

• No technology category name is ever perfect.
• It’s particularly hard to describe NoSQL (Not Only SQL) accurately, given the basic confusion as to what NoSQL is all about.
• That said, it seems pretty clear that NoSQL is about making big websites (and perhaps other cloud-like installations) run and scale.
• Dwight Merriman (founder/CEO of MongoDB vendor 10gen) is heading in the right direction when he says that the unifying ideas of NoSQL are that you do away with transactions and joins. But if he’s ever said something like “NoSQL is Foo without joins and transactions,” I don’t know what Foo is.
• Actually, I do know what Foo is – Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. I just don’t know what Foo is called.
• Obviously, Foo is a lot like OLTP (OnLine Transaction Processing). However, it would be pretty silly for Foo to actually be OLTP, given that one of the core points of NoSQL is that you don’t have transactions.
• It not just the “T” part of OLTP that’s fried. Calling something “OnLine” only makes sense as long as offline is an option, and offline transaction processing has been obsolete for a very long time.*

*Sure, if you strain you can talk yourself into exceptions. But the point stands.

So we need a name for Foo, where Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. Thus, three major subcategories of more-or-less disk-based Foo are:

• No-compromises ACID-compliant relational OLTP
• Sharded MySQL
• NoSQL

There may be some more purely memory-centric versions too, but let’s put those aside for the moment.

Absent a better idea, I can squeeze Foo into yet another four-letter acronym:

HVSP (High-Volume Simple Processing)

That’s as imperfect as any other category name, and an awkward mouthful to boot. So I’d love to hear a better one; if you have such, please share it! In the mean time, I think “HVSP” has merit because:

• The “Processing” part should be noncontroversial.
• “High-Volume” is inherent to the challenge. If RDBMS scale well enough for your use case, using something less powerful is probably silly.* Similarly, while Oracle shines at high-volume OLTP workloads, there are many cheaper DBMS that do a fine job of OLTP at lower volumes.
• “Simple” is the core principle of NoSQL systems, which drop joins and transactions as being too much foofarah. That only makes sense at all under the assumption that you have bone-simple queries and updates, so that programming around the lack of joins and transactions isn’t all that much of a burden.
• Something similar is true of sharded MySQL.
• Less obviously, “simple” is a core principle of relational OLTP as well. The point of the relational model is to cap the complexity of data operations, or more precisely to hide that complexity from programmers.
• And overloading the word “simple” a bit, it’s fair to say that if you’re reading or writing one record at a time, you’re doing something relatively simple, at least as opposed to what you do in analytic processing. The OLTP vs. OLAP distinction is preserved in this name change.
• The whole thing matches my definition above, namely “what happens when lots of people want to get small amounts each of information in or out of a database at the same time.”

*Assuming, of course, that rows-and-tables are a good metaphor for your data structure in the first place.

Systems I’m leaving out of the HVSP and hence also NoSQL categories include:

• Hadoop and other batch-oriented MapReduce. Hadoop isn’t part of NoSQL. I’m pretty sure that Cloudera CEO Mike Olson agrees with me.
• More generally, non-SQL data stores that don’t meet the HVSP criteria. Dave Kellogg stretches things when he claims that MarkLogic is a NoSQL system. (But then, that was in a post where he seemingly praised a train wreck of an article.)

But hey – what good is a categorization if it doesn’t leave some things out?

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

• Facebook invented and is adopting Cassandra.
• Twitter is adopting Cassandra.
• Digg is adopting Cassandra.
• LinkedIn invented and is adopting Voldemort.
• Gee, it seems as if the super-scalable website biz has moved beyond MySQL/Memcached.

But in addition, he provides a lot of useful links, which DBMS-oriented folks such as myself might have previously overlooked. Following those trails gets one to, among other things:

• A September, 2009 post outlining Digg’s reasons for moving to Cassandra. The core idea is that joining two tables is expensive; it’s cheaper to store the results prejoined on disk. Details are provided.
• A February, 2010 post outlining Twitter’s reasons for moving to Cassandra. They boil down to “sufficiently scalable, sufficiently simple, sufficiently robust, robustly open source.”
• A Flickr slide presentation saying “normalization is for wimps”. They seemed to be staying with MySQL, but lusting after XPath.
• A nice Cassandra technical overview by Evan Weaver of Twitter.

I also recall seeing something that said “We have 13X as many queries as updates, so of course we should optimize for reads,” but I can’t find that now. The classical OLTP answer to that would probably be “Yeah, but by the time you’re two-phase-committing and integrity-checking all the part of that update, it turns out updates are still what you should optimize for.” Well, what if the update is so simple that that’s no longer a valid argument?

There certainly seem to be some non-obvious technical choices being made here, with options being conflated that perhaps shouldn’t be. In particular, I wonder whether things are being written to cheap disk in a really fast way when it might be better to keep them in more expensive RAM or, perhaps better yet, solid-state memory. Perhaps then the functionality/performance tradeoff wouldn’t be so painful.

On the other hand, the designers of the world’s most scalable websites — e-commerce sites perhaps excepted — seem pretty unanimous in thinking it’s best to bake some database/integrity management into the applications, rather than offload it all to an RDBMS. Why? Because the transactions are so simple that hand-coding all that isn’t prohibitive. And of course because of their extreme performance and scalability needs.

I’m not sure on what basis one could argue that they’re wrong.

The article makes the same conflation, but while reading it I noticed something interesting. The concrete examples cited are of clever consultants who crafted innovative data visualizations on the fly, to make conclusions patently apparent to even mathematically-challenged decision-makers. That kind of thing is important, and has been going on for over 140 years.*

*Yes, I’m trotting out the Florence Nightingale example again. I continue to be in awe of her.

What worries me is the article’s suggestion that the best data visualizations are done by visualization experts, as ways of making information apparent to other people. For as long as data visualization relies on hotshot visual-design experts doing one-off projects, its impact on enterprises overall will remain extremely limited. In other words, to the extent it is incorrect to conflate data visualization and data exploration, data visualization will remain a fringe technology.

To be fair, a primary decision support/business intelligence usage cycle has always been — where by “always” I mean “for at least the past 35+ years” –

• Data exploration. Power user uses technology to find something interesting.
• “Look what I found!” Power user then shows a report, chart, or other summary/representation to colleagues.

So to the extent modern interactive data exploration/visualization technology fits that paradigm, great. But to the extent that visualization experts are somehow integral to the technology’s use, it will remain stuck on the analytic fringe.