• No technology category name is ever perfect.
• It’s particularly hard to describe NoSQL (Not Only SQL) accurately, given the basic confusion as to what NoSQL is all about.
• That said, it seems pretty clear that NoSQL is about making big websites (and perhaps other cloud-like installations) run and scale.
• Dwight Merriman (founder/CEO of MongoDB vendor 10gen) is heading in the right direction when he says that the unifying ideas of NoSQL are that you do away with transactions and joins. But if he’s ever said something like “NoSQL is Foo without joins and transactions,” I don’t know what Foo is.
• Actually, I do know what Foo is – Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. I just don’t know what Foo is called.
• Obviously, Foo is a lot like OLTP (OnLine Transaction Processing). However, it would be pretty silly for Foo to actually be OLTP, given that one of the core points of NoSQL is that you don’t have transactions.
• It not just the “T” part of OLTP that’s fried. Calling something “OnLine” only makes sense as long as offline is an option, and offline transaction processing has been obsolete for a very long time.*

*Sure, if you strain you can talk yourself into exceptions. But the point stands.

So we need a name for Foo, where Foo is what happens when lots of people want to get small amounts each of information in or out of a database at the same time. Thus, three major subcategories of more-or-less disk-based Foo are:

• No-compromises ACID-compliant relational OLTP
• Sharded MySQL
• NoSQL

There may be some more purely memory-centric versions too, but let’s put those aside for the moment.

Absent a better idea, I can squeeze Foo into yet another four-letter acronym:

HVSP (High-Volume Simple Processing)

That’s as imperfect as any other category name, and an awkward mouthful to boot. So I’d love to hear a better one; if you have such, please share it! In the mean time, I think “HVSP” has merit because:

• The “Processing” part should be noncontroversial.
• “High-Volume” is inherent to the challenge. If RDBMS scale well enough for your use case, using something less powerful is probably silly.* Similarly, while Oracle shines at high-volume OLTP workloads, there are many cheaper DBMS that do a fine job of OLTP at lower volumes.
• “Simple” is the core principle of NoSQL systems, which drop joins and transactions as being too much foofarah. That only makes sense at all under the assumption that you have bone-simple queries and updates, so that programming around the lack of joins and transactions isn’t all that much of a burden.
• Something similar is true of sharded MySQL.
• Less obviously, “simple” is a core principle of relational OLTP as well. The point of the relational model is to cap the complexity of data operations, or more precisely to hide that complexity from programmers.
• And overloading the word “simple” a bit, it’s fair to say that if you’re reading or writing one record at a time, you’re doing something relatively simple, at least as opposed to what you do in analytic processing. The OLTP vs. OLAP distinction is preserved in this name change.
• The whole thing matches my definition above, namely “what happens when lots of people want to get small amounts each of information in or out of a database at the same time.”

*Assuming, of course, that rows-and-tables are a good metaphor for your data structure in the first place.

Systems I’m leaving out of the HVSP and hence also NoSQL categories include:

• Hadoop and other batch-oriented MapReduce. Hadoop isn’t part of NoSQL. I’m pretty sure that Cloudera CEO Mike Olson agrees with me.
• More generally, non-SQL data stores that don’t meet the HVSP criteria. Dave Kellogg stretches things when he claims that MarkLogic is a NoSQL system. (But then, that was in a post where he seemingly praised a train wreck of an article.)

But hey – what good is a categorization if it doesn’t leave some things out?

• A little over a year ago, Julian Browne put up a great post on Eric Brewer’s CAP conjecture/theorem, which provides much of the impetus to relax the traditional requirement for atomicity/consistency.
• Even more directly inspirational to NoSQL technology development were two seminal papers: Google’s on BigTable and Amazon’s on Dynamo. (That said, I’m having trouble getting myself to actually read them from start to finish, especially since they’ve been superseded by subsequent technology development.)
• 10gen (the MongoDB guys) hosted a NoSQL conference yesterday. Much blogging has ensued. The best post I’ve seen so far was by Adam Marcus. I find the graph database notes near the bottom particularly interesting.
• Mark Callaghan hit back against the NoSQL movement hype, and in particular against the MySQL/memcached is passe‘ meme. On the other hand, he also bemoaned many failings of MySQL. On the third hand, he praised or at least expressed hope for a variety of MySQL-related technologies, including Tokutek’s TokuDB and Continuent’s Tungsten.
• In connection with that debate, Mark Rendle offered a funny rant, mainly pro-NoSQL, in the style of a Socratic dialogue.
• John Quinn of Digg recently described Digg’s move from MySQL to Cassandra, and outlined a lot of features Digg was adding to Cassandra, all of which it is open-sourcing.
• The NoSQL guys maintain their own long list of NoSQL-related links.

• Facebook invented and is adopting Cassandra.
• Twitter is adopting Cassandra.
• Digg is adopting Cassandra.
• LinkedIn invented and is adopting Voldemort.
• Gee, it seems as if the super-scalable website biz has moved beyond MySQL/Memcached.

But in addition, he provides a lot of useful links, which DBMS-oriented folks such as myself might have previously overlooked. Following those trails gets one to, among other things:

• A September, 2009 post outlining Digg’s reasons for moving to Cassandra. The core idea is that joining two tables is expensive; it’s cheaper to store the results prejoined on disk. Details are provided.
• A February, 2010 post outlining Twitter’s reasons for moving to Cassandra. They boil down to “sufficiently scalable, sufficiently simple, sufficiently robust, robustly open source.”
• A Flickr slide presentation saying “normalization is for wimps”. They seemed to be staying with MySQL, but lusting after XPath.
• A nice Cassandra technical overview by Evan Weaver of Twitter.

I also recall seeing something that said “We have 13X as many queries as updates, so of course we should optimize for reads,” but I can’t find that now. The classical OLTP answer to that would probably be “Yeah, but by the time you’re two-phase-committing and integrity-checking all the part of that update, it turns out updates are still what you should optimize for.” Well, what if the update is so simple that that’s no longer a valid argument?

There certainly seem to be some non-obvious technical choices being made here, with options being conflated that perhaps shouldn’t be. In particular, I wonder whether things are being written to cheap disk in a really fast way when it might be better to keep them in more expensive RAM or, perhaps better yet, solid-state memory. Perhaps then the functionality/performance tradeoff wouldn’t be so painful.

On the other hand, the designers of the world’s most scalable websites — e-commerce sites perhaps excepted — seem pretty unanimous in thinking it’s best to bake some database/integrity management into the applications, rather than offload it all to an RDBMS. Why? Because the transactions are so simple that hand-coding all that isn’t prohibitive. And of course because of their extreme performance and scalability needs.

I’m not sure on what basis one could argue that they’re wrong.

The article makes the same conflation, but while reading it I noticed something interesting. The concrete examples cited are of clever consultants who crafted innovative data visualizations on the fly, to make conclusions patently apparent to even mathematically-challenged decision-makers. That kind of thing is important, and has been going on for over 140 years.*

*Yes, I’m trotting out the Florence Nightingale example again. I continue to be in awe of her.

What worries me is the article’s suggestion that the best data visualizations are done by visualization experts, as ways of making information apparent to other people. For as long as data visualization relies on hotshot visual-design experts doing one-off projects, its impact on enterprises overall will remain extremely limited. In other words, to the extent it is incorrect to conflate data visualization and data exploration, data visualization will remain a fringe technology.

To be fair, a primary decision support/business intelligence usage cycle has always been — where by “always” I mean “for at least the past 35+ years” –

• Data exploration. Power user uses technology to find something interesting.
• “Look what I found!” Power user then shows a report, chart, or other summary/representation to colleagues.

So to the extent modern interactive data exploration/visualization technology fits that paradigm, great. But to the extent that visualization experts are somehow integral to the technology’s use, it will remain stuck on the analytic fringe.

I just don’t see Oracle abandoning HPC entirely. I think it may call it by some other name or describe it differently, but it will be in the high throughput computing business for the foreseeable future.

There are some interesting angles for it to pursue. Many of its best commercial customers have sizeable HPC or HPC-like workloads that Oracle can now (with the addition of Sun) compete for. I don’t see it passing up those opportunities.

Oracle can also look to specialize on certain subsets of the market and provide more of a solution rather than piece parts. I wouldn’t be surprised to hear of it offering an Exadata-like system that is optimized for, say, seismic or financial services. In fact, Exadata as it stands today is a decent fit for financial service analytic workloads.

HPC can be a profitable business and, in a lot of organizations, it’s growing faster than traditional business processing. From Oracle’s perspective, what’s not to like?

Now, except for the Exadata-in-financial-services comment, that’s not directly an argument for the convergence of number crunching and data management.  However, I think Netezza and Aster Data are showing the way for that convergence. So, up to a point, is the scientific-research community. And of course the Hadoop guys think they have the best way to that convergent future.

But if Dan Olds is right that the best technologies for Oracle to pursue HPC and big-data processing with aren’t all that far apart, then the chances that Oracle will indeed pursue their convergence are pretty high. And that would amount to critical mass for the trend.

• One of the most confusing things about Sybase ASE is its version numbering. In particular,
  • Sybase ASE 15.5 went GA in December, 2009. (But the clustered version is just coming out in March.)
  • The prior version of Sybase ASE was 15.03.
  • Sybase ASE 15.0 came out in September, 2005.
  • The version of Sybase ASE before that was 12.5.
  • And by the way, Sybase System 10 came out in 1994 or so.
• Sybase ASE 15.0 was a major rewrite. In particular, Sybase ASE 15.0 had a “brand new” optimizer and query processing engine, based on the Volcano model. The main driver of the rewrite was to make Sybase ASE suitable for mixing OLTP and some level of decision-support workloads. (Not on the order of what Sybase IQ can handle, but at least operational reporting and so on.)
• I haven’t looked up Volcano in more detail than to confirm that what I thought Raj said made sense, but as he characterized it, it’s a lot more modular than what Sybase had in ASE 12.5. For example, substantially the only join algorithm in Sybase ASE 12.5 was nested loop – no hash or sort/merge.
• As you might imagine, a lot of things one might regard as core modern DBMS features were only added to Sybase ASE once 15.0 came out. Examples include:
  • Various forms of partitioning at the storage level.
  • User-defined functions (UDFs).
  • A clustering offering that competes with Oracle RAC. (100 or so customers are on that so far.) Absent clustering, Sybase ASE is limited to a single SMP (Symmetric Multiprocessing) box.
  • Shared disk. Amazingly, it seems that before 2008, every node in an SMP box running Sybase ASE had its own private partition (maybe not the right word) of data.
• In Sybase ASE, you have lots of databases managed by one database server. You can write SQL statements that span multiple databases, but they have to reference database names as well as table names.
• There are several ways to get data from one place to another in Sybase’s technology and nomenclature, specifically including. Replication Server, Incremental Data Transfer, and “proxy tables.” (Other than the fact that Replication Server is a separate, chargeable product, I don’t really have these straight.) In addition, there’s a hand-coded one in Sybase RAP, which will get a planned 5-6X performance improvement later this year when it is replaced by Incremental Data Transfer.

And in what basically sounds like a very cool approach, Sybase ASE has a lot of memory-centric aspects. That said, Sybase’s in-memory ASE story is still incomplete (wait until the next release) and confused (I think in part because of what’s missing in the current release).  Also, this is one area where the non-technical nature of the briefing got in my way. So here’s some of what I do and don’t know about Sybase’s memory-centric ASE strategy:

• Sybase lets you mix and match on-disk and in-memory databases under one instance of Sybase ASE. To a programmer, it all looks like ASE.
• I don’t know exactly what the limitations are on what you can do with in-memory databases, how you can use them in tandem with on-disk databases, etc.
• You can replicate data from disk to an in-memory Sybase ASE database today. (Hello caching, ala Oracle Times Ten or IBM DB2/solidDB.)
• Replicating from memory to disk is a near-term future capability. (So Sybase does not yet have a hybrid memory-centric story ala solidDB Classic.)
• I have no clue as to what kinds of in-memory data structures Sybase ASE uses.

One of the two posts inspiring me to mention this is a high-level one on “technical debt“, reminding us why things don’t always get done right the first time, and further reminding us that circling back to fix them sooner rather than later is usually wise. The other connects two observations that individually have great merit (at least if you don’t take them to extremes):

• Update-in-place is passe’
• So is elaborate up-front database design

Specific points of interest here include:

• Most data never gets changed after being written. Update-in-place doesn’t save all that much in storage hardware.
• Update-in-place interferes with a lot of modern optimizations in analytic DBMS design.
• Knowing what values data had in the past is interesting in and of itself.
• So, potentially, is knowing what “dirty” data end-users — especially customers and prospects — decided to enter.
• The “right” amount of data validation is application-dependent. For example, if data validation involves torturing your customers, maybe it’s not such a good idea. (Great observation by Chris.)
• If you have the old data as well as the new, the harm of having “bad” updates is lessened. (Central connecting observation by Chris.)
• People enter data inconsistently. MDM (Master Data Management) and data cleansing tools fix much (admittedly not all) of the harm. Computers are cheaper than people. You do the math.
• Data is increasingly being managed in non-relational and/or non-persistent ways. Get used to it.
• As the NoSQL guys point out, some of today’s most demanding applications have extremely simple schemas.

*I probably have it worse than most people in that regard, because my clients run tentative feature lists and announcement schedules by me well in advance, which may get changed multiple times before the final dates roll around. I also occasionally miss some detail, if it wasn’t in a pre-briefing but gets added at the end.

Anyhow, the three big themes of this month’s announcements are probably:

• Integrating different kinds of analytic processing into databases and DBMS.
• Taking advantage of hardware advances.
• Playing catchup in areas where small vendors’ products weren’t mature yet.

For example, the three biggest data warehouse DBMS product announcements this month are probably:

• Aster Data nCluster 4.5. Much like Aster’s prior release — Aster Data nCluster 4.0 – Aster Data nCluster 4.5 has a major focus on integrating analytics and database processing. This time, the emphasis is on application development tools and pre-built analytic packages. In addition, Aster’s management tool GUIs have been upgraded, building on catch-up functionality in the Aster Data nCluster 4.0.
• Netezza’s “i” add-on to its existing TwinFin products. With Netezza TwinFin(i), Netezza becomes the second MPP RDBMS vendor with a comprehensive “Big Data Analytic Platform” kind of strategy. (Netezza would surely argue that it was the first, but that depends on how seriously one took Netezza’s prior attempt.) Many of the details are different from Aster’s, of course, but the general philosophy is similar. So far, Netezza has announced one interesting proprietary library of analytic packages (for linear/matrix algebra), plus the port of 4,000 or so functions in open source libraries.
• Vertica 4.0. Vertica has had a highly innovative columnar DBMS architecture from the getgo, but at the cost of some restrictions or awkwardness in the relationship between data layout and SQL processing. Vertica says that Vertica 4.0 fixes all that. In addition, it has some analytic processing enhancements, especially in the time series area, where Vertica doesn’t vigorously dispute that Sybase IQ previously had an advantage.

In addition,

• Teradata is announcing its Data Warehouse Appliance 2580, the successor to the Teradata 2550. This is purely a hardware refresh; Teradata’s hardware and software upgrades are not generally synced. The Teradata 2580 upgrades CPUs from Harpertown to Nehalem, includes 3X the RAM of its predecessor, and offers an option for 1 TB disks (thus lowering the bottom price/TB a lot, to $31K list).
• Aster, Vertica, and ParAccel have all called attention to the fact that, if solid-state drives have interfaces like those of disk drives, and if a DBMS supports disk drives, then a DBMS also supports solid-state drives as well. At least Aster and ParAccel have signaled that they have at least one customer or prospect each interested in Fusion I/O’s solid-state technology, especially in the retail sector. This is basically a hardware matter as well, and a big deal only for those who were somehow unaware of the impending dominance of solid-state memory technology.
• Sybase announced its Aleri acquisition earlier this month.
• Various vendors have bragged about various rankings, awards, or benchmarks, or – sometimes less tediously — about last year’s sales results.

• Netezza’s software runs on well-designed proprietary hardware. Aster runs on hardware that’s more off-the-shelf.
• Aster was first to ship, and will also be first to ship an IDE (Integrated Development Environment).
• MapReduce is central to Aster’s approach. Netezza TwinFin(i) supports MapReduce too, specifically a Hadoop implementation, but I don’t get the sense that everything Netezza does is built on MapReduce underpinnings.
• Both Aster and Netezza try to provide rich functionality for creating in-memory data structures parallel analytic programs can use. Both seem to let you escape from the pure relational-table paradigm more easily than, say, Teradata’s new persistent memory capabilities do.
• Aster and Netezza have made different choices about what kinds of prebuilt analytic packages to offer. Netezza could actually leapfrog Aster in this regard, but let’s see where each vendor is by, say, mid-year. If you care about the details of built-in analytic functions, you really should consider executing non-disclosure agreements with both those companies.
• Both Aster and Netezza stress that you can run analytic functions out-of-process, greatly reducing the chance that they crash the database. Netezza and I’m pretty sure also Aster also retain the option of running in-process, which provides maximum performance. (In Netezza’s case C++ is the only in-process language supported, and I think Aster has a similar limitation.)
• Like Aster, Netezza is integrating SQL queries and other analytic processing under the same workload management rubric.
• Much like Aster, Netezza is tap-dancing by implying much richer forthcoming SAS support than anything currently announced. (The crunch-per-paragraph ratio in either vendor’s SAS-related press releases to date is distressingly low.)

More specifically, here are some highlights of what I know, am guessing, and/or am allowed to say about Netezza TwinFin(i) at this time.

• The foundation for the analytic add-ons in Netezza TwinFin(i) is some sort of low-level “analytic executables.” Not understanding exactly what these are is my biggest area of confusion in the whole TwinFin(i) stack. Are they all C++, with everything translated into same? Is there Java all the way down as an alternative? (E.g., Hadoop is written in Java.) Anyhow, whatever it is, it’s surely a big improvement on Netezza’s prior Verilog-based generation of analytic extensibility technology.
• The announced list of languages supported in Netezza TwinFin(i) is Java, Python, Fortran, R, and C/C++. More are coming.
• Netezza has named a lot of analytic functions it is adding, and hinting about more to come. It has named CRAN/R and GNU libraries, saying those have 1900 or more functions each. Netezza has also built its own linear algebra library for TwinFin(i), called nzMatrix. And as previously noted, TwinFin(i) also boasts a Hadoop implementation.
• I haven’t heard about much in the way of TwinFin(i)-specific IDE support.
• I don’t really have details as to what kinds of in-memory data structures Netezza TwinFin(i) does or doesn’t support.

• Aster Data Analytic Foundation, a set of analytic packages prebuilt in Aster’s SQL-MapReduce
• Aster Data Developer Express, an Eclipse-based IDE (Integrated Development Environment) for developing and testing applications built on Aster nCluster, Aster SQL-MapReduce, and Aster Data Analytic Foundation

And in other Aster news:

• Along with the development GUI in Aster nCluster 4.5, there is also a new administrative GUI.
• Aster has certified that nCluster works with Fusion I/O boards, because at least one retail industry prospect cares. However, that in no way means that arm’s-length Fusion I/O certification is Aster’s ultimate solid-state memory strategy.
• I had the wrong impression about how far Aster/SAS integration has gotten. So far, it’s just at the connector level.

Aster Data Developer Express evidently does some cool stuff, like providing some sort of parallelism testing right on your desktop. It also generates lots of stub code, saving humans from the tedium of doing that. Useful, obviously.

But mainly, I want to write about the analytic packages. I’m not convinced that they’re a big deal in themselves yet, or that a whole lot of person-months have gone into their combined development. Still, I think they provide a great indication of one direction in which analytic functionality is going. And by the way, Aster promises to release a lot more of that kind of thing over the next 12 months.

Aster’s flagship analytic package is nPath, which is like a regular expression matcher, but for (time) series of data rather than for character strings. The main use for nPath is in pulling specific kinds of event sequences out of web or network event logs. However, one could imagine uses in other sectors that focus on temporal or sequential data (e.g., trading, intelligence, other sensor analysis), should existing SQL- and/or CEP-based technologies not prove sufficiently flexible. Aster 4.5 adds some new aggregation capabilities around nPath.

Other not-wholly-new packages in the Aster Data Analytic Foundation announcement are for sessionization (of clickstream data and the like) and tokenization (of text/character string data). While sessionization can be done in SQL, Aster thinks its MapReduce-based version is faster, since it doesn’t require self-joins. Makes sense. Aster’s tokenization sounds lame, however – text analytics in MapReduce tends to reinvent simplistic wheels for no clear reason, and Aster doesn’t seem to be an exception. (Aster would argue, however, that anything it does in SQL-MapReduce is more flexible than pure SQL or pure MapReduce alternatives.)

Another example of better-living-without-self-joins is Aster’s new market basket package. This lets you look at a set of point-of-sale data, pick a small integer N, and pull out all the sets of N things that were bought by the same person at the same time. I haven’t probed the claim in detail, but Aster implies there’s less combinatorial explosion in its approach than it is in the self-join alternative.

Note: Gartner highlighted self joins as a performance challenge in its recent Data Warehouse Magic Quadrant.

Aster is also releasing a few statistical and general analytic functions — specifically (and I quote a slide):

• exponential moving average
• weighted moving average
• simple moving average
• volume-weighted average price
• correlation
• linear regression
• logistic regression
• approximate_percentile
• approximate_count_distinct

The point of the last two items on the list is that if you set a non-zero tolerance for error, you can you can count things or order them into bins very efficiently – especially in terms of RAM — while being guaranteed not to exceed your error tolerance.

Note: One obvious inference from this list — which Aster gladly confirms — is that Aster has high hopes of selling to the financial services industry.

Finally, Aster is releasing its first pure graph-analytic function, for finding the shortest path between a given pair of nodes.

While I had the Aster folks on the phone anyway, I also took the opportunity to ask about the Aster nCluster 4.0 capability to create fairly persistent non-relational in-memory data structures. Specifically, I asked whether different users could access the same in-memory structure, and was told that this is a little klugey but not too horrendous. That suggests Aster’s capability may be a strict superset of UDF-based (User-Defined Function) approaches to meeting the same need, at least from a functionality standpoint. However, ease of creating those in-memory structures may still be better in the more SQL/UDF-centric approach favored by Teradata.