• Dwight Merriman of 10gen (MongoDB)
• Damien Katz of Couchio (CouchDB)
• Matt Pfeil of Riptano (Cassandra)
• Todd Lipcon of Cloudera (HBase committer)
• Tony Falco of Basho (Riak)
• John Busch of Schooner
• Ori Herrnstadt of Akiban

By no means do I have time to do these conversations justice, in terms of giving them the write-ups and/or immediate follow-up that they deserve. Indeed, I’ll leave for vacation Saturday morning with my 2000-word NoSQL article still unwritten. So I’ll dump as many observations as I can into one or a few posts now, and play catch-up later as circumstances allow.

In no particular order:

• A number of NoSQL offerings have had more uptake to date than most of the scale-out SQL offerings have.
• “Document-oriented” NoSQL projects CouchDB and MongoDB have probably had the most users get into production, but perhaps for pretty small systems.
• Cassandra and Hbase — the column-group-architecture guys — have probably had the most bang-in-lots-of-writes HVSP production uptake.*
• I didn’t talk customer count with Schooner, but the decently-stocked Schooner customer page suggests Schooner may be something of an exception to these generalities.
• A lot of these companies are in the low-to-mid-teens of employees.
• The SQL-oriented companies, despite having fewer or no customers, often seem to have more money. (One reason I get the impression SQL guys have more money is, frankly, that more of them are talking about engaging my services.)
  • Schooner cites $20 million in VC.
  • Clustrix cites a figure close to that.
  • Basho cites $10 million, plus a new round of $1.5 or $2 or $2.5 million. The new round is at a  lowered valuation.
  • That same site says Tokutek finally was able to raise some VC. Congrats!
• It’s only a two-company trend, but I was pleased to hear that both 10gen/MongoDB and Akiban were seeing Drupal as a major use case or potential use case. No word on rescuing WordPress from its MySQL implementation, alas, but it seems that a Drupal site typically has 40-200+ tables, while a WordPress one has 10ish.
• Another trend I think I’m seeing is serious object-oriented apps banging things straight into a simple back end. Workday is a huge example of that. Akiban hopes to do something similar with Hibernate.
• Stability and maturity are still issues for many of these products. E.g., HBase isn’t even in Release 1.0 yet. Ditto Cassandra, and surely many of the others. Unsurprisingly, making Cassandra stable is still a challenge.

*As is common for terms I suggest, the “HVSP” name is not getting any traction. What do you think of Marton Trencseni’s suggestion of OLRP, for OnLine Request Processing?

One thing that makes following this area interesting is that so many projects are open source, leading there to be a lot of information in the wild. I hardly have time to read the mailing list for each project; but the people I talk with do, and often they may sorta kinda remember something somebody else posted one or several months back. As just one example, the mailing lists are said to confirm:

• Contrary to rumor, Facebook hasn’t moved in-box search off of Cassandra.
• Apparently, however, it’s true that Cassandra inventor Facebook has stopped working on Cassandra, and Facebook’s core Cassandra developers have shifted over to HBase.

Also, figuring out usage of open source software can be … interesting.

• People who use open source software don’t have to reveal themselves, as there’s no purchase transaction to kick things off.
• On the other hand, if they’re serious enough in their use, they often do.
  • There are two main ways to get tech support for open source software — the community or a company that sells support — and both ways let the main support-selling company know that one is a user.
  • Some folks even add themselves to open lists of users, for example these rather long lists for HBase and CouchDB.
  • Or they show up at conferences. For example, two tweets from Riptano founder Jonathan Ellis suggest at least 30 production Cassandra users were represented at a recent event. That’s more detail than his colleague Matt Pfeil wanted to give me when talked.

OK. This post has gotten pretty long, even without me saying anything resembling an overview of any of the seven companies I listed up top, or of their products’ adoption. So I’ll just publish this now, and edit in links below to follow-on posts if and when they become available.

I would say thousands. The object model for our applications consists of over 2000 classes. On average these classes have multiple relationships with other classes so that would have some kind of multiplicative effect when it came to using tables.

One example of where you’d be proliferating tables (and not getting as satisfactory of a solution relationally) is worktags. Currently we have a class for worktags. Instances of this class can point to various instances of detail lines (expense lines, po lines, invoice lines, etc…). A detail line can have many worktags pointing to it. To model this relationally you’d need either a separate table for each type of detail line in the system to store the tags associated with it or a single worktag for detailed line table that could be foreign keyed for all types of detail lines that would store their worktag. Either way involves more tables and more clunkiness.

Another example of where our oo designs wouldn’t directly translate is our ability to describe to shared part of a detail line in one class and have all instances of detail lines inherit the fields that are shared. To do this relationally you’d probably replicate the shared fields in each table representing the various kinds of transactional details (again lines, po lines, invoice lines, etc…). You’d lose the ability to maintain and change the shared fields (and the processing logic for those fields) in one place.

Anyway, I’d go with “thousands” as our answer. I do think this is an interesting question and wish we had more time to figure out a more accurate answer.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

• Workday has forward-thinking ideas about SaaS enterprise applications and the integration of business intelligence into same.
• Workday has highly innovative ideas in how it manages data.
• Companies founded by Dave Duffield tend to feature smart, likeable people who talk to one pleasantly and forthrightly. Workday is no exception; CTO Stan Swete and the other Workday folks present were a delight to talk with.
• I’d invited Merv Adrian to come along with me. He asked great questions, and I could gather myself a bit despite how sleep-deprived I was for the first part of that trip.

Workday kindly allowed me to post this Workday slide deck. Otherwise, I’ve split out a quick Workday, Inc. company overview into a separate post.

The biggie for me was the data and object management part. Specifically: 

• Workday’s applications run entirely in-memory, in a highly object-oriented structure. Persistence is mainly for the sake of data safety …
• … but not entirely. In earlier releases, Workday kept absolutely everything in RAM. However, certain things are kept only on disk, such as:
  • Audit files.
  • Certain documents (notably resumes).
• Workday’s whole database – data and metadata alike – is persisted to disk in <10 MySQL/InnoDB tables. MySQL is basically just being used as a key-value store, albeit one with ACID transactional support.
  • There are 3 main tables: attributes, relationships, instances.
  • When I suggested this might be like an entity-attribute-value model, Workday said it would be even better to think in terms of instanceID-attribute-value.
  • As you might expect for a database that simple, its schema doesn’t change much.
  • By way of comparison, Workday estimates that if its software were written relationally, there would be 1000s of tables, which would take up 10-100X as much disk space.
• All write transactions are banged immediately into the MySQL database. I.e., RAM and disk are never allowed to get out of sync.
• Workday’s database is append-only. This is exploited for effective dating (pretty heavily, it seems, perhaps because that’s a useful concept in human resources) and snapshotted reporting.
• Workday’s built-in BI doesn’t have a lot of choice but to do scans, traversing the object model. This turns out to be fast enough.

Other notes on Workday’s data and object management strategy include:

• Workday is object-oriented through and through – no object-relational mapping – turtles all the way down. On average, a class has about 2 attributes.
• 94% of requests are reads, traversing the object hierarchy.
• Workday databases are pretty small.
  • The biggest database Workday supports uses 17 gigabytes of RAM.
  • Workday databases are much smaller on disk than in RAM.
• Workday’s “dream” is to move from disk to solid-state memory.
• Workday uses GPLed MySQL/InnoDB. So there’s no software license reason to ever move away (e.g., to a pure key-value store).
• Disaster recovery is based on local and remote MySQL slaves.

Obviously, serious apps have been built before in object-oriented and/or key-value ways, with the resulting objects then being banged to disk (or in some cases kept in memory). Examples include:

• Numerous applications are built on object-oriented DBMS. Generally they go against disk, although memory-centric implementations can save a lot of pointer-chasing. Often they’re queried via SQL.
• Basho’s website says that its key-value store Riak was originally conceived in connection with a planned salesforce automation product, but I don’t think that the application part of that plan ever got built.
• SAP has longstanding doubts about relational dogma, although not nearly to Workday’s extreme.
• Obviously, some major internet applications just bang data into key-value stores.

Still, perhaps because it wholly object-oriented yet doesn’t even bother with anything like a real object-oriented DBMS, Workday’s approach seems particularly cool.

Other highlights of Workday, Inc.’s technical story include:

• Workday has settled into a schedule of three releases per year, and has pretty much lived up to that for >2 years.
  • Every user is always on the latest Workday release.
  • You can delay turning on significant new Workday software functionality if you want to.
  • Pure UI changes to the Workday software are handled much as they are on various websites today. Sometimes you have no choice but to live with them; sometimes the prior version of the UI remains available to you for a while.
• Workday’s navigational approaches look pretty cool.
  • The core concept is a list of actions you can perform now, rather than more standard menus.
  • Roles/permissions are of course central to this.
  • Reports have lots of actionable links in them. (More than just drilldown, although specific examples have slipped my memory.)
  • Alternatively, you can navigate via a search box, searching both on names of objects (e.g. users, divisions) or on names of tasks. This is somewhat reminiscent of an approach SAP was considering a few years ago.
• Workday says it has four key design premises:
  • Web-Familiar Experience. I’d say that’s true to to the extent it makes sense. In many ways, the web needs to catch up to Workday.
  • Enterprise Reporting. The idea is that you get a report, then take actions based on it. Hence the report-centric options for navigation.
  • Integration On-Demand. That’s a fancy way of saying “Plays nicely with others.”
  • Configurable Business Processes. Duh. That’s pretty essential if you want to do serious SaaS applications.
• Workday maintains a strong separation between application logic and UI development. Developer do no screen layouts. Instead, Uis are automatically generated for:
  • Flash/FLEX
  • iPhone
  • Mobile HTML
  • PDF export
  • Excel export
• Workday only talks to the outside world via web services.
  • Workday is heavily into SOAP (Simple Object Access Protocol).
  • The acquisition of OEM partner CapeClear gave Workday an Integration Service (i.e., enterprise service bus) that translates SOAP into whatever else might be needed for integration, and also does reliable delivery.
  • All that said, Stan Swete sees integration among various SaaS offerings as an area needing significant future attention.
• Workday’s business intelligence ideas are interesting, but I think there’s a long way for that technology still to go.
  • Workday’s BI seems to be focused on report/drilldown kinds of functionality.
    • You can slice by up to 2 dimensions at once.
    • Then you can keep slicing, however, by more dimensions, as many times as you like.
  • While you can take actions straight from reports, some of the specific BI/app integration ideas we discussed are still futures. (E.g., analyzing spend at the time of expense report data entry or approval.)
  • Of course, Workday’s web services interface lets you export Workday data into 3rd-party tools. Indeed, if you want to integrate data from Workday and some other source(s), that’s your only choice.
• Workday offers a clever metaphor to illustrate that your data may be more secure offsite than on – the bank vault. (I have no idea whether that’s a SaaS industry standard, but I hadn’t heard it before.) Of course, that metaphor does beg some issues specific to the remote data case, such as:
  • When your data is on premises, you know whether the government has insisted on looking at it.
  • More than cash, data keeps traveling back and forth to the remote location, which creates at least a theoretical risk of interception.
• Workday says the toughest part of globalization is the issue of which personal data is or is not maintained. For example, in the US you’re not allowed to not ask a job applicant’s religion, but in the UK you’re not only permitted but indeed required to.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

Workday, Inc. was founded by Dave Duffield and Aneel Bhusri, who’d previously worked together at PeopleSoft. It is generally the case that the companies Dave starts: 

• Develop application software for large or fairly large enterprise customers.
• Build those applications in/on their own platform technology, which is meant to be cutting-edge in its day. (For example, PeopleSoft was early in building an RDBMS-based client/server application suite, and did so with the help of a clever technology called PeopleTools I nonetheless helped talk PeopleSoft out of further commercializing.)
• Focus first on human resources software (Dave had another HR company before PeopleSoft).
• Move fairly early into non-profit/higher-education accounting (Dave had a company in that area before PeopleSoft, and PeopleSoft was fairly active in the area too).
• Emphasize a pleasant corporate culture.

All these things seem true of Workday Inc., although the non-profit/higher-ed move is just underway now. Specifically:

• Workday was founded in 2005, starting with an asset buy of some platform software a key PeopleTools developer had been working on for years.
• Workday has had multitenant SaaS offerings from the getgo. (And that’s all Workday does.)
  
• Workday has around 150 customers.
• Workday sells mainly to multinational corporations, generally based in North America. Efforts in the UK are beginning to ramp up.
• Workday has six core application modules, among which are:
  • Workday Human Capital Management (almost all the customers).
  • Workday Payroll (a little under 50 customers). Workday is partnered with local providers for payroll in 20 countries, and is building its second inhouse version (Canadian) now.
  • Workday Financial Management (a little under 20 customers, for what is far from a complete system).
  • Workday Benefits Network, providing connectivity to benefits providers (that’s the only Workday module that isn’t straight software).
• Workday, Inc. has around 500 employees, mainly in Pleasanton, CA. About 20 are in Dublin, Ireland, courtesy of the acquisition of CapeClear. About 1/3 are in development.

Workday prices its services based on metrics for the overall client business, not per-Workday-user. (Actually, the metric is basically headcount, which makes sense given Workday’s application focus.)

Some of these points are covered in more detail in a Workday Inc. slide deck.

This post is part of a three-post series

• Workday Inc. company overview (brief)
• Workday Inc. technology overview (detailed)
• Workday Inc. CTO Stan Swete’s comments on database strategy

Edit: Also, there’s a Workday blog with only a few posts, which nonetheless seems to flesh out a few of the ideas in this post series.

That said, I think I found four sensible points in Pentaho’s Hadoop strategy, namely:

1. If you use an ETL tool like Pentaho’s to move things in and out of HDFS, you may be able to orchestrate two more steps in the ETL process than if you used Hadoop’s native orchestration tools.
2. A lot of what you want to do in MapReduce is things that can be graphically specified in an ETL tool like Pentaho’s. (That would include tokenization or regex.)
3. If you have some really lightweight BI requirements (ad hoc, reporting, or whatever) against HDFS data, you might be content to do it straight against HDFS, rather than moving the data into a real DBMS. If so, BI tools like Pentaho’s might be useful.
4. Somebody might want to use a screwy version of MapReduce, where by “screwy” I mean anything that isn’t Cloudera Enterprise, Aster Data SQL/MapReduce, or some other implementation/distribution with a lot of supporting tools. In that case, they might need all the tools they can get.

The first of those points is, in the grand scheme of things, pretty trivial.

The third one makes sense. While Hadoop’s Hive client means you could roll your own integration with your own favorite BI tool in any case, having somebody certify it for you themselves could be nice. So if Pentaho ships something that works before other vendors do, good on them. (Target date seems to be October.)

The fourth one is kind of sad.

But if there’s any shovel-meet-pony aspect to all this — or indeed a reason for writing this blog post — it would be the second point. If one understands data management, but is in the “Oh no! Hadoop wants me to PROGRAM!” crowd, then being able to specify one’s MapReduce might be a really nice alternative versus having to actually code it.

It also is understood that, realistically, I can’t be expected to know and mention the very latest news for all the many products in the categories. Even so, I think this would be fine time to check just where NoSQL and HVSP adoption stand. Here is most of what I know, or links to same; it would be great if you guys would contribute additional data in the comment thread.

In the NoSQL area: 

• Back in April, the VoltDB guys told me they thought Cassandra and HBase were the two NoSQL systems with the most momentum.
• I know distressingly little about HBase adoption, but a source who may or may not wish to remain anonymous was kind enough to alert me that Twitter and StumbleUpon each have ~30 node deployments, for analytics and analytics/HVSP respectively.
• I wrote in detail on Cassandra adoption last month. News since then includes:
  • Facebook is rumored to have dropped Cassandra completely.
  • Twitter clarified that it may not be quite as lovestruck by Cassandra as before, but they’re still very close friends.
  • It’s not obvious that the Cassandra Summit unveiled a lot of new adoption stories.
• Northscale’s Membase is still in its early days.  Zynga is bought in, however, as is something called NHN Korea. (Edit: I subsequently saw NHN Korea on a prominent SEO expert’s list of the top half dozen or so search engines in the world. Who knew?)
• Basho has listed a few Riak customers. If memory serves (I haven’t spoken with Basho for a while, and some of my notes are misplaced due to some computer sloppiness), Basho has a few dozen customers in total.
• Mozilla has a 4 machine, 64 core Riak cluster in production.
• Hypertable has a few users/project sponsors, Baidu being the biggest name among them.
• I don’t really know how the MongoDB/10gen guys are doing. I think this is at least as much my fault as theirs. Anyhow, they seem to have links to a couple of folks who have written about MongoDB usage.
• NimbusDB is still in stealth mode. I’d be surprised if they had users for a while yet, since in January they didn’t yet sound as if development was very far underway. (Actually, I forget whether NimbusDB is supposed to be SQL-based or not.)

Among the SQL or SQL-friendly guys:

• Clustrix says it has a few production users, some big-name, but is not disclosing them yet.
• dbShards has around 6 customers, including Facebook. (Facebook may outpace even Twitter and Zynga in using the most products mentioned in this post.)
• As of May, VoltDB had one paying customer, plus 150 beta customers who weren’t in production yet.
• Akiban says they’ll get me up to speed on Thursday.
• ScaleDB seems to be pedaling along in perennial beta. Whether ScaleDB has any actual beta users is less clear. On the plus side, checking that out uncovered a pretty funny April Fool blog post.
• Groovy Corporation seems to have disappeared, or morphed into something called uCirrus, or something like that.

To review, Membase is a reasonably elastic persistent data store, sporting the memcached API, making memcached/Membase an attractive alternative to memcached/sharded MySQL. As of now, Membase is a pure key-value store.

Northscale defends pure key-value stores by arguing, in effect: 

• You can do a lot with entity-attribute-value triples.
• If your key looks like an entity-attribute concatenation, then your entity-attribute-value triple can be transformed into a key-value pair.

Northscale has a point. Still, I think that in most use cases you’ll want a data model and/or data access methods that are at least a little richer than pure entity-attribute-value.

Fortunately, that’s the direction Northscale is taking Membase. I don’t get the impression that the details have been worked out yet, but the general idea is:

• Northscale is putting a publish-subscribe interface into Membase it calls “tap,” useful for replication, node rebalancing, etc.
• Tap will also serve to connect Membase data to a Membase feature Northscale calls “Node Code,” which will be code that runs in a separate process on each Membase node.
• Node Code will include things like:
  • Language run-times
  • Standard libraries for things like index-building

Will Membase Node Code be a close substitute for relational DBMS functionality, or even the Cassandra architecture? I doubt it, especially at first. But at least it will keep Membase developers from getting locked in to a very simple and restrictive data management paradigm.

• If your goal is to keep a certain class of queries from taking too many resources, Tim thinks a great way of doing that is to control how many of them are allowed to run concurrently.
• By way of contrast, Tim is cautious about the common approach of just lowering a query’s priority. His concern is that a long-running query could linger even longer, creating a long-lasting bottleneck in, for example, temp space.
• When running over (I believe) Linux and AIX, DB2 workload management is integrated with operating system workload management. I.e., the same “service class” or “workload class” (at a guess, the former is the official term and the latter is the term that makes sense) of queries and associated processes gets the same treatment in both DB2 and the OS.
• DB2’s workload management extends to buffer pools, to inhibit low-priority queries from evicting a higher-priority query’s data from cache.
• Sometimes, workload management doesn’t throttle a query, but just decides to collect stats for future analysis. (This is on the eminently reasonably theory that the best stats to collect are the ones that are live when performance problems are actually occurring.)

Finally, Tim spoke of what I regard as the weirdest workload management requirement, one I also heard about from Netezza (but didn’t explicitly mention) in June. Sometimes, it seems, you simply don’t want queries to finish too fast. Why? Because if you give great performance when the machine is lightly loaded, then business users might expect that performance too when the machine is heavily loaded and you can’t deliver it. Apparently, in some environments it’s better to never deliver great query performance than it is to do so only inconsistently.

• You really, really want to have multiple data streams coming out of temp space, as close to simultaneously as possible.
• The storage performance characteristics of such a workload are more reminiscent of “random” than “sequential” I/O.

If everybody else is cool with it too, I can live with that.

Meanwhile, I talked again with Tim Vincent of IBM this afternoon. Tim endorsed the temp space/Flash fit, but with a different emphasis, which upon review I find I don’t really understand. The idea is:

• Analytic DBMS processing generally stresses reads over writes.
• Temp space is an exception — read and write use of temp space is pretty balanced. (You spool data out once, you read it back in once, and that’s the end of that; next time it will be overwritten.)

My problem with that is: Flash typically has lower write than read IOPS (I/O per second), so being (relatively) write-intensive would, to a first approximation, seem if anything to disfavor a workload for Flash.

On the plus side, I was reminded of something I should have noted when I wrote about DB2 compression before:

Much like Vertica, DB2 operates on compressed data all the way through, including in temp space.

• Technology it already has released*, but has not published any reference architectures for
• A Barney partnership**

In other words, Vertica has succumbed to the common delusion that it’s a good idea to put out half-baked press releases the week of TDWI conferences. But if we look past that kind of all-too-common nonsense, Vertica is highlighting an interesting technical story, about how the analytic DBMS industry can exploit solid-state memory technology.

*Upgrades to Vertica FlexStore to handle Flash memory, actually released as part of Vertica 4.0

** With Fusion I/O

To set the context, let’s recall a few points I’ve noted in the past:

• Solid-state memory’s price/throughput tradeoffs obviously make it the future of database storage.
• The Flash future is coming soon, in part because Flash’s propensity to wear out is overstated. This is especially true in the case of modern analytic DBMS, which tend to write to blocks all at once, and most particularly the case for append-only systems such as Vertica.
• Being able to intelligently split databases among various cost tiers of storage – e.g. Flash and disk – makes a whole lot of sense.

Taken together, those points tell us:

For optimal price/performance, analytic DBMS should support databases that run part on Flash, part on disk.

While all this is a future for some other analytic DBMS vendors, Vertica is shipping it today.* What’s more, three aspects of Vertica’s architecture make it particularly well-suited for hybrid Flash/disk storage, in each case for a similar reason – you can get most of the performance benefit of all-Flash for a relatively low actual investment in Flash chips: 

• Vertica lets you split tables by column, and Vertica FlexStore is versatile enough to let you put only the most-used columns in Flash. (Vertica offers a figure that 85% of usage calls on only 15% of columns, but I don’t know how rigorously grounded those numbers are.)
• To the extent that Vertica data is more compressed than many of Vertica’s competitors’ (which it probably is, debates over the magnitude of Vertica’s advantage notwithstanding), the total storage-hardware cost of sticking stuff in Flash is less when you use Vertica than with other systems.
• Vertica has relatively less need for temp space than some other systems. (Vertica uses figures of <20% of total storage, vs. 30%+ for some other systems.) If you want to use Flash for temp space, so as to accelerate your toughest queries, that can save you some cash …
• … and by the way, temp space is an especially good use of Flash, because temp space is accessed in a less sequential manner than data storage is.

The least obvious of those points are about temp space; I only understood the particulars when Vertica development chief Shilpa Lawande explained them to me Thursday.

* At least in theory; customer adoption may be a different matter.

But before drilling down on temp space, let me first note that there’s one offsetting factor to all those “We need somewhat less Flash than the other guys” Vertica advantages. Like all serious databases, a Vertica installation keeps two or more copies of all data, to that there’s no storage single point of failure. In a flexible system like Vertica, you can put one copy on Flash and one on disk. But if you do that in Vertica, you forgo fully exploiting one possible benefit of Vertica’s architecture – the ability to store different copies of a column in different orders, which are beneficial for accelerating different groups of queries.*

*More precisely, you don’t get the full benefits of Flash acceleration for every query touching those columns.

OK. Back to temp space. There are four kinds of things you can put in storage if you’re running a database management system:

• The software itself.
• Persistent data. (I.e., tables, if the DBMS you’re running is relational.)
• Metadata, especially the kind that lets you find data – indexes, zone maps, catalogs, etc.
• Temporary data constructs built as part of, say, a sort-merge join. These, by definition, are what populate temp space.

Just to be clear, those constructs are NOT temporary tables of the sort created by, say, Microstrategy; such tables are handled like any other data. Rather, they are ephemeral creations and, so far as I can tell, not tables at all.

Vertica offered two theories as to why its DBMS requires less temp space than competitors do:

• To the extent data is decompressed before being operated on in memory by the DBMS, that decompression would of course also apply to temp space as well. Vertica prides itself on keeping data compressed all the way through, and seems to get away with smaller temp space allocations as a benefit.
• Since Vertica can store columns in expedient sort orders, it does less sorting overall, and sorting is a big use of temp space.

Obviously, no matter which DBMS you use, the amount of temp space you need is surely workload-dependent. Even so, Vertica’s claim to something of an advantage seems legit.

Truth be told, I’m not convinced the savings involved are great enough to matter a whole lot – but it’s a fun subject to think through.

And finally: One of my biggest surprises since starting to look at analytic-DBMS-on-Flash has been the centrality of temp space. Talking to Vertica Thursday, I finally uncovered a key reason why: Temp space tends to be accessed via multiple streams of data at once. I’m still struggling with WHY that is true, with two reasons suggested being:

• Temp space can be accessed by multiple operations at once. (But isn’t that also true of the rest of storage?)
• Merge sorts, a common use of temp space, read multiple streams of data. (Couldn’t you tweak your software to make that not be true?)

But if we grant that temp space naturally is accessed in multiple places at once – well, that’s a lot like random I/O, and if you’re doing a lot of random reads, you’d love to use something other than spinning disk.

]]> http://www.dbms2.com/2010/08/16/vertica-flash-temp-space/feed/ 10 http://www.dbms2.com/2010/08/16/vertica-flash-temp-space/