To understand Sybase’s plans for Aleri and CEP, it helps to understand Sybase’s current CEP-oriented offering, Sybase RAP. So far as I can tell, Sybase RAP has to date only been sold in the form of Sybase RAP: The Trading Edition. In that guise, Sybase RAP has been sold to >40 outfits since its May, 2008 launch, mainly big names in the investment banking and stock exchange sectors. If I understood correctly, the next target market for Sybase RAP is telcos, for real-time network tuning and management.

In addition to any domain-specific applications, Sybase RAP has three layers:

• CEP (Complex Event Processing). Sybase RAP CEP is based on a version of the Coral8 engine Sybase licensed and has been subsequently developing.
• In-memory DBMS. Sybase’s IMDB is part of (but I guess separable from) and has the same API as Sybase’s OLTP DBMS Adaptive Server Enterprise (ASE, aka Sybase Classic).
• Sybase IQ. Actually, Sybase used the phrase “based on Sybase IQ,” but I’m guessing it’s just Sybase IQ.

In theory, there could be a DBMS other than Sybase IQ, such as Sybase ASE or even Oracle, because Sybase IMDB can talk to a variety of DBMS. I didn’t get the impression, however, that in practice there were any Sybase RAP installations whose persistent DBMS was anything other than Sybase IQ.

Aleri had all along had something called Project Ohio, to merge Coral8 with Aleri Classic. Now Sybase’s own CEP engineering team is being added to the mix, schedules are being reconsidered and haven’t been disclosed yet. (If one woman can produce one baby in nine months, how long does it take nine women to produce a baby?) Apparently Sybase has a dozen programmers in the CEP area, plus ~20 more on Sybase RAP, not counting QA, documentation, etc.; that represents a significant bump to the overall Aleri development team.

Sybase doesn’t seem to have decided what to do yet with the various business intelligence/real-time OLAP engine products and technologies it is inheriting from Aleri.

And finally, some metrics:

• The Sybase/Aleri guys estimate that 1/3 of of Aleri’s customers and even less of its revenue came from outside the financial services sector. They did say the non-financial-services business was “starting to pick up,” but not very convincingly.
• Sybase IQ is now up to >1800 customers, with >200 new ones in 2009.
• Sybase IQ indeed has users taking in market feeds up to 3 terabytes a day, so it probably matches Vertica in having at least several-hundred-terabyte databases in the financial sector.

• The folks at Sybase still haven’t figured out when to prebrief me. (Edit: I’ve been briefed subsequently.)
• Sybase/Aleri is a potentially powerful combination, if they can effectively address the point I just made about integrating disparate latencies. That said, I’m not expecting a lot, because the CEP industry always disappoints me.
• Microsoft, IBM, and (somewhat less clearly) Oracle are all trying to do CEP inhouse. Sybase is making a good choice in having serious CEP inhouse itself
• Surely the main focus and financial justification for the Sybase/Aleri acquisition is the financial services market.
• Specifically, I expect the focus of technical integration between Aleri and Sybase’s DBMS products to start with Sybase IQ.
• Coral8 had some interesting ideas about how to integrate CEP with OLTP/operational BI, but I’m not aware that they got much traction.
• I bet there are use cases where Sybase tries and fails to sell Adaptive Server SQL Anywhere that CEP would be a better technical fit, but I don’t immediately see much practical business significance to that observation.
• While this deal could easily strengthen the Vertica/StreamBase partnership, I don’t see any reason why it would lead those two companies to actually merge.

Related link

• Thinking about analytic latency

• The most important issue in database and analytic technology, in my opinion, isn’t technological at all – rather, it’s the legal and political steps needed to preserve liberty in the face of advancing, intrusive technology.
• Another important issue for society – and this one does involve a lot of technology – is scientific number crunching. In particular, database technology for scientific computing needs to be developed much further. I’ll have more to say on all this soon.
• More generally, technology needs to keep advancing for parallel analytics. Fortunately, it is. Watch this space over the next few weeks.
• Oracle has said, in effect, that its most important technological challenge of the decade is getting solid-state memory right. I agree.
• Data volumes will keep going up, up, up. Technology needs to keep evolving accordingly. Much of what I write is on that subject.
• Data needs to be processed and analyzed at very different latencies. And there’s much further to go in integrating disparate latencies.
• Analytic database management in the cloud hasn’t been solved yet, especially for Big Data. Among the reasons are the difficulty of moving data into the cloud (unless it originated there), the slowness of moving it from node to node in shared-nothing architectures (which reduces the elasticity benefit), and above all the long and unpredictable latencies of interprocessor communication while queries are running (a key subject of discussion at the Boston Big Data Summit).
• Better business intelligence user interfaces are increasingly available. I’m thinking particularly of approaches with buzzwords like visualization/interactive exploration or faceted. But they aren’t well-integrated into the overall analytic stack, as big BI vendors are trailing the smaller ones in this regards. (Part of the problem relates to my previous point.)
• Application development over text search isn’t in the same league as application development over relational DBMS. The choices are mainly XML (e.g., MarkLogic), SQL for text integrated into RDBMS (limited by the weakness of those integrations), and something like Attivio’s Java SDK. There’s a major conceptual barrier in building those apps, namely the unpredictability of query results. Still, it should be possible to do better.
• Similarly, text analytics and conventional analytics exist well side by side. They can even be in the same database and/or dashboard, although in practice that is limited by the strong SaaS focus of text mining vendors and users. But analytic integration of them is really hard. Linguistic imprecision is, in my opinion, only the #2 reason for this difficulty. The #1 reason is that trends detected by text analytics are much less precise than trends on tabular data – e.g., a 50% increase in a certain kind of complaint may be no more significant than a 5% change in a revenue variable.
• I’m increasingly persuaded that graph analytics can be handled without a graph-centric data model. But right now, it isn’t being handled well at all. Lots more needs to be done – although when it is, it will just exacerbate the privacy/liberty dangers that so concern me.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology

One section covered recent/ongoing/near-future trends that I thought were particularly interesting, including:

Simpler database technology, by which I mean DBMS that are:

• Easier to administer than market-leading systems …
• … even if at the cost of being special-purpose
• E.g.,
  • MySQL and older mid-tier RDBMS such as Progress
  • Many analytic DBMS and appliances, most notably Netezza’s

For general purpose or OLTP uses, I’m not a big fan of MySQL (not enough progress in making it industrial-strength), PostgreSQL (no good company behind it – I’m a non-fan of EnterpriseDB), or Ingres (open source or not, it’s an antiquated system that hasn’t been invested in as much as Oracle, DB2 or SQL Server).

But I get the impression there are a lot of contenders among small startups, featuring very new architectures for OLTP or general-purpose database management. VoltDB comes to mind. NimbusDB is finally within range of getting funded. Dan Weinreb told me Friday he knows of a bunch of others as well. And that’s all before we even get into the NoSQL kind of alternative.

Flexible storage architectures. That’s starting out with an emphasis on hybrid columnar, as in the examples of Vertica and Greenplum. Oracle (to whom I’m under no NDA obligation) and other vendors (to whom I am) are going that way as well.

Multi-tier database architectures, by which I mean at least two things:

• The database tier/server tier split of Exadata
• Hybrid RAM/disk architectures, examples of which include
  • Vertica’s RAM-based write-optimized store
  • Sensage’s CEP-in-the-DBMS
  • This in-memory analytics stuff we keep hearing about from the BI vendors
  • Any true in-memory/disk hybrid, such as the regrettably sidelined solidDB
  • Smart thinking by numerous DBMS vendors about optimizing the use of RAM and/or Level 2 cache

Netezza is particularly interesting to watch in this regard because it:

• Had a pretty strict storage/other processing split in prior product generations and …
• … ditched that in its latest generation …
• … which however is focused on optimizing the use of RAM cache

Also noteworthy is Petascan, the stealth-mode –and therefore harder to watch right now – company I keep teasing about, which makes a strong case for carrying the database/storage tier split into the flash/solid-state memory technology generation. Calpont also has a server/storage tier split, but that’s of mainly theoretical interest unless and until Calpont actually ships an MPP version of InfiniDB.

Cheaper parts, which have of course been a huge trend for decades. Solid-state memory will soon conquer the world. Meanwhile, cheaper sensors drive that machine-generated data I keep talking about.

An ever-better understanding of scale-out technology, in several respects, including:

• Query, notably data movement for MPP DBMS
• Update, especially minimalistic DBMS approaches, be they sharded MySQL or more NoSQLish
• Number-crunching, especially via MapReduce and/or parallel analytic libraries integrated into DBMS

Cool trends I touched on more briefly include:

• More data being available for analysis. This was a core theme of my Enzee Universe keynote speeches; there are also some notes on it in my post based on my Boston Big Data Summit talk.
• More users being served by analytics. Ditto.
• Data exploration/visualization, ala QlikView, Spotfire, or Tableau, and also the faceted stuff.
• The democratization of data mining. But I’m not as sure of that one as of the others…

One area I flat-out forgot to mention is easy data mart spin-out.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Flash, other solid-state memory, and disk
• Open issues in database and analytic technology

• Solid-state memory will soon be the right storage technology for a large fraction of databases, OLTP and analytic alike. I’m not sure whether the initial cutoff in database size is best thought of as terabytes or 10s of terabytes, but it’s in that range. And it will increase over time, for the usual cheaper-parts reasons.
• That doesn’t necessarily mean flash. PCM (Phase-Change Memory) is coming down the pike, with perhaps 100X the durability of flash, in terms of the total number of writes it can tolerate. On the other hand, PCM has issues in the face of heat. More futuristically, IBM is also high on magnetic racetrack memory. IBM likes the term storage-class memory to cover all this — which I find regrettable, since the acronym SCM is way overloaded already.
• Putting a disk controller in front of solid-state memory is really wasteful. It wreaks havoc on I/O rates.
• Generic PCIe interfaces don’t suffice either, in many analytic use cases. Their I/O is better, but still not good enough. (Doing better yet is where Petascan – the stealth-mode company I keep teasing about – comes in.)
• Disk will long be useful for very large databases. Kryder’s Law, about disk capacity, has at least as high an annual improvement as Moore’s Law shows for chip capacity, the disk rotation speed bottleneck notwithstanding. Disk will long be much cheaper than silicon for data storage. And cheaper silicon in sensors will lead to ever more machine-generated data that fills up a lot of disks.
• Disk will long be useful for archiving. Disk is the new tape.

*When the first three people to the question microphone include both Mike Stonebraker and Dave DeWitt, your thinking tends to clarify in a hurry.

Related links

• A slide deck by C. Mohan of IBM similar to the one he presented at the NEDB Summit about storage-class memories.
• A much more detailed IBM presentation on storage-class memories.

Other posts based on my January, 2010 New England Database Summit keynote address

• Data-based snooping — a huge threat to liberty that we’re all helping make worse
• Interesting trends in database and analytic technology
• Open issues in database and analytic technology

The first hard disks ever were introduced by IBM in 1956. They rotated 1,200 times per minute. Today’s state-of-the-art disk drives rotate 15,000 times per minute. That’s a 12.5-fold improvement since the first term of the Eisenhower Administration. (I understand that the reason for this slow improvement is aerodynamic — a disk that spins too fast literally flies off the spindle.)

Unfortunately, random seek time is bounded below, on average, by 1/2 of a disk’s rotation time. Hence disk seek times can never get below 2 milliseconds.

From that, much about modern analytic DBMS design follows.

• Database and analytic technology, as they evolve, will pose tremendous danger to individual liberties.
• We in the industry who are creating this problem also have a duty to help fix it.
• Technological solutions alone won’t suffice. Legal changes are needed.
• The core of the needed legal changes are tight restrictions on governmental use of data, because relying on restrictions about data acquisition and retention clearly won’t suffice.

But this time I don’t plan to be so quick to shut up.

My best writing about the subject of liberty to date is probably in a November, 2008 blog post. My best public speaking about the subject was undoubtedly last Thursday, early in my New England Database Summit keynote address; I got a lot of favorable feedback on that part from the academics and technologists in attendance.

My emphasis is on data-based snooping rather than censorship, for several reasons:

• My work and audience are mainly in the database and analytics sectors. Censorship is more a concern for security, networking, and internet-technology folks.
• After censorship, I think data-based snooping is the second-worst technological threat to liberty.
• In the US and other fairly free countries, data-based snooping may well be the #1 threat.

Below are the actual notes I spoke from. (Not typed out in the notes is the part about how I think there are a lot of precedents in US law for my suggestions, specifically in how the Fourth and Fifth Amendments are handled, but you can find that part in the blog linked above.) My call to action is still pretty vague, along the lines of “Think about this stuff, and talk about it too.” (And please don’t be as pig-headed as the Slashdotters mentioned here. ) Beyond that, simpler and more immediate – well, this is one blog post I’m REALLY hoping will get a rich comment thread. Please pitch in!

My notes, with minor edits for blog-posting clarity:

• Tremendous amounts of information are being or can be electronically recorded about everybody
  • All our credit card purchases
  • All our web access activity
    • For at least two different reasons – child porn and internet file sharing — governments around the world are finding reasons to mandate that this be stored …
    • … even as governments also sometimes try to mandate that it be deleted
  • All our email and other internet communication information, either in full content or at least from a traffic analysis standpoint
    • Many of our movements
    • Electronic tollbooths
    • Location-aware mobile devices
    • Police cameras
    • I doubt much of this is retained today except the camera part, but it could be
  • And by the way, it is not illegal in the US for the government to put cameras or other electronic sensors outside your house to peer in, which offers all sorts of other intrusive possibilities as that technology improves.
• Analysis tools are improving in steps
  • Data mining
  • Social graph analysis
• If misused, this technology is an enormous threat to liberty …
  • … and we’re all complicit in it (because we help advance the technology).
• So it’s our duty to also work to forestall the threat.
• Unfortunately, most technological solutions have no hope of working
  • The data WILL be retained.
  • The data WILL be possible to correlate and de-anonymize.
  • Any technology that purports otherwise is merely a stopgap at best
    • But that doesn’t meant such stopgap technology isn’t worth developing and deploying
• The only realistic long-term solution is legal, which is to say political
  • If regulating information gathering and retention is doomed to failure …
    • … regulate information use instead
      • Make it illegal to admit this kind of information in court
      • Make it illegal to use this kind of information to even track all but the most terrorist of criminal suspects
      • Certainly make it illegal to reveal much of this information to the private sector, or in many cases for the private sector to use it
  • Obviously, there have to be huge carve-outs for the ordinary business of marketing, government, medicine, whatever
  • But that’s the direction we need to go
  • And I’m convinced that we in the technology community need to show the way

More technical posts based on the same talk

• Flash, other solid-state memory, and disk
• Interesting trends in database and analytic technology
• Open issues in database and analytic technology

Related links (updates)

• CNet reports that the FBI is asking internet service providers in the US to keep website access data for 2 years
• CNN reports that US states are storing DNA for all babies born

That said, highlights of my Netezza Skimmer briefing included:

• In essence, Netezza Skimmer is 1/3 of Netezza’s previously smallest appliance, for 1/3 the price.
• I.e., Netezza Skimmer has 1 S-blade and 9 disks, vs. 3 S-blades and 24 disks on the Netezza TwinFin 3.
• With 1 disk reserved as a hot spare, that boils down to a 1:1:1 ratio among CPU cores, FPGA cores, and 1-terabyte disks on Netezza skimmer. The same could pretty much be said of Netezza TwinFin, the occasional hot-spare disk notwithstanding.
• Netezza Skimmer costs $125K.
• With 2.8 or so TB of space for user data before compression, that’s right in line with the Netezza price point of slightly <$20K/terabyte of user data.
• That assumes Netezza’s usual 2.25X compression. I forgot to ask when 4X compression was actually being shipped.
• I forgot to ask, but it seems obvious that Netezza Skimmer uses identical or substantially similar components to Netezza TwinFin’s.
• Netezza Skimmer is 7 rack units high.
• In place of the SMP hosts on TwinFin Systems, Netezza Skimmer has a host blade.
• Netezza (specifically Phil Francisco) mentioned that when Kalido uses Netezza Skimmer for its appliance, there will be an additional host computer, but when it uses TwinFin for the same software, the built-in host will suffice. (Even so, I suspect it might be too strong to say that Skimmer’s built-in host computer is underpowered.)
• Netezza also suggested that more appliance OEMs are coming down the pike specifically focused on the affordable Skimmer.

Obviously, Netezza Skimmer isn’t breaking any new technical ground. If Netezza had just called Skimmer “TwinFin 1,” nobody should have objected. So the main news here is that you can buy a Netezza box for $125K, plug it in, load a few terabytes of data, and be good to go with a pretty solid data warehouse.  For enterprises and data mart outsourcers with databases of the appropriate size, that could be a pretty attractive deal.

Is Netezza Skimmer as cheap as buying your own hardware and putting (free) Greenplum Single-Node Edition software on it? Not even close, especially since Greenplum’s free option limits you to lower overall compute power. Does Netezza Skimmer have as high availability as more expensive alternatives? In some cases, surely not. Skimmer is neither the cheapest thing around nor an utterly high-end product.

But Netezza Skimmer belongs on a lot of short lists even so.

• Oracle thinks flash memory is the most important hardware technology of the decade, one that could lead to Oracle being “bumped off” if they don’t get it right.
• Juan believes the “bulk” of Oracle’s business will move over to Exadata-like technology over the next 5-10 years. Numbers-wise, this seems to be based more on Exadata being a platform for consolidating an enterprise’s many Oracle databases than it is on Exadata running a few Especially Big Honking Database management tasks.

And by the way, Oracle doesn’t make its storage-tier software available to run on anything than Oracle-designed boxes.  At the moment, that means Exadata Versions 1 and 2. Since Exadata is by far Oracle’s best DBMS offering (at least in theory), that means Oracle’s best database offering only runs on specific Oracle-sold hardware platforms.

*E.g., I was sitting upstairs in my parents’ apartment in Columbus, OH having the call while their doctor, who I’ve never met, was visiting downstairs. He offered to make a special trip back Saturday afternoon because he missed me Wednesday, but he’s notorious for not coming when he says he will. Update: He didn’t come Saturday. On Saturday he said he’d come Sunday. He didn’t do that either.

Other high- and lowlights of our conversation included:

• Flash is the main new hardware element in Exadata Version 2. Otherwise, Exadata 2 is just an annual refresh of Exadata Version 1 to include updated components (Nehalem chips, bigger disk drives, etc.)
• Juan thinks it’s suboptimal to use flash memory through the bottleneck of disk controllers, favoring PCIe cards instead. (I emphatically agree.)
• Juan resolutely ducked questions about actual Exadata production deployment. Literally the only fact he shared in that regard is that there are at least 2 Exadata production systems running that each have 2 or more racks cabled together.
• Juan stressed that Exadata runs apps written over Oracle DBMS unchanged.
• When making mixed-workload claims for Exadata 2, Juan stressed consolidation of multiple databases, some OLTP and some analytic. He didn’t really argue with my skepticism about integrating OLTP and analytics in the same database, with one exception:
• Juan pointed out that in major OLTP apps such as ERP systems, there often is actually more processing going on in reporting and other batch stuff than there is in true OLTP.
• Exadata 2’s flash memory is designed as a disk cache, smarter than LRU (Least Recently Used). The two examples Juan gave of “smarter than LRU” are that backups and table scans don’t flush the cache.
• I forget whether this is new in Exadata 2 (I think it is), but anyhow – Exadata has a “Storage Index” that’s a lot like a Netezza zone map. I.e., for each megabyte or so of data it stores the min and max value of every column; if a query predicate rules out those ranges, that megabyte is never retrieved.
• Oracle has long offered what sounds like flexible workload management capability, and this has now been extended to specifically include I/O resources on the storage tier.
• This isn’t Exadata-specific, but Oracle has built a file system on top of its DBMS, optimized for speed, which helps with, e.g., ELT (Extract/Load/Transform). Evidently, it’s not at all the same thing as Mark Benioff’s 1990s Microsoft-annoying IFS (Internet File System) project, which seems to have morphed into a content management SDK.

Highlights specifically in the area of parallelization included:

• Juan stressed that all databases consolidated onto an Exadata machine are/should be striped across all storage units.
• On the other hand, Juan said that different databases should be confined to specific cores or CPUs on the database tier.
• But on the third hand, Juan also stressed – in what could be called a “private cloud” pitch – that there’s great elasticity as to which databases are matched to which server CPUs.
• Contrary to what I thought he and/or his colleagues told me a year ago, Juan said RAC (Real Application Clusters) is a big part of Oracle’s data warehouse processing.
• However, Juan says that what I regard(ed) as a major objection to Oracle’s database-tier parallelization — the need to manually specify “degrees of parallelism” — has now been obviated by automation. Juan thinks that few data warehouse DBAs will now need to manually tune parallelism, with minor exceptions. One exception he cites is that if a nightly report really is non-urgent, it can just be forced to run on a single core with no chance to grab more resources. (However, Juan thinks manual tuning of parallelism will continue to play a greater role in OLTP.)

OK. That’s all I can get done tonight (see above re: inconvenience of timing). Follow-on subjects I’d like to and indeed plan to post about include:

• What Juan said about hybrid columnar compression
• Oracle’s delightfully non-confidential slide deck, and a few comments about same

• Traditional data of the sort that’s stored in relational databases, aka “structured.”
• Everything else, aka “unstructured” or “semi-structured” or “complex.”

There are plenty of problems with these formulations, not the least of which is that the supposedly “unstructured” data is the kind that actually tends to have interesting internal structures. But of the many reasons why these distinctions don’t tend to work very well, I think the most important one is that:

Databases shouldn’t be divided into just two categories. Even as a rough-cut approximation, they should be divided into three, namely:

• Human/Tabular data –i.e., human-generated data that fits well into relational tables or arrays
• Human/Nontabular data — i.e., all other data generated by humans
• Machine-Generated data

Even that trichotomy is grossly oversimplified, for reasons such as:

• These categories overlap.
• There are kinds of data that get into fuzzy border zones.
• Not all data in each category has all the same properties.

But at least as a starting point, I think this basic categorization has some value.

By human-generated data that fits well into relational tables or arrays, what I really mean is: the input from most conventional kinds transactions – purchase/sale, inventory/manufacturing, employment status change, etc. This is the core data managed by OLTP relational DBMS everywhere. It is also the core data in analytic relational or MOLAP databases. The vast majority of what we think or know about “database management” applies primarily to data of this kind, in large part because of two fundamental properties of this information:

• It is meaningful to contemplate this data as being 100% accurate and complete (even if that goal is difficult to achieve in the real world).
• This data is precise – i.e., one can check predicates against it and (give or take regrettable data imperfections) get inarguable yes/no answers.

For most enterprises, this is the most important data they have. It was created as a result of expensive business activities. It deals directly with money, employees, physical goods, and the rest of the things that make an enterprise go. It can be fruitfully analyzed in ever more ways, which is why it should never be thrown out or even entirely relegated to tape, now that data warehouse software, hardware, and storage has become so cheap. (“Disk is the new tape.”) And because of the importance of both preserving and accessing it, it should often be stored in multiple copies – OLTP, data warehouse, data mart, in-memory analytics, near-line quasi-archive, MOLAP cubes (if you must) and so on, plus of course replicas for high throughput and availability.

But humans generate many other kinds of data as well, especially in a form directly suitable for communication – text (in many formats), documents (text or otherwise), pictures, videos, etc. Traditional relational databases are a poor home for this kind of data because:

• This data often deals with opinions or aesthetic judgments – there is little concept of perfect accuracy.
• Similarly, there is little concept of perfect completeness.
• There’s also little concept of perfectly, unarguably accurate query results – different people will have different opinions as to what comprises good results for a search.
• Queries don’t lend themselves to binary answers; rather, documents can have differing degrees of relevancy.

Systems for managing this kind of data are much less advanced than relational database managers. Nobody knows how to get all the information out of a text document, or query all of it if they could, and the story is even worse for non-text examples. The systems that give the best query results aren’t necessarily the same ones that have the best database administration features. Basically, this area is still a mess, and it’s a mess that consumes a huge fraction of all the data storage products sold today.

But give or take questions of storage efficiency and deduplication, if humans created that kind of data, they put a lot of effort into it, so it’s worth keeping. Besides, compliance regulations commonly mandate that we do so – except, perhaps, when they mandate that we throw it away.

Machine-generated data is a whole other can of worms. Paradigmatic examples of what I mean by “machine-generated data” include:

• Computer, network, and other equipment logs
• Satellite and similar telemetry (whether for espionage or science)
• Location data such as RFID chip readings, GPS system output, etc.
• Temperature and other environmental sensor readings
• Sensor readings from factories, pipelines, etc.
• Output from many kinds of medical device, in hospitals and (increasingly) homes alike

Unlike human-generated data, whose growth is constrained by macro factors such as population and total level of economic activity, machine-generated data will continue to grow as fast as Moore’s Law lets it. That fact has two profound consequences:

• It is unrealistic to hope ever to keep most or all machine-generated data, whereas I think that’s exactly what should and will happen with human-generated data
• Before long, most data (by volume) will be machine-generated

And so it is not really an exaggeration to say that machine-generated data is the future of data management.

I’d like to close this long post by immediately pointing out some of the flows in this simple trichotomy. One obvious gray area lies in hybrid human/machine-generated data, three big examples of which are:

• Web clickstreams
• Call detail records (CDR)
• Stock trades

In all three cases, we are quickly getting to the point where this data is preserved in its entirety (even if the network event data associated with the web logs is reduced before storage). And in each case it fits pretty well into RDBMS, although Hadoop has a role to play as well. So pretending it’s purely human-generated probably isn’t all that misleading.

Another gray area lies in text that gets linguistically processed – i.e. via text-mining tools – with the output placed into a relational database. I don’t immediately see a workaround for that flaw in my labeling scheme.  So let’s just say no taxonomy is perfect.*

*Come to think of it, that’s one of the problems holding back text-mining technology.

And of course some of the NoSQL folks would note that I was oversimplifying when I tied my first category specifically to relational DBMS. So would the folks at Intersystems.

But the biggest oversimplification stems from this:

As Mike Stonebraker* and I argued a couple of years ago, I really think that database management technologies should be divided into 10+ categories.

*Note: The links to Stonebraker’s own posts will be broken until Vertica’s webmaster gets his/her act together. But you can find them under other URLs via web search.)

]]> http://www.dbms2.com/2010/01/17/three-broad-categories-of-data/feed/ 6 http://www.dbms2.com/2010/01/17/three-broad-categories-of-data/