2. Numerous vendors are blending SQL and JSON management in their short-request DBMS. It will take some more work for me to have a strong opinion about the merits/demerits of various alternatives.

The default implementation — one example would be Clustrix’s — is to stick the JSON into something like a BLOB/CLOB field (Binary/Character Large Object), index on individual values, and treat those indexes just like any others for the purpose of SQL statements. Drawbacks include:

• You have to store or retrieve the JSON in whole documents at a time.
• If you are spectacularly careless, you could write JOINs with odd results.

IBM DB2 is one recent arrival to the JSON party. Unfortunately, I forgot to ask whether IBM’s JSON implementation was based on IBM DB2 pureXML when I had the chance, and IBM hasn’t gotten around to answering my followup query.

3. Nor has IBM gotten around to answering my followup queries on the subject of BLU, an interesting-sounding columnar option for DB2.

4. Numerous clients have asked me whether they should be active in DBaaS (DataBase as a Service). After all, Amazon, Google, Microsoft, Rackspace and salesforce.com are all in that business in some form, and other big companies have dipped toes in as well.

I’m skeptical that one can succeed both in that market and in selling database software, for reasons including:

• Nobody I can think of has done so.
• The value propositions are different.
  • DBaaS is about having administration be so easy that you the customer doesn’t need to worry about it.
  • Database software is about one or more of:
    • Development ease.
    • Price/performance/throughput.
    • Big-enterprise/legacy-vendor considerations.

I’m also skeptical about service-only DBaaS strategies, because users will naturally resist vendor lock-in.

But despite all my skepticism, DBaaS is an area I should probably learn more about.

5. I plan to spend more time looking at machine learning and other advanced analytics. I doubt they’ll soon match the past few years’ hype about “big data analytics”, but even the reality of modern analytics looks like it’s getting more interesting. Ditto if somebody has an interesting twist on more traditional predictive analytics.

6. Three years ago,  I wrote:

• It is inevitable* that governments and other constituencies will obtain huge amounts of information, which can be used to drastically restrict everybody’s privacy and freedom.
• To protect against this grave threat, multiple layers of defense are needed, technical and legal/regulatory/social/political alike.
• One particular layer is getting insufficient attention, namely restrictions upon the use (as opposed to the acquisition or retention) of data.

*And indeed in many ways even desirable

It is now frighteningly obvious that the US is becoming a high-surveillance society. The Boston Marathon bombing added three new elements to an already snowballing trend:

• A revelation that the FBI could track Tamerlan Tsarnaev’s communication content without any known warrant.
• A further revelation that the police know how to put on large paramilitary displays of force (and that the public generally approves).
• An increased belief that widespread video surveillance of public places is a Good Thing.

I need to write more about privacy.

Yes, I’m asking via a blog post, even thought the core problem is that people who want to see my posts via e-mail aren’t getting them. Please work with me on this anyway.

My two basic questions are:

• What should be the frequency of delivery? To date, it’s been nightly (at least in theory).
• What delivery technology should be used? To date, it’s been FeedBlitz.

1. The nightly scheduling has been an artifact of an RSS-to-email link that no longer seems stable. So I’m thinking of just manually pasting each post into a list email, in which case:

• Posts could be sent without delay.
• Every post would be delivered by separate mail. (As opposed to having only one post per night be mailed, while others just get linked to.)

It’s a bit more work for me, but probably nothing dire. Does lower latency sound good to everybody?

2. The main technical options seem to be:

• Free services oriented to discussion lists, such as Yahoo Groups, but set to announce-only. These have very basic functionality.
• Commercial services oriented to marketing email lists, such as Aweber or MailChimp. Does anybody have favorable or unfavorable experience with particular services? Most vendors surely use one or another, but it’s tough to guess which they’ve selected just based on their spam and pabulum informative communications, given the customizability those services provide.

Any thoughts would be most welcomed.

3. And while I’m at it — what I should I do for social/sharing buttons? Presumably, if I included buttons that made it easy for you to tweet links to my posts, submit them to Hacker News, etc., more of you would do so. Which specific options would you like to use?

• Twitter?
• LinkedIn?
• Google +?
• Facebook?
• Slashdot?
• Hacker News?
• dzone?
• Digg?

Anything else? I’d like to omit the more dubious possibilities, as offering everything could be a lot of clutter …

Amazon Redshift

Amazon did a deal with ParAccel that amounted to:

• Amazon got a very cheap license to a limited subset of ParAccel’s product …
• … so that it could launch a service called Amazon Redshift.
• Amazon also invested in ParAccel.

Some argue that this is great for ParAccel’s future prospects. I’m not convinced.

No doubt there are and will be Redshift users, evidently including Infor. But so far as I can tell, Redshift uses very standard SQL, so it doesn’t seed a ParAccel market in terms of developer habits. The administration/operation story is similar. So outside of general validation/bragging rights, Redshift is not a big deal for ParAccel.

OEMs and bragging rights

It’s not just Amazon and Infor; there’s also a MicroStrategy deal to OEM ParAccel — I think it’s the real ParAccel software in that case — for a particular service, MicroStrategy Wisdom. But unless I’m terribly mistaken, HP Vertica, Sybase IQ and even Infobright each have a lot more OEMs than ParAccel, just as they have a lot more customers than ParAccel overall.

This OEM success is a great validation for the idea of columnar analytic RDBMS in general, but I don’t see where it’s an advantage for ParAccel vs. the columnar leaders.

Concurrency

As I admitted in the comment thread to my first Actian/ParAccel post, I’m confused about what kind of concurrent usage ParAccel can really support. The data I have, e.g. in the link immediately above, is not conclusive. Googling suggests that VectorWise was at one user per core a couple of years ago, supportive of my hypothesis that it doesn’t have some big concurrency edge on ParAccel. But to repeat — I don’t really know.

DBMS acquisitions in the past

My history blog on DBMS acquisitions yielded more favorable examples than I was expecting. (Of course, I omitted a lot of small and boring failures.) And DBMS conglomerates are the rule more than the exception, with IBM, Sybase, Teradata and Oracle all adopting acquisition-aided multi-DBMS strategies, at least to some extent.

That said, Sybase is the main example of a vendor of a slow-growth DBMS (Adaptive Server Enterprise) doing well with a faster-growing one (Sybase IQ). Perhaps not coincidentally, Actian’s latest management team draws significantly on Sybase. So yes; ParAccel is now owned by a company run by guys who know something about selling columnar DBMS.

But the whole thing would be more convincing if Ingres had shown more life under Actian’s ownership, or indeed at any point in the past 20 years. My bottom line is that Actian was floundering badly in the DBMS market 1 1/2 years ago, and not a lot of favorable news has emerged in the interim — except, quite arguably, for the management changes and acquisitions themselves.

• ParAccel scales out.
• ParAccel has added analytic platform capabilities.
• I don’t know for sure, but I’d guess ParAccel has more mature management/plumbing capabilities as well.

One might conjecture that ParAccel is bad at highly concurrent, single-node use cases, and VectorWise is better at them — but at the link above, ParAccel bragged of supporting 5,000 concurrent connections. Besides, if one is just looking for a high-use reporting server, why not get Sybase IQ?? Anyhow, Actian hasn’t been investing enough in VectorWise to make it a major market player, and they’re unlikely to start now that they own ParAccel as well.

But I expect ParAccel to fail too. Reasons include:

• ParAccel’s small market share and traction.
• The disruption of any acquisition like this one.
• My general view of Actian as a company.

2 years after being acquired, Vertica — which conceptually has always been ParAccel’s closest competitor — has finally taken major hits on engineering staffing. Even so, I expect HP Vertica to reopen what was once a large technology and momentum gap vs. ParAccel.

My views on Actian start:

• Actian is attempting to build a database software conglomerate on the cheap, starting with Ingres, ParAccel, VectorWise, Pervasive (itself a small conglomerate) and Versant.
• Actian hasn’t accomplished much with Ingres, its original acquisition.
• Actian hasn’t accomplished much with VectorWise.
• Actian’s brief, embarrassing pivot away from database software was a joke. (The comments at that link also show VectorWise’s positioning as very different in September, 2011 than it is now.)
• I’ve had some very bad experiences with Actian management, although it seems to have largely turned over since then.
• I can’t identify the folks to make this work at the acquired pieces either (even though I think well of a few of them, e.g. Mike Hoskins and Rick Glick).

I.e., building a database conglomerate is hard, and Actian isn’t up to the challenge.

Actian has three main paths it can follow for synergy:

• Acquire a lot of pieces and flip the whole thing for more money to a foolish buyer. This strategy worked splendidly for Autonomy, and to some extent for Sybase as well. But it’s a longshot, and not necessarily a win for customers even if investors do well.
• Sell a bunch of disparate products through the same sales force. Tough to execute. And at best it raises sales coverage up to the level of that for the most successful product — and Actian doesn’t really have successful new products.
• Integrate the technologies. Blech. You don’t integrate DBMS with wildly different architectures, as Informix died trying in the 1990s.

I don’t see enough opportunity there for the whole thing to work out, with sales synergy being the best opportunity to prove me wrong.

Related links

• Doug Henschen and Derrick Harris offer quotes and numbers about the deal.
• VectorWise’s academic founders Peter Boncz and Marcin Zukowski seem to have left the company.

1. So far, the buzzword of the year is “real-time analytics”, generally with “operational” or “big data” included as well. I hear variants of that positioning from NewSQL vendors (e.g. MemSQL), NoSQL vendors (e.g. AeroSpike), BI stack vendors (e.g. Platfora), application-stack vendors (e.g. WibiData), log analysis vendors (led by Splunk), data management vendors (e.g. Cloudera), and of course the CEP industry.

Yeah, yeah, I know — not all the named companies are in exactly the right market category. But that’s hard to avoid.

Why this gold rush? On the demand side, there’s a real or imagined need for speed. On the supply side, I’d say:

• There are vast numbers of companies offering data-management-related technology. They need ways to differentiate.
• Doing analytics at short-request speeds is an obvious data-management-related challenge, and not yet comprehensively addressed.

2. More generally, most of the applications I hear about are analytic, or have a strong analytic aspect. The three biggest areas — and these overlap — are:

• Customer interaction
• Network and sensor monitoring
• Game and mobile application back-ends

Also arising fairly frequently are:

• Algorithmic trading
• Anti-fraud
• Risk measurement
• Law enforcement/national security
• Healthcare
• Stakeholder-facing analytics

I’m hearing less about quality, defect tracking, and equipment maintenance than I used to, but those application areas have anyway been ebbing and flowing for decades.

3. Much of customer interaction revolves around recommendation and personalization. In connection with that I’ll remind you:

• Multiple sources say that 5 millisecond response is a real need. Srini Srinivasan explained why in a January comment.
• The results of the recommendation and personalization can be delivered in many different ways — product recommendations, ads, special offers, email, snail mail, call center scripts and more. This is the paradigmatic example for my skepticism about complete analytic applications.

4. Networks and sensors emit the epitome of machine-generated data. Data sources include web logs, network logs (in the IT sense), telecommunication networks, other utilities (e.g. electric), vehicle fleets, and more. Application themes include:

• Human monitoring, via some kind of real-time business intelligence view. I hear about that a lot.
• Various kinds of automated response. (Security is an obvious example.)
• Integration with other kinds of application, data source, or use case.

As one example of the last point, Oliver Ratzesberger told me years ago that eBay had up-to-the-minute BI cubes integrating customer response and log data, for the purpose of quickly detecting technology problems. Acunu recently told me that similar applications are one of their sales focuses.

5. In another example, games and mobile applications can be a lot like websites in terms of the analytics that support them (all the more so if we’re talking about games with in-app purchases). Two special features come up repeatedly, however — leaderboards for games, and geospatial data sent by mobile devices.

6. Algorithmic trading is flashy because of the sums of money involved, and because of what is often hyper-low latency; I’ve even heard 50 microseconds, and that’s a slightly out of date figure for a sequence of several atomic operations. But otherwise it’s not one of the more interesting areas to me, for at least two reasons:

• It depends on a lot of latency-specific stuff, such as hand-crafted hardware.
• The participants are secretive — understandably so as they’re literally in a race with each other –and don’t reveal much.

Another reason I don’t study it much is that high-frequency trading could be devastated at any time by some simple regulatory changes.

7. I finally figured out one of the big drivers for better risk analysis. Banks need to keep capital lying around to cover a fraction of the risk they take on. If they can estimate the risk more precisely, and come up with a lower number, then they need to keep less capital. That’s a lot like finding large bags of money.

8. Anti-fraud applications arise in many industries, with many different kinds of data and latency requirement. For example:

• Insurers don’t want to pay bogus claims. They usually have weeks to think about that problem.
• Telcos don’t want to provision services for customers who will defraud them. They have to decide at call-center speed.
• Similarly, retailers don’t want to accept bogus returns.
• Stockbrokers don’t want rogue traders to defeat their controls. A lot of data and analysis go into that mission, as billions of dollars — literally — can be at stake.

9. And finally, the recent Boston Marathon bombing has brought law-enforcement/anti-terrorism applications to the fore. The Boston Globe criticized difficulties in information sharing, but the money quote is:

The FBI followed up by checking government databases and looking for things such as “derogatory telephone communications, possible use of online sites associated with the promotion of radical activity, associations with other persons of interest, travel history and plans, and education history,” according to FBI Supervisory Agent Jason J. Pack. “The FBI also interviewed Tamerlan Tsarnaev and family members. The FBI did not find any terrorism activity.”

Neither the telephone intercept nor the web-surfing tracking is a capability the government routinely admits, unless there was something like a wiretap order that I so far haven’t seen reported.

Related links

• Government surveillance is even more inevitable than when I wrote in 2010 that freedom can only be preserved by limiting government USES of data.
• Stakeholder-facing analytics isn’t much better understood than when I wrote about it in 2010.
• I wrote up a different list of analytic use cases back in 2006.
• The continued drop in high-frequency trading latency strengthens my 2009 contrast between the speed of a turtle and the speed of light; we’re now over a 3 * 10^10 difference between the speed of trading and the speed of generic planning, and many turtles walk well faster than 1 cm/sec.

MemSQL’s flagship reference is Zynga, across 100s of servers. Beyond that, the company claims (to quote a late draft of the press release):

Enterprises are already using distributed MemSQL in production for operational analytics, network security, real-time recommendations, and risk management.

All four of those use cases fit MemSQL’s positioning in “real-time analytics”. Besides Zynga, MemSQL cites penetration into traditional low-latency markets — financial services (various subsectors) and ad-tech.

Highlights of MemSQL’s new distributed architecture start:

• There are two kinds of MemSQL node — “aggregator” and “leaf”.
  • Aggregators are a kind of head node. You can have a bunch of them.
  • Leafs run full single-server MemSQL. You can have a bunch of them too.
• MemSQL has two query optimizers. One kind runs on the aggregator nodes, and thinks about the whole cluster. The other runs on the leafs, and only thinks about its own node.
• Much of the join and aggregation work is done on the aggregator nodes, but I didn’t pursue that issue in much detail.
• It is good policy — and supported — to replicate small dimension/reference tables across the cluster. These are replicated to aggregator and leaf nodes alike. (This tells us that some joins are indeed done on the leafs. )
• MemSQL replication can be synchronous or asynchronous. It can be used for high availability.

Also:

• MemSQL writes (whether primary or replicated) go to a buffer. The buffer size can be 0 or positive, in a tradeoff of durability vs. the likelihood of a disk I/O bottleneck.
• MemSQL has many virtual nodes on each physical (leaf) node. (This is pretty much an industry-standard best practice, as it helps with elasticity, recovery from node failure, and so on.)
• Compression is still a future feature.
• So is online schema change.
• Leaf nodes have cost-based optimizers.
• MemSQL’s aggregator (cluster-wide) optimizer is mainly heuristic, but is supposed to get more cost-based in future releases.
• In some releases it will be possible to keep MemSQL running while upgrading the software. But that’s not a promise for releases that change how replication works.

And which not-easily-parallelized aggregate did MemSQL implement first? The same one Platfora did — COUNT DISTINCT.

Tokutek’s news is that they’re open sourcing much of TokuDB, but holding back hot backup for their paid version. I approve of this strategy — “doesn’t lose data” is an important feature, and well worth paying for.

I kid, I kid. Any system has at least a bad way to do backups — e.g. one that involves slowing performance, or perhaps even requires taking applications offline altogether. So the real points of good backup technology are:

• To keep performance steady.
• To make the whole thing as easy to manage as possible.

GenieDB is announcing a Version 2, which is basically a performance release. So in lieu of pretending to have much article-worthy news, GenieDB is taking the opportunity to remind folks of its core marketing messages, with catchphrases such as “multi-regional self-healing MySQL”. Good choice; indeed, I wish more vendors would adopt that marketing tactic.

Along the way, I did learn a bit more about GenieDB. In particular:

• GenieDB is now just backed by a hacked version of InnoDB (no more Berkeley DB Java Edition).
• Why hacked? Because GenieDB appends a Lamport timestamp to every row, which somehow leads to a need to modify how indexes and caching work.
• Benefits of the chamge include performance and simpler (for the vendor) development.
• An arguable disadvantage of the switch is that GenieDB no longer can use Berkeley DB’s key-value interface — but MySQL now has one of those too.

I also picked up some GenieDB company stats I didn’t know before — 9 employees and 2 paying customers.

Related links

• Recent posts about TokuDB and GenieDB

• Teradata tends to get 35-55% (roughly speaking) annual performance improvements, as measured by its internal blended measure Tperf. A big part of this is exploiting new-generation Intel processors.
• This year the figure is around 40%.
• The 6700 is based on Intel’s Sandy Bridge.
• Teradata previously told me that Ivy Bridge — the next one after Sandy Bridge — could offer a performance “discontinuity”. So, while this is just a guess, I expect that next year’s Teradata performance improvement will beat this year’s.
• Teradata has now largely switched over to InfiniBand.

Teradata is also talking about data integration and best-of-breed systems, with buzzwords such as:

• Teradata Unified Data Architecture.
• Fabric-based computing, even though this isn’t really about storage.
• Teradata SQL-H.

The upshot is that Teradata has at least 6 kinds of rack or cabinet it wants to sell you — along with software to connect them — of which it really thinks you should get at least 3:

• The 4 main Teradata-software appliances:
  • Active Enterprise Data Warehouse (the new 6700). Teradata thinks every sufficiently large enterprise should have one of these.
  • Extreme Performance Appliance (Teradata 4xxx), based on solid-state drives (which are also used in the 6xxx systems). At least I think so; the 4xxx wasn’t in the most recent slide deck I saw.
  • Data Warehouse Appliance (Teradata 2700).
  • Extreme Data Appliance (Teradata 1650).
• The Teradata Aster Big Analytics Appliance, running Aster and Hadoop software. Teradata basically thinks everybody should have one of these too.
• A separate cabinet for special-purpose “Teradata Managed Servers”. While there’s some space for Managed Servers in other Teradata appliances, Teradata now offers so many such capabilities that it thinks you will likely need a separate rack for those as well. These include (partial list):
  • Viewpoint system management.
  • Backup.
  • Teradata Unity.
  • Data movement, which is not the same thing as Teradata Unity.
  • Data loading, which is yet something else.
  • Generic compute (notably, to run SAS).

Even that doesn’t exhaust the possibilities:

• The 36 InfiniBand ports Teradata can fit into a cabinet aren’t enough, it suggests and presumably will sell you free-standing Mellanox switches as an alternative.
• That slide deck split the Big Analytics Appliance back out into Aster and Hadoop options.
• There also seems to be a SAS-specific modeling appliance.

And you can have — or in some cases must have — Teradata Managed Server nodes in other kinds of Teradata appliance.

Finally, Teradata also offers a stand-alone single- or several-node Teradata 670 Data Mart Appliance, notes on which include:

• The Teradata 670′s entry price is under $1/2 million, if you want to use it as your first Teradata system (something that evidently is happening, mainly outside the Americas).
• Another use for the Teradata 670 is for physical — as opposed to virtual — data mart spin-out.
• The primary use for the Teradata Data Mart Appliance, however, seems to be test/development for larger Teradata systems.
• The Teradata Data Mart Appliance is one of the options for placing in a separate managed-server Teradata rack.

Related links

• My recent musings on the variety of clusters and appliances an enterprise could have.
• A March, 2012 post on various vendors’ admissions that multiple analytic database systems are needed.

*Speaking of confusion — Teradata Aster seems to use the spellings SQL/MR and SQL-MR interchangeably.

… except that now there’s also a SQL-H for regular Teradata systems as well. While it has the same general features and benefits as SQL-H for Teradata Aster, the details are different, since the underlying systems are.

I hope that’s clear.

• DeepDB’s indexes can help you with analytic queries; hence, DeepDB is marketed as supporting OLTP (OnLine Transaction Processing) and analytics in the same system.
• DeepDB is marketed as “designed for big data and the cloud”, with reference to “Volume, Velocity, and Variety”. What I could discern in support of that is mainly:
  • DeepDB has been tested at up to 3 terabytes at customer sites and up to 1 billion rows internally.
  • Like most other NewSQL and NoSQL DBMS, DeepDB is append-only, and hence could be said to “stream” data to disk.
  • DeepDB’s indexes could at some point in the future be made to work well with non-tabular data.*
  • The Deep guys have plans and designs for scale-out — transparent sharding and so on.

*For reasons that do not seem closely related to product reality, DeepDB is marketed as if it supports “unstructured” data today.

Other NewSQL DBMS seem “designed for big data and the cloud” to at least the same extent DeepDB is. However, if we’re interpreting “big data” to include multi-structured data support — well, only half or so of the NewSQL products and companies I know of share Deep’s interest in branching out. In particular:

• Akiban definitely does. (Note: Stay tuned for some next-steps company news about Akiban.)
• Tokutek has planted a small stake there too.
• Key-value-store-backed NuoDB and GenieDB probably leans that way. (And SanDisk evidently shut down Schooner’s RDBMS while keeping its key-value store.)
• VoltDB, Clustrix, ScaleDB and MemSQL seem more strictly tabular, except insofar as text search is a requirement for everybody. (Edit: Oops; I forgot about Clustrix’s approach to JSON support.)

Edit: MySQL has some sort of an optional NoSQL interface, and hence so presumably do MySQL-compatible TokuDB, GenieDB, Clustrix, and MemSQL.

Also, some of those products do not today have the transparent scale-out that Deep plans to offer in the future.

Among the 10 people listed as part of Deep Information Sciences’ team, I noticed 2 who arguably had DBMS industry experience, in that they worked at virtualization vendor Virtual Iron, and stayed on for a while after Virtual Iron was bought by Oracle. One of them, Chief Scientist & Architect Tom Hazel, also was at Akiban for a few months, where he did actually work on a DBMS. Other Deep Information Sciences notes include:

• Deep has 25 or so people in all.
• Deep had a recent $10 million funding round.
• Deep Information Sciences is the former Cloudtree, which as of February, 2011 was pursuing quite a different strategy. (Evidently there was a pivot.) Deep was founded in 2010.
• There are 2 paying customers for DeepDB, even though it’s still in beta, and 8 trials. A similar number of trials and strategic partners are queued up.
• DeepDB general availability is expected later this quarter.

Although our call was blessedly technical, we didn’t have a chance to go through the DeepDB architecture in great detail. That said, DeepDB seems to store data in all of 3 ways:

• An in-memory row store.
• An on-disk row store with a very different architecture.
• Indexes, which can also serve as a column store.

Notes on that include:

• DeepDB’s in-memory row store is designed to manage single rows as much as possible, rather than pages. Indeed, there are “aspects of tries”, although we didn’t drill down into what exactly that meant.
• Indexes are streamed to disk no less than once every 15 seconds, by default, and perhaps with latency as low as 10 milliseconds.
• Perhaps the most important point I didn’t grasp is “segments”. The data and indexes on disk are stored in segments, which can be of different sizes, and which may each carry some summary data/metadata/whatever. Somehow, this is central to DeepDB’s design.
• In what is evidently a design focus, DeepDB tries to get the benefit of “in-memory data” that isn’t actually taking up RAM. B-trees can point at rows that aren’t actually in memory. Segments evicted from cache can leave some metadata or summary data behind.
• DeepDB’s compression story seems to be a work in progress.
  • There’s prefix compression already, at least in the indexes, which Deep just calls “compaction”.
  • Other compression is working in the lab, but not scheduled for Version 1.0.
    • Block compression seems to be in play.
    • Delta compression was mentioned once
    • Dictionary compression wasn’t mentioned at all.
  • DeepDB apparently will keep compressed data in cache, then decompress it to operate on it.
  • Different segments can be compressed/uncompressed differently.
• DeepDB’s on-disk row store is append-only. Time-travel is being worked on. While I forgot to ask, it seems likely that DeepDB has MVCC (Multi-Version Concurrency Control).

And finally: DeepDB in its current form is a “drop-in” InnoDB replacement, but not necessarily bug-compatible.