• SAS faces a number of challenges, not unlike those faced by other high-priced legacy technology vendors.
  • It is used by organizations who have large budgets to pay for the product and to pay people to be expert on the product’s intricacies.
  • SAS has not integrated with scale-out analytic DBMS technologies as well or quickly as had been hoped, or as earlier marketing suggested was likely.
  • SAS has not been strong in helping its users do agile predictive analytics.
• SAS’ strengths are concentrated in product breadth:
  • Lots of statistical algorithms.
  • Various vertical products that make the modeling techniques more accessible in specific application domains.
  • Various approaches to engineering for scalability — no one of those has been a table-thumping success to date, but SAS has the resources to keep trying.
  • Some level of integration with its own business intelligence and text analytics products.
• For any particular use case, the burden of proof is on SAS alternatives to show that they have enough pieces in the toolkit to meet the needs.
  • SPSS (now owned by IBM) also has legacy issues.
  • KXEN is focused on marketing use cases.
  • Mahout has been one of the less successful Hadoop-related open source projects.
  • R-based technology is still maturing.
  • The modeling capabilities (as opposed to just scoring) bundled into RDBMS and well-parallelized tend to be pretty limited. Apparent exceptions tend to just be R repackaged.

• In general, I regard Gartner Magic Quadrants as a bad use of good research.
• Illustrating the uselessness of — or at least poor execution on — the overall quadrant metaphor, a large majority of the vendors covered are lined up near the line x = y, each outpacing the one below in both of the quadrant’s dimensions.
• I find fewer specifics to disagree with in this Gartner Magic Quadrant than in previous year’s versions. Two factors jump to mind as possible reasons:
  • This year’s Gartner Magic Quadrant for Data Warehouse Database Management Systems is somewhat less ambitious than others; while it gives as much company detail as its predecessors, it doesn’t add as much discussion of overall trends. So there’s less to (potentially) disagree with.
  • Merv Adrian is now at Gartner.
• Whatever the problems may be with Gartner’s approach, the whole thing comes out better than do Forrester’s failed imitations.

*At the time of this posting, I don’t yet have a link. However, I expect that to change quickly, and I plan to edit this paragraph accordingly. If nothing else, I hope people will drop links into the comment thread.

Specific company comments, roughly in line with Gartner’s rough single-dimensional rank ordering, include:

• The Gartner Magic Quadrant’s comments on Teradata seem pretty fair. I don’t think I’m much in disagreement when I say:
  • Teradata has the richest, most mature analytic DBMS offering.
  • Teradata has an outstanding track record both for managing large data volumes and for high-concurrency mixed workloads.
  • Aster Data was a cool Teradata acquisition, even if Teradata/Aster synergies or integration have been nominal to date.
  • Teradata still needs to get out of its own way in marketing, positioning, packaging, and/or defining its premium-priced system vs. its more moderately-priced alternatives. Indeed, as necessary as this approach may have been to fending off encroachments by Netezza and others, what Teradata really needs to do is evolve to a more pick-your-own-node-combination mix-match kind of offering.
• Gartner has talked with a lot of Oracle Exadata users who say that the product works; Gartner has also stopped beating Oracle up for its previous policy of almost never doing onsite POCs (Proofs of Concept); both parts of that ring true with me. But Gartner also rightly dings Oracle for various issues in cost and cumbersomeness. Overall, while I agree there are organizations for which Oracle should indeed be a top-ranked choice, there are many others who shouldn’t put Oracle on their short list.
• Third in the Gartner MQ rankings is IBM.
  • Gartner gets so caught up in reciting the names of various IBM product offerings that it neglects to say much good about DB2 itself. (I tend to have a similar problem.)
  • But Gartner does mention concurrency as a strength. I agree, especially if we presume that that was a reference to DB2 rather than Netezza.
  • Gartner cites Netezza’s post-acquisition annual growth rate as 30%. Gartner seems to think this is a good number. I disagree, but in Netezza’s defense, it has had to endure IBM’s post-acquisition on-boarding process.
• Arguably fourth in the Gartner Data Warehouse Magic Quadrant rankings is EMC/Greenplum.
  • In general, Gartner likes the taste of Greenplum Kool-Aid.
  • Gartner neglects to ding Greenplum for concurrency challenges, which I view as an oversight given Gartner’s general stress on that area.
  • Gartner does ding Greenplum for support challenges.
  • Gartner neglects to praise Greenplum for true hybrid row/columnar data management, a feature shared by Teradata and Vertica, among others, but not by Oracle, DB2, or Netezza.
  • Gartner located a half-petabyte Greenplum database. This doesn’t surprise me, even though Greenplum has frequently made exaggerated claims about large-size database successes in the past.
  • Gartner reports a >400 figure for Greenplum customers, which is plausible.
• In its first deviation from strict one-dimensional rank ordering, the Gartner Magic Quadrant ranks Sybase ahead of Greenplum in completeness of vision but behind in “ability to execute”.
  • If that were the other way around, it might make more sense. Greenplum promises anything and everything you might ever want for analytic data management or the associated analysis; but Sybase has vastly more analytic DBMS users than Greenplum does, running a variety of demanding workloads.
  • Gartner appears to think that Sybase IQ requires less database administration than I do.
  • Gartner seems concerned that SAP will position HANA and Sybase ASE as, between them, the only DBMS you’ll ever need, casting doubt on Sybase IQ’s future. I wouldn’t worry about that if you have a problem you want to solve today.
• The Gartner Magic Quadrant for Data Warehouse Database Management Systems ranks Microsoft sixth overall, despite noting that there isn’t a single production reference for Microsoft’s Parallel Data Warehouse. In support of this ranking, it for example cites the compression feature, which distinguishes Microsoft SQL Server from no other product on the list except Kognitio. If you have such an undemanding data warehousing problem that many different analytic DBMS could meet your needs, there’s a good chance Microsoft SQL Server can also do the job; and if you’ve bought into the Microsoft technology stack, you might as well keep going down that path. Otherwise, I don’t know why somebody should adopt Microsoft’s offering at this time.
• Seventh along the main diagonal path in the Gartner Magic Quadrant is HP Vertica. I’d rank Vertica higher than that, but in fairness I note two execution concerns. First, HP has a lousy track record, both in acquisitions and in data warehousing/analytics. Second, Vertica is bad about answering my email. Anyhow, Gartner doesn’t seem to have given Vertica credit either for its full customer count or for the multiple petabyte-scale databases Vertica runs.
• 1010data is an outlier, with Gartner noting that it only partly fits in with other “Data Warehousing Database Management” companies, and hence kind of confessing that 1010data on the Magic Quadrant is somewhat arbitrary. Stuff like that is bound to happen, given the inherent difficulties of defining market categories. Anyhow, my thoughts on 1010data include:
  • I’m nervous about the fact that 1010data doesn’t actually control its own DBMS technology, but rather relies on old code from the small private company KX Systems.
  • There are three main reasons to consider 1010data:
    • You want to enter the data mart outsourcing business in a casual way, and you like its SaaS offering.
    • You want to engage in stakeholder-facing analytics in a casual way, and you like its SaaS offering.
    • You love 1010data’s particular set of interactive analytic features and performance.
• Back to the main path winding along the Gartner Magic Quadrant main diagonal — next up is ParAccel. While I question some of the peripheral comments, I agree with Gartner’s core messages that:
  • ParAccel, the product, is blazingly fast in certain use cases.
  • ParAccel, the company, is dangerously small.
• Eighth on the Gartner MQ’s main path is Kognitio. This is too high. Kognitio positions itself as offering in-memory DBMS, yet stubbornly refuses to do any kind of data compression. That’s an awful combination of choices. As for using Kognitio’s data warehousing SaaS offering — why would you do that, when more modern products are available on a SaaS/cloud basis as well?
• Ninth in the Gartner Magic Quadrant main rankings is SAND.
  • The SAND section is not a triumph of Gartner accuracy. For example:
    • Gartner completely missed the errors in SAND’s reported customer counts.
    • Gartner refers to SAND as being “in existence for approximately nine years”, which is too low by at least a factor of 2.
    • Gartner says “SAND is a privately held company”, even though Merv knows better than that.
  • Otherwise, Gartner’s opinion on SAND seems to boil down to “Interesting technology and ideas, but dangerously small company.” I agree.
• Tenth and too low in the Gartner MQ main rankings is Infobright.
  • At least by some metrics (e.g. customer count), Infobright isn’t as dangerously small as ParAccel, SAND, Kognitio, et al.
  • That said, Infobright is small and focused on machine-generated data. So I wouldn’t be confident in Infobright’s future technology path for human-generated data use cases.
  • Infobright’s performance is uneven — blazing in cases where the Knowledge Grid helps, but not necessarily stellar by analytic DBMS standards when full table scans are called for.
  • I agree with Gartner that the possibility of Oracle/MySQL future shenanigans is a concern. But while the energy behind MySQL forking efforts doesn’t seem too great right now, I’d expect them to revive and offer a successful escape path if it seemed Oracle was going to indeed play hardball.
  • Also, given that it’s already an open source vendor, there are various kinds of assurances Infobright could give that would also help alleviate customer concerns.
• Actian, formerly Ingres, took a big tumble in Gartner’s rankings versus last year, when I simply wrote “What Gartner said in connection with Ingres is too inaccurate to deserve detailed attention.” I’m even a little harsher about Ingres/Actian’s DBMS products and prospects than Gartner is, but at least now we’re in the same ballpark.
• Along with Infobright, ParAccel, and SAND, Exasol appears to be another of the “good columnar technology/small company” crowd. As with other such products, one should be careful about fit-and-finish features that are missing today, as there is no assurance they’ll be added in a timely manner going forward.
• illuminate Solutions, which was on last year’s Gartner list, now appears to be an ex-company.

• “Hadoop applications” is a catch-all category. Since Derrick offered suitable caveats around the label, I’m fine with what he said.
• Hadoop management software commonly comes in the form of suites. Derrick’s discussion was solid.
• Derrick seems to want to define “Hadoop” as being whatever is in the relevant Apache projects. Cool. He does seem to wind up on both sides of the “MapR and DataStax put Hadoop MapReduce on top of something that isn’t HDFS — so is that Hadoop or isn’t it?” question, but that’s a tough ambiguity to avoid.
• Derrick could have been a little clearer on the subject of Hadoop distributions.

Let’s drill down into that last one. Derrick refers to Hadoop distributions as “products” that:

package a set of Hadoop projects (MapReduce, Hive, Sqoop, Pig, etc.) in a way that in theory makes them integrate more naturally, and to run both smoothly and securely.

While that’s a reasonable recitation of the idea’s benefits, I’d rather say that a “distribution” of open source software comprises:

• Open source software, in selected versions.
• (Possibly) additional code.
• (Likely) documentation.
• (Possibly) legal assurances such as intellectual property indemnification.

In the case of Hadoop:

• The version selection is a relatively big deal. There are a lot of Hadoop sub-projects. There’s been some splitting and forking and recombination. Testing a specific set of  point releases for integration and bugs is a non-trivial user benefit.
• The additional code is generally focused on installation or whatever, because the rest is bundled into separately identified management software. Even so, because of the large number of moving parts, this is a good thing to have.
• What’s more, in the case of Cloudera, using a particular distribution (theirs) is a prerequisite to getting the most widely adopted Hadoop management software (also theirs), which in turn is required if you want the industry’s most widely adopted Hadoop support (ditto). Similar things are apt to be true of rival distributions.

Interesting quotes include:

WibiData is designed to store … transactional data side-by-side with profile and other derived data attributes.

… the ability to add new ad-hoc columns to a table enables more flexible analysis: output data that is the result of one analytic pipeline is stored adjacent to its input data, meaning that you can easily use this as input to second- or third-order derived data as well.

schemas can vary over time; you can easily add a field to a record, or delete a field. … But even though you start collecting that new data, your existing analysis pipelines can treat records like they always did; programs that don’t yet know about the new cookie are still compatible with both the old records already collected, and the new records with the additional field. New programs fill in default values for old data recorded before a field was added, applying the new schema at read time.

schemas for every column are stored in a data dictionary that matches column names with their schemas, as well as human-readable descriptions of the data.

Interesting aspects of the post that don’t lend themselves as well to being excerpted include:

• How the Produce-Gather “analysis calculus” — i.e. framework — works.
• How this all ties into Apache projects (and sub-projects) such as Hadoop, HBase, and Avro.

• Sumo Logic does SaaS (Software as a Service) log management.
• Sumo Logic is text indexing/Lucene-based. Thus, it is reasonable to think of Sumo Logic as “Splunk-like”. (However, Sumo Logic seems to have a stricter security/trouble-shooting orientation than Splunk, which is trying to branch out.)
• Sumo Logic has hacked Lucene for faster indexing, and says 10-30 second latencies are typical.
• Sumo Logic’s main differentiation is automated classification of events.
• There’s some kind of streaming engine in the mix, to update counters and drive alerts.
• Sumo Logic has around 30 “customers,” free (mainly) or paying (around 5) as the case may be.
• A truly typical Sumo Logic customer has single to low double digits of gigabytes of log data per day. However, Sumo Logic seems highly confident in its ability to handle a terabyte per customer per day, give or take a factor of 2.
• When I asked about the implications of shipping that much data to a remote data center, Sumo Logic observed that log data compresses really well.
• Sumo Logic recently raised a bunch of venture capital.
• Sumo Logic’s founders are out of ArcSight, a log management company HP paid a bunch of money for.
• Sumo Logic coined a marketing term “LogReduce”, but it has nothing to do with “MapReduce”. Sumo Logic seems to find this amusing.

What interests me about Sumo Logic is that automated classification story. I thought I heard Sumo Logic say:

• It’s largely unsupervised machine learning.
• It’s specific to a particular user/data set.
• It can be up and running and classifying things effectively almost instantly (i.e., on seconds’ or minutes’ worth of data).
• It’s informed by what different users tag as false positives. (Or maybe that is planned for future versions.)

I have a little trouble seeing how all those points fit exactly together, so perhaps I got some details wrong.

The payoff is that machine learning directly informs the Sumo Logic user interface. In particular, large numbers of events are bundled into a small number of categories, hopefully making it much easier for network operations types to scan the UI and pick out what’s important.

In general, the idea of machine-learning informing analytic UIs via some sort of classification is common in text-oriented technologies, notably in:

• Good ol’ text search.
• Text mining vendors’ approaches to clustering hits on words or phrases that say substantially the same thing.

But otherwise it seems kind of rare, if we stipulate that ad-serving/general internet personalization isn’t really an analytic UI — but I’d love to hear of any interesting examples I’ve overlooked.

• The Forrester Wave’s relative vendor rankings are meaningless, in that the document compares apples, peaches, almonds, and peanuts. Apparently, it covers any vendor that includes a distribution of Apache Hadoop MapReduce into something it offers, and that offered at least two (not necessarily full production) references for same.
• The Forrester Wave for “enterprise Hadoop” contradicts itself on the subject of Hortonworks.
  • The Forrester Wave for “enterprise Hadoop” is correct when it says “Hortonworks … has Hadoop training and professional services offerings that are still embryonic.”
  • Peculiarly, the Forrester Wave for “enterprise Hadoop” also says “Hortonworks offers an impressive Hadoop professional services portfolio”. Hortonworks will likely win one or more nice partnership deals with vendors in adjacent fields, but even so its professional services capabilities are … well, a good word might be “embryonic”.
• Forrester Waves always seem to have weird implicit definitions of “data warehousing”. This one is no exception.
• Forrester gave top marks in “Functionality” to 11 of 13 “enterprise Hadoop” vendors. This seems odd.
• I don’t know why MapR, which doesn’t like HDFS (Hadoop Distributed File System), got top marks in “Subproject integration”.
• Forrester gave top marks in “Storage” to Datameer. It also gave higher marks to MapR than to EMC Greenplum, even though EMC Greenplum’s technology is a superset of MapR’s. Very strange. (Edit: Actually, as per a comment below, there is some uncertainty about the EMC/MapR relationship.)
• Forrester gave higher marks in “Acceleration and optimization” to Hortonworks than to Cloudera and IBM, and higher marks yet to Pentaho. Very odd.
• I’m not sure what Forrester is calling a “Distributed EDW file store connector”, but it sounds like something that Cloudera has provided via partnership to a number of analytic DBMS vendors.
• Forrester’s “Strategy” rankings seem to correlate to a metric of “We’re a large enough vendor to go in N directions at once”, for various values of N.
• Forrester is correct to rank Cloudera’s “Adoption” as being stronger than EMC/Greenplum’s or MapR’s. But Hortonworks’ strong mark for “Adoption” baffles me.

• Give or take minor tweaks, what I wrote in my August, 2010 Couchbase updates still applies.
• Couchbase now and for the foreseeable future has one product line, called Couchbase.
• Couchbase 2.0, the first version of Couchbase (the product) to use CouchDB for persistence, has slipped …
• … because more parts of CouchDB had to be rewritten for performance than Couchbase (the company) had hoped.
• Think mid-year or so for the release of Couchbase 2.0, hopefully sooner.
• In connection with the need to rewrite parts of CouchDB, Couchbase has:
  • Gotten out of the single-server CouchDB business.
  • Donated its proprietary single-sever CouchDB intellectual property to the Apache Foundation.
• The 150ish new customers in 2011 Couchbase brags about are real, subscription customers.
• Couchbase has 60ish people, headed to >100 over the next few months.

If you previously heard the brand names Couchbase Single or Couchbase Mobile, pay no further attention to them. Couchbase Single was CouchDB; Couchbase Mobile is part of Couchbase’s feature set.

The current product is Couchbase 1.8, which is a whole lot like what previously was called Membase. New features in Couchbase 1.8 (versus prior versions of Membase) were concentrated in client libraries/SDK (Software Development Kit). Not coincidentally, Couchbase has hired developer evangelists who are in charge of making Couchbase play nicely with various specific languages (e.g. C/C++)

Drilling down further into the CouchDB part of the story:

• Couchbase 2.0 will replace Couchbase 1.8/Membase’s SQLite back-end with CouchDB.
• Parts of CouchDB that do things like read, write, or compact data have been rewritten from Erlang to C.
• Couchbase still uses other Erlang parts of Apache CouchDB, and would be delighted if the community were to usefully enhance them.
• Couchbase’s heavy contributions to development of open source CouchDB will, for the most part, continue.
• CouchDB stuff donated to the Apache Foundation includes:
  • Documentation
  • Packaging
  • Performance enhancements

There’s at least one Couchbase user with >1000 nodes (at a guess, Zynga).  More typical might be 20 nodes or less. This led me to wonder how much data one puts on a Couchbase node anyway. The answer turns out to vary widely, in that you want your working set to be in RAM, and whether that’s your entire database or just a slice of it depends on the nature of the application.

James echoed a trend I’ve heard elsewhere as well, in which products one things of as being internet-specific are also sold in a few cases to conventional enterprises for — you guessed it! — their internet operations. I also asked him about competition, and he asserted:

• MongoDB is the big competition. He believes Couchbase has an excellent win rate vs. 10gen for actual paying accounts.
• DataStax/Cassandra wins over Couchbase only when multi-data-center capability is important. Naturally, multi-data-center capability is planned for Couchbase. (Indeed, that’s one of the benefits of swapping in CouchDB at the back end.)
• Redis has “dropped off the radar”, presumably because there’s no particular persistence strategy for it.
• Riak doesn’t show up much.

Three things that absolutely should NOT be obstacles to these ends are:

• Corporate DBMS standards.
• Corporate data governance processes.
• The difficulties of ETL.

Reasons they shouldn’t or don’t need to be obstacles include:

• Analytic DBMS are often vastly more cost-effective than general-purpose ones.
• In particular, analytic DBMS are often much easier to install and manage than general-purpose ones.
• Heavy data governance bureaucracy is often unnecessary because:
  • The department should know what the limitations on the data’s accuracy are.
  • The department should know how much data accuracy is required.
  • The side-effects on other departments of any data inaccuracy would be minimal.
• There are multiple good schemes for populating data marts, managed by cost-effective analytic DBMS, with data from integrated data warehouses.
  • ELT (Extract/Load/Transform) almost always works, because data cleaning/data quality was handled at or before the IDW level, and because the analytic DBMS has the processing power to pull it off.
  • ETL (Extract/Transform/Load) should be easy as well. (If isn’t, something may be lacking in your ETL set-up.)
  • Analytic DBMS are increasingly adding capabilities for easy spin-out of real or virtual data marts. Other kinds of technology (e.g. virtualization) are having their database spin-out capabilities upgraded as well.

One point to remember in support of departmental autonomy is that departments’ views of what data to use may be more expansive than central IT’s. One reason is that important data may be external to the company, outside IT’s natural realm  of concern. Examples of this include but are hardly limited to:

• Anything like “market data”.
• Anything like “sentiment analysis”.
• Data owned by supply chain partners.

Further, even the more innovative internal data sources are commonly departmental, for example various kinds of multi-structured data (text verbatims from customers, log file data, and so on).

Whatever is true of data management (and ETL) is true for metadata management, even if it’s done by some kind of business intelligence tool. What I mean by that is:

• Whoever manages data is also responsible for ingesting and emitting it …
• … and specifically for emitting it in understandable, well-organized, well-named formats, …
• … so that departments can take responsibility for what amounts to lightweight analytic application development.

As for the “application development” itself, I’m envisioning at least three things:

• Math.
• Sophisticated relational query.
• Data visualization.

I.e., I’m talking about what “analysts” and “quants” do. So to put the point even more simply:

• Analysts and quants should be able to consume data that’s organized in a friendly manner.
• Central IT should be friendly in how it serves data.

One corollary of this approach is that departments should try to adhere to corporate BI standards, at least for routine dashboard and reporting. Indeed, if a department brings in a business intelligence tool different from the corporate standard, there are three main possibilities:

• The tool is integrated with something else it makes sense to bring in, such as a third-party data supply or application.
• The tool has an important capability the corporate standard doesn’t have, such as more flexible visualization and drilldown.
• Central IT screwed up, making things much more difficult than they needed to be.

Reporter: [Care to comment]?
CAM: SQL Server is an adequate product if you don’t mind being locked into the Microsoft stack. For example, the ColumnStore feature is very partial, given that it can’t be updated; but Oracle doesn’t have columnar storage at all.

Reporter: Is the lock-in overall worse than IBM DB2, Oracle?
CAM: Microsoft locks you into an operating system, so yes.

Reporter: Is this release something larger Oracle or IBM shops could consider as a lower-cost alternative a co-habitation scenario, in the event they’re mulling whether to buy more Oracle or IBM licenses?
CAM: If they have a strong Microsoft-stack investment already, sure. Otherwise, why?

Reporter: [How about] just cost?
CAM: DB2 works just as well to keep Oracle honest as SQL Server does, and without a major operating system commitment. For analytic databases you want an analytic DBMS or appliance anyway.

Best is to have one major vendor of OTLP/general-purpose DBMS, a web DBMS, a DBMS for disposable projects (that may be the same as one of the first two), plus however many different analytic data stores you need to get the job done.

By “web DBMS” I mean MySQL, NewSQL, or NoSQL. Actually, you might need more than one product in that area.

• Many analytic challenges are inherently departmental.
• In many cases, central IT control of analytics isn’t needed.
• Departments move ahead without central approval or involvement because they can.

That said, arguments for centralizing analytic technology include:

• A lot of data is used by more than one department, for example:
  • Financial transactions (one or more affected departments and also the central accounting group).
  • Web logs (marketing and IT/web operations).
• Departments may not have the requisite technical expertise (and it may be redundant/cost-ineffective for them to acquire it).

What’s more, there are IT best practices to support department-level analytics. Some of the key ones boil down to:

• Be flexible in your analytic DBMS support.
• Be responsive to requests for ETL.

My conclusion is that central IT should encourage (and aid) departmental analytics. Let’s look at some details.

I think two huge categories of analytic problem are inherently departmental:

• Investigative analytics (pretty much all of it).
• Routine monitoring/dashboarding if the data is tracked just by one department.

Investigative analytics is a kind of research activity — you’re looking to discover previously unrecognized patterns. There are two approaches to this — you can do it in the department that has the relevant business knowledge, or you can outsource it to a special group of “discoverers” (commonly statisticians).* Either way, this is a small team/departmental kind of activity.

*Combining the two approaches is common — a department can have its own analytically adept discoverers, whether they’re call “quants” or just “business analysts”.

Reporting/monitoring BI at least has the potential to be enterprise-wide — but commonly it isn’t, as each department has its own operational data sources and metrics. Marketing departments may watch external data that the rest of the company doesn’t worry about. But it can be true across the board. Factory operations folks may track machine tool data the rest of us barely understand.

Even if a business need is strictly departmental, there can be at least two reasons to centralize technology implementation:

• The department doesn’t have the critical mass of IT expertise.
• Departmental IT has side effects on the rest of the company.

Whether those reasons hold up depends a lot on what kind of analytic scenario we’re talking about.

Let’s organize that part of this discussion in line with the taxonomy from my eight kinds of analytic database posts last July.

• Enterprise data warehouses fall under the purview of major IT organizations. That remains true even if we pivot to the more realistic concept of integrated data warehouse. However, less stuff needs to be protected in an EDW/IDW than some data authoritarians like to think.
• I wrote that the stresses on traditional data marts were “performance, concurrency, TCO.” This is a clue that the more demanding examples are right in IT’s wheelhouse. As for the less demanding cases — IT should be able to meet those needs without breaking a sweat.
• Agile investigative data marts are inherently departmental. If you have the talent to use one, you also have the talent to, for example, train into being a part time Netezza DBA. Who cares if you don’t have the expertise to do sophisticated tuning? Analytic DBMS are fast enough — and hardware is cheap enough — that you don’t that skill set anyway.
• Big investigative data marts can go either way. They’re technically challenging, so IT certainly has a claim on them. But in cases where the data, while big, is fairly homogeneous, it’s also not unrealistic for departments to handle the mart themselves.
• Bit buckets are often departmental today, with the department in question happening to be central IT. And central IT is where they’re likely to flourish, as the data they hold becomes ever more diverse.
• Archival data stores are a central IT matter. Nobody else is likely to care enough to do it right.
• Outsourced data marts, by definition, don’t live inside conventional enterprises. But they are often a way for business units to get access to data and analytics without relying on central IT.
• Operational analytics servers are likely to be sufficiently mission-critical that you want them handled by IT.

So in most cases I’d say: Departments can manage their own investigative data marts, and so of course can SaaS vendors and third-party data providers; other analytic databases should be run by central IT. (And of course, large departments with serious local IT can fuzz those distinctions up.) Beyond that, it would seem that whoever administers the database should administer the rest of the analytic stack as well.

That still leaves us with some practical questions, such as:

• Exactly what products should IT departments buy for which purposes? I hope a lot of posts in this blog are helpful in that consideration.
• How should development tasks be split between departments and central IT? It may take me a while to get a post together on that the subject, since in general the analytics-development picture is pretty complicated to lay out.
• How should departments and central IT work together to manage departmental investigative data marts? I hope to post on that subject soon.