Four of us are presenting, here’s the dates/times (all ET) and themes:

• Nov 3, 3.30-4.15: Matt Aslett - Open source to the rescue?

Can open source really help enterprises cut costs and ride out the economic storm? What has been the impact of current conditions on open source adoption? How is this being reflected in the business strategies of vendors – both open source specialists and traditional proprietary vendors?

• Nov 4, 11.00-11.45: Nick Patience & Kathleen Reidy - E-Discovery to Information Governance: From Reactive, Unavoidable Cost to Proactive Cost-Avoidance.

E-discovery is a market without a lot of discretionary spending – legal events and investigations occur, and require that organizations produce relevant electronic information, no matter the difficulties or costs. This fact has driven lots of vendors from various sectors to the e-discovery (also known as e-disclosure) market: it is driving business in the archiving, enterprise content management and enterprise search markets, as organizations want to figure out how to better prepare for litigation before it occurs.

• Nov 4, 11.45-12.30: Simon Robinson -  Storage Technology Is Thriving in the Economic Downturn

The economy is shrinking, but data is growing. Almost universally, storage vendors claim they can help IT ‘do more with less’ by squeezing more value out of storage assets to meet rampant data growth and stiffer retention criteria. This presentation will examine how three key trends in storage innovation – optimization, unification and the cloud – are helping some storage vendors thrive in this uncertain climate. The session will conclude with a vendor panel discussion.

Henry Baltazar is also attending and we’re all avaiable for 1:1s, though some of our days are getting pretty near to full. Contact your account rep about booking a slot.

If you are a client and you’re not attending then you’re missing out on one of the key beneifts of being a client!

If you’re not a client and you wish to attend, you can do that too, only you’ll have to pay to get in. Either way, you can register here.

Beyond information management all our other themes will be address including cloud (a lot!), security, virtualization, eco-efficient IT and our popular M&A panel, which always comes right before cocktails on day 1.

See you there!

First came the formation of Dynamo Business Intelligence Corp, (aka Dynamo BI), a new commercially supported distribution, and sponsor, of LucidDB. Then came the launch of InfiniDB Community Edition, a new open source analytic database based on MySQL from Calpont.

We actually included Calpont in our report but its product plans at that time looked precarious to say the least as the company found that its plans to launch a data warehousing platform based on MySQL were overshadowed by Oracle’s acquisition of Sun.

We were somewhat sceptical about whether Calpont - which has had a couple of false starts in the past - would find a way to bring something to market and we are impressed that the company has reached a licensing agreement with Sun that supports its open source and commercial aims.

Specifically the company has arranged an OEM agreement with Sun for the MySQL Community Server version that enables it to be used with both Calpont’s open source and commercially licensed products. The first of those is InfiniDB Community Edition, a column-oriented, multi-threaded data warehouse platform which acts as a storage engine for MySQL.

The GPLv2 Community Edition will only be available for deployment on a single-server and without any formal support from Calpont and is primarily aimed at raising interest among MySQL developers. A fully certified and supported commercial version will follow, although Calpont is reticent about providing details on that at the moment other than that it will make use of Calpont’s massively parallel processing capabilities and modular architecture to scale out as well as up.

Calpont faces some competition in the MySQL segment from Kickfire and Infobright, particularly the latter given their similar open source software strategies (Kickfire is a MySQL appliance). Infobright has has grown rapidly since going open source and now boasts more than 100 customers, although Calpont maintains that leaves plenty of opportunities amongst MySQL users.

We would agree with that, and also with the company’s claim to offer something different from Infobright technologically. Infobright also offers column-based storage but not massively parallel processing (although it is working on a shared-everything, peer-to-peer architecture). We should note that InfiniDB Community Edition is also restricted to a single server but this is the result of a strategic decision, rather than a technical limitation. The commercial version will be fully MPP.

We recently noted that LucidDB is another open source database that is often overlooked since the LucidDB code is not commercially supported.

Any concern over the future of LucidDB following the demise of LucidEra should be put to bed by the formation of Dynamo BI with the intention to provide a commercially supported distribution of LucidDB.

As LucidDB project lead John Sichi wrote:

“This is an offering which has been completely missing up until now, and which I and others such as Julian Hyde believe to be essential for accelerating adoption of LucidDB. LucidEra provided much of the critical development effort, but never offered commercial support on LucidDB since that was not part of its software-as-a-service business model. Eigenbase provides community infrastructure and development coordination, but a commercial offering is not part of its non-profit charter. So in the past, when individuals and companies have asked me whom they should talk to in order to purchase support for LucidDB, I have never had a good answer. “

Meanwhile Nicholas Goodman revealed that the company has acquired the commercial rights to LucidDB and plans to offer DynamoDB as a prepackaged, assembled distribution. It will also be fully open source and all new features will be contributed to LucidDB.

It is very early days for Dynamo BI, which doesn’t even have a website as yet, so it’s difficult to judge the company’s plans, but with some of the lead LucidDB developers involved and a solid starting project - “the best database no one ever told you about” - it has every chance. We’ll be looking to catch up with the company just as soon as it gets up and running.

The data warehousing sector is extremely crowded and we continue to believe that there will be a shakeout in the near future, but there are opportunities for companies that are able to differentiate themselves from the pack. Starting a data warehousing company is generally not something that we would recommend right now, but both Calpont and Dynamo BI have opportunities to establish themselves.

We look at the reasons for that projected growth, identifying 10 drivers overall, one of which is the rise of search-based applications. At some point in the future we’d like to try and size that market, although it’s too nascent to put a number on it just yet.

You can download an executive summary or find out more about the report here.

Suffice to say I’m very excited about this new addition to our coverage, adding the quantitative element to our many years of analyzing the market on a qualitative basis.

This report will be updated every six months with new figures and every 12 months with new analysis an figures. We provide analysis of the industry throughout the year through our Market Insight Service in shorter, more regular form.

This is not only the fist in a series of reports on the enterprise search business, but also the first in a series of market sizing reports within information management. The next will be on the data warehousing business, due in early 2010, written by Matt Aslett.

With choice comes potential confusion, since users have to identify and compare different products and features, as well as vendor viability, to ensure they are investing their IT budgets wisely – especially in the current economic climate.

Our latest special report, Warehouse Optimization - Ten considerations for choosing/building a data warehouse, is designed to help reduce that confusion and is now available for existing 451 Group clients to download and non-clients to purchase. An executive summary is also available.

The report provides an overview of the data-warehousing vendor landscape, as tracked by The 451 Group, and examines the business and technology trends driving this market. It identifies 10 key technology trends in data warehousing and assesses how they can be used to choose the technologies and vendors that are best suited to a would-be customer and its specific application.

The report is not designed to make recommendations on particular vendors or technologies, but to provide an independent overview of the sector, which could be used by customers as part of a vendor-evaluation process. The report also examines the potential for consolidation and identifies some potential merger and acquisition drivers, as well as providing profiles of the data-warehousing vendors being tracked by The 451 Group as part of its ongoing coverage of this sector.

Look out also for a forthcoming webinar in which we will present the key findings and implications. We’ll keep you posted on the details.

Q: Can you talk to any trends you see in terms of who in an organization is purchasing governance/e-discovery tools?

This is something covered in some detail in the report itself.  In general, there’s some difference in terms of purchasing between “governance” and “e-discovery.”  If the use case being addressed in a particular procurement process is specifically for reactive e-discovery - meaning, the ability to respond to a specific legal discovery request - then the process is likely to have heavy involvement from the legal department if not full ownership by that team with IT involvement.

Governance is generally broader and is likely to involve more underlying pieces of technology (e.g., archiving, records management, indexing tools for distributed data and e-discovery / early case assessment).  There’s certainly no single approach to governance and most organizations are in the earliest of stages in terms of putting in place some kind of broader governance strategy.  Procurement is still likely to be tied to more tactical requirements and the specifics of those requirements will dictate who’s involved (e.g., e-discovery is more likely to be run by legal, as noted above, while an email archiving decision is more likely to be led by IT with legal involvement).  Generally speaking, hashing out broader governance strategies may well involve IT (email management, storage, ECM and search folks), legal, compliance officers, records managers and security personnel, among others.

Q: What are your thoughts about how far right along EDRM the big ECM vendors will move?

So far, ECM vendors are focusing on the far left of the electronic discovery reference model (EDRM).  This has expanded in the last twelve months or so from a far more limited focus solely on the “information management” process step to greater capabilities for data identification, collection, preservation, and some review and analysis.  This is likely to continue, though I’d be surprised to see ECM vendors move beyond this.  Identification, collection and preservation will be key areas in the short term (EMC’s recent Kazeon buy is a good example of how ECM vendors will look to better handle distributed data).  Review and analysis capabilities are likely to remain in the area of early-case assessment, with the expectation that a winnowed-down set of data is still likely to be turned over to external counsel for further review and analysis. That’s likely to be where most ECM vendors stop, though not all; Autonomy, for example, plays specifically in the legal market as well with iManage and Discovery Mining.

Q: Can you explain a bit more what you mean by “litigation readiness”?   What processes does this cover?

I guess this is a phrase I use a lot when talking about information governance and perhaps I didn’t explain it well enough on the webinar.  Litigation readiness is really just one reason organizations are interested in information governance.  Poor information governance makes it difficult to respond efficiently and cost effectively to e-discovery.  There are a number of processes involved in better preparing for litigation, but ideally, organizations need to have some high-level understanding of what data exists, where it is and who has access to it.  That’s a whole lot easier said than done of course, particularly when you need to include data on desktops, laptops, shared file drives and so forth.  The processes generally need to encompass maintaining some kind of index of what resides on all those devices and how that data will be captured and secured if needed.  That needs to be combined of course with more formalized management of data in archives and records management systems, with some consistency in terms of retention and disposition policies (that are standardized and enforced) across sources.  Few organizations have a very good handle on this sort of thing across repositories and unmanaged devices today, but those that are more often involved in litigation are likely to be more litigation-ready.

Q: Is Information Governance of primary interest in the US or are companies in Europe also concerned? I.e. is there an opportunity for vendors beyond the US?

Information governance as it relates primarily to litigation readiness is of primary interest to those in the US and in parts of Europe that have similar discovery or disclosure requirements for electronic information.  In geographies that don’t yet have as strict requirements for electronic discovery, governance may still be an interest but may be for different reasons.  Compliance with specific regulations (e.g., privacy-related legislation) can be a concern, for example, as can IP protection or other types of security.   So there is certainly opportunity for vendors in specific markets, such as archiving, but the drivers might be different.

That’s probably enough for one blog post.  Again, those interested in the full webinar can find it here.

The webinar is free and open to anyone, so please feel free to join if you’re interested in this topic.

During the webinar, I’ll outline some of the trends we uncovered while doing our research for this report, look at the vendor landscape and M&A activity in this area, and briefly discuss some of the technologies that we think will be important in this sector moving forward.

Here’s the info and registration link:

The Rise of Information Governance webinar

Thursday, September 24, 2009

12:00 - 1:00 PM EDT

Register here

Recorded versions of our webcasts are available on our site a short while after the events are over.

The quote from CEO Mike Lynch grandly proclaims this to be Autonomy’s “second fundamental technology” - IDOL itself being the first. That’s quite a claim and we’re endeavoring to find out more and will report back as to exactly how it works and what it can do.

Overall though this is part of a push by companies like Autonomy, but also Attivio, Endeca, Exalead and some others into the search-based application market. The underlying premise of that market is database offloading; the idea of using a search engine rather than a relational database to sort and query information. It holds great promise, partly because it is the bridge between enterprise search and business intelligence but also because of the prospect of cost savings for customers as they can either freeze their investments in relational database licenses, reduce them, or even eliminate them.

Of course if the enterprise search licenses then get so expensive as to nullify the cost benefit, then customers will reject the idea, which is something of which search vendors need to be wary.

Users can apply to joint the beta program at a very non-Autonomy looking website.

Ever since Mike Stonebraker and fellow researchers at MIT, Brandeis University, the University of Massachusetts and Brown University presented (PDF) C-Store, a column-oriented database at the 31st VLDB Conference, in 2005, the database industry has debated the relative merits of row- and column-store databases.

While row-based databases dominated the operational database market, column-based database have made in-roads in the analytic database space, with Vertica (based on C-Store) as well as Sybase, Calpont, Infobright, Kickfire, Paraccel and SenSage pushing column-based data warehousing products based on the argument that column-based storage favors the write performance required for query processing.

The debate took a fresh twist recently as former SAP chief executive, Hasso Plattner, recently presented a paper (PDF) calling for the use of in-memory column-based storage databases for both analytical and transaction processing.

As interesting as that is in theory, of more immediate interest is the fact that Oracle - so often the target of column-based database vendors - has introduced a hybrid column-oriented storage option with the release of Oracle Database 11g Release 2.

As Curt Monash recently noted there are a couple of approaches emerging to hybrid row/column stores.

Oracle’s approach, as revealed in a white paper (PDF) has been to add new hybrid columnar compression capabilities in its Exadata Storage servers.

This approach maintains row-based storage in the Oracle Database itself while enabling the use of column-storage to improve compression rates in Exadata, claiming a compression ratio of up to 10 without any loss of query performance and up to 40 for historical data.

As Oracle’s Kevin Closson explains in a blog post: “The technology, available only with Exadata storage, is called Hybrid Columnar Compression. The word hybrid is important. Rows are still used. They are stored in an object called a Compression Unit. Compression Units can span multiple blocks. Like values are stored in the compression unit with metadata that maps back to the rows.”

Vertica took a different hybrid approach with the release of Vertica Database, 3.5, which introduced FlexStore, a new version of the column-store engine, including the ability to group a small number of columns or rows together to reduce input/output bottlenecks. Grouping can be done automatically based on data size (grouped rows can use up to 1MB) to improve query performance of whole rows or specified based on the nature of the column data (for example, bid, ask and date columns for a financial application) to improve query performance.

Likewise, the Ingres VectorWise project (previously mentioned here) will create a new storage engine for the Ingres Database positioned as a platform for data-warehouse and analytic workloads, make use of vectorized execution, which sees multiple instructions processed simultaneously. The Vectorwise architecture makes use of Partition Attributes Across (PAX), which similarly groups multiple rows into blocks to improve processing, while storing the data in columns.

Update - Daniel Abadi has provided an overview at the different approaches to hybrid row-column architectures and suggests something I had suspected, that Oracle is also using the PAX approach, except outside the core database, while Vertica is using what he calls a fine-grained hybrid approach. He also speculates that Microsoft may end up going the third route, fractured mirrors - Update

Perhaps the future of the database may not be row- or column-based, but plaid.

I’ve covered data warehousing before but taking a fresh look at this space in recent months it’s been fascinating to see the variety of technologies and strategies that vendors are applying to the data warehousing problem. It’s also been interesting to compare the role that open source has played in the data warehousing market, compared to the database market.

I’m preparing a major report on the data warehousing sector, for publication in the next couple of months. In preparartion for that I’ve published a rough outline of the role open source has played in the sector over on our CAOS Theory blog. Any comments or corrections much appreciated.