“…produce and permit the requesting party or its representative to inspect, copy, test, or sample the following items in the responding party’s possession, custody, or control.”

The key phrase “possession, custody or control” is something to be examined more closely in the context of Cloud Computing environments, where typically the cloud customer is the party in control and the cloud service provider is the party in possession and custody. In cases where the cloud customer is the party in litigation, it is natural to serve pre-trial a discovery request under Rule 26 (b) to the cloud customer and expect that since they are the party in control, and can therefore instruct the cloud provider to perform at least some form of collections. Now the question that remains is whether the same request can be made of the cloud provider, since they are the party in possession and/or custody. It is evident that requesting the cloud provider to perform a discovery request on behalf of their customers is impractical since any assertion of privilege or confidentiality would require the cloud customer to be involved in the discovery request. Besides, the cloud provider producing documents without consent from the customer of the cloud would run afoul of the Stored Communications Act (SCA). For these reasons, the broader three-pronged test of “possession, custody or control” embodied in Rule 34 (a)(1) should be revised to mean only “party in control”.

This view is supported in the seminal decision on Flagg v. City of Detroit, Slip Copy, 2008 WL 787061 (E.D.Mich.) . A great analysis of this case by Timothy Ackermann is available in The Federal Lawyer, November/December 2009 article, titled Consent and Discovery under the Stored Communications Act . As stated, when it comes to the application of “possession, custody or control”, the most significant test for cloud based deployments is “control”. For the cloud customer, “possession and custody” are not relevant, because in a strict sense, it is the cloud provider that can claim possession and custody. However, the cloud customer is clearly in “control” of the data, as evidenced by pretty much every service contract that gives the customer “the legal right to obtain the documents on demand”. Also, the cloud customer has the right to give “consent” to the cloud service provider to make the documents available. Thus, a cloud customer cannot claim that since they did not have “possession or custody”, the e-discovery obligations cannot be waived.

From the cloud provider’s perspective, the mere fact they have “possession or custody” does not require them to produce documents, unless the cloud customer gives lawful consent per the Stored Communications Act. Yet again, for the application of FRCP 34(a)(1) , we find that the party in control over the data is the one that determines discoverability of data in the cloud. In contrast, the third party that is merely in possession or custody is not required to produce responsive ESI, given the provisions of the SCA. As noted in Flagg v. City of Detroit, the district court did not find the need to consider the issue of having a subpoena issued to the cloud provider (SkyTel Communications), since the required evidence was more easily acquired by an e-discovery request to the cloud customer (City of Detroit). A similar argument is made in the Crispin v. Audigier Inc., a case involving postings on familiar social networking sites, Facebook and MySpace. Here, District Judge Margaret M. Morrow goes to great lengths explaining why the provider is not required to produce documents based on protections offered by the SCA.

In summary, the nature of cloud deployments and their usage redefines the scope of ESI to those that the customer has control. Regardless of the interpretation of Rule 34, common sense dictates that the cloud provider and cloud user cooperate when it comes to e-discovery requests. Of course, one of the challenges with cloud deployments is the SCA and its interpretation for cloud-resident ESI. This will be the subject of my next post.

Many enterprises clearly understand the benefits of in-house discovery that include lower processing and review costs, earlier access to case facts, better control over the processes, etc.  But some struggle with how to begin their journey, for any number of reasons (lack of knowledgeable staff, failure to get executive buy-in, inability to build a compelling business case, etc.).  Fortunately, the folks at Home Depot have recently completed their journey and have offered to share secrets they leveraged throughout the process.

In a similar fashion to best selling author’s Stephen R. Covey’s “The 7 Habits of Highly Effective People” David Steel, Sr. Counsel and Barbara Squires, Paralegal at The Home Depot will host a web seminar to walk us through the some of the e-discovery habits that helped them successfully navigate their way through the process.  The web seminar is titled “5 Habits to Create a Highly Effective In-House E-Discovery Process” and it’s free to attend. Since we don’t want to steal their thunder, we won’t divulge their habits now, but suffice it to say that every company can learn from their experiences.  And, after the web seminar I’ll devote more blog time to further expansion of each habit.

Since it’s our raison d’être to help companies complete their e-discovery journey, we’re excited to have The Home Depot on to share stories from their journey, all in the hope that others, just embarking on their own expedition, can be just as successful.

Several people have pointed to poor marketing as the problem, and it’s true most customers are confused. Summation’s products all have different names (iBlaze, Discovery Cracker, CaseVault, CaseVantage), and it is unclear how they relate to one another. But the problem is more fundamental than just marketing. There has been no innovation from Summation for years; its products are difficult to use; and, they don’t integrate with each other. So, naturally, customers switch to more compelling solutions, revenue declines, management cuts costs, talented people leave, service levels deteriorate, more customers defect, and the cycle repeats.

As the management teams at Silicon Graphics, Siebel, or Yahoo! can tell you, once a technology company faces this death spiral, it’s very, very hard to turn things around. But that’s exactly what AccessData must do for its recent acquisition of Summation to work.

On the face of it, you would not expect AccessData to be capable of addressing Summation’s problems. As the #2 player in the forensics market to Guidance Software, it has no experience in legal review. Its customers are enterprises and government agencies, not law firms or litigation support service providers. Its headquarters is in Lindon (Utah), whereas Summation based is in San Francisco. But AccessData has a capable team, and must have some plan in mind. What is it likely to do? My guess is as follows:

• Claim “end-to-end” in the enterprise market: AccessData will likely bundle the iBlaze review platform with its own forensic collection products (FTK) and claim end-to-end coverage of the EDRM model. The products obviously don’t integrate with one another, or even have the same UI, but some customers may not realize how important that is until after they have purchased. This is the same strategy used by Autonomy, which also puts together disparate products (Aungate, Introspect, etc.) and markets them as an integrated package.

• Promote CaseVault and CaseVantage in the law firm market: These hosted review platforms are not widely used. AccessData will be hoping that with better marketing and sales execution, it can drive adoption of them by law firms and litigation support service providers. But most providers today seem pretty happy with Clearwell and/or kCura Relativity, so it’s unclear why they would switch away to CaseVault / CaseVantage.

• Cut costs: On the day the acquisition closed last month, AccessData fired most of Summation’s engineers. That’s understandable, given the shrinking revenue. But it only accelerates the death spiral. With no engineers, it’s impossible to innovate or improve the products.

• Sunset iBlaze product lines: This sounds radical since, according to Katey Wood at the451 Group, iBlaze accounts for 70% of Summation’s revenue. But AccessData may decide to focus its development efforts on CaseVault and CaseVantage, ceasing all investment in iBlaze. Effectively, this means it would “milk” the law firms using iBlaze, and pitch enterprises a product with no real roadmap for improvement. Given how far iBlaze has fallen behind, there is a strong argument that further investments are probably just throwing good money after bad.

It will take a few months before we can say for sure whether these, or other, changes will make any difference. If the experience of other companies is any guide, they may slow the decline for a while, but not reverse it. After all, there may be some people out there using Silicon Graphics computers to access their Siebel CRM systems or search the web on Yahoo, just like there will be some using Summation’s products for document review. But there are fewer and fewer every day.

Even with the lure of the big waves beckoning you out into the water, if you follow electronic discovery you likely have a hard time pulling yourself away from the report, and this year is no exception. In fact, this year’s report is especially insightful, as George and Tom seem to have done a particularly impressive job of getting the pulse of not just what’s going on in the law firm and service provider parts of the market, but the enterprise as well.

This is a big change from just a couple of years ago. Go back and review the executive summary from 2008, and you’ll notice a very different feel to the findings. In 2008, much of the talk was around the dynamics of the service provider market, with relatively little discussion of trends related to the e-discovery process and technological innovation in the space. In 2008, it felt like e-discovery was something you had other people do for you: the word “consumer” appeared 12 times in the executive summary. In 2010, two short years later? Just five times. Why? The language may be telling. “Cost” appeared seven times in the 2008 report. In the 2010 report? 16… more than twice as often.

What seems to have happened is that the recession has been something of a refining fire for the electronic discovery market. In order to reduce costs and manage risks, enterprises are behaving much less like consumers and more like real customers with skin (and money) in the game. Not surprisingly, they’ve gotten extremely aggressive about bringing  innovative cost-containing measures to bear on the process. Socha and Gelbmann highlight three:

• More targeted preservation and collection of ESI
• More focused review and analysis of the data
• More effective use of technology to speed up the efforts, improve quality, and reduce costs

This is great news for innovative software companies in the e-discovery space — and their customers. What one would expect to occur in a maturing market is that it would move from a period of rapid innovation to a lower-innovation, consolidation phase. However, that’s not the case here. While there is consolidation occurring,  what’s remarkable about e-discovery right now isn’t really all the acquisition press releases in your twitter feed (mainly from vendors saddled with prior-generation point solutions who are trying to acquire their way toward a complete offering). Rather, it’s how leading enterprises are increasingly seeking, and finding, cutting-edge solutions to solve cost, efficiency, and risk management problems associated with e-discovery that simply weren’t available prior to the meltdown.

As in-house legal and IT e-discovery spending starts to gain steam, look for enterprises purchasing in-house solutions to demand many of the innovations that have been developed over the last couple of years (most of which are highlighted by the Socha-Gelbmann survey):

• Targeted collection: Products better able to strategically target the collection of ESI, rather than attempting to boil the ocean, are more suited to the mindset and approach of cost-conscious enterprises
• Iterative discovery: Products that are able to provide “to the left” functionality while still providing enterprise-class, intuitive processing, analysis, review, and production functionality
• Support for small and big cases: In discussing “small is the new big”, Socha and Gelbmann highlight how “the aggregate of small cases dwarfs the combined large cases.” Successful products must simultaneously handle high numbers of smaller cases while still scaling to the largest matters
• Integrated analytics: Products must bring to bear powerful analytics across all stages of the e-discovery process, focused not just on document review, but also looking at aggregates of data from many different angles and allowing you to see the big picture across the entire case for effective information and cost management

Is the EDD space maturing? Yes, as Socha and Gelbmann rightfully point out. But it’s doing so in surprising, innovative ways that, when it’s all over, may well prove to be a silver lining to the cloud of challenges the industry has faced over the last two years.

The overview paper is now available and covers the design of the two tasks within the track – the Interactive task and the Batch task. The Interactive Task is very relevant for the legal community, since it is designed specifically for analyzing the task of producing specific records in response to a “discovery request”. As noted in the paper, 15 teams participated, including 10 commercial teams, up from three teams in 2008. The 2009 study was also the first time an email collection (based on Enron emails released by FERC) was used.

The Interactive Task involves a “mock complaint” and seven different topics, with each topic described in the form of a general information request. Several teams participated by choosing one or more topics and submitting responsive documents for each.  These were then assessed using a mathematically sound sampling and estimation methodology, and effectiveness metrics were computed for each team.

The critical summary measure is F1, a combination of precision (estimate of false positives) and recall (estimate of false negatives). Overall, the highest F1 measure achieved across six of the seven topics was very good, as evidenced by values from 0.614 to 0.840. As an example, an F1 measure of 0.840 was achieved with a Recall of 0.778 and Precision of 0.912. This implies that the information request was satisfied with very few false positives (8.8%) and few false negatives (22.2%). Having a high precision implies that your reviewers will be reviewing fewer irrelevant documents, hence reducing your review workload and review costs.  A high recall ensures that very few documents were missed, so your case teams can be confident that all the facts of the case are examined.

It’s always important to look not only at the results, but the costs incurred when achieving said results.  We can break this into the costs that each team incurred, and the costs that assessment and topic authorities incurred. Unfortunately, the study did not track the amount of resources each team expended, so we will have to leave that as a possible improvement for a future study. To get a view of the second cost, a review of the tabulation of team interactions with topic authorities (Figure 3 of the overview paper) is helpful. In this study, the topic authority plays the role of a case expert. The numbers show that for some topics, a highly acceptable F-measure (over 0.75) was achieved even with interactions of 100 minutes, well below the 600 minutes allocated for each team. This would indicate that the teams were able to understand and construct meaningful searches with very reasonable amount of involvement of a case expert.

The other interesting conclusion is that there is value in selecting a corpus containing attachments. The study found that attachments increased the value of responsiveness by measuring the “document to attachment” ratios. For the responsive set, this ratio was a significantly higher value of 4.8 (i.e., responsive document families had, on average, one message and 3.8 attachments), while the entire population had this ratio at 2.2. This suggests that using the Enron corpus that contained attachments was a very good decision.

Of course, the most revealing, controversial finding is with respect to the Assessment and Adjudication phase of the project. As noted in the overview paper’s section 2.3.4, the rate of success of appeals was significant, ranging from 82% to 97%. In other words, the initial sample assessments were reversed in an astonishingly large number of cases. One could argue that the appealed documents were carefully selected, but that argument is weakened by the varying number of appeals by participating teams, and the success rate for even the teams with larger number of submissions. As noted in the paper, the teams that invested greater amounts of resources in the appeals phase benefited proportionately in the levels of improvement of their final precision and recall numbers. I know that constructing appeals can consume a lot of resources since, in addition to the normal information retrieval task, you are required to provide a convincing argument for reversing an initial judgment. This becomes very much a review exercise, not unlike the traditional manual review that the broader legal industry has been struggling with. For example, our own appeals budget was limited, forcing us to sample the appealed documents and select only a few. The outcome of this is that un-appealed documents are all assessed as relevant, which is unsubstantiated by the large number of appeals. In the final analysis, section 2.4.2 illustrates a salient indicator of success – teams that had a positive and useful interaction with the topic authority had the greatest success of initial assessments as well as success in appeals, and the ones that leveraged this for the greatest number of appeals had reported the greatest F-measure.

The 2009 study saw a significant increase in participation from commercial teams. My own personal observation is that unlike academic teams, commercial teams tend to evaluate their participation in TREC projects through the narrow prism of short-term return on investment. While there is value in contributing to the community, I am sure each team is asked to justify the benefits of participation to their management. Some would argue that the full benefit is not realized because of the restrictions placed on dissemination of results within the broader community, especially in the area of marketing the results. I am sure every commercial participant would want to promote their performance, and highlight how their technology and methodology was superior. Given that such direct comparisons are not permitted, the ability to market your results is severely curtailed. The potential for comparative analysis could be a powerful motivator for all participating teams to invest more in the exercise, with the final outcome that the community benefits.

As I noted in my previous post, the legal e-discovery profession needs an independent authority that can challenge vendor claims and provide objective validation of one of the most complex areas of e-discovery – search and information retrieval. TREC has stepped in and served that need very effectively. And, this has been deservedly noticed by the people that matter – Justices of cases involving electronic discovery, expressing their opinions regarding “reasonableness” with respect to cost-shifting, adverse inference, motion to dismiss and other judgments.

A study of such magnitude is bound to have certain flaws, and these are documented in Section 2.5. Leaving aside these shortcomings, the TREC Legal Track effort is immensely useful for both participants and consumers/users of legal technologies and services. The value offered to the community by such studies is well captured in the companion report, titled the Economic Impact Assessment of NIST’s TREC program. As the TREC coordinators are rolling out their new TREC 2010 Legal Track tasks, it is obvious that continued improvements in both the design and execution will make it even more attractive for all participants. Clearwell Systems is committed to the overall goals of TREC and intends to continue their involvement in the TREC 2010 Legal Track projects.