ESI Gladiator

Steve Susman - Tell His Story

2020-07-15T05:53:00.000-07:00

The trial bar lost a giant on July 14, 2020 - "Big Daddy" Steve Susman has left us for that big courtroom. Ellen lost a loving husband. Harry, Stacy, and family lost a great Dad and Grandpa. The SG family and the entire legal industry lost a great trial lawyer and innovator of many things that we take for granted today. I lost a father figure and mentor. As anyone who spent any time in Steve Susman's orbit will tell you – friend or foe – he was not just a great trial lawyer. Arguably the greatest. He was a great man. In the months and years to come, much will be written about Steve’s body of work. To borrow a theme from the play “Hamilton” – Those of us who spent any time in his orbit, must “tell his story”. I personally owe much of who I am today to Steve’s teachings and early influence on me as a young man.

Much has been and will be written about this trial or that. Tales will be told of Steve’s bravado, hubris and daring methods. Stories like the famous Samsung vs. TI trial that settled on the eve of closing arguments with Samsung (the PLT) agreeing to pay TI $1.1 B for a new licensing agreement, settling multiple cases filed all over the world. Steve was so disappointed he would not be able to deliver that closing argument we spent literally all night preparing, that he tried to get opposing counsel to agree to let the case go to the jury just to “see who would have won”. Oh, and there was a small wager that would have been involved. Of course, the judge would not allow it even if opposing had agreed to the wager. Or the Sunrise vs. Xerox case where Steve’s direct examination of our client was so persuasive that at the conclusion of 2 days of direct and cross examination, that Federal Judge asked our PLT rep a single question: “So, Mr. Pfeffer, to sum up your testimony; Xerox got you out on a limb and sawed it off behind you?” Mr Pfeffer, of course, answered - “Yes.” The balance of the trial did not go well for Xerox. Or, the time Steve had me “fax” opposing counsel his cross-exam Q&A because they had complained the prior day about “being surprised”.

Those stories are not myth. I was there. They really happened. What made Steve a great trial lawyer came down to honesty. Yes, Steve was brilliant, and highly skilled at the mechanics of trial work. But Judges and Juries liked Steve because he was honest. When he made a mistake in front of a judge or jury, he would immediately admit that mistake once known. He was adamant with everyone around him the importance of transparency and honesty in everything you do. You cannot fool most juries, Steve would contend. Juries loved Steve.

The story I will remember most, however, and the one that to me tells most about who he was, had nothing to do with a trial directly. It is a story I retell frequently to demonstrate “innovation”. One day around 1990 I was in Steve’s office. Like most days then, I was surrounded by boxes of paper over in the corner, staying busy, and waiting on Steve to complete a revision to a Q&A, so I could do my thing with his “document chron”. I sat quietly as few words are spoken during these prep sessions. There was no need for words as everyone knew precisely what was expected. Everything was scripted by that Q&A.

“Walker, do you know how to type?”, came a question from Steve’s desk. Taken aback by the odd question, I answered: Yes, I took typing in the 7^th grade and was a “Clerk Typist” in the Army. Well, I need to learn how to type, he exclaimed. Is there something wrong with David, I say. (David was his secretary at the time). No, he replied, this process is very inefficient. Too slow. I am wasting time, he says. Alright, I say, as busy as you are, how does you taking the time to learn to type going to make YOU more efficient? Our job is to make YOU more efficient, not the other way around. He explains the current process for preparing a Q&A.

I “dictate” my Q&A into a device and hand a tape to David.
David listens to the tape and transcribes my words into a document.
David prints the document and places in my in-box.
I then take a red pen and make handwritten revisions.
I give the marked-up paper document back to David.
David makes revisions, prints a new copy and hands to me for another round of revisions.
After multiple revision rounds, a final product is created.

If I learn to type, Steve opines, I can eliminate 6 steps. I do make myself more efficient and save a ton of turn time. What did Steve do? Not only did he learn to type, he issued a proclamation to the firm that all lawyers and legal assistants would learn to type, and everyone will have computers on their desks. Secretaries and only some lawyers had computers on their desks in those early days. The Dictaphone eventually went the way of the dinosaurs. I do not know for sure, but I would be willing to bet that every lawyer at SG today types their own first draft. Does anyone really use a Dictaphone anymore?

That story is a great example of how Steve thought, taught and innovated. He did not just talk efficiency, he lived it and expected everyone around him to live it as well. It did not matter whether the case compensation was hourly, or contingent, Steve handled a case the same way – get in front a judge or jury just as fast and efficiently as possible. Out lawyer and out work the opposition. Avoid discovery disputes and go to trial “by agreement”.

Those of us who spent any time in Steve Susman’s orbit must tell his story and pass down what he took the time out of his busy days to teach us. I personally owe it to Steve to show others what I have come to call the “SDS way”. Do you have story to tell?

Rest in Peace Big Daddy. You WILL be remembered. We are better today because of you.

2017-11-03T12:25:00.004-07:00

Relativity Fest 2017 Review – Part II
Rise of the Machine

Part I was of the R Fest 2017 review – "The Popsicle Moment" - can be found here.

Introduction

History in this industry teaches us that we can’t place all our eggs in a single basket. As noted below, been there and still have the T-Shirt to remind me. After all, iControl ESI has been stepping out on our own technology for years. We are agnostic. I didn’t attend R Fest simply because we have clients that love Relativity. No, I attended primarily because of history. I personally made a big mistake in 2008, as did many. I try not to make the same mistake twice.

My History without Relativity – When you can’t beat them, join them

Rewind to August 25, 2008, the 31^st annual ILTA conference at the Gaylord, Texan near Dallas, Texas. I was at FCS at the time and a crew of us were attending ILTA. FCS, as some might recall, was the 2^nd Attenex partner when Attenex built out their channel. In fact, FCS was an award winning Attenex partner. Yet, we went to ILTA that year looking to replace Attenex. Why would an award winning Attenex provider be seeking a replacement? Well, in June of that year, FTI announced that it was acquiring Attenex! Despite receiving calls and assurances that “nothing will change” at Attenex, we knew better. FTI had acquired Ringtail in 2005 and, of course, said nothing will change then. Well, as we all know, a lot did change and those of us at FCS knew there would be changes at Attenex that would have a dramatic negative impact on our business. History did in fact repeat itself and most of the Attenex crew left, including Joe White who joined FCS as president in June of the following year (2009).

So, in 2008, the FCS team was on the hunt for new technology. We encountered these folks from “kCura” pitching this product Relativity. By this time, we had heard of Relativity because they had been making some noise, but Relativity wasn’t as mainstream as it is today. Relativity was unproven in 2008. We went to this little suite at the Gay Lord and sat down with Andy Sieja and others. I can’t remember all the players from that side, but it was a crowded suite. Andy made the pitch. BTW, Andy, if your reading this, your sales skills have improved dramatically since that Gay Lord presentation, but I digress. We listened, and I remember walking out of that suite thinking….hmmm…these guys might be on to something. They were talking all about openness, integration, cloud and a lot of things that have since transpired. Yet, if we went with Relativity, we would be making a bet with folks that we didn’t really know. They didn’t seem to have a lot of capital, but we really didn’t know what was going on with them as a company. Flash in the pan? We had seen so many others with great product ideas come and go.

At the time, we were talking to Lateral Data – makers of Viewpoint. We knew the folks at Lateral Data, some of which were with me at RLS a few years before. Viewpoint was also new, but we had a client using it because one of the firm partners had a relationship with the Lateral Data owner. One of my close friends at the time was there. It felt comfortable. We were sold on the fact that with Lateral Data, we would get a lot of voice into the road-map, and we did. We even did some dev work to help. Since we had a strong dev team at that time, we saw Viewpoint as an opportunity to help make an existing product better. FCS became Lateral Data’s first channel partner. Life was good, but it didn’t last. We made a BIG mistake choosing Viewpoint over Relativity, as history would teach us.

You see, we didn’t expect what would happen next to Lateral Data. Lateral Data sold to Xerox in 2012. Of course, Xerox said nothing would change. But, dramatic change ensued. Xerox yet again fumbled the future. Yes, some may see that last sentence and recognize a book: Fumbling the Future: How Xerox Invented, then Ignored, the First Personal Computer. That’s right, I bet many of you did not know that not only did Xerox invent the first personal computer, they also invented, and mass produced, the first lap top. The battery got so hot, they didn’t call it a lap top because it would burn your leg. It was called a “battery powered PC”. I know because I worked on the resulting litigation when Xerox gave up and cancelled the manufacturing contract with Sunrise Systems. But, again I digress. Bottom line, we at FCS bet on the wrong horse when we bet on Viewpoint.

As previously mentioned, it has only been the last two years where I am somewhere that offers Relativity. Prior to that time, I’ve spent this entire time essentially selling against Relativity.

Place Multiple Bets

So, how does one learn from and avoid those prior mistakes? Well, you spread your risk. You bet on multiple horses and hedge your bets. You make sure one of your horses allows you to add your own secret sauce. iControl ESI, where I am now, offers several technology options, including our own technology, Recenseo, that we’ve had for well over a decade. We also offer IPRO’s Eclipse as well as Relativity. Our advanced technology, Envize, integrates with Relativity, and hopefully one day soon, Eclipse. Neither Relativity nor IPRO can or will devote as much time as we do in the machine learning area, and why should they when companies like iControl are years ahead of them? Products like Relativity and Eclipse are simply platforms that do a lot of the things that the “mainstream” wants and needs. For those that are doing more advanced work, we don’t really need to focus on the “mainstream” basic functions of a review tool, because we can integrate into the tools that customers want, including our own review tool. But, we are going to date multiple partners because our client’s tastes vary.

Relativity continued to push the open platform message throughout the conference.

One of our biggest fears about Relativity is that, because of Relativity Assisted Review, Relativity would become closed. Turns out, that is not their vision now, and never has been. Relativity will remain open to those of us who want to use our own secret sauce. API’s for Relativity are very mature and improving with each new release (saw the road-map).

Oh, there was another concern also resolved at the conference. There have been “rumors” that Relativity is looking to sell to Microsoft. Andy shot that notion down. Not happening anytime soon, he says.

Analytics

Conference speakers did a great job of finally explaining the difference between the broader “Analytics” vs. more specific “Predictive Analytics”. When most think about analytics, they think of basically three things, which all leverage a form of Machine Learning.

Email Threading and visualization
Near Dupe Identification and visualization
Concepts clustering and visualization

Machine Learning base technology that works best for those three forms of “analytics” does not necessarily work best for “Predictive Analytics” which is closer to actual “Artificial Intelligence” than the technology used for the above three “analytic” features. Those older methods simply require too many computing resources to support true machine learning where humans teach the machine and the machine learns, real-time, as the review progresses. Grossman/Cormack call this “Continuous Active Learning” (CAL).

Rise of the Machine

My focus at R Fest this year was advanced analytics. It’s the place I live and breathe every day. There is a great deal of new interest in advanced analytics. Those of us in the software world need to know what direction the market is headed to confirm we are headed in the right direction, both as a software and services company. I do not want to bet on the wrong horse again. Machine Learning was indeed a major topic at R Fest with panels of judges, scientist, legal professionals and integrator in attendance, talking about advanced technology. If I had to pick two R Fest themes, I’d say they were “Integration” and “security”.

I attended several sessions that discussed the “machine learning” road-map of several products, not just Relativity. A major topic, and floor discussion among those of us focused on the rise of the machine, was around who’s secret sauce is the right direction. You see, there has been for years a raging debate between the various scientist around which “algorithm” or “method” works best. The three dominant camps are:

Latent Semantic Indexing (LSI) camp, focused on “visualization” of concepts, and not as much on the “Predictive Analytics” capabilities of machine learning.
“Logistic Regression” camp that believes their methods (and custom algorithms) are more “accurate”.
Support Vector Machine (SVM). The dominant method (but not the only) being adopted by most of the integrators at R Fest.

SVM vs. The World

Relativity announced at R Fest 2017, that they too have chosen the SVM camp and did a very good job of explaining why.

The “accuracy” difference between the various methods in our world is, well, not material. Although, the Yang et al 2017 paper – Effectiveness Results for Popular e-Discovery Algorithms - seem to suggest that SVM performs better than other methods in many areas, see Tables 1, 2 and 3 in that study.
Again, SVM is the dominant base method that most at R Fest are using. Other methods, such as Logistic Regression, were in the minority at R Fest. The SVM selection is validated by our study - Batch-Mode Active Learning for Technology-Assisted Review* that was accepted by IEEE in 2015. Despite all this external validation of our chosen method, I wanted to make sure we sought some validation from other integrators and not just the academics. I personally am interested in workflow and the “does it work” test preferring to let the real scientist duke it out around the back-end science. That’s not to say that I don’t engage in debates with those PhD types. Those that know me, know that I enjoy nothing more than a good debate about machine learning. I frustrate the scientist, however, because I say things like – too many clicks. Too much effort to build classifiers. We must review too many documents with your method. Lawyers must learn math, and wait too long for the leavers to get pulled. “Looks great” but too much flash, bells and whistles we don’t need, and not enough of those things that work practically...and other such “logical” arguments. This kind of discussion often makes the technical types uncomfortable, because they usually have never actually worked the business end of litigation. What works in the lab doesn’t always work in reality, I say.

If the academics want to spend their days splitting hairs around precision and recall, I say knock yourselves out, but the “math” is never going to work. Nobody wants to review 20% of anything so the sampling math works on low prevalence populations, which is our world and not the laboratory. My job is to take what the scientist creates in the lab and see if it works in the field, so to speak.

Accuracy is not the primary reason that a SVM based algorithm(s) is the better choice. Flexibility and performance drove our choice. Like some other methods, an SVM based method requires fewer computing resources. Classification and rescoring can be done much faster, making a continuous review workflow possible. Also, with SVM, you don’t have to spend a great deal of time manually creating “classifiers” and filtering out noise. In the machine learning environment, SVM based methods are very good at filtering out the “false positives”. “Stabilization”, that point where the machine learning is stabilized, and the machine isn’t getting any smarter, is achieved much faster than some other methods. In other words, you review fewer training documents. With SVM, you arrive at the same train station as other methods, you just get there earlier and with less effort.

Conclusion

As Andy Sieja likes to say, we must all figure out how we can work together. Machine Learning in the legal space has arrived. The benefits can no longer be ignored by the mainstream. I know, I know, I and others have been saying that for years. But, to be fair, the industry has spent too many years talking about the underlying science, which has been around for decades, and not enough time talking about how to use the machine in a familiar workflow. Federal and State court judges have for years said, “you don’t need our blessing” to use advanced analytics. Most lawyers just want to review relevant documents without a bunch of waste. Most don’t want to spend time with charts and graphs, math and science, bells and whistles. Those that do, well, there’s plenty of that in the eDiscovery software world.

So, if you’re not using advanced analytics and are waiting on something else to happen, I’d sure like to know what exactly is keeping you from leveraging the machine. I’d love to hear your objections and be given an opportunity to address all concerns. Trust me, your unlikely to come up with a new reason not already addressed over the past decade, so we can certainly help address pretty much any concern. If not me, contact somebody expert in this area. IF you don’t, you are going to wake up one day in a world you don’t recognize.

2017-10-25T17:30:00.002-07:00

Relativity Fest 2017 Review – Part I

The Popsicle Moment

This is the first in a two-part review of Relativity Fest 2017. Part two is titled – Relativity Fest 2017 Review – Part II – The Rise of The Machine. Subscribe to ESI Gladiator to receive notification when it's published tomorrow.

Introduction

The verdict - TWO THUMBS UP!

This was my very first Relativity Fest! That’s right, I went kicking and screaming into the den of the dragon I have somehow avoided all these years. Avoidance of Relativity has not been by accident. Like other old industry war horses, I’ve used virtually every “eDiscovery” application that’s hit the market over the past 25 years, but not Relativity. Not until past few years. I am no different than the average consumer – I have my favorites. My favorite historically has not been Relativity, for reasons that are unimportant here.

Like many service providers in the industry these days, the company I work for offers clients options. A few years back, a client pulled us into both Eclipse AND Relativity. That law firm client has an even mix of Relativity and Eclipse cases. Even though we have our own review software, Recenseo, and highly advanced “predictive analytics” application, Envize, we had to bring in the technology the customer wants. Of course, we believe our technology is just as good as those other guys. Yet, we now support substantial matters in all three review tools. Clients have choices. We are not alone; many providers now offer multiple options. FTI, one of the largest companies in our space, recently announced that it has rolled out Relativity and spoke about why during the keynote speech at R Fest 2017. Yes, that’s the same FTI that bought both Attenex and Ringtail. More on that in Part II.

We are rapidly reaching the point where you can get a wide variety of technology on any street corner. So how does a consumer differentiate between “providers”? Service – create those Popsicle Moments. What the heck is a Popsicle Moment? Read on.

The Power of Moments

There was a very thought-provoking “moment” at the conference. Chip & Dan Heath were keynote speakers. They just published their 4^th New York Times best seller – “The Power of Moments”. I’ve attended dozens of these talks over the decades. I’m not known as a reader of self-help books. In the legal technology business, there isn’t much time for sitting on the park bench, smelling roses, and reading self-help books. This talk and this book, however, was different. I was so moved by the two brothers that I read the book on the plane home.

The Heath brothers cover a lot of ground. If I had to sum it up with one story and theme, however, their message is about creating “moments” for your clients/customers. In litigation, all cases have peaks and valleys in terms of the experience, especially when dealing with ESI and technology. There have been many studies over the years (discussed in the book) that show that consumers (even lawyers) are more focused on the overall experience than the valleys. If you create “peaks” in the client’s experience, the valleys tend to be overshadowed and the overall client experience is positive. The brothers provided a perfect example.

The Magic Castle Hotel

How do you create that magic for your clients and customers? Those “peak” moments that they will remember? The brothers used a very specific and mind-boggling example. There is a “hotel” in LA called the The Magic Castle Hotel. It is award winning. It is one of the “three top-rated hotels in Los Angeles over competition like the Four Seasons Hotel at Beverly Hills and the Ritz-Carlton Los Angeles.”

Their customer reviews on Travelocity and Hotels.com are, well, remarkable. Out of 3,129 Travelocity reviews currently, 85% rate Magic Castle as very good or excellent, with the clear majority rating them as excellent. Customers that booked through Hotels.com rates the hotel a 4.6 out of 5! If you want to stay there, you need to book well in advance during peak times.

By comparison, the Ritz-Carlton does carry a high rating, but far fewer reviews (633) and a Travelocity rank of #3 in LA – just BEHIND Magic Castle! The Ritz is known for creating a unique customer experience at 3X the rate of Magic Castle. Yet, as the brothers point out, Magic Castle creates a customer experience that beats the Ritz at their own game.

The Magic Popsicle & Free Stuff

So, what’s so remarkable about Magic Castle? It’s a MOTEL! Yes, they do have “Hotel” in the name, but it’s essentially a two-story motel. Painted bright yellow with a pool no bigger than most backyard pools. Yet, reviews of the “Pool” are “GREAT”.

The rooms are unremarkable. Clean, with just the basic amenities inside the room.

Hotel management has obviously figured out how to create those “peak moments”. Here are just a few of the clever customer experiences they create. I’ll then relate that to the legal business. There are many more than these, so these are just examples.

1. Snacks and drinks are free. That’s right, free. No catches.

2. Laundry.They do your laundry every day, returned to your room, pressed and folded.

3. Ice cream! Self-service ice cream machines. Who doesn’t like ice cream!

4. They Celebrate YOU. If the staff knows about a birthday or anniversary, you are surprised with gifts in your room upon arrival.

5. Actual Magic. Many of the hotel staff are magicians and entertains guests across the complex, and there are formal magic shows in the lobby – hence the name “Magic Castle".

6. Magic Popsicle. But, those are not the crowd favorite amenity. By far, the most mentioned, most popular amenity is the Popsicle Hotline.

Pick up the red phone next to the pool and ask for Popsicle. In short order, a hotel staff member shows up - pool side - with white gloves, and a tray of individually wrapped Popsicle.

Smiles. Happy customers. That Popsicle Moment they will always remember. That stopped up toilet on day two is long forgotten.

How to Create those Magic Popsicle Moments

In the Litigation Support world, those on the front lines are the Paralegals, Project Managers and other staff in supporting roles. We support trial lawyers. Those jobs are among the most difficult jobs on the planet. Ask anyone in LitSup and they will tell you just that. Lawyers are our clients and customers. But, we are not a hotel! But wait, many law firms and corporate legal departments do in fact already provide soft drinks and coffee, although some have mused that there is a reason that caffeine is free, but I digress. At our company, we have ice cream, Popsicle, drinks, snacks and the like. Studies show that “creature comforts” can dramatically improve morale and attitude in the work environment.

Early in my career, I spent a decade or so bouncing around the country supporting trial lawyers at trial. Like many paralegals today, my responsibility was vast. I was responsible for all logistics inside and outside the court room. Everything from making sure all demonstratives were ready and presented at the right time during trial. Helping prepare Q&As, assisting during examination. Outside the courtroom, I was responsible for making sure all those “creature comforts” were there so the lawyers could focus on their work and that next witness. I was the gate keeper and the client handler, responsible for making sure the firm’s client was taken care of, but didn’t interfere with the lawyers' trial prep. One great trial lawyer I used to support was made happy when he had four things at the end of the trial day:

1) Document Chron of exhibits;

2) A bottle of good wine (intake managed by me);

3) A good cigar; and

4) A nice dinner, but not too heavy to avoid a sleepy lawyer who will need to prepare most of the night.

Before I went to trial with a lawyer for the first time, I made a point to find out what they liked and disliked. I made it my mission to make sure they stayed focused and didn’t have to worry, or even think about anything but the case. I ran interference for all access to whatever lawyer was “on” with the next witness. Nobody got to that lawyer without going through me. I delivered to their room, so they didn’t waste time coming down to the war room. Without really realizing it at the time, I was creating “Popsicle Moments”. Project Managers that support a wide variety of “clients” are those with the best opportunity to create those memorable moments and do so all the time. At least, those organizations that are highly focused on “customer service” do. Great PMs are there when you need them – whatever you need. Great PMs are focused on making sure you are trained to do things yourself if you want, or doing whatever you need for you when you can't. They “show up”. There is always “free stuff” with any great service provider. They don’t nickel and dime you for everything. A customer centrist provider won't consume you with the same questions over and over. Great PMs make it their mission to determine your likes and dislikes and execute. You trust them. When your supported by the right provider, your service level is the same whether you are sending them a million dollars of work, or a thousand.

Conclusion

As “eDiscovery” pricing becomes a commodity, and profit margins of service providers continue to shrink, it is easy for an organization to make up the difference by cutting wages of workers. When an organization does that, the obvious result is a less qualified staff. “Free” project management might be a sign that your service with that organization may not be what you expect. Nothing is ultimately free.

What are you doing to create those Popsicle moments? Is your service provider creating Popsicle moments for you?

2017-08-30T09:32:00.001-07:00

Do You Agree to Untested Search Terms?

Introduction

Unless you are using a proven key term validation methodology that utilizes sampling, you are almost certain to agree to terms that are likely overly broad. Obviously, agreeing to terms that are overly broad is costly, but common. The greater risk - what are you missing?

Too many declare that search terms "look good" without the appropriate testing.

Victor Stanley v. Creative Pipe, Inc. (2008)

I recently ran across a Latham & Watkins “Client Alert” from July 2008 - The Importance of Victor Stanley v. Creative Pipe, Inc. for Electronic Discovery. The Victor Stanley opinion was the first prominent opinion involving search terms and ESI. In that case, the context was the use of search terms to screen documents for privilege before production. The terms the defendant used were, well, untested, poorly crafted and as a result missed privilege documents that were ultimately produced. Bad things ensued. The Victor Stanley opinion is now almost a decade old, yet many still agree to terms (or use them unilaterally) without a defensible methodology. Judge Grimm gave some sound advice back then that still applies today. The Latham white paper summarizes Judge Grimm’s insight thus:

“Search Methodology In evaluating the reasonableness of the defendants’ methodology, the court was critical of the fact that the keyword search terms were selected by the party and its attorneys and were not disclosed to the court, noting that, “while it is universally acknowledged that keyword searches are useful tools for search and retrieval of ESI, all keyword searches are not created equal.”²³ The court suggested that proper selection of keywords involves “technical, if not scientific knowledge.”²⁴ The court further cautioned that the choice of terms is important because “simple keyword searches end up being both over- and under-inclusive in light of the inherent malleability and ambiguity of spoken and written English (as well as all other languages).”²⁵ The court also noted that the defendants did not assert that any sampling was done of the text-searchable ESI files that were determined not to contain privileged information on the basis of the keyword search to see if the search results were reliable.²⁶ The court’s order suggests that sampling is a prudent way to test the reliability of keyword searches and verify that the searches are neither over-inclusive nor under-inclusive.” [emphasis added]

The Latham "alert" also provided some sound “practice pointers”

Meet and Confer with the Opposing Parting Regarding Search Terms
Determine Whether to Ask the Court for a Discovery Order Compelling Production
Be Prepared to Defend the Search Methodologies.

Key Term Testing

Figure 1 Recenseo Search Term Manager

That 3^rd Latham practice tip is where most struggle getting it right. Too many use the wrong tools, the wrong methodology and/or get poor technical and workflow advice. Running a search and reviewing the result without testing isn’t the way to go. To properly sample and test search terms, requires solid search term hit reporting. Figure 1 above is reporting from analytics software Recenseo. The Recenseo Search Term Manager interface allows the user to revise terms on the fly and immediately update the hit rates. When used in conjunction with “Predictive Analytics”, one has powerful information about how well search terms are performing across the entire corpus, or limited to a specific scope. For more on how to combine Search Term validation with Predictive Analytics, see “Combining Predictive Coding and Search Term Classification in 5 Easy Steps”

Automated sampling functionality within Recenseo makes it easy to pull random samples of key term hits, review them, and then analyze the “responsive rate” because reporting can be narrowed to a specific scope such as those documents tagged, or "predicted", as relevant. This is a critical distinction from some applications on the market where key term hit reporting is either not present at all, or only available across an entire corpus. There are also performance issues with some applications where getting the critical hit reporting requires, search construction expertise, long waits and substantial computing resources. Searches are slow! While the overall hit rate is important, It is also important to know the hit rate on tags, or maybe a specific custodian of interest, so you can properly evaluate the effectiveness of key terms before you agree to them. See Figure 2 below.

Figure 1 - Recenseo Hit Report Export to Excel

Conclusion

Do not agree to terms until you know the relevancy rate for each term. Do not agree to a term if it is hitting only 30% responsive, for example. Pick the threshold that makes you comfortable. While it’s better to be overly broad than too narrow, being too overly broad costs time and money. At a 30% responsive rate, 70% of your effort is wasted on documents of no value. Get help. Settle on a provider, methodology and technology that provides you with the proven workflow, tools and proper advice.

Be careful with those search terms!

2017-08-14T06:34:00.001-07:00

ePizza as a Service – Cloud Explained (OP, IaaS, PaaS and SaaS)

By Mark G. Walker, VP Advisory Services and Susan Kavanagh, Senior Advisor -iControl ESI

Introduction

We recently came across again this LinkedIn post from 2014 by Albert Barron, a software architect at IBM. His analogy does a great job of explaining the various “… as a service” models using everyone’s food favorite – PIZZA!!!. We are taking a few liberties with Albert’s original version and hope he doesn’t mind.

Making Pizza

Figure 1 above assumes that your product is a Pizza and makes a comparison between three levels of Cloud services and an on premises model, to the pizza industry. Which model works best depends upon the organization and is almost always a cost vs. benefit analysis.

On-Premises - All you can eat

With the on-premises approach, you can make your pizza at home. You are responsible for purchasing all the ingredients including making the dough and sauce from scratch.

Explanation: To justify the cost of this approach, versus an outsourced approach, you will need a steady demand for pizza with great frequency, meaning someone is ordering pizza in large quantities every day. With this model, you will need a dedicated staff for everything and operate as though you are essentially running a pizza shop.

This model is the equivalent of bringing all of your eDiscovery services in house, whether you are a law firm, corporation or other entity.

Infrastructure as a Services (IaaS)

With this model, you can buy some of the ingredients prepackaged (dough, sauce, toppings and cheese), but everything you need to cook the pizza (like the stone, oven, etc.) is rented from a vendor. In other words, you’re cooking a pizza using the providers ingredients in the providers kitchen.

Explanation: Like the on-premises model, you may need a significant demand for pizza, but maybe not in large volumes and not as frequently. Since what you need to cook the pizza is only rented – using only what you need – you can weather peaks and valleys in demand. If your cooking the pizza, of course, you must be able to justify the dedicated cooks required to do the work.

Think of this approach as an extension of internal practice support and IT teams. A hosting partner builds the infrastructure, configures systems and sets up data transfers, but the internal team will still manage most of the work.

Platform as a Service (PaaS)

With this approach, the provider makes the Pizza using all their ingredients and equipment and delivers to your home.

Explanation: This model might benefit those that enjoy pizza delivered from a pizza shop but prefer to dine at home, rather than go out to a restaurant.

Due to its limitations, Platform as a Service is not a widely-used model in eDiscovery.

Software as a Service (SaaS)

With this model, you would go to a physical location, order your pizza and eat it at the restaurant.

Explanation: This is usually associated with those that don’t have a high demand for pizza. While that could be one justification, many are moving to SaaS models even when volume and demand are high, or one big event is calling for a large volume of pizza over a short period of time.

This approach compares to eDiscovery clients who prefer to have all their data maintained by a centralized service provider. These providers host and manage a software solution, allowing user access to an application on-demand, over secure networks to work with their data.

Figure 2 above depicts how delivering a product in a data environment corresponds to the components of delivering a pizza.

Conclusion

Which of the four models works best for your organization will depend on many factors. While cost vs. benefit is a consideration, efficiency is another component that should be reviewed when deciding on the best approach. There is every indication that most in the eDiscovery space that cannot justify on-premise models, are moving toward a SaaS based approach as opposed to IaaS models.

Consider how much pizza you consume on a regular basis, then decide if it’s more cost effective and efficient to dine-in, carry-out or simply do-it-yourself.

Just a little tip: it’s always better with extra cheese!

2017-07-25T13:45:00.006-07:00

Top Adviser warns of out-of-control autonomous robot lawyers

Mark Walker, an eDiscovery adviser, author and frequent commentator, advocated Wednesday for "creating rules that govern how we use Artificial Intelligence (AI) with legal technology lest we unleash on humanity a set of autonomous robot lawyers that we don't know how to control.”

“I’ve spent my entire adult life serving lawyers. Non-lawyers like myself have a hard-enough time now getting the human kind to listen", Walker stated.

Walker was responding to the mention of numerous articles, blog posts and commentary around the rise of AI in the business world, military and more recently the legal technology field. A little-known fact – the military has a rule in place, “due to expire later this year” that restricts the use of autonomous drones to take human life. Under the current rule, the drone that delivers the lethal munition must be manned by a human.

"I don't think it's reasonable for us to put robots in charge of complex legal advice without some measure of restriction, for example," Walker told a colleague during a trip to the water cooler.

That conversation also included discussion about a wide range of topics, including North Korea, Iran, defense budget issues, and how we may want to consider changing where we get those breakfast tacos. Full disclosure and complete transparency: No one else attended the impromptu water-cooler session. The “colleague” wishes to remain anonymous. During that same water-cooler conversation, Walker also opined:

"There will be a raucous debate in the industry about whether we take humans out of the decision to decide whether a document is relevant or not, for example," but added that he was "an advocate for keeping human lawyers involved in those decisions – for now."

Walker further said humans need to remain in the decision-making process "because we take our values and subjective judgement into account. AI’s don’t" He pointed to the rules of civil procedure and the need to consider issues like proportionality and discriminate action against doing something that might make perfect sense to an AI, but doesn’t to the human lawyer.

His comments come as the US military has sought increasingly autonomous weapons systems and fears that corporate GC’s may begin doing the same with robot lawyers. As noted by a recent CNN Politics article US general warns of out -of-control killer robot -

“In July 2016, a group of concerned scientists, researchers and academics, including theoretical physicist Stephen Hawking and billionaire entrepreneur Elon Musk, argued against the development of autonomous weapons systems. They warned of an artificial intelligence arms race and called for a "ban on offensive autonomous weapons beyond meaningful human control."

OK,….by now hopefully you know this post is a parody of that CNN article referenced above about autonomous drones the military wants to deploy. Few in the legal technology industry believe that jobs are going to be taken away by autonomous robots that think and make decisions on their own. I must say “few” because we really don’t know what those IBM Watson types have on the drawing board, but we can be confident in our narrow “eDiscovery” world there won’t be any “autonomous” technologies anytime soon.

Most “Predictive Analytics” technology that we use to reduce the number of electronic documents needing review at its root is indeed based on complex technology that does use Machine Learning, which is a form of AI. The underlying technology is so complex that most layman don’t understand how “it” works with any depth. That lack of understanding isn’t due to a lack of aptitude – lawyers and their staff are obviously smart folks. Rather, that underlying “math” is so mature that a deep understanding by someone at that level simply isn’t necessary. The root technology has been around for decades and is very mature. How you train the technology and what you do with the results is the more important place to gain an understanding. From a workflow standpoint, different applications work in different ways. We don’t need technologist over complicating what happens under the hood. We didn’t do that when “search and retrieval” technology was developed and rolled out. Everyone just assumes that if I search for “dynamite”, I will retrieve all the documents that contain the word dynamite.

Machine Learning, the form of AI we use in “Predictive Analytics” technology, doesn’t just look for single words, it “classifies” all documents into conceptual buckets by considering all the words in a document, usually focused on noun and noun phrases. The technology also compares the frequency of certain words and how words and phrases in one document compares to sentences with similar words in another document. The right technology does this continuously across millions of documents as humans review. With the right technology, humans review “samples”, either selected by humans, the technology, or both. The technology presents sample documents that contain similar content to the human for consideration. In other words, humans, not the technology, decide what’s relevant. The technology simply places documents into conceptual buckets and then finds documents that are similar in content to documents identified by the humans as either relevant or not relevant. It really is just that simple.

Conclusion

As previously noted, no one is seriously staying away from “Predictive Analytics” technology because they fear that we are going to weaponize autonomous robot lawyers that are going to take their job. Rather, the problem with adoption of this technology has been a lack of basic understanding because WE (that royal industry we) have made this technology sound too complex and risky. The reality is that the risk in using this technology resides with the humans, just as it does with search and review the result workflows. Predictive Analytics technology doesn’t make autonomous decisions. If there is a flaw – something gets missed – it’s the human usually at fault, not the technology. Garbage in, garbage out. Sure, some technology stacks are better than others. The differences are usually around workflow and feature set, not the quality of the classifier method. There are differences in performance where a technology is using an old method or algorithm that can only utilize simple passive learning as opposed to constant re-ranking of documents as document review progresses. For that continuous re-ranking, you need active learning capability.

We are finally seeing adoption of “Predictive Analytics” technology on the rise.

Are you at least exploring the use of Predictive Analytics?

2017-07-06T11:05:00.000-07:00

Is AI Replacing Lawyers and their Staff?

By Mark G. Walker, VP Advisory Services and Susan Kavanagh, Senior Advisor -iControl ESI

Introduction
Don’t shoot the messenger! You can’t make this stuff up. Or can you? There have been several articles recently suggesting that Artificial Intelligence (AI) is creeping into the legal arena and that “Lawyers could be the next profession to be replaced by computers”. That’s literally the title of a recent CNBC “Future of Work” article.

“The legal profession — tradition-bound and labor-heavy — is on the cusp of a transformation in which artificial-intelligence platforms dramatically affect how legal work gets done.”

To be fair, despite the catchy title, that article - like this one - didn’t conclude that AI was a threat to the legal profession in that it would eliminate jobs. In fact, the opposite is true. "I think we will see a rise of more jobs in the legal market as a result of AI", said Andrew Arruda, CEO of Ross Intelligence.

Sofia Lingos, a lawyer and board member of the Legal Tech Resource Center of the American Bar Association, when asked if lawyers should be afraid or encouraged by AI said:

“Both. It is wise to embrace it now so that it can be a tool as opposed to an impediment. No one wants to be competing against Watson," Lingos said, referring to IBM's cognitive computer system.

"But if you can't beat 'em, join 'em!"

The Future Work article discusses more than just using machine learning (a sub science of AI) to reduce the number of documents needing human review. The article also explores other areas where AI is beginning to creep into traditional corporate legal work such as contracts, wills, real-estate and general, repetitive legal work.. AI allows legal teams to do more interesting work they were trained to do and “automate” much of the drudge work. Lawyers didn’t necessarily go to law school to spend their first 5 years in document reviews, so you would think that the legal community would be flocking to use this technology. This however, has not been the case.

To understand how AI is indeed changing how we manage ESI specifically, we need to understand the context.

Image Credit: TheMerkie.com

AI and Document Review
So why aren’t more lawyers adopting this game changing technology? Truthfully, the burden of explaining how machine learning works rests squarely on the shoulders of services providers who create, deliver and operate this technology. Too often the discussion has been lost in endless technical debates about whose product is the best on the market. What follows is our attempt at an intentionally oversimplified explanation of how machine learning functions in document review.

Five Stage Process

AI is not going to replace lawyers anytime soon. In fact, think of the technology in the context of Electronically Stored Information (ESI) as a sophisticated search and find tool. How technology performs is much more sophisticated in “classifying” and retrieving information than a simple search or near dupe identification. A good sales approach by a service provider can convince any end-user that their technology is the best. However, in their simplest form, all technology is reading words and phrases, and classifying documents by creating unique conceptional buckets and then placing related documents into those conceptual buckets. Humans then teach the technology which buckets are relevant or not relevant by reviewing samples presented to them by the technology. For a more detailed 5 step process involving search term validation, see our five part series –Combining Predictive Coding and Search Term Classification in 5 Easy Step.

Stage 1 – Judgmental Seeds

A typical document collection usually contains a very low percentage of relevant documents. Usually between 1 - 5 out of every 100 documents reviewed will be relevant. We are, in most cases, looking for the needle in a haystack. So, the first step in teaching the machine what is relevant is to increase the odds that relevant examples will be selected for consideration. We provide the technology with "seeds". These seeds can be examples of known relevant and not relevant documents. Or, more commonly provide a list of search terms that will help find a larger number of relevant documents - roughly 20 or 30 out of 100. Key terms do not have to be tested, and if search terms are being exchanged with the opposition, the technology will help validate those terms by deploying our 5 Step Processes referenced above.

Stage 2 - Training and Uncertainty Sampling

Reviewers tag each document in the seed set as relevant or not relevant (if not already known). It is really that simple. Lawyers will then review documents the traditional way they have always conducted document review.

Stage 3 – Predictive Analysis
The algorithms within the software analyze the human decisions from Stage 1 and offers predictions based on comparing the content from the seed documents to other documents with similar content. The software will then classify each document available into one of three buckets:

Potentially Relevant
Potentially Not Relevant
Uncertain - reviewers did not review any samples like these

If active or continuous active learning workflows are being used, this predictive analysis is performed by the software continuously, or near continuously. It is important to know whether your software is using “passive” or “active” methods and algorithms.

Stage 4 – Sample, Train and Stabilize

If you’re using active learning technology, the software will feed examples of documents it thinks are relevant, not relevant and “uncertain” documents to the reviewer. The uncertainty sampling is critical. With software that uses only passive learning there are many more steps that need to take place. Therefore, before you engage machine learning, make sure you know whether you have active learning available to you. With passive learning workflows, while the end result will be the same, it will take much longer to get there.

As the iterative review of training samples continues, the software should provide stabilization indicators that allow those administering the review the ability to determine when the machine learning has stabilized. Different software packages measure stabilization differently, so it is important to establish your stabilization goal. For example, iControl ESI’s ENVIZE solution, has its own stabilization indicators that tell us how often the human agrees or disagrees with the software predictions as the primary, but not exclusive indicator, of when stabilization has occurred and the software is not gaining anymore knowledge about what is relevant.

Stage 5 – Complete Review

Once -stabilization has occurred and the software has a good idea what is or is not relevant, you can then proceed with review. With some software, you can continuously review only the most highly ranked relevant documents at any stage in the process. This is often referred to as “Continuous Active Learning” or CAL.

Conclusion

Predictive Coding is growing rapidly. While it indeed uses machine learning, which is a subset of AI science, we are not deploying sentient computerized beings that think for themselves. The AI we use today to reduce document review does not decide what is relevant on its own. The technology must be taught how to perform correctly. That said, it is entirely possible that perhaps one day Watson will replace living, breathing and thinking lawyers, but that is the reality of today. At present, the technology is continuing to mature and the subset of AI we prefer to refer to as Machine Learning, is the future, and it is a very powerful tool to lawyers and legal staff.
Tags: #machinelearning, #predictiveanalytics, #ediscovery , #predictivecoding, #bigdata, #AI

2017-06-12T13:59:00.000-07:00

How do I prove that TAR makes sense?

By Mark Walker, VP Advisory Services at iControl ESI

Please post a response with your thoughts, especially if you disagree with any of this, or I get anything wrong. This post is intended to prompt discussion on this topic.

Introduction

When does it make sense to use a TAR workflow? Those of us that work with predictive analytics (a/k/a predictive coding) and TAR workflows have been asked this question more times than we can count. The answer is usually the unpopular “it depends” response. At the end of the day, there should be a cost vs. benefit math exercise. However, non-monetary factors can also impact the decision. The time allotted to get the production out the door, resources available, and the budget are also factors. Even case strategy can factor into the equation. There are a lot of variables. Virtually everyone agrees that in most cases we simply cannot review everything. Most just resort to using date, file type and search term filters. We can do better.

Those of us that have been using TAR workflows for years know that a well-planned TAR workflow using machine learning (preferably active learning) will save both time and money. We know that using the right technology is highly accurate when based upon sound sampling methods where humans teach the technology to find what they seek. But how do we prove it to someone who has never traveled that road? Lawyers are all about proof. That’s what they do. We have a tough audience..

Defining the Problem

A few weeks ago, I reconnected with a LitSupport manager at a major law firm. He has been in the industry a very long time and closely follows the most cutting-edge technology. As a LitSup manager, he has had success convincing lawyers within his firm to use TAR workflows. Well, some of them. This time, I asked him the dreaded question, but in a slightly difference way – “What kind of cases should your lawyers consider using predictive analytics.” His answer, tongue in cheek: “Every Case!” We both got a good chuckle out of that answer. While we chuckled, he is exactly right. But, like everyone else in the industry, he is also frustrated with the industry as a whole’s inability to make the argument in a way that resonates with lawyers. Some use fear to convince – if you don’t do it, others will. Lawyers like litmus tests. Bright lines. They don’t like grey. Lawyers don’t react well to threats and attempts to invoke fear.

When reviewing documents, lawyers want documents that are relevant. Sure, good lawyers are concerned about cost and one would think would be interested in anything that will make them more efficient. But, they are also concerned about risk and trust.

Here’s the root of the problem: Relevancy rates in collected documents are often as low as 1% in most cases. That means 99 out of every 100 documents collected have no value. Sure, there are exceptions, but it is rare that a document review relevancy rate is above 50% using traditional search and review work flows. No matter how you cut it, when 50% of what you review (best case) is wasted effort, there is an expensive problem that needs to be solved. By the way, a search and review workflow that achieves a 50% reduction and relevancy rate is a phenomenal achievement. We traditionally see closer to 30% without leveraging a TAR workflow. We can do better! We must get as close to the 1% we seek as possible.

Using a document count litmus test to determine whether to use predictive analytics doesn’t work. For example, “use predictive analytics when you have 10,000 documents to review”. The average single custodian (witness) has on average 10,000 documents collected. If 1% of that is what we expect to be relevant, then out of 10,000 documents your seeking 100 that are relevant. There are too many other factors that might make it more cost effective to just review the 10,000 documents. Document count is not the right litmus test.

Solving the Problem - Do the math

Using our 10,000 document, single custodian example, we arrive at a conservative 50% relevancy rate litmus test. That is, if you expect that whatever method you use to filter down before review will yield less than a 50% relevancy rate during review, then it makes sense for you to deploy TRUSTED predictive analytics technology to your review, often in conjunction with validating search terms to exchange with the opposition. See Combining Search and Predictive Coding in 5 Easy Steps. While you can’t really know for certain what the actual relevancy rate will be up front, obviously, we can usually have a pretty good idea if it’s going to be above 50%.

In our 10,000-document example using a traditional filter, search and review methods, one might cut the review in half and only review 5,000 documents. At a billing rate of $250 per hour, and a typical review rate of 55 docs per hour, the cost to review 5,000 documents is $22,727.27. $250 an hour is low compared to the market rate for associates. Make your estimates conservative.

If predictive analytics rate is $0.06 per document, the cost to classify with predictive analytics the 10,000 documents available for review is $600. All other technology costs such as processing and hosting will be incurred no matter what review method you chose.

Leveraging predictive analytics, you should typically see an 80% or above relevancy rate during review. If you only achieve 50% using traditional search and review, then spending $600 on analytics achieves at least 30% improvement, which is very conservative. Therefore, in this very conservative example you reduce the review by 1,500 documents and avoid 27.7 hours of review time. At $250 per hour, that’s $6,818.18 of review cost avoided. Since the analytics cost just $600, the net savings is $6,218.18. How can anyone ignore that advantage?

Ah, naysayers might say, we are going to use contract reviewers at $55 per hour! Even with the dramatically reduced billing rate, there is still a net savings of $900, and don’t discount speed either.

Predictive Analytics is not just for Big cases anymore.

In the example above, we’ve used a very small case - a 10,000 document case hosted in a review platform is, well, rare these days. Many of the cases we deal with are multi-million document cases. 100,000 hosted is common. Using the same modeling as outlined above, the savings achieved on a 100,000-document population is persuasive and undeniable.

At a $250 per hour review rate

At a $55 per hour review rate

Conclusion

With very few exceptions, leveraging a TAR workflow that includes predictive analytics (a/k/a predictive coding) will save considerable time and money. The courts have been encouraging lawyers to leverage technology. Clients are demanding their outside counsel reduce costs. Fixed fee arrangements are becoming common place where lawyers have skin in the game to keep the time they spend on matters low. For contingent fee lawyers, time really is money.

Do the math yourself. Apply whatever assumptions you feel appropriate. Increase document decisions per hour, lower hourly rates, increase the per doc cost of analytics. What you will find is that using even the most extreme and efficient methodology, leveraging predictive analytics simply makes financial sense for everyone involved. Reach out to me and I’ll provide you with a calculator so you can input your own assumptions.

So, what’s keeping you from leveraging predictive analytics? Inquiring minds want to know.

2017-03-06T13:51:00.000-07:00

Parts 4 & 5: Combining Predictive Coding and Search Term Classification in 5 Easy Steps

By Mark G. Walker, VP Advisory Services and

Robin Athlyn Thompson, VP Marketing | Business Development

By popular demand, we are releasing Steps 4 & 5 together. In case you missed Part 1, you can find it here. You can find part 2 here, and part 3 here.

Introduction to Steps 4 & 5.

Steps 4 & 5 are frequently performed in parallel. When available, predictive coding is beneficial in validating key terms.

Step 4: Validate Key Terms Before You Agree to Them

There are those of us who have spent decades developing key term validation protocols, keeping the attorneys involved on task, and hopefully convincing them not to agree to poor key terms. Poor key terms can, and frequently do, return 70%, 80%, even more than 90% documents that have little or no value to the case. Key terms are usually overly broad. In the search-world we call this “over-fitting,” A certain amount of over-fitting is desirable, as you don’t want to be too narrow with key terms as something can be missed. On the other hand, you don’t want to be too broad, because the more you must review, the greater the cost and the more likely it will be that the opposition will fuss about dumping. Not that dumping ever happens in this business! Just like Goldilocks and the three bears, we’re aiming for key terms that are just right.

There are entire protocols and technology features dedicated to validating search terms. Oversimplified, a search term validation process is one that is repeatable and contains quality control measures. Documents hitting a proposed set of search terms are “sampled” and those samples are reviewed and scored.

Key Term	Hits Sampled	Tagged Relevant	% Relevant
Diamond	100	20	20%
Joe	100	10	10%

Imagine a case about a fictional restaurant called Diamond Joe’s. The restaurant chain is owned by the fictional company Diamond Joe Holding. The majority shareholder is the fictional Joe Diamond. Joe owns an interest in many companies, some completely unrelated to the subject of the litigation, the restaurant chain. Joe owns a diamond mine in South Africa – Joe’s Diamond Mines. Joe also owns a chain of jewelry stores in South Texas and Mexico. Finally, Joe owns a minor-league baseball team named, you got it – The Diamondbacks. As you might imagine, searching Joe Diamond’s email collection along with 50 of his employees will yield a great number of “false positives” using the terms diamond and Joe. Of course, that seems obvious in this example, but there are many terms that have multiple meanings and depend on context. Sampling hits of those terms, along with any others you have, will eventually ferret out which terms can be changed by, dropping some terms like Joe and diamond, and/or adding other terms, proximity connectors and other tweaks to existing and new terms. Search term validation protocols are very effective in doubling and even tripling the relevancy rate of documents that you ultimately must review. The cost savings is dramatic because even without leveraging advanced technology outlined in Step 5, far fewer documents are reviewed and of those reviewed; far fewer are of no value.

On large projects, search term validation protocols can be tedious, but are necessary. Your protocol must be repeatable, reportable, and iterative with validation and verification. While sound key term validation protocols get you to the same place, the road is much shorter when you measure key term effectiveness as you conduct your sampling using the advanced analytics and strong key term reporting as outlined in Step 5.

Step 5: Leverage Smart Technology

Before classifying ESI in an analytics engine, perform any additional objective filtering that you can to eliminate ESI that has no value in a text classification engine, or is known to be irrelevant. As previously discussed, audio and video files, image only file formats can often be eliminated from classification. Eliminate ESI that may have survived prior filters, and sometimes can more easily be identified once in the review platform where predictive coding is delivered and available. Establish a separate work flow for files that can’t be classified. If your using the right technology and provider, this will be part of their standard process, but be certain.

Advanced analytics, such as predictive coding or machine learning, is not new. The technology and methods that underlay analytical engines has been in use, well, since computers to run them have existed. In eDiscovery and Information Governance software platforms, predictive coding technology has been available for well over a decade. However, it is only recently that lawyers and judges have truly begun to become comfortable with Predictive Coding technology and associated workflows. Predictive Coding is a large bucket of all types of analytics tools, all of which are useful for different reasons. Here, however, we are focused solely on machine learning. Machine learning (ML) is the sub-field of computer science that gives computers the ability to learn, without being explicitly programmed (Arthur Samuel, 1959). (Samuel, 2000) ML evolved from the study of pattern recognition and the computational learning theory in artificial intelligence. (Encyclopedia Britannica, n.d.) Sounds a bit like rocket science? Well, at its core, technology built on machine learning is full of complex algorithms, equations, hyper-planes and all kinds of complex things that frankly none of us really need to understand. To someone like me, it is rocket science. What we do need to understand is this: ML allows you to review samples of documents, mark them relevant or not relevant, and the technology will classify everything based upon human review of those exemplars. The technology finds everything that is like those documents that are marked as relevant or not relevant. Like any evolving technology, however, you must make sure you have a basic understanding of the technology you intended to use.

Many of the ML engines used for predictive coding today were not originally built for predictive coding. They were in fact built on methodologies and algorithms intended for concept classification analytics and visualization (reporting) of concepts. The clear majority of the predictive coding engines on the market today, are passive learning applications. Passive learning applications classify ESI as a snapshot in time. You then review representative conceptual samples from the target population that are randomly selected by the application you are using. Once the sample is reviewed, the ML engine determines what it thinks is relevant or not relevant based on that snapshot. Many samples are reviewed in this process, and sometimes many re-classifications must occur. Because a passive engine is a static snapshot of the data, samples must be larger in number, and there are many starts and stops as you train the machine to determine what is relevant as opposed to what is not relevant. Like search term validation protocols without ML, with passive ML you get to the same spot down the road as an active learning ML, it just takes you longer to get there. One has to review dramatically more samples and you must have substantial assistance to conduct reclassification and to measure stability.” Stability is that point where you know that the machine has learned all it is going to learn from samples, and it is time to stop training and conduct quality control audits. Determining stabilization in a passive learning based tool can be challenging.

Active learning ML-based technology is different. Active learning engines are usually based upon binary methods and algorithms such as Support Vector Machine (SVM), for example (Saha, Hasan, Burgess, Habib, & Johnson, 2015). Active learning changed the game with respect to speed and efficiency. The biggest advantage to the consumer, is that the engine continually and “actively” reclassifies what is relevant as the sample review is being conducted. With the right active learning engine, this reclassification happens virtually in real time no matter the number of reviewers. Feedback on how you are doing is also immediate and continuous.

So how does ML help with the all-important key term validation? Simple: because the classification engine is classifying all documents in a targeted ESI population, allowing you to grade the effectiveness as you go, you have real-time feedback on search term effectiveness - assuming, of course, that the technology you are using has strong key term hit reporting. With ML you are not limited to just the sample documents that you review. The machine takes what has been reviewed, and then extrapolates that to the entire population of data. Your search term hit report can then provide a relevancy hit rate across all data, not just what has been reviewed. As learning stabilizes, so too do the key terms, allowing you to quickly determine which terms need work. The technology will often suggest terms by showing you those terms that are most common in relevant documents.

Once learning has stabilized, follow a well-established audit sample review to make sure that you agree that the learning has stabilized. It is then time to move on to privilege review and production.

Conclusion

Well-established filtering, key term validation and machine learning workflows are becoming common place and for very good reason – combining the two has proven over and over to save considerable time and money by eliminating ESI that has no value. In our world, time is indeed money.

References

Enclycopedia Britannica. (n.d.). Machine Learning. Retrieved from Britannica: http://www.britannica.com/EBchecked/topic/1116194/machine-learning

National Institutes of Standards and Technoloy. (n.d.). National Software Reference Library. Retrieved from National Software Reference Library: https://www.nist.gov/programs-projects/national-software-reference-library

Saha, T., Hasan, M., Burgess, C., Habib, M., & Johnson, J. (2015). Batch-mode active learning for technology-assisted review. Big Data (Big Data), 2015 IEEE International Conference on (pp. 1134-1143). Santa Clara, California: IEEE.

Samuel, A. (2000). Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research & Development, 44(1/2), 207.

2017-03-01T13:37:00.001-07:00

Part 3: Combining Predictive Coding and Search Term Classification in 5 Easy Steps

By Mark G. Walker, VP Advisory Services and

Robin Athlyn Thompson, VP Marketing | Business Development

This week, part 3 of our 5-part series on combining search term classification and predictive coding. In case you missed Part 1, you can find it here. You can find part 2 here.

Step 3: Process the Good Stuff

Once you’ve eliminated everything that you can objectively eliminate, it’s time to process. Processing is the act of extracting metadata, content, indexing, analyzing and staging ESI for review/production. Some steps, such as indexing content, can be a second or third stage, depending on the service provider’s capabilities. The first stages of ingesting ESI is often referred to as pre-processing. As noted in Step 2, all container files are opened, and individual files are created during processing. Emails and attachments, for example, are pulled from the PST container and presented as individual files rather than a single container file.

Once processing is complete, apply your “objective” filters identified in Step 2 again so that you can identify files coming from containers that can be suppressed from downstream processes.

Unlike prior workflows centered on applying search term filters at this stage, you SHOULD NOT filter by search terms during processing, unless you’re using terms that are validated using a process outlined in Step 4 and will not change going forward. Even those of us expert at developing search terms should remember that using those search terms during processing may result in pulling a large percentage of irrelevant documents. The fact is we can’t be certain how well search terms perform until we perform sample review and testing. At minimum, we encourage you to perform these minimum tasks discussed here.

Finally, as processing extracts domains, we recommend you seek a report of domains present in the ESI and filter-out emails from domains that are clearly junk. Emails from cnn.com, for example, may be clear spam emails. Some processing applications have rudimentary review and tag functions designed precisely for this purpose. Be careful, however, as anything you do in terms of filtering during processing can have a negative impact downstream. Regardless of whether you filter out junk domains during processing, you will want to do that step (again if you did so during processing) once the ESI resides in the review/analysis platform. Here are a few things to consider during processing. This is not intended to be an exhaustive list.

Apply Objective Filters – Apply again any objective filters that where applied during Step 2.
Consider “Pre-Processing” steps – It may dramatically speed up processing to utilize a multi-stage processing work flow. For example, you may not want to extract text and conduct indexing on files that may be filtered out.
Be Careful with Search Terms - Before applying search term filters during processing, consider very carefully the consequences. There are serious ramifications to deduplication, for example, if your search terms change and you receive new data that may apply a different set of terms.
Domain Filters – identify junk domains and eliminate files associated with clearly junk emails.

Stay tuned next week for Part 4: Validate Key Terms Before You Agree to Them

2017-02-22T08:31:00.001-07:00

Part 2: Combining Predictive Coding and Search Term Classification in 5 Easy Steps

By Mark G. Walker, VP Advisory Services and

Robin Athlyn Thompson, VP Marketing | Business Development

This week, part 2 of our 5-part series on combining search term classification and predictive coding. In case you missed Part 1, you can find it here.

Step 2: Dump the Junk

ESI collections include acquisitions of ESI from laptops, 3rd party sites, file servers, wherever users keep potentially relevant ESI resides. In some cases, entire user hard drives are collected. In other cases, just user files are collected. Whatever the collection method, thousands, millions, even billions of files are collected. Experience teaches us that less than 1% of information collected will prove to be valuable to your case. There are an enormous number of collected files that are of no value. Here are three common objective filters that can be applied to eliminate known garbage before you do any downstream indexing, analysis or classification. This is not intended to be an exhaustive list.

De-NIST – NSIT is an acronym for National Institute of Standards and Technology. The National Software Reference Library (National Institutes of Standards and Technology, n.d.) is a sub-project of NIST which collects a master list of known computer applications to help maintain the known list of application and system files. To De-NIST means you use these resources to eliminate what are known application or system files that have no value in most cases.
File Type Filter - Eliminate known file types outside of NIST. In most cases, an inclusive file filter ingests into processing only specific file types of interest. Audio, video, image and other specific file types may be set aside, or not used at all. These file types are very heavy, driving up cost, contain little or no text content and are difficult to analyze, often requiring a different process and workflow. Create a special process for audio/video files that may be relevant. Your eDiscovery budget will thank you.
Date Range Filter – We would urge caution when applying a date filter BEFORE processing. Processing is the act of extracting metadata and content. This process also expands container files such as email archive PSTs and ZIP files. If you apply a date filter before processing, and container files are being processed, you are virtually guaranteed to miss files of interest. By way of example, if you create a PST archive of my email today, it will contain months and even years of email, yet the date of the PST will be today’s date. If your date range filter does not include today’s date, that PST will be eliminated from processing consideration, even though email within the date range are inside the email archive.

Next week: Part 3 "Process the Good Stuff"

2017-02-15T13:18:00.002-07:00

Combining Predictive Coding and Search Term

Classification in 5 Easy Steps

By Mark G. Walker, VP Advisory Services and

Robin Thompson, VP Marketing and Business Development

iControl ESI

This is the first step in a 5 step series...Stay tuned for Step 2 next week.

Introduction

So many of our colleagues in this industry have spent decades persuading lawyers and their staffs to adopt technology and use powerful search and conceptual classification to efficiently and effectively manage eDiscovery projects. Just last week, an attorney referred to predictive coding as a “new thing,” saying “So, Walker, you’ve been hammering home that we should do a better job with search terms, now comes this new thing “predictive coding”– what is this again and why do you recommend we change gears?” That became the writing prompt to distill our advice on this subject to these 5 simple steps to make your team a pro.

Those of us who support leveraging advanced analytics technology, such as predictive coding, are not suggesting a shift of the gears. Well, at least not some of us. Technologists (primarily) have been trying to move lawyers away from the use of key terms as “objective” filters for some time now. A decade or so ago, conceptual search emerged in legal software. In 2017, the most common exchanges of filters in agreement among the parties are date range, file type and search terms. These search terms are tangible and objective things that the parties can exchange, and will behave essentially the same irrespective of what platform is being used by the parties to perform search. For example, a search for the term “diamond” should yield the same number of documents hit whether your platform uses DT Search, Apache Solr, or some other search engine. For better or worse, we’ve made lawyers like search terms. Convincing attorneys to like search terms has taken us decades, so let’s not waste the effort! The use of advanced machine learning technology, reporting, and – yes - math, will make those objective terms better. These steps are not intended to be a comprehensive list of every step and task that should be performed on cases involving Electronically Stored Information (ESI), as there are literally entire books devoted to this topic. Rather, this is a short list of those mandatory tasks that should be performed on virtually any case of any size.

Step 1: Identify the Witnesses, Preserve and Collect the ESI

The process begins with legal hold to make certain that relevant documents/ESI doesn’t disappear. Legal holds can be issued across an organization, or to only those that are anticipated to have relevant information and facts. There are web-based legal hold solutions that will not only automate the process of creating legal hold, but will also help monitor compliance and profile what information the witness(es) might have that is relevant BEFORE you must gather their files.

The exercise of “profiling the data” of a custodian is a great way to determine what a specific witness has on their computer, inside email stores, file shares or where ever relevant ESI may exist. Data profiling applications work by reading metadata to determine file type, size, counts and so on. This information can be very handy in determining the cost of collecting, processing, and reviewing, and helps in forecasting a budget, defending against a potential motion to compel, or seeking protection against overly broad requests - as if that never happens! Step 1 should include the following, at a minimum:

1. Identify witnesses that may have relevant information;

2. Issue a Legal Hold across the organization or specific to witnesses that are known to have relevant information. Many legal hold applications also provide the ability to create, send and track customizable fact based questionnaires;

3. Monitor Compliance with the issued legal hold. We highly recommend using an automated notification and monitoring;

4. Profile the Data of witnesses who may have relevant ESI. Profiling the data will determine what ESI witnesses have, or have access to and will help with precise cost estimates.

5. Preserve and/or Collect ESI. If an organization has "preserve in place” capability, preserve ESI for those witnesses that are expected to have relevant ESI. Preserve in place is the ability to prevent the deletion of email for specific custodians, for example. Some companies use “journaling” as a way to preserve in place. If the ability to preserve in place isn’t available, collect the ESI as quickly as possible. The longer you wait, the more likely it is that relevant information may disappear, increasing the risk of spoliation. The delete key is not your friend! Preserve and collect broad -- collecting ESI is the least expensive part of the process.

Next week we will cover Step 2: Dump the Junk. Your comments and opinions are welcome.

2016-11-10T14:20:00.000-07:00

Why the Pundits Failed to Predict Trump

Contributing bloggers: Susan Kavanagh and Mark Walker

Democrats and Republicans alike are looking back at this election and saying “how did we miss this?” Democrats took great pride in their “analytics,” and much of how Hillary positioned herself to the public was based upon polling and analysis of polling. Obviously, the polling was not accurate, so the entire Democratic strategy based on these analytics was – well -- way off base. In the Trump camp, Donald J. simply ignored the polling, and went with his gut. Turns out Trump’s gut was right and the Democratic pundits were wrong -- really wrong! Here’s what happened.

In order to understand at a very high level how this technology works, some background is necessary. In our world of litigation, we use analytical algorithms that were originally developed for political polling. In fact, the FBI used the very analytics we use here every day to analyze email in connection with the Clinton email investigation. That is called “eDiscovery.”

Those of us in the eDiscovery industry have struggled for years with telling family or friends “not in the business” what it is that we do for a living. Our elevator pitch for our customers, mostly attorneys, goes something like this: “We help pull ‘Electronically Stored Information’ (ESI) from your client’s servers, process it, analyze it and then use analytics so you can decide what to produce, or, if you are the receiving party, decipher what is important. Well, actually, most lawyers just know what eDiscovery is, albeit at a very high level. Actually, most don’t really want to know how the sausage is made. They are simply interested in the result.

Explaining to family and friends what eDiscovery is, is an entirely different matter. Sometimes I just say I do data forensics with stuff like email. “You do what with email?” they ask. Then the explanation would go something like “we collect electronic data from corporations, process that information and then using ’high-tech’ we help the lawyers find what’s important among what is usually stored in a great sea of information.” Most folks don’t really have any point of reference because they almost never deal with lawyers and certainly don’t have anyone looking at their email (or at least they think they don’t).

Enter the Hillary Clinton email scandal. Now we have a point of reference that pretty much everyone has heard of and directly involves eDiscovery, identical to what I do day in and day out. At some point, someone – likely a service provider like the company I work for – collected Hillary’s email from her private email server. That service provider then processed the email. When ESI is processed, information about that data is extracted from those files. A great deal of information is extracted beyond just the text of the email.

So, how is this relevant to the email scandal and the new information from the FBI? Reports vary, but Hillary appears to have produced approximately 30,000 emails to the FBI. The FBI apparently reviewed those, and decided not to recommend prosecution to the DOJ. Then, on Friday October 28, the FBI announces that it has found more email (roughly 650,000) that “might be relevant” to the Clinton email investigation, and that it “needs time” to sort it all out. This email was found on the laptop of a Hillary Clinton adviser in connection with an unrelated matter. We won’t give that unrelated matter any new press here as it is of no consequence. Of particular interest is whether there are any new emails on that laptop that are relevant to the Clinton investigation. In addition, of those that are relevant, are they new or are they just duplicates?

The FBI initially indicated that it might take months to review the data, those of us that live in the eDiscovery world know that this is a small universe of information and with the appropriate technology, should only take about 24 hours to analyze and review. As it turns out, it took the FBI only a few days to determine that there simply wasn’t anything there that changed the recommendation not to prosecute. The FBI didn’t review 650,000 emails. They simply applied analytics.

With that oversimplified explanation of how analytics are used to analyze information in litigation, how does this tell us that the pundits got it wrong? The error has to do with how those analytics were used, not a problem with the underlying algorithms. The algorithms are based upon proven mathematical science that has been used for decades. If you provide the technology with the right inputs, you get the right answer. Just like “garbage in equals garbage out,” bad input equals a bad result. When we first began using analytics, as the FBI did during its email investigation, some of us that grew up in the legal world (as opposed to the technical world) began asking questions about how the math was being applied. Specifically, some of our questions were about sampling sizes. Are we getting the appropriate samples that will let the technology “learn” and model whatever it is we are trying to attain?

ESI Advantage wrote about this problem in May 2012 – “Are your samples Random? Are you just getting random results?”

The problem is very simple in both the legal world and in the world of political polling. It is very easy to get your inputs wrong. The problem has to do with both the sample size and the actual nature of the sample. As explained more fully by reading a Ralph Losey work, or the many posts on ESIAdvantage, the problem is with how sampling is being performed. In political polling, pollsters randomly select potential voters to call and as ask a very short and list of questions that require a “Yes”, “No”, or “Undecided” answer. The “math” tells the pollster how large the sample size needs to be to meet a mathematical margin of error. Based upon the size of the population, the math tells the pollster how many potential voters need to be polled. Pollsters collect additional information about the profiles of those that were interviewed such as race, religion, how they have voted in the past, and so on. That information is input into the technology and modeling is created that should predict who’s leading the polls and the demographic of voters who are likely to vote for one candidate or the other. Those analytics help decide how to message issues directly to those that were polled.

Sample size calculation to meet a specified margin of error – say +/-3% - is based upon a large percentage of those sampled. 20% – 50% will answer Yes or No, with the remaining being categorized as “Undecided”. The composition of the sample is critically important. The sample size in political polling is usually a few thousand out of many millions, so the nature and input of those sampled is critically important. All downstream analytics are based upon those answers. Again, a bad sample equals a bad result.

In the litigation world, we are usually also sampling many millions of “documents”. In the early days, some of us immediately saw the problem: the sampling math doesn’t work. Why? The likelihood of a positive answer from a purely random sample had about a 1% chance of being relevant, if the sampling size of what the algorithm uses relies on a high “richness” of positive answers. In litigation, our “richness” is very low and the sample size calculation doesn’t work and the likelihood of getting relevant documents in your sample is very low. The technology therefore has a much harder time determining what is relevant in the population as a whole because the algorithm is making those predictions based upon the textual content and concepts contained within the documents being sampled. The legal experts have literally spent years debating this problem with the technology experts who design the tools and tweak the algorithms being used to fit our needs. Many argued early on that we need to “stack the deck” and raise our ability to locate relevant documents in our sample so that the math will work. Many technologists debated that it does not fit how the technology is designed. The approach we recommended is now referred to as “judgmental.” We select known relevant and irrelevant samples. Very often we will use search terms that will raise the level of relevant documents in our sample. This allows us to sample the right number of documents to “train” the technology about a relevant document profile, so that the analytics work. Today, the debate over “judgmental” vs “statistically random” sampling is over with just a few technologists still holding out, largely because they are still peddling outdated technology and methods.

Of course, you rely on the answers you get during sampling to be truthful, accurate and know that it doesn’t change until you take a new sample. This is because the person reviewing the sample, usually a lawyer at a very high hourly billing rate, changes their mind about what is relevant as they learn from reviewing. What is relevant can and does sometimes change dramatically. If we are using what’s called an “active learning” approach, the technology adjusts the kinds of samples being presented to the lawyers with each new sample reviewed, determining what it needs to learn based on how the system decides what changes in relevance.

So, what went wrong with the how pundits projected the election? As it turns out, there were many more “undecided” voters that the pollsters predicted and the wrong profiles were being sampled. Many of those polled either didn’t reveal that they were going to vote for Trump, or they simply changed their mind. Like we did in the early days of using analytics in litigation, the pollsters used the wrong methodology, not the wrong technology.

In litigation, most of us have resolved the problem. We were driven to do that sooner rather than later, because if we have a miss, it costs our clients millions of dollars in review costs, because we are sending too many documents deemed relevant to lawyers who are billing by the hour. We have to use the latest technology and do not have the luxury to be wrong in litigation. More importantly, we have to use the right methodology.

In summary, the Democrats were using the wrong consultants, and the wrong approach. In the months and years to come, we hope to see vast improvements in how political polling is conducted.

2016-07-12T13:53:00.000-07:00

TAR - Not Just For Big Data Volume Cases

The events of the last couple of weeks have given me a great real-life example to share with you regarding Technology Assisted Review (TAR). These use-case anecdotes are right in line with our educational program this month providing education with TAR. It’s our duty to continue to educate ourselves on the technology available, and the risks and benefits of its use, and below are two great examples of instances demonstrating that TAR is not only valuable, delivering ROI, in big data volume cases, but in small ones as well.

The use of TAR and its work flows is nearly a common practice (and in fact almost mandatory in BIG data volume cases). Indeed, in our shop, we just completed a large 8.5-million record case where the lawyers reviewed only 6,000 (less than 1%) documents to achieve technology training stabilization. What is stabilization? Stabilization is the point where stability scores tell us that the technology has learned all it is likely going to learn from a sample review. Because of how well TAR worked in that case, we measured over $1.4 M in ACTUAL review cost savings just based upon what TAR indicated would not be relevant documents. The vast majority of what was identified as relevant by this process was produced without review – over 350,000 (a claw back agreement was used to protect any privileged documents produced). There were about 30,000 documents for priority custodians that had to be reviewed before production. The legal team chose to review only what TAR determined as relevant. Precision was measured at 77%. What does that mean? 77% of what the TAR process deemed relevant was in fact relevant, confirmed by human review. This precision rate is very good, and the savings remarkable, right?

Well, that wasn’t the only remarkable thing we learned about TAR this week. I ran into a lawyer at an event a few weeks back and we exchanged greetings. I gave him my business card and told him “call me if you ever need help with eDiscovery.” A week later, my phone rings and the conversation begins “I have your card here, and remember that you said call me if I need help with this “eDiscovery stuff.” He needed help indeed, and fast. He represented a client who has been sued over a trademark issue. They were sitting on the other side of a motion to compel ruling that required them to collect, filter, review and produce in less than two weeks. The attorney had a 3-person staff to get the work done and knew that the normal approach would not meet the deadline and an extension was not available. He asked if I had any idea what he should do. We were looking at a situation most shops would consider a small case with one custodian which traditionally is not a great number of documents. The attorney was from a small firm, with limited resources and budget, and limited time. I decided to advise that we treat this matter as if it were the 8.5-million record case I talked about above, and use TAR and its work flows. I am sharing with you below the steps we took. Again, this feeds directly back to my opening paragraph: Some lawyers today are not familiar with technology, which is one of the primary drivers behind the amendment to the ABA Model Rules of Conduct. In those cases, we use a defined step-by-step process to educate and inform how the process works.

The upshot in this “small” case is that the deadline was met. In fact, we were a day early. Documents reviewed – 650. Documents produced 12,211.

Step 1: Collect Data. Ooooops – we discovered the custodian in this small case had much more data than expected -- more than 300 GB! Finding more data was not conducive to meeting the tight deadline in a standard approach!

Step 2: Filter out all the file types we do not want or need – the lawyer decided to focus on a few very specific file types. Process and deduplicate. Weed out whatever we can by other judgmental means. The result – 210,000 documents remain. OK, that is better than the original collection, but way too much to review!

Step 3: The lawyer indicated he wanted to try using search terms. Result? 28,000 documents came back as hitting the terms, surprising the lawyer. What surprised us even more was that it would take north of 250 hours to review those documents. There was neither the time nor the money to follow that process. What now? Step 4!
Step 4: Enter TAR and EnvizeTM, our machine-learning tool with Active Learning. We will use the initial (completely untested) terms and run analytics just on the 28,000 documents hitting those terms. We create a few “Judgmental” random samples and launch into review/training. No control batch because EnvizeTM doesn’t need one, at least not at this stage.

Step 5: Terms are not bad – about 30% of the training documents reviewed in the first judgmental random sample were actually responsive. That is about what we expect with untested terms and exactly what we hope for to train the technology – a good mix of relevant and not relevant docs. We next created judgmental random samples to start and then use Active Learning to feed the reviewers what EnvizeTM said it needed to learn – that is the beauty of active learning.
Step 6: Stabilization occurred very quickly. Figure 1 above shows the result after 815 documents. At this point, we switch to Continuous Active Learning (CAL) to feed the reviewers highly relevant content – documents that have the highest relevance scores.
Step 7: After just a few hundred CAL docs reviewed, lawyers report that they have become confident that the technology has done its job and ask that we run priv screen and produce. We suggest QC and audits. Lawyer says – not looking for precision, just looking to make sure we are not missing anything and don’t care if we are a bit over inclusive. We ultimately review a random sample of the “left behind”, just to make sure we were not missing anything. We had not.

Step 8: DONE – everybody is happy.

Conclusion? TAR has utility beyond big-data volume cases. Almost any case of any size that has ESI can benefit from using machine learning technology and a sound TAR work flow.

Want to learn more? See the July Webinar replay here.:
TAR: A Peek Inside the Black Box.

2016-05-18T09:20:00.000-07:00

What's inside the Black Box?

A recent study reported that more than half of Fortune 1000 and American Lawyer 200 attorneys noted concern about effectively defending the results of predictive coding. Predictive Coding to many is a black box. Envize^TMis the latest from iControl ESI, publisher of Recenseo. Envize^TMwill change the way you use Predictive Coding. Envize^TMallows you, the user, to see inside the black box and control the process yourself with a UI unlike any on the market. You won’t need a PhD to guide you. Envize^TMis self guided and you can use Recenseo OR your existing review tool.

iControl Intellectual property utilized in this tool is not new. iControl has been using the underlying technology in our own software for several years, only recently giving it a name and Productizing the technology for use by anyone. Envize^TMutilizes either passive or active learning, allowing you to have visibility into exactly where you are in the process and how well the technology is learning what documents you think are important. Envize^TMis based on sound and scientifically scrutinized underlying technology that has been accepted by the academic community. In 2015, the Computer Science department of Indiana and Purdue Universities co-authored and published an academic paper on iControl ESI's methods. This impressive academic paper - Batch-Mode Active Learning for Technology-Assisted Review* - was submitted to, accepted by and presented at the : IEEE Big Data 2015 Industry & Government Conference - Submission: N216. The underlying technology is the work of years of research and testing. Below is an abstract of that academic paper.

"Abstract—In recent years, technology-assisted review (TAR) has become an increasingly important component of the document review process in litigation discovery. This is fueled largely by dramatic growth in data volumes that may be associated with many matters and investigations. Potential review populations frequently exceed several hundred thousands documents, and document counts in the millions are not uncommon. Budgetary and/or time constraints often make a once traditional linear review of these populations impractical, if not impossible—which made “predictive coding” the most discussed TAR approach in recent years. A key challenge in any predictive coding approach is striking the appropriate balance in training the system. The goal is to minimize the time that Subject Matter Experts spend in training the system, while making sure that they perform enough training to achieve acceptable classification performance over the entire review population. Recent research demonstrates that Support Vector Machines (SVM) perform very well in finding a compact, yet effective, training dataset in an iterative fashion using batch-mode active learning. However, this research is limited. Additionally, these efforts have not led to a principled approach for determining the stabilization of the active learning process. In this paper, we propose and compare several batchmode active learning methods which are integrated within SVM learning algorithm. We also propose methods for determining the stabilization of the active learning method. Experimental results on a set of large-scale, real-life legal document collections validate the superiority of our method over the existing methods for this task."

You don’t need a PhD behind the scenes working levers. Envize^TMallows multiple sampling methods and easy setup.

Envize^TMprovides multiple ways to keep score, including our own Envize^TMScore that tells you exactly where you stand at any given moment.

So, What Makes This Different?

Start Training Faster (with or without control set)
Finish Training Faster (with or without control set)
Better handling of rolling population changes
Envize Automated Project Analysis and Recommendations
Better performance measure
Better Review Quality Estimates

Software and Services Since 1999

To Learn More....

Contact: mwalker@icontrolesi.com

2015-05-01T14:08:00.000-07:00

Why Can't Lawyers Go Paperless?

Short Answer: The bates number. Although, there are a variety of other reasons, including the fact that some lawyers simply prefer to print and review paper.

The Paperless Office

There was a time as a naive young man I believed that lawyers would one day "go paperless". The year was 1982. The place - Harris County, Texas District Clerk's Office. Long time District Clerk Ray Hardy and Deputy District Clerk Hank Husky had a dream to go paperless. Back then, the district clerk's office occupied a few floors of the old Harris County courthouse on Fannin Street. That building was literally overrun with paper. I was a young man just starting out as a court clerk and had the honor to serve on the freshly formed "Justice Information Management System" committee (JIMS) on behalf of the 125th District Court. Then young Judge Michael O'Brien was very interested in automation. At that time we had more than 3,500 pending active civil cases in the 125th and we were just one of many courts. We were literally being overrun by paper. Filing cabinets were located in the court room - space was at a premium. Ray Hardy dubbed the solution the "Paperless Office" and the committee undertook what became a more than 30 year mission to find a solution to the problem. While I have long since realized that lawyers will never in my lifetime give up paper, I am pleased to say that Ray Hardy's dream of the "Paperless Office" in Harris County is very close to reality.

“In Harris County, [Ray Hardy is] also known for his crucial role in development of the Justice Information Management System, the automation of the office and his vision of the ‘paperless office’ – a vision that today is very close to reality.” [See remembering Ray Hardy - 1927 - 2013]

The Bates Number

While courts and clerk's offices around the country are moving to electronic filings, we are still a very long way away from being truly paperless. To be truly paperless, evidence must be "produced" in electronic form, and therein lies the rub. While there may be many valid reasons why some lawyers don't want to produce in electronic form, the primary reason is the need to "petrify" a produced document into a format that can be labeled, tracked and exchanged. Accomplished blogger Mike McBride describes this problem best.

“One of the real challenges in moving firms to native production is the Bates Label, so I’ll be interested in seeing how you deal with that. Discussions I’ve had about native production typically start and end with “how would I keep track of which document is which if it’s not labeled?” Unfortunately, that question exposes the larger issue behind it, namely that attorneys are still printing their documents and reviewing them in that format, even when they have a sophisticated tool available to them to do review. When you print out a copy of a document, removing all of the available metadata, the label becomes the only way to truly tie it back to its electronic version. Until we get them to stop printing, I’m afraid native production is still a step too far.” [Master Bates Numbers in E-Discovery | Ball in your Court]

Craig Ball, in his December 2012 post - Master Bates Numbers in E-Discovery - provides the answer that many of us have been "preaching for years".

"...you produce in native forms and simply emboss Bates numbers when the evidence is downgraded to paper or TIFF for use in proceedings. That is, you add Bates numbers only where and when they’re worth the trouble."

Craig even provides language and a protocol that the parties can agree upon. I have used similar protocols for native productions successfully many times, by the way, so it works and saves significant time and money.

Unique Production Identifier (UPI)

a. Other than paper originals, images of paper documents and redacted ESI, no ESI produced in discovery need be converted to a paginated format nor embossed with a Bates number.

b. Each item of ESI (e.g., native file, document image or e-mail message) shall be identified by naming the item to correspond to a Unique Production Identifier according to the following protocol:

i. The first four (4) characters of the filename will reflect a unique alphanumeric designation identifying the party making production;
ii. The next nine (9) characters will be a unique, sequential numeric value assigned to the item by the producing party. This value shall be padded with leading zeroes as needed to preserve its length;

iii. The final five (5) characters are reserved to a sequence beginning with a dash (-) followed by a four digit number reflecting pagination of the item when printed to paper or embossed when converted to an image format for use in proceedings or when attached as exhibits to pleadings.

iv. By way of example, a Microsoft Word document produced by Acme in its native format might be named: ACME000000123.doc. Were the document printed out for use in deposition, page six of the printed item must be embossed with the unique identifier ACME000000123-0006.

Conclusion

While many lawyers are indeed adopting native production formats, we are still a long way from this becoming common practice. The next evolution will be to produce in the "cloud" without the need to even exchange electronic copies, much less wholesale printing to TIFF or PDF. The bates number will live on. I don't expect to see electronic productions become common practice in my lifetime, but I will continue to carry the torch of the "Paperless Office" in honor of Ray Hardy's dream. Never give up, never surrender.

2015-04-20T08:40:00.001-07:00

Do Lawyers Not Get Technology, or Do Technologist Not Get Lawyers?

The short Answer: A little of both. Here's why the answer is complex...or is it?

The Great Transparency Debate

No question that lawyers need to keep up with technological advances that are rapidly changing the practice of law. Comment on ABA Rule 1.1: Competence reads in part:

"To maintain the requisite knowledge and skill, a lawyer should keep abreast of changes in the law and its practice, including the benefits and risks associated with relevant technology, engage in continuing study and education and comply with all continuing legal education requirements to which the lawyer is subject. [emphasis added]"

It is true that some litigators don't understand technology at the level of the technologist. It seems like just yesterday that we had those CRT monitors on our desks and paper documents were everywhere. But, should we expect litigators to gain a deep understanding of technology? Have the technologist in the process of peddling their goods over complicated things by trying to make their special sauce too special?

What litigators do understand is this - the vast majority of all ESI preserved, collected, filtered, reviewed and produced will never see the light of day outside some hosted environment. Most ESI will never even receive a bates number, much less get used at a deposition, in a court filing or in court during trial. In fact, in most cases, less than 1% has any value at all.

“I saw one analysis that concluded that .0074% of the documents produced actually made their way onto the trial exhibit list-less than one document in ten thousand. And for all the thousands of appeals I've evaluated, email appears more rarely as relevant evidence.” DCG Sys., Inc. v. Checkpoint Techs, LLC, 2011 WL 5244356 at *1 (N.D. Cal. Nov. 2, 2011) (quoting Chief Judge Rader)

So, lawyers know that they are looking for but a handful of documents in a vast sea of mostly useless information. Throw in a bunch of technologist, many of which have never darkened the doors of a courtroom in a professional capacity. Add a desire for complete transparency and "court approval" by the technologist, and some lawyers, and you have a great deal of confusion. Much of the concern about transparency is misplaced.

John Tredennick, of Catalyst Repository Systems, just this weekend published an article Using CAL to Solve the Transparency Problem in TAR, that correctly concludes (among other things) that it's not the technology that needs to be defended. Although, John limited his conclusions to Continuous Active Learning (CAL) (a/k/a TAR 2.0) applications. Lawyers and technologist are focused on the wrong part of the TAR story. ESI Advantage here and in the past continues to submit that whether your using TAR 1.0 or 2.0, the technology should not be put on trial, so to speak. Technology is simply a tool.

The Solution

ESI Advantage presented the solution to the misconception that court approval is needed for litigators to use technology back in February 2012 - "Technology Assisted Review ("TAR") - We have been expecting you! What follows is the solution proposed then and still valid today.

_____________________________________________

Some continue to think that courts are going to somehow sign off on “Predictive Coding”.

Given the significant benefits that technology-assisted review can bring to e-discovery from the dual perspectives of quality and cost, expert commentators have asked a key question: Why isn't everyone using it? Of the 11 e-discovery vendors surveyed by Kershaw & Howie:

“[t]he most mentioned reason cited by respondents was uncertainty or fear about whether judges will accept predictive coding.” Kershaw & Howie,Crash or Soar (supra).

For the love of Pete, please stop thinking of this technology as something that the courts are going to “sign-off” on!! They will not. Judge Peck said just that in the aforementioned Forbes article.

What Technology-Assisted Electronic Discovery Teaches Us About The Role Of Humans In Technology.

“Until there is a judicial opinion approving (or even critiquing) the use of predictive coding, counsel will just have to rely on this article as a sign of judicial approval. In my opinion, computer-assisted coding should be used in those cases where it will help “secure the just, speedy, and inexpensive” determination of cases in our e-discovery world.”

The courts have for years encouraged the parties to cooperate and come to agreement on the filtering methods. This attitude of expected cooperation on the discovery process is nothing new. It is not something that the technologist dreamed up. When you read the rules of any state or federal court, you will find agreement and cooperation mentioned frequently. While most “predictive coding” applications are built upon the same basic statistical algorithms and linguistic pattern detection, they don’t all always achieve the same result. This is because that how you conduct the analysis – the workflow - is critical to the end result. TAR is not the silver bullet alone. Lawyers SHOULD NOT have to defend the technology – how the algorithms work, or what the technology does or does not do. Instead, we should focus on the workflow and the end result. The technology being utilized should be transparent. The workflow and end result should be based upon what a human decides is relevant. For us at [ESI Advantage], there are really only two types of filters that need be explained [when transparency is desired].

Objective Filters: Objective filters are essentially filters that can be applied without any “subjective” reasoning. File type and date range are two common examples (not an exhaustive list) of filters applied in most cases. These filters should not be considered work product. Objective filters are often disclosed and agreed upon between the parties.

Subjective Filters: Subjective judgment is applied to document decisions in most cases. When a lawyer reviews a document, the “mental impressions” that lawyer makes is generally considered work product. In the old days, we did not disclose how many “boxes” we reviewed to arrive at a production that may only be a few documents, or a few boxes out of hundreds. We did not disclose what file cabinets were opened, or who was interviewed. Then as is the case today, that production would occur in one of two forms (generally speaking). Documents are either produced in their “usual course of business”, or “by request.” In the “old days” we avoided producing “by request” as if it were the plague.

Search terms today are often used, [and will be continued to be used for some time to come...] A term is indeed an objective filter, but you may arrive at the use of a term using subjective factors. You may even use “predictive” technology that, in its simplest form, looks for common words, concepts and patterns. Either a document contains that word (or some variation) or it does not. How you arrived at the term, however, may well be subjective and protected work product (some lawyer will make that call). After all, attorneys are officers of the court and when responses are provided, the attorney for each party is representing that a diligent search has been conducted and that there has been compliance with the requests, or the appropriate objections have been lodged.

Conclusion

Nothing has changed since that February 2012 post, except that we would have expected lawyers to have adopted TAR methods in greater numbers than we've seen over the past several years. In our humble opinion, the pure and simple reason is that technologist and litigators are not always on the same page about what is important. As outlined above, we need to be focused less upon how the sausage is made (the math behind TAR) and more on how the sausage tastes. The parties should agree upon filters that they can exchange such as validated search terms, date and file type filters. Whether your using a TAR 1.0, 2.0 or the yet to come 3.0 method is of little consequence to the courts. The courts simply want everyone to get along.

2015-04-14T06:49:00.001-07:00

Why the Cloud is Cool!

Introduction

According to Wikipedia, "Cloud computing is a computing term or metaphor that evolved in the late 2000s, based on utility and consumption of computer resources. Cloud computing involves deploying groups of remote servers and software networks that allow different kinds of data sources be uploaded for real time processing to generate computing results without the need to store processed data on the cloud. Clouds can be classified as public, private or hybrid.^[1]^[2]"

Many use Cloud computing every day without even realizing it. Examples include web-hosted email (such as Gmail) and social media sites (such as LinkedIn and Facebook). For Cloud-based email, the email application and the actual messages are hosted on servers sometimes across the country from the user. Users log-in via a web browser or PDA app, and email is displayed on the device but the user's reply, send or forward commands are executed on a server, or several servers across town, or across the country.

In my world - the legal field - lawyers are literally being overrun with data - Electronically Stored Information (ESI). Legal software has historically followed the traditional software model: lawyers purchase a software license and install on individual computers or on servers in the law firm's data center. Hardware and software following the traditional model can be expensive.

More recently, a new software model has emerged: Software as a Service (or "SaaS"). SaaS is different from traditional software. Software is no longer installed on individual computers or the firm's server. Software is accessed via a web browser over the Internet. Data is transferred encrypted and stored securely in the service provider's data center. Updates are rolled out by the service provider or software company continually. SaaS is usually sold on a subscription basis, usually for a flat monthly or annual fee.

Here are just a few reasons why the legal community should give SaaS and Cloud Computing a serious look.

Spend Less and Do More

With no servers to maintain, or storage to add, hardware costs are reduced. Software costs are also reduced because with most ESI software cost models, you only pay for what you use. While it is critically important for law firms and corporate legal departments to have some level of support staff, staffing needs are obviously less because there are fewer things to manage from an IT perspective. Your technical staff spends more time with the higher level functions of helping you learn and use technology and less time managing hardware, software, security and infrastructure.

You Get the Good Stuff Sooner

Because the software provider only needs to update it's own environment, and not thousands of installs across the country, you receive the latest and greatest updates as soon as the provider makes them available. No more waiting on the firm's IT department to test and then execute a roll-out plan. By the way, roll-out plans are critical, so this should not be a knock on IT departments. Lord knows this old war horse has been through countless upgrades and roll-outs and knows the importance. Something always goes wrong - 100% of the time.

One Neck to Choke

The last thing you need is to choke your own neck. Since it is the SaaS provider conducting roll-outs on their own servers and not yours, it is up to the SaaS provider to make sure that the roll-out goes smoothly. When it doesn't, you have one neck to choke. Most software contracts have guaranteed up time provisions.

Your Compatible

Traditional legal software is often limited by hardware compatibility, operating system, and licensing constraints. For example, traditional ESI software may only be compatible with a PC running Windows XP or later. There are usually also certain hardware requirements that must be met, and then updated when the provider issues new software. It is not uncommon for users to have to upgrade, as an example, RAM to take advantage of new features. Because SaaS solutions are web-based, they are usually compatible across platforms and web browsers. If the hardware requirements change with a new release, it's the SaaS provider's responsibility to make sure your compatible and not your IT department.

Your Data is Safe - Guaranteed

Lawyers are understandably concerned about security. However, a legitimate argument can and has been made, that files stored on a provider's servers are more secure than those located on a typical attorney's PC or in a law firm's data center. SaaS service providers often employ elaborate security measures and multiple redundant backups in their data centers. Obviously ask a lot of questions about a service provider's security and make sure you get assurances in writing. If there is a breach, and you've done your due diligence, you've got a neck to choke and it is the provider that is responsible for the cure, whatever that might be.

Conclusion

There are a great number of additional considerations when deciding whether a SaaS based model is right for you and these are but a few. There are many considerations not addressed here. Get help from an expert. Ask questions, get comfortable with your choice, but continuing to ignore cloud based solutions is without question a mistake. Don't get left behind.

Are your samples random, or are you just getting random results? (Revisited)

2015-04-06T07:07:00.000-07:00

This is the second in a series of posts where we will revisit ESI Advantage articles and posts from the past. We will update and critique in places. As always, your input and comments are welcome. This post is from May 13, 2012.

Introduction

There has been much discussion of late about a variety of "emerging" technologies. Predictive coding, err um, Technology Assisted Review (TAR), statistical random sampling (SRS), what some algorithm does or does not do, complete with panels of experts to explain this that or the other. Entire days of testimony are being devoted to peeking behind the curtain of “predictive coding” algorithms. The Digital Advantage continues to ask, why all the fuss? Shouldn't we be focused on the merits some say? Why all this math? After all, most lawyers will tell you they are lawyers today because of the supposed lack of math, but we digress.

Comment: Since this article, Continuous Active Learning (CAL), or TAR 2.0, has emerged as sound option over previous methodologies, but only in some cases. Prior to the emergence of CAL, Simple Passive Learning (SPL) and Simple Active Learning (SAL), are typically associated with early approaches to predictive coding, which are often now referred to TAR 1.0.[i]

It’s Just Math

Ralph Losey in his most recent treatise on random sampling is quite the read. Ralph predicts (trumpets sound)…

“….in the year 2022 a random sample polling of American lawyers will show that 20% of the lawyers in fact use random sampling in their legal practice. I make this prediction with an 95% confidence interval and an error rate of only 2%. I even predict how the growth will develop in a year by year basis, although my confidence in this detail is lower.” (Random Sample Calculations And My Prediction That 300,000 Lawyers Will Be Using Random Sampling By 2022)

Ralph’s prediction, of course, is tongue-in-cheek. Well, sort of. The math behind sampling is serious. When you consider that unlike the broader world where statistical models are intended in studies or polls, in the document world the all important base line changes from one matter to the next. Put another way, statistical modeling is only as good as the information you feed into it and any assumptions that you apply. By way of example, Ralph’s conclusion that 300,000 lawyers will utilize random sampling by 2022 may be flawed.

Assuming that by the year 2022 there are 1.5 Million lawyers (the ABA estimated there were 1,128,729 resident, active lawyers in 2006), I predict that 300,000 lawyers in the U.S. will be using random sampling by 2022. The confidence interval of 2% by which I qualified my prediction means that the range will be between 18% and 22%, which means between 270,000 lawyers and 330,000 lawyers. I have a 95% level of confidence in my prediction, which means there is a 5% chance I could be way wrong, that there could be fewer than 270,000 using random sampling, or more than 330,000. (Id.)

Flawed you say? The equation for determining sample size based upon known and desired factors such as population size; tolerable margin of error and other factors are well settled. No, the math is not questionable. It works. We have seen it over and over. Trust those math folks. Be careful with your inputs.

In terms of the numbers you selected above, the sample size n and margin of error E are given by

x = Z(c/100)2r(100-r)

n = N x/((N-1)E2 + x)

E = Sqrt[(N - n)x/n(N-1)]

where N is the population size, r is the fraction of responses that you are interested in, and Z(c/100) is the critical value for the confidence level c.

Rather, the math only works if your inputs and assumptions are sound. Here, not all 1.5 million lawyers Ralph assumes will exist in 2022 are litigators. We don’t expect that real-estate and tax lawyers will be utilizing random sampling related to document review. Those contract lawyers are unlikely to be interested in sampling as well. So, the population Ralph starts with may be far less than 1.5 million. Although, we have not audited Ralph’s results and that is not really the point. Here, the result would be sampling more than you need, which is not a bad thing necessarily. The result would be better. But, that simply re-enforces Ralph’s over arching point and ours here – it is not precise statistics that are important. Statistical sampling is a tool among many other reinforcing tools. You don’t have to be a Ralph Losey type lawyer and gain an understanding of statistical sampling (the underlying math), or hire an expert to explain it to a judge or jury. Sample size is important so that you are gathering enough inputs and that those inputs carry the least amount of risk you are going to miss important information. The process used should measure objective information. Results should be validated and audited, so getting a precise sample size is not as important as using some rule of thumb that is repeatable. Statistical sampling is simply a method by with you are organizing documents upon which to gather what to tell the machine. When you consider that less than 1% of all documents that have any value at trial, reviewing everything simply is not possible, nor necessary in virtually all cases.

“I saw one analysis that concluded that .0074% of the documents produced actually made their way onto the trial exhibit list-less than one document in ten thousand. And for all the thousands of appeals I’ve evaluated, email appears more rarely as relevant evidence.” DCG Sys., Inc. v. Checkpoint Techs, LLC, 2011 WL 5244356 at *1 (N.D. Cal. Nov. 2, 2011) (quoting Chief Judge Rader)

Follow a Simple Process

Unlike the use cases for which random sampling models were built, in the document review and production world, we are not shooting in the dark. While it is true that in almost all cases the relevant material is very small in proportion to the amount of material available for analysis, we have a pretty good idea what words and phrases appear in relevant documents, providing at least a start. The subject matter is known. Filter parameters can be narrowed by date, authors and recipients and any number of other known factors. In the old days – those days before technology – we just knew where to look. Today is no different except that we now have technology to help us. Technology helps us increase the odds in our favor. Audits will identify new terms, phrases and concepts for the technology to use to find new sources. Sampling is not so random.

It is becoming common place to agree upon and use search terms, often without any testing or validation of those terms what-so-ever. Wouldn’t it be important to know for certain, say with a 2% chance of error, that term you chose would return relevant documents? Don’t you want to know what percentage of your effort will be wasted if you review all documents hitting a specific term? Why not take a “statistical” sample of all documents hitting that term and measure the relevancy rate? You don’t need to prove what’s statistically appropriate, there are ample calculators that will “do the math” for you. The math has been proven. See sample size calculator by RAOsoft.com and Ralph Losey’s “worst case scenario” sample predictor. Using statistical sampling calculators inside a well reasoned process to, as an example, test the validity and recall rates of terms that are being contemplated is not something that should have to be defended. You are simply using a calculator to help you find the best possible samples upon which to measure a term(s) effectiveness. Ultimately, it is the term (along with other objective filter parameters) that are agreed upon, not what constitutes a sound statistical sample. In other words, the result and the confidence in that result, not necessarily how that sausage was made. Humans, not machines, are deciding what terms, topics and concepts are relevant. The technology simply finds documents with content similar to that which a human has decided as relevant. That’s why some call this emerging technology “machine learning”.

Today, agreeing upon a set of term or phrases remains the only reliable objective filter that can be agreed upon and easily transferred from one technology to the next. Terms that are validated utilizing a repeatable and quantifiable methodology is going to make it much easier to defend choice of terms. And oh by the way, these are not things about which we are guessing. Don’t guess, get help.

Comment: Since this article appeared in ESI Advantage, two significant studies have been the topic of many commentators. These studies codify our conclusions in two ways. First, the vast majority of discovery costs are document review (Rand Report). Among the many conclusions in both the Rand report and The Crossman/Carmack study, is that a hybrid approach to document review is the favored approach – Judgmental sampling using validated search terms coupled with Technology Assisted Review (TAR).[ii]
____________________________________________________________

[i] “Continuous Active Learning for Technology Assisted Review – How it Works and Why it Matters for E-Discovery”. (John Tredennick; August 13, 2014)

[ii] “Where the Money Goes – Understanding Litigant Expenditures for Producing Electronic Discovery” (Rand Institute For Civil Justice; by Nicholas M. Pace and Laura Zakaras; 2012)

“Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery” (Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH. 11 (2011))

2015-02-17T11:26:00.001-07:00

TAR - Is Your Seed Sound?

This is a republish of a TDA post from June 2012 and has been slightly revised. Not much has changed over the past 3 years, except that conventional wisdom now is that seed sets do not have to be created by "subject matter experts". See Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery, SIGIR’14, July 6–11, 2014 (The Grossman Cormack Study). Indeed, The Grossman Cormack study validates what many commentators have been writing for years - purely random sampling is rife with risk and error. A combination of judgmental (using search terms) and random sampling to create seed sets is the superior approach.

Those that are using Technology Assisted Review (“TAR”) already know that the technology is sound. As The Digital Advantage (TDA) has written before, it is not the technology we should question necessarily. Rather, success relies upon how the technology is used and what one does with the results. Most applications deploying TAR use sophisticated “find similar” algorithms to compare content patterns of one group of documents to another. Using sampling techniques, small groups of documents are reviewed by investigators/reviewers and QC’d by subject matter experts. The results are then compared to the larger corpus by the technology. The corpus is defined by the investigator, whoever that might be. The technology then ranks each document in the corpus against the small reviewed sample. Some have referred to this as a Google ranking of sorts. This small sample is generally referred to as a “Seed”. There may be many seeds used in large projects. Seeding, by the way, is a technique that has been used for just about as long as we have had search and review technology. What’s different today? The technology has gotten much better both in terms of sophistication as well as our ability to define and implement workflows.

So, the seed is critical right? Bad seed, bad result. And when we say the quality of the seed is important, we are not just talking about the quality of the seed reviewed and QC’d by the subject matter expert. The make-up of the seed can make or break the result in terms of quality and/or time. TAR technology and sampling methods are based upon random sampling a percentage of the target population of documents. Traditional sampling methods assume a high relevancy content within the overall population being investigated. The lower the relevant content, the larger your sample size should be, technically. For example, some sampling calculations assume that 50% of the sampled population is relevant. A relevancy rate that high in our world is seldom the case. Relevancy rates below 1% are common and rarely over 10%. A purely random sample of 1,000,000 universe, as an example, is very unlikely to yield very many relevant documents using a purely random approach. So, your purely random sample is a bit like a shot gun blast in the dark.

Does that mean that we should have less confidence in TAR and sampling mythologies? No, it most certainly does not in our view. Rather, doesn't it make sense to create a better seed and increase the accuracy? Utilizing a proven search term validation methodology so that seeds rich in relevant content are reviewed is the better course. By the way, even a bad seed is superior to the common practice of throwing terms at the wall and seeing what sticks, but we digress.

As TDA has opined many times, outside of date and file type filers, search terms are the only truly transportable objective content filters that the parties can agree upon. Using seeds constructed and reviewed from validated search terms increases dramatically the impact of the seeds and the success of any TAR workflow. Far fewer documents will be reviewed. Do you have sound auditing methodologies? Are you just throwing technology at the problem, or are you using tried and true workflows?

2013-10-11T11:25:00.000-07:00

Parkinson’s First Law: "Work expands to fill the time available for its completion"

British historian Cyril Northcote Parkinson (1909–1993) first coined this proverb in 1955 which came to be known as his most famous “Parkinson's Law”. A more succinct phrasing commonly used is "work expands to fill the time available." The phrase first appeared as the first sentence of a humorous essay published in The Economist in 1955, later reprinted together with other essays in the book Parkinson's Law: The Pursuit of Progress (London, John Murray,1958). He derived the dictum from his extensive experience in the British Civil Service.

Parkinson's First Law points out that people usually take all the time allotted (and frequently more) to accomplish any task.

Parkinson's First Law is applied in many areas of life.

In Project Management, individual tasks with deadlines rarely finish early because the people doing the work expand the work to finish approximately at the end-date. This, of course, is counterproductive to proper planning. In the eDisco arena, planning is frequently a challenge. No matter how many things one has on their plate, they all tend to get done. "If you want something done, give it to a busy person". The perception is that busy people are better at "time management” than those who are not busy. While it may be certainly true that busy people are indeed better at time management, the reality is that they are just doing more and the work is not expanding indefinitely to fill non-busy time.

Parkinson's Ten Laws

There are actually ten (10) Parkinson’s Laws! Here they are. In the coming weeks, we will explore some of these. Which of these do you think apply most to the practice of law and eDisco?

"Parkinson's Law" = Work expands to fill the time available.

"Parkinson's Second Law" = Expenditures rise to meet income.

"Parkinson's Third Law" = Expansion means complexity; and complexity decay.

"Parkinson's Fourth Law" = The number of people in any working group tends to increase regardless of the amount of work to be done.

"Parkinson's Fifth Law" = If there is a way to delay an important decision the good bureaucracy, public or private, will find it.

Parkinson's Sixth Law = The “Law of Science" - The progress of science varies inversely with the number of journals published.

Parkinson's Seventh Law = The “Law of Delay" - Delay is the deadliest form of denial.

"Parkinson's Eighth Law = The “Law of Meetings" - The time spent in a meeting on an item is inversely proportional to its value (up to a limit).

"Parkinson's Ninth Law = The “Law of 1000" - An enterprise employing more than 1000 people becomes a self-perpetuating empire, creating so much internal work that it no longer needs any contact with the outside world.

"Parkinson's Tenth Law = The “Law of Data" - Data expands to fill the space available. Buying more memory encourages the use of more memory-intensive applications. Gordon Moore observed in the mid-1980s that the memory usage of evolving systems tends to double roughly once every 18 months. Evolving systems create more data. More data requires more space.

ESIAdvantage is back!

2013-07-24T08:28:00.002-07:00

ESIAdvantage is Back!

ESIAdvantage (formerly TDA) is back by popular demand! We have changed the blog name. Don't ask why, just go with it! It has been a year since our last post, but we are back tomorrow with a guest post you don't want to miss. Subscribe to get the notice when we post tomorrow morning.

Join us in welcoming ESIAdvantage back!

TAR - Is Your Seed Sound?

2012-06-22T12:18:00.002-07:00

Those that are using Technology Assisted Review (“TAR”) already know that the technology is sound. As The Digital Advantage (TDA) has written before, it is not the technology we should question necessarily. Rather, success relies upon how the technology is used and what one does with the results. Most applications deploying TAR use sophisticated “find similar” algorithms to compare content patterns of one group of documents to another. Using sampling techniques, small groups of documents are reviewed by subject matter experts and then the results are compared to the larger corpus by the technology. The corpus is defined by the investigator, whoever that might be. The technology then ranks each document in the corpus against the small expert reviewed sample. Some have referred to this as a Google ranking of sorts. This small sample is generally referred to as a “Seed”. There may be many seeds used in large projects. Seeding, by the way, is a technique that has been used for just about as long as we have had search and review technology. What’s different today? The technology has gotten much better both in terms of sophistication as well as our ability to define and implement workflows.

So, the seed is critical right? Bad seed, bad result. And when we say the quality of the seed is important, we are not just talking about the quality of the seed review by the expert. The make-up of the seed can make or break the result in terms of quality and/or time. TAR technology and sampling methods are based upon random sampling a percentage of the target population of documents. Traditional sampling methods assume a high relevance content within the overall population being investigated. The lower the relevant content, the larger your sample size should be, technically. For example, some sampling calculations assume that 50% of the sampled population is relevant. A relevancy rate that high in our world is seldom the case. Relevancy rates below 1% are common and rarely over 10%. So, your random sample is a bit like a shot gun blast in the dark.

Does that mean that we should have less confidence in TAR and sampling mythologies? No, it most certainly does not in our view. Rather, doesn’t it make sense to create a better seed and increase the accuracy? By the way, even a bad seed is superior to the common practice of throwing terms at the wall and seeing what sticks, but we digress.

As TDA has opined many times, search terms are the only truly transportable objective content filters that the parties can agree upon. Using seeds constructed and reviewed from validated search terms increases dramatically the impact of the seeds and the success of any TAR workflow. Far fewer documents will be reviewed. Do you have sound auditing methodologies? Are you just throwing technology at the problem, or are you using tried and true workflows?

Deleted Email. Are you looking in the right place?

2012-06-04T14:41:00.000-07:00

Those of us who live in the forensic world know that when you delete an email, it is not really deleted. Not right away. Most, however, don’t know where to look and may be looking in the wrong places.

Let’s start with where one won’t find deleted email. Individual deleted emails usually don’t reside the computer’s hard drive as individual emails. Not as single files. So, if you are conducting a forensic recovery of a hard drive and are looking for individual emails, you might be looking in the wrong place. The operating system keeps track of where files are placed on the hard drive. In the case of email, the hard drive MAY contain a single file containing many emails. Where to look is largely going to depend upon what email system is being used and how that system is configured. For the purposes of this discussion, we will assume the email platform is Microsoft Outlook. In this case, we will use the example of a laptop computer that is configured to manage email within an OST file. An Outlook Storage Table (.ost) file is an offline folder file utilized by Microsoft Outlook. Offline folders make it possible for the user to work offline and then to synchronize changes with the Exchange server the next time they connect.

Another type of file in Outlook is the PST file, which is simply stored on the client or a server other than the Exchange server. A Personal Storage Table, better known as the “PST” (.pst), is an archive file that Microsoft Outlook users use to manage email messages, calendar items and other things normally managed by Outlook. A PST file is usually stored on a user’s hard drive, or on a network share, as opposed to archived email managed by Microsoft Exchange. A PST email archive is usually created by individual users to store email outside of the corporate email environment and circumvent server storage quotas. What is the difference between an OST and PST file? An OST file starts as a mirror image of a user’s folders on the Exchange Server, and works in conjunction with the Exchange Server during synchronization. A set of PST files, on the other hand, is simply a storage location on the hard disk or a server other than the Exchange Server. When a user works offline, that user can add, delete, and modify the contents of an OST file exactly as that user would with a folder on a server. For example, a user can change and move items in the offline Inbox, send messages that are placed in the offline Outbox, and read offline public folders. In the meantime, information on the server is still processing. The user continues to receive new messages in the mailbox while other users can add, delete, and change items in public folders. However, the user is not aware of these changes on the server until a connection to the network is reestablished. It should be noted that while both OST and PST files are commonly referred to as "container" files, they are actually tables containing database entries (email text and metadata) and objects (attachments).

When a user is managing email using an OST or PST and deletes an email, the pointer to that email (and any attachments) are first moved to the “deleted items” folder within the tables. The pointer and the email will remain in that folder until the user “dumps” the deleted file folder. This act simply removes the pointer to that email from the “deleted items” folder within that PST or OST file. The pointer to the table entry containing the information relating to the email is removed and not the information itself (the text, metadata and associated objects). The email and the attachment (objects) will remain within the “slack space” of that OST or PST file until the user “compacts” that file, which will physically remove the information for those records within the tables. Most users don’t know how to perform a compact, so it is likely that any and all deleted email will remain in that file until the entire container file is deleted. There is no individual email file managed outside that OST or PST, so an attempt to recover from the hard drive of the device you are examining will not recover individual emails directly. Rather, what one would recover is, for example, an entire OST or PST file that a user has deleted. It is important to look for and recover entire OST and PST files from the hard drive, by the way, so don’t skip that step. If you do, you will be missing potentially important information.

Recovering email from the slack space of an OST or a PST recovers those orphans and is a step that is routinely missed by most in this business. Most tools and processes don’t look in the right place. Are you missing important evidence by not looking in the right place?

Are your samples Random? Are you just getting random results?

2012-05-13T15:22:00.000-07:00

Introduction

There has been much discussion of late about a variety of "emerging" technologies. Predictive coding, err um, technology assisted review (TAR), statistical random sampling (SRS), what some algorithm does or does not do, complete with panels of experts to explain this that or the other. Entire days of testimony are being devoted to peeking behind the curtain of “predictive coding” algorithms. The Digital Advantage continues to ask, why all the fuss? Shouldn't we be focused on the merits some say? Why all this math? After all, most lawyers will tell you they are lawyers today because of the supposed lack of math, but we digress.

It’s Just Math

Ralph Losey in his most recent treatise on random sampling is quite the read. Ralph predicts (trumpets sound)…

“….in the year 2022 a random sample polling of American lawyers will show that 20% of the lawyers in fact use random sampling in their legal practice. I make this prediction with an 95% confidence interval and an error rate of only 2%. I even predict how the growth will develop in a year by year basis, although my confidence in this detail is lower.” (Random Sample Calculations And My Prediction That 300,000 Lawyers Will Be Using Random Sampling By 2022)

Assuming that by the year 2022 there are 1.5 Million lawyers (the ABA estimated there were 1,128,729 resident, active lawyers in 2006), I predict that 300,000 lawyers in the U.S. will be using random sampling by 2022. The confidence interval of 2% by which I qualified my prediction means that the range will be between 18% and 22%, which means between 270,000 lawyers and 330,000 lawyers. I have a 95% level of confidence in my prediction, which means there is a 5% chance I could be way wrong, that there could be fewer than 270,000 using random sampling, or more than 330,000. (Id.)

In terms of the numbers you selected above, the sample size n and margin of error E are given by

x = Z(c/100)2r(100-r)

n = N x/((N-1)E2 + x)

E = Sqrt[(N - n)x/n(N-1)]

where N is the population size, r is the fraction of responses that you are interested in, and Z(c/100) is the critical value for the confidence level c.

“I saw one analysis that concluded that .0074% of the documents produced actually made their way onto the trial exhibit list-less than one document in ten thousand. And for all the thousands of appeals I’ve evaluated, email appears more rarely as relevant evidence.” DCG Sys., Inc. v. Checkpoint Techs, LLC, 2011 WL 5244356 at *1 (N.D. Cal. Nov. 2, 2011) (quoting Chief Judge Rader)

ESI Gladiator

Steve Susman - Tell His Story

Relativity Fest 2017 Review – Part IIRise of the Machine

Part I was of the R Fest 2017 review – "The Popsicle Moment" - can be found here.

Introduction

My History without Relativity – When you can’t beat them, join them

Place Multiple Bets

Analytics

Rise of the Machine

SVM vs. The World

Conclusion

Relativity Fest 2017 Review – Part I

The Popsicle Moment

Introduction

The Power of Moments

The Magic Castle Hotel

The Magic Popsicle & Free Stuff

How to Create those Magic Popsicle Moments

Conclusion

Do You Agree to Untested Search Terms?

Introduction

Victor Stanley v. Creative Pipe, Inc. (2008)

Key Term Testing

Conclusion

ePizza as a Service – Cloud Explained (OP, IaaS, PaaS and SaaS)

Introduction

Making Pizza

On-Premises - All you can eat

Infrastructure as a Services (IaaS)

Platform as a Service (PaaS)

Software as a Service (SaaS)

Conclusion

Top Adviser warns of out-of-control autonomous robot lawyers

Is AI Replacing Lawyers and their Staff?

Five Stage Process

Stage 2 - Training and Uncertainty Sampling

How do I prove that TAR makes sense?

Introduction

Defining the Problem

Solving the Problem - Do the math

Predictive Analytics is not just for Big cases anymore.

Conclusion

Parts 4 & 5: Combining Predictive Coding and Search Term Classification in 5 Easy Steps

Introduction to Steps 4 & 5.

Step 4: Validate Key Terms Before You Agree to Them

Step 5: Leverage Smart Technology

Conclusion

References

Part 3: Combining Predictive Coding and Search Term Classification in 5 Easy Steps

Step 3: Process the Good Stuff

Part 2: Combining Predictive Coding and Search Term Classification in 5 Easy Steps

Step 2: Dump the Junk

Why the Pundits Failed to Predict Trump

TAR - Not Just For Big Data Volume Cases

What's inside the Black Box?

Why Can't Lawyers Go Paperless?

The Paperless Office

The Bates Number

Conclusion

Do Lawyers Not Get Technology, or Do Technologist Not Get Lawyers?

The Great Transparency Debate

The Solution

Conclusion

Why the Cloud is Cool!

Introduction

Spend Less and Do More

You Get the Good Stuff Sooner

One Neck to Choke

Your Compatible

Your Data is Safe - Guaranteed

Conclusion

Are your samples random, or are you just getting random results? (Revisited)

Introduction

It’s Just Math

Follow a Simple Process

Parkinson’s First Law: "Work expands to fill the time available for its completion"

Parkinson's Ten Laws

ESIAdvantage is back!

ESIAdvantage is Back!

TAR - Is Your Seed Sound?

Relativity Fest 2017 Review – Part II
Rise of the Machine