Critical Thought in Legal, Compliance and Investigative Analytics

Resistance to Predictive Coding: Discretion is Indeed the Better Part of Valor

2012-02-07T12:37:00.000-08:00

In an interesting north o’ the border article citing a field survey done by Symantec at the recent Legal Tech legalpalooza, the following:

In the survey, the most commonly cited reasons for not adopting it[predictive coding], among those familiar with their company’s stance on predictive coding, were: concerns about accuracy (62 per cent); difficult to defend (57 per cent); cost (57 per cent); concerns about privilege/confidentiality (54 per cent), and difficult to understand (53 per cent).

Seven out of 10 respondents said predictive coding technology will go mainstream if it becomes easier to use, more transparent, and less expensive. Only eight per cent disagreed.

Putting aside the validity (and not to mention the irony) of such a “survey”, it does illustrate a reasonable concern that has taken on “elephant in the room” status amidst the drum beat for acceptance.

By that I mean an affirmative condescension over the reasonable fear of just what’s missed when predictive coding is used. The two highest ranking concerns, accuracy and defensibility, really revolve around the issue. Despite the pronouncements to the contrary, in the context of predictive coding, many a practitioner continues to fret: how can I, the lawyer and/or the client, attest to having done a reasonably diligent job identifying and producing documents that are material to the litigation at hand? By using this process, am I at any risk of having not turned over something critical?

That’s the fear that threads through much of the survey’s findings.

And it is a valid concern.

As I have argued previously, random sampling alone simply will not suffice. And it appears that none of the studies done regarding technology-assisted reviews have actually measured the probability that TARs QAed by random sampling have missed important documents.

The common saw made in response to this concern is that random sampling, coupled with some form of judgmental sampling will do the trick.

Easy-peazy-lemon-squeezy.

Not so valid an answer, and if the survey results are any indication, a good part of the crowd having been sourced says it may prove to be an unwise one. Judgmental sampling can often mean nothing more than a fancy way of saying a guess. A guess that has much in common with the guessing that is still done all too often by attorneys when formulating search terms.

To break through to the larger customer base, the industry will have to offer “better”. For transparency, accuracy and defensibility concerns to be allayed, there will need to be better quantification of what is highly relevant, and, based upon the evidence derived from the process itself, a statistical underpinning to the supplemental investigations to unearth any highly relevant materials.

It's Apparently Technology Assisted Review Sweeps Week!

2012-01-31T07:20:00.000-08:00

Predictivus for the rest of us! (or something like that)

Given the opening flood gates, here in no particular order, are links to related media:

Orcatec: http://bit.ly/xpkvfS

Symantec: http://bit.ly/A6UVjY
   http://bit.ly/A2NtbC

Equivio: http://bit.ly/xuclqr

FTI: http://bit.ly/vZ9CeQ

eDJ Group: http://bit.ly/wpSiVC

Autonomy: http://bit.ly/yXxZHQ

Recommind: http://bit.ly/x88p4r

Exterro:    http://bit.ly/zPXZuy

Catalyst:    http://bit.ly/zxwOCH

D4Discovery: http://bit.ly/zPzB6R

Above the Law: http://bit.ly/zr7WWv

LTN: http://bit.ly/xn5aLM

Law.com: http://bit.ly/wqYhaB

Forbes: http://onforb.es/yQJMAl

Metro Corp Counsel: http://bit.ly/yLloqT

TheEdgeRoom: http://bit.ly/zkJ4vG

2012 Legal Tech Prediction: Technology-assisted eDiscovery is the “Now” Meme; Technology-assisted Rational Matter Management is the Next Meme

2012-01-26T07:58:00.000-08:00

Technology-assisted review processes are poised to make inroads this year and this is reflected in the current Legal Tech press releases. It’s the electric car of the e-discovery industry. Hi-tech and more efficient. So does their mainstream adoption signal the end of history for the e-discovery review industry?

Not by a long shot.

Successful TAR adoption will a next wave of necessary changes, this time directly affecting litigators themselves, and necessitating the emergence of -- perhaps finally – an ECA approach that is really about case assessment and not just the narrow subset of e-discovery challenges. Why?

Because successful (mind I said successful) TAR review process will be more complex than existing processes. This means much more involvement by substantive legal personnel as part of the upfront strategy team. The practice of attorneys passing search terms down from Olympus to be executed by lit support personnel will need to go the way of Druid Solstice rituals.

TAR processes will also drive the much earlier crystallization of how litigators analyze the merits of the case and how to articulate how particular facts are relevant. Best practices will require that litigators do much better job of formulating a granular road map of what information is relevant and how it becomes that way – even for first pass review architectures.

The combination of earlier key evidence identification (both party and opposing party documents) and earlier maturation of issue understanding by the litigation team, both necessitated by the best practices of technology-assisted ediscovery, provides circumstances very close to the Goldilocks conditions for a much more fertile evidence-based platform. A platform from which many benefits derive, including better tactical and strategic decisions, more reasonable settlement decisions, safer alternative fee arrangements, and cross litigation cost/benefit analyses.

The Bigger Picture: How Predictive Coding in eDiscovery Can Change Information Search in Big Data

2012-01-15T20:17:00.000-08:00

It’s possible that the processes developing in the e-discovery predictive coding arena have broader implications in the age of “big data”, as it relates to information consumers and search evolution.

Predictive coding as developed in ediscovery to allow users to iteratively get better search results. It does this by allowing the user to tailor “searches” by: having access to the content of each free text item; enabling users to iteratively train predictive modeling algorithms based upon the entire content of the item using robust feedback from the user’s review of search results.

When a predictive coding process is used, pre-determined structured information from each unstructured item in a corpus is extracted and an index is created. Terms, phrases, entities, etc are obtained. Further, in many processes, additional structured information is derived – term frequencies, co-occurrence measures etc. The predictive model algorithm understands the format of the resulting extracted information precisely because each process was developed with the other in mind.

Contrast this to current internet searches. They are essentially Boolean broad-stroke searches. Users must manually screen results. There is no individualized tagging or feedback. The process cannot employ predictive coding analytics against any corpus, because the user does not access to any extracted structured information the contents and metadata of items in the “corpus”. And in fact no corpus exists except as a result of an initial search. Users operate on a query-response basis.

But does internet search have to be this way?

Perhaps not. One can envision a world akin to ediscovery predictive coding, only writ large.

A world where there are agreed upon rules that enable the automatic creation of a structured information extraction output table (like a text extraction XML rule set) for every item presented through the web. What the table consists of would be determined by industry standards.

Users in this world would have not only a search interface but also a consumer-oriented interface to manage the review process.

In this imaginary world, let’s say a user started with a standard query: “What is the likelihood that social security will still exist when I retire in 2040?”

Instead of simply a display page of possible hits, the search result would make available to the user’s predictive “app” the “XML” information for each of the first N hits.

The user would review these items and have the ability to code each as to responsiveness level for his other query, in this case the issue of the future of Social Security viability. The resulting data would indicate which items were on the money, and which were not.

The search engine would not (at least exclusively) use a Boolean search but would upon user initiation upload the review result data, such as an item identifier and the reviewer’s responsive/non-responsive information. This information would be merged with the item’s extraction record. The search engine would then utilize its own matching algorithms to compare hits and non-hits against its immense internal database of extraction results to refine the next iteration of displayed results.

This kind of approach would require a lot of coordination, and changes to the way searches happen, but could help make much more efficient (just like it does in e-discovery) the search process for billions of people.

Weekly GRC and E-discovery Analytics News 1-15-12

2012-01-15T20:07:00.001-08:00

eDiscovery

What Technology-Assisted Electronic Discovery Teaches Us About The Role Of Humans In Technology

http://onforb.es/xSo6r7

Bottom Line Driven Proportional Review

http://bit.ly/ApO5J6

eDJ Survey On Predictive Coding And Debate Over PC-TAR Term

http://bit.ly/A0FY8e

Use of "Predictive Coding" to Limit Cost and Improve Efficiency in Healthcare E-discovery: The Light Is Green, but Proceed with Caution, as appeared in AHLA's Healthcare Liability & Litigation

http://bit.ly/x04EfX

What is predictive coding and can it help me?

http://bit.ly/wb8Lie

Analytics

Data Analytics Can Backfire Without Experts

http://bit.ly/wLEaI8

Peirce: A high-tech revolution for world cities

http://bit.ly/AwpYoX

Ten significant business analytics technology events

http://bit.ly/xWxoky

Big Data, analytics get even bigger, hotter in 2012

http://bit.ly/wHg81f

BI’s Dirty Secrets

http://bit.ly/xIdoNn

Information technology forecasts for 2012

http://bit.ly/zEi1vu

Big Data Analytics: Trends to Watch For in 2012

http://dthin.gs/wkhQK6

Forbes Echoes PostModern EDiscovery Blog On The Problem of Hot Documents in Assisted Reviews

2012-01-09T12:31:00.000-08:00

It's nice it be validated. (A little credit would be even better.)

Forbes just came out with an article about predictive coding. In it, the authors discussed some validation issues related to idiosyncratic highly relevant documents...

Like the ones I discussed as critical to address back in April, Of Dark Swans and Smoking Guns: Confronting Large-Bore Risk in Predictive Coding, as well as The Allegory of The Devil and the Wise Bean Counter: Why Automated Review Testing is Like an Audit, not an Election Poll in June, and Sedona Fumbling Inside the Ten: Quantifying Sampling Judgment Calls in July.

Defensibility will require more than random sampling, that's now not so heretical a notion. What it will take, especially for communicative documents like emails, is not just linguistically-based experts, but also metadata analysis - and a formal data mining model at that. Not just lawyers taking guesses at where to stratify. (How's that any different than having them select search terms?)

This is so because some communications the relevance of some important communications is based tangentially, or not at all on the text, but on other factors. Those have to be identified and quantified, consistent with Valora Technologies' metadata approach, which I mentioned earlier today in a post.

Data Driven: eDiscovery Journal's Pioneering Benchmarking Efforts and Valora Technologies' Sampling Rationale

2012-01-08T19:48:00.000-08:00

I just finished one of the best pieces I’ve seen on the emerging TAR industry. It’s from Barry Murphy at eDiscovery Journal. It’s the first time I’ve seen a description of real relevant robust testing and a candid assessment of results. Also, the findings are consistent with my experience. And given my interest in Kaggle competitions, I appreciated the research effort that was made.

As I have discussed previously, predictive modeling algorithms are not magical. They tend to locate very content similar documents. That’s one reason near-duplicate clustering should be performed before any modeling engine is employed. First, “NDuping” greatly reduces the size of the document sets to be analyzed, (predictive values can be gathered for one document from each near duplicate cluster and applied mechanically to the rest); second, it avoids internal inconsistencies in coding as described in the eDiscovery Journal piece; and, third, it promotes valid random sampling later in the process by ensuring each distinct (in terms of content) document only gets one chance at selection.

Text mining algorithms can be customized so that the predictive “net” successfully collects more varied relevant documents. The problem here is that this generally requires the ad hoc creation of custom rule sets, dictionaries and libraries, to inform the model of relationships between the pertinent terms and concepts. The alternative route, simply using statistical correlations among words and terms without domain input about what’s material, is problematic in e-discovery, and the results can be disappointing. This is so because often it is only bits of language in a document that make it relevant, or it’s in fact a combination of metadata, such as email parties and bits of information that create relevance. Humans do this well, almost unconsciously. But without instruction to the contrary, the machines will look at everything.

So these human enhancements can be embedded in the machine, but it can be a labor intensive process and must be done for each litigation. A practical approach is to include and constrain the tweaking process.

And this leads to an important and overlooked requirement for any TAR process: robust sample testing to uncover missed “important documents”.

An acknowledgement has to be made that in any TAR process, there is the real probability that some relevant documents will be missed. An under-appreciated but very true fact is that most relevant documents are of at best marginal value. There are though always small subsets of relevant documents that matter to actual litigators. It is the possibility that one or more of these has been missed (and this is a double edged sword – smoking guns can point in either direction) that makes people rightfully wary of TAR methods.

So, the solution to the problems mentioned in the eDiscovery Journal blog, as well as the wariness by litigation participants, is targeted robust sampling

Serendipitously, I spoke recently with Sandy Serkes and Tara Lamy from Valora Technologies, located out of Beantown, about their TAR process and during that talk learned that they apply a particular kind of stratified sampling that is consistent with what I believe to be a rational mitigation of the risk of missing important “stuff” in a TAR review.

The key to their stratification process derives in part from their approach and in part from their evidence-based rationale. Their TAR approach generates large sets of content-based and derived metadata. These attributes are key to the stratified selection process. Using a data-driven evidence-based approach, they identify rules corresponding to relevant metadata components of already coded “important” documents. The evidence-based approach assumes, quite reasonably I would argue, that undiscovered important documents will share metadata attributes with already discovered important documents. The derived rules are then applied against the larger TAR “predicted” non-relevant population. The resulting “hit set” is then randomly sampled. The selected samples are reviewed to determine if any previously undiscovered documents exist.

This approach provides, I believe, the best practicable assurance that important documents have been identified.

I also believe that this approach will be borne out by testing involving completely coded documents sets. That’s one reason I put a call out for them recently. Completely coded sets will provide much better metrics on TAR performance, and overcome some current testing limitations that have been noted by Ralph Losey:

“There is one more fundamental thing you need to understand about the TREC tests, indeed all scientific tests, one which I suppose you could also call a secret since so few people seem to know it, and that is, no one, I repeat, no person, ever sat down and looked at all of the 685,592 documents under consideration in 2010 TREC Legal Track interactive tasks. No one has ever looked at all of the documents in any TREC task. No person, much less a team of subject matter experts with three-pass reviews as I discussed in Part One[of Ralph’s blog], has determined the individual relevancy, or not, of all of these documents by which to judge the results of the software assisted reviews. All that happened (and I don’t mean that as a negative connotation), is that a random sample of the 685,592 documents were reviewed by a variety of people.”

Weekly GRC and E-discovery Analytics News 1-8-12

2012-01-08T19:34:00.000-08:00

Text Mining

Research on Key Problems in Text Sentiment Analysis

http://bit.ly/wHMG2q

Sentiment Analysis in Practice

http://web.cs.dal.ca/~yongzhen/publication/paper/ICDM2011_SentimentAnalysisInPracticeTutorial.pdf

Opinion Mining: Exploiting the Sentiment of the Crowd

http://bit.ly/yKzC6F

e-Discovery

The ESI Maven: A New Breed of ESI Practitioner

http://bit.ly/waNPSz

Research Evolves On Predictive Coding-Technology Assisted Review (PC-TAR)

http://bit.ly/xLGEie

How accurate can manual review be?

http://bit.ly/AiqQJC

So let’s get to the heart of this Predictive Coding – Technology Assisted Review thing

http://bit.ly/yOjlIW

Costs Versus Quality In Electronic Discovery Document Reviews

http://bit.ly/zIqo5a

Fraud

Application fraud: enemies at the gate

http://bit.ly/ymdMl9

A Predictive Analytics Arsenal

http://bit.ly/A6obI2

Adaptive case management: Process takes a detour

http://bit.ly/zFeE5L

The Search For Completely Coded Sets - A New Year's Resolution

2011-12-25T17:30:00.000-08:00

The new year should begin the quest for large, completely coded document sets for testing of predictive techniques as well as robust sampling methodology testing.

Here's to a productive year!

Technology Assisted Reviews: What Exactly Do the Studies Measure

2011-12-21T12:44:00.000-08:00

Does anyone know whether there exist any publicly-available reports outlining systematic testing of and results for technology-assisted review solutions where a significant document corpus was comprised of a complete set of manually coded documents that (as a consequence) contained a complete known (to the testers)subset set of relevant documents?

Do any existing results in the e-discovery industry purport to provide metrics as to whether documents were missed, as well as the nature of the missed documents?

If so, are there links to the reports?

Trade Article: Few Canadian Companies Use Fraud Detection Software

2011-12-21T09:35:00.000-08:00

Canadian Underwriter reports on a SAS-sponsored study outlining executive views on insider/outsider fraud threats and the disconnect between risk and advanced detection resourcing.

Slouching Toward Kaggle - LTN Article:E-Discovery Pitches Meet Sabermetrics

2011-12-19T19:43:00.000-08:00

From the article:

"However, in 2011, many e-discovery companies appear to have saved their resources for the more pragmatic E-Discovery Institute project. The latter study is limited to businesses and is expected by organizers to field up to 20 teams. "I'm more interested in the much bigger policy decisions that the judiciary is making," EDI leader and LTN advisory board member Patrick Oot said. "When litigants choose one form of technology over the other, what are the ramifications of it? It's very difficult to apply real science to a loosey-goosey standard like reasonableness."
In addition, the EDI project is sponsored by Oracle, which may give contracts to top-performing teams, event staff said. The test will begin early in 2012 because of personnel changes after its announcement last February, Oot said. Beyond measuring document recall and precision, teams will also try to determine which documents are privileged, a step TREC does not take."

Starting to sound like Kaggle which I pitched recently.

A good idea; though I think there are advantages to having an independent platform for competitions.

Secrets from the e-Discovery Team: Through the Risk Looking Glass

2011-12-19T11:54:00.000-08:00

Ralph Losey, in his recent blog posts (there are Parts I and II) has done the e-discovery world a great service. They are long reads, and reference a report from information scientist William Webber. All are well, well worth the read by anyone doing the risk profile around TAR adoption. The points and issues raised are some of the most germane to automated review evolution that I’ve seen.

Some observations from a risk perspective:

1. Result sampling methodologies have a long way to go. Metrics need to be developed that describe the probability of highly relevant “unlike” documents having been missed by a TAR.

This need is brought into stark relief by this nugget from Ralph Losey’s post:

“There is one more fundamental thing you need to understand about the TREC tests, indeed all scientific tests, one which I suppose you could also call a secret since so few people seem to know it, and that is, no one, I repeat, no person, ever sat down and looked at all of the 685,592 documents under consideration in 2010 TREC Legal Track interactive tasks. No one has ever looked at all of the documents in any TREC task. No person, much less a team of subject matter experts with three-pass reviews as I discussed in Part One, has determined the individual relevancy, or not, of all of these documents by which to judge the results of the software assisted reviews. All that happened (and I don’t mean that as a negative connotation), is that a random sample of the 685,592 documents were reviewed by a variety of people.”

Wow. So apparently no one has quantified what would get missed if the predictive engine was run against the whole set? This type of risk should flash red on the "Litigation Risk Management Dashboard." One missing "hot" document, produced independently can have exponentially large consequences.

Standard random sampling alone doesn’t solve this problem of quantifying the risk of these missed “hot” items. Random sampling results could only be relied upon as a useful metric if all relevant documents have roughly the same import or if all of the relevant documents are comprised of linguistically similar features. Neither fact can be assumed in the general case. See The Allegory of The Devil and the Wise Bean Counter: Why Automated Review Testing is Like an Audit, not an Election Poll

I think the posts’ point about further testing is on the money. What should be the context of that testing?

2. As for the question of how and/or who decides what is relevant, that is a non-technology problem that will require a primarily non-technology solution. I would argue that reasonableness requires the ability to establish the consistent application of an articulable rationale for decision-making on disputed items, whether by committee, crowd sourcing results or autocrat.

Given the likelihood that responsive/non-responsive disagreements are much more likely to occur at the margins of potentially useful information, anything more is navel-gazing. Do we really need to create a sub-industry to calculate how many marginally relevant documents can dance on the edges of a production?

3. The risk of reviewer attentiveness can be reduced, while increasing the effectiveness of predictive engines, by enhancing the reviewer interface and information payload provided by reviewers through the interface.

4. The risk of intra and inter-reviewer inconsistencies among highly similar documents can be greatly reduced or eliminated by the application of near-duplicate cluster assignment to reviewers.

5. For random errors, in addition to the dual assessments described in the second, or in place of it, risk can be mitigated by using predictive modeling engines similar to the ones used for predictive pruning by having them serve up variance warnings either to reviewers themselves or to review supervisors. Similarity clustering scores would alert personnel when a reviewer codes a document inconsistently with already coded same-cluster documents. This variation on crowd sourcing would both: bring to the fore documents about which there is real question as to relevance; and identify reviewer inattention errors. This is a really good application for analytics.

Weekly GRC and E-discovery Analytics News 12-18-11

2011-12-18T20:20:00.000-08:00

FCPA and Fraud Analytics

Anti-Corruption Programs: Is Yours Up To Snuff?

http://bit.ly/vRmL5n

Mortgage schemes resurfacing to plague shaky housing market

http://bit.ly/t5QilB

Drawing the Line Between Fraud and Abuse in PIP Claims

http://bit.ly/t08Z69

e-Discovery

What Do You Want to See From CorpCounsel in 2012?

http://bit.ly/uedr9S

Secrets of Search – Part One

http://bit.ly/tJj2xJ

Secrets of Search – Part II

http://bit.ly/tWNitx

Text Analysis: The next step for eDiscovery, Legacy Information Clean-up and Enterprise Information Archiving – Electronic Discovery

http://bit.ly/tBsRXl

Predictive Analytics

Coding wizards offered X Factor-style competition

http://bbc.in/tsl9yc

Companies Want Better Data Analysis for Decision-making in New Wave of Technology Upgrades, Says KPMG Survey

http://bit.ly/rT2sLB

Who Shall Judge the Judges? When Reviewers Disagree

2011-12-17T15:38:00.000-08:00

Great post in the e-Discovery Team blog that addresses in part the problematic coding phenomenon that is exposed when multiple reviewers look at the same document. Discrepancies arise as to relevance, and often consensus is never achieved even after debate. Note: this is why anarchies don't work.

Sometimes, you need a boss.

I increasingly think that trying to overcome the difficulty inherent in accurately determining relevance requires that the measurement be finer. "Yes" or "No" buries a lot of information that could possibly solve this problem, at least as a practical matter.

I'd be curious to know what the "materiality" level breakdown is for the disputed documents, the non-overlaps as it were.

Are they composed of marginally responsive documents, representative of a class of documents that are routinely missed -- in whole or in part -- in all manner of reviews?

If so, perhaps scaled relevance value tags could be employed to ameliorate the problem. That is, if it is shown that there is less disagreement about the important documents than the trivial ones..

EDiscovery Journal: Predictive Coding Will Not Replace the "Linear Review" in 2012

2011-12-15T12:12:00.000-08:00

Barry Murphy, in a Forbes piece, How Will eDiscovery Impact Businesses In 2012?
on anticipated 2012 e-discovery trends states, inter alia:
"eDJ does not see predictive coding gaining widespread adoption as a replacement for linear review in 2012, though we are beginning to hear stories of this occurring."

Do practitioners/vendors view this as an accurate statement?

Feds Charge Former Siemens Executives With Bribery

2011-12-13T13:16:00.000-08:00

Just in from Forbes.

A Holiday Editorial on Getting Litigants to Play Nicely: "No, Virginia, There is No Cooperation Fairy."

2011-12-11T21:31:00.000-08:00

It’s safe to say that judicial urging and the 2009 Sedona Conference “proclamation” calling for cooperation in the discovery process apparently has not had quite the effect of a Mafia Commission edict.

Surprising?

Litigators are adversarial. Trial attorneys more so. Civil litigation often involves issues that are complicated. Many times parties cannot be sketched in black or white. Charcoals are needed.

And yet, litigation lawyers, often unconsciously, adopt a “there’s us, and there’s them” outlook, adopting their clients, and saying things like “we didn’t do anything wrong” when discussing the actions of their clients.

It’s stitched into the litigation fabric and into attorneys. Not to mention the fundamental mandate about zealous advocacy.

So, how realistic is it to expect attorneys to compartmentalize in such a way?

Sure, in some cases real cooperation does—and will-- happen. But as someone more cynical than myself might point out, those cases probably never should’ve gotten to a judge anyway.

But for the greater number of Hatfield and McCoy litigations (and given the lack of civility in legal practice more and more mundane fare takes on the appearance of a blood feud), will judicial or policy group proselytizing alone prompt any measure of voluntary compliance?

True, I concede, there is the apocryphal Christmas Truce during WWI. A better lesson for the judiciary probably comes from Cnut. One can’t simply order back the tide.

Asking lawyers to -- from their perspective and possibly that of their clients -- forego their deep-rooted sense of responsibility to the best interest of their clients, not to mention the likely reaction of clients themselves to their lawyers making concessions, is proving to be too much. There is to some extent, as they say in business compliance circles, a misalignment of incentives. And when this happens, best laid plans “gang aft agley”.

A fundamentally different approach has to be considered. Think honest broker.

Now, the holiday is upon us, and it’s a natural time to sentimentalize… to wax hopeful. But as Franklin Pierce Adams of the Algonquin Roundtable observed about December 26^th: “Christmas is over, and business is business.”

So with the end of the holidays soon upon us, let me bluntly argue for the end of the age of innocence, for the acceptance of a certain reality, and invite a discussion on addressing the problem differently:

“No Virginia, there is no cooperation fairy.”

Weekly GRC and E-discovery Analytics News 12-11-11

2011-12-11T14:05:00.000-08:00

Fraud and FCPA Analytics

Verizon Uses Predictive Modeling To Detect Health Fraud

http://bit.ly/rRyE50

SEC turns to data analysis to draw out offending advisers

http://bit.ly/v3GpnH

Dongbu Insurance Detects Fraud With SAS Software

http://bit.ly/tUmDrv

Data And Legal Clarity Can Improve Compliance With Health Care Fraud Laws, Panelists Say (“The first day that we turned this on, we identified 300 known mobsters in the Medicare system,” Smolonsky said. “We discovered that providers who had been debarred years ago were still billing” for services. “And that was low-hanging fruit,” he added.)

http://www.mainjustice.com/2011/12/09/data-and-legal-clarity-can-improve-compliance-with-health-care-fraud-laws-panelists-say/

e-Discovery

Making e-discovery faster, cheaper: Users see big potential for predictive coding

http://bit.ly/uv5XhO

eDiscovery: Clustering Technology & Document Review

http://bit.ly/uZ5SV0

Text Mining

Digital Reasoning gets more dough for big data intelligence push

http://bit.ly/vunoO3

Datamining Twitter

http://bit.ly/t1NDtA

Compliance and Risk Analytics

Data as Regulatory Inoculation

http://bit.ly/usZllG

Predictive Analytics in Workers’ Comp Arrives

http://bit.ly/uklsXq

In-House Compliance Requires Company-Wide Efforts

http://bit.ly/vp4wpp

IRS Using Advanced Analytics

2011-12-08T07:26:00.000-08:00

Government trade piece describes the IRS adoption of a SAS advanced analytics platform to detect errors and fraud; interestingly, the solution includes SAS's text mining platform.

The incorporation of free text analytics into traditional structured data mining and modeling represents the evolution of useful analytics.

My Observations on Digital Reef eDiscovery Analytic Technology Videos

2011-12-07T11:02:00.000-08:00

Assuming you have actually watched the videos (and they are well, well worth the watch if you're looking for information on predictive methods), here are my observations, in no order:

· Native text “normalization” is absolutely required. In fact, good metadata and content ETL, like that provided by vendors such as Digital Reef, is a necessity for good document analytics.

“Impedance mismatch” was used to describe the phenomenon whereby people speak about a particular topic or issue in linguistically distinct ways. Example: I hit a home run; I homered; I hit it out of the park; I sent one air mail; I got a hold of all of one. BTW, ironically, I think the phrase impedance mismatch is a bit self-referential given its traditional meanings.

· Latent semantic analysis, creating “vectors” based on tokenized terms in documents. Here, “impedance mismatch” has to be remembered. You have to keep in mind that the methods aren’t magic. Token vectors from two documents discussing the same topic but using different terms (see home run above) won’t be recognized as similar … unless

o there are adequate existing thesaurus libraries that associate disparate terms with each other in a taxonomy group (given the idiosyncrasies of litigation documents and issues, pre-canned libraries have definite limits);

o probabilistic LSA is employed (this helps but experience dictates that unsupervised PLSA results for idiosyncratic issues and corpi can be both over-inclusive and under-inclusive, and really not good when language is opaque i.e. fraud and deception), or,

o most effectively, an interactive supervised process whereby people code not documents but terms and/or metadata into the larger related categories of relevance. This was alluded to as “derived features”. I’d refer to it as semi-manual feature selection, wherein the subject matter expert teaches the system which bits of information matter in determining relevance, and which don’t.

· “You can’t just push a button” was a very important point. Subject matter experts are vital to process. Teams that include people who understand both the technological capabilities and limitations, as well as the legalities of each case are critical for success. Also, best tagging involves tagging relevance to the actual terms that are relevant from within the communication (content and/or metadata).

· The mention of using entities, capitalization, punctuation, etc as part of the analysis was fascinating. I’d add pronoun usage, language changes, sentiment, and passive language usage to the list. But whether these have a use in generic e-discovery remains to be shown.

· I think the panel missed the point of the question relating to data within documents --essentially spreadsheets. This can be a common issue. The answer depends. It depends for instance on whether you want let’s say all daily revenue reports, because that category of document is relevant, or whether you want all documents where, e.g., total revenue listed on the sheet exceeds a certain amount? The first problem often can be solved with a search term combination, the second requires creative pre-processing, transactional analysis and perhaps advanced analytics, but is doable.

· Supervised/unsupervised analysis -using probabilistic LSA methods in an unsupervised process to find what goes together with communication documents, especially if metadata will be utilized, can lead to highly UN-valuable results. To use metadata in a valid way, separate machine learning is needed to determine which metadata to use (if any) for specific classes of documents. Supervised (semi-manual) learning is probably the most practical solution for e-discovery.

· I didn’t like the fact that near-dupe tagging was listed in ECA only. Its use is absolutely not limited to the ECA function. Employing near-dupes effectively is highly effective in collapsing the review process and associated costs without incurring risk.

Although techniques can be developed to do many cool things, there will likely be a vendor cost associated with each, making many useful ad hoc approaches impracticable unless the ability to operate in-house has been developed.

System would monitor feds for signs they're 'breaking bad'

2011-12-07T07:36:00.000-08:00

Interesting piece about a DARPA-backed system being developed with the goal of identifying and ranking information and/or employees who are (or intend on) engaging in "improper behavior in the federal government."

Technically, the proof's in the pudding. Using advanced analytics to accurately decipher free text is at the opposite pole from trivial,

Legal, social issues abound.

Pfizer FCPA/Qui Tam Woes: Case in Point for a Unified Cross-Purpose Analytics-based Compliance Platform?

2011-12-06T09:05:00.000-08:00

In November, the WSJ reported: Pfizer Near Settlement on Bribery

The article stated in part: "Spurred by the investigations, companies throughout the industry are developing multimillion-dollar compliance programs and are spending more time trying to identify foreign doctors and health-care workers that could qualify as foreign officials, said Jay Perlman, director of global investigations and compliance at Navigant Consulting Inc. "We're seeing a real sea change in the attitude," he said."

In 2009, it was reported that Pfizer agreed to pay $2.3 billion to settle qui tam and criminal charges. As part of that settlement, Pfizer "[...] entered into a Corporate Integrity Agreement (CIA) with the Office of Inspector General (OIG) of the U.S. Department of Health and Human Services (HHS). The CIA institutes certain new measures and requires Pfizer to continue maintenance of a corporate compliance program for a period of five years." Pfizer reported that "[a]although Pfizer was subject to a prior CIA, it has consistently and voluntarily strengthened its compliance program beyond those requirements, including enhancements to the company’s comprehensive product risk assessment and mitigation process and compliance monitoring operations."

Business decisions in this space are complex, but where there's not one but two substantial regulatory birds of prey perched overhead, patently a strong driver exists to at least consider a comprehensive cross-function (closer-to-real-time) analytics-based monitoring system to mitigate two significant points of risk at reduced cost (as compared to silo solutions).

I have no idea how Pfizer is approaching the issue, but it serves as a good case study on how strategic thinking about compliance analytics might pay off.

Breaking Spoliation Claim: BP says Halliburton destroyed Gulf spill evidence

2011-12-06T05:48:00.000-08:00

CNN reported in an article today that BP is claiming Halliburton's computer model tests were destroyed specifically because it was concerned about their use against the company in litigation.

"BP is seeking to have a 'third-party specialist" examine a Halliburton computer, saying "such an examination might well recover the missing modeling results, or shed light on the circumstances of their apparent disappearance.'"