“What’s it going to take to move the data quality space forward in the future? That’s the question recently addressed by Ted Friedman of Gartner as reported in an article in destinationCRM.com. He suggests that the real answer may be messaging.”

CiOZone: Master Data Management Ready For Prime Time

“As with many application areas, Microsoft’s sweeping move into MDM signals a mainstreaming of the field, according to Aaron Zornes, founder and chief research officer of the The MDM Institute, Burlingame, Calif. ‘As a practice, MDM has been going on in some industries since 1980s, but it’s only been formalized with a growing, purpose-built vendor base in recent years,’ he says. ‘In the time since the institute was founded in 2004, the industry has matured considerably.’”

data quality PRO: Data Quality Blog Roundup - June 2009 Edition

“Another marked increase in online publishing this month for the data quality sector. A smattering of new entrants means there is a steady flow of fresh ideas and insight in this months blog roundup.”

Government Security News: OPINION / Analyzing intelligence data: Matching information in foreign languages

“While the challenge is great, there is technology specifically designed to “connect the dots” among persons, places and things of interest. Called “entity resolution,” this technology is coming into the mainstream, specifically in light of the growing urgency to track down terrorists and stop terrorist threats before they happen.”

Life as a Healthcare CIO: International EHR Adoption

“The most widely implemented are England, Denmark, Netherlands, and certain regions of Spain which are close to 100%. Sweden, Norway are at 80% and behind and Germany/France are at 50%. The US is somewhere between 2 and 20%, depending on how you classify a comprehensive EHR.”

Share This

What’s it going to take to move the data quality space forward in the future? That’s the question recently addressed by Ted Friedman of Gartner as reported in an article in destinationCRM.com. He suggests that the real answer may be messaging.

“Vendors have done a reasonably poor job in that they could get better at articulating the true business value [of data quality solutions],” he says. The Gartner analyst notes that vendors tend to talk about functionality in terms of technological advances, rather than conveying how that technology actually supports the business infrastructure. Friedman also notes that, in general, vendors could get better at articulating how tools support initiatives such as information governance and regulatory compliance — two notable industry trends.

Ted is right that to call out vendors to improve our messaging. At Identity Resolution Daily, we get down into technical details fairly often. For example, our bloggers have talked about data matching, its relationship to identity resolution, critical requirements for identity resolution, and we’ve had a series on data quality. Professor John Talburt of UALR’s Center for Advanced Research in Entity Resolution and Information Quality (ERIQ) is a regular contributor who talks about technical definitions and issues surrounding these topic areas.

That’s not to say that business issues around entity resolution and information quality have been ignored. Real world problems like lottery retailer fraud have been a frequent topic, as has organized retail crime. Another business problem we’ve talked about is employers trying to cheat workers compensation laws, and we’ve actually discussed regulatory compliance (OK, so we did get a bit technical on that one).

A huge issue related to information governance is preserving the rights of individual privacy. Because of our involvement in TSA’s Secure Flight program, we’ve written about this issue repeatedly since Identity Resolution was created. A recent post captures the essence of the issue.

So at best, I’d have to say we get a C+ or a B- on our messaging. With our upcoming release of IRE 2.2, we’ll make every effort to respond to Ted’s constructive criticism of the data quality space.

Share This

“In my last post I discussed my definitions for entity resolution, entity identification, entity disambiguation, and anonymous entity resolution.  (And I reiterate that these are just my definitions and are not binding on anyone except possibly my students.) Let’s go back to the overarching term entity resolution (ER).  In its broadest sense, I see ER as encompassing three major activities…”

Article Marketing: Electronic Medical Records – Are There Reasons For Low Implementation?

“Doctors may soon have little choice but to implement computerized medical billing and patient record systems. HIPAA’s scope recently expanded to health care providers with less than $5 million in revenue.”

OCDQ Blog: Worthy Data Quality Whitepapers (Part 1)

“It is about the data – the quality of the data… This is the subtitle of two brief but informative data quality whitepapers freely available (no registration required) from the Electronic Commerce Code Management Association (ECCMA): Transparency and Data Portability.”

SmartData Collective: Moving BI Into The Cloud Part 1

“Over the last year I have been reading a lot about cloud computing and trying to predict how this can be used in business intelligence and analytics. I believe that the cloud is becoming relevant for a number of reasons…”

Share This

Share This

In my last post I discussed my definitions for entity resolution, entity identification, entity disambiguation, and anonymous entity resolution.  (And I reiterate that these are just my definitions and are not binding on anyone except possibly my students.)

Let’s go back to the overarching term entity resolution (ER).  In its broadest sense, I see ER as encompassing three major activities:

1.    Extracting or collecting of entity references from sources
2.    Linking references to same entity
3.    Exploring networks of entity associations.

In this post let’s focus on the first process, extracting and collecting entity references.  For many academic researchers, this is what entity resolution is all about, i.e. it represents the really interesting and challenging part. An extensive body of research literature discusses the methods and techniques for finding entity references in unstructured information, especially unstructured textual information (UTI).

There are many other ER participants for whom this process holds little or no interest.  These are primarily the commercial ER processors who expect that the information they begin with is already structured.  For them, the starting point is a record or database instance assumed to relate to an entity (e.g. customer) and that has well-defined fields or columns.  Their game is all about the process of linking these records.

However, there’s a growing realization that most of an organization’s information assets reside in unstructured data stores such as emails, reports, spreadsheets, photos, graphs, comments, notes, and other sources that are not only unstructured but may not even be in computer readable format.  The consensus is that the 80-20 rule applies: 80% unstructured to 20% structured.  The actual proportion will vary from organization to organization, but there is no denying that a tremendous amount of information is tucked away in unstructured formats.  Consequently, the text miners, law enforcement, intelligence community, and other old hands at entity extraction are now being joined by the commercial world in the rush to exploit this new source of information and potentially business intelligence (BI).

Researchers in image processing have long recognized the process of “feature extraction” where the parts of an image of interest (such as a human face) are located within the larger image. Thus I like the term “entity extraction” to describe this process in a broader sense that also includes text, audio, and other media, not just images.

The level or degree to which an entity reference is classified is another important issue in entity extraction.  Entities are just the people, places, or things we are interested in for a given application, and as we have learned from object-oriented analysis, these entity/objects often exist within a logical hierarchy.  The level of classification often impacts the strategy and the complexity of the extraction process.

To illustrate levels of classification, consider the following example of unstructured text that might appear in a newspaper announcement:

“On July 21, 2008, Mary Jo Smith, daughter of Sam and Sue Smith of Ft. Worth, was married to John Doe, son of Bill and Mary Doe, in a ceremony at the St. Joe Church in Dallas.”

At the highest level, several parsers could read the text and classify most of these references as people (e.g. Mary Jo Smith, John Doe), places (e.g. Ft. Worth, Dallas), and dates (e.g. July 21, 2008).  At a deeper level, however, we are interested not just in entity class, but more particularly in the entity’s sub-class or role.  The context is often given in the form of an “ontology” that specifies the entities and roles within a given context.  In this case, a “marriage ontology” would have roles for Bride, Groom, Date-Of-Marriage, Parent-of-Bride, Parent-of-Groom, and Place-of-Marriage. In our example above, determining that the reference Mary Jo Smith is not only to a person, but to the person in the role of “bride” in the context of a marriage announcement is a more demanding problem than simply discovering that Mary Jo is a person.

Even from this simple example, it is clear that developing a general solution for extracting and classifying entity references is a formidable challenge.  Another growing area of ER research is the new focus on moving beyond linking a reference to the same entity to networks of linkages, a topic for my next post.

Share This

Francine Hardaway’s Blog: Are There Economies of Scale in Medicine?

“The efficiencies come when a group of physicians are all responsible for a patient’s continuity of care, and when they share information such as that possible with electronic health records (EHRs).”

Insurance & Financial Advisor: Poizner, industry oppose California downgrading of insurance fraud felonies

“‘Reclassifying 73 crimes including ‘false insurance claims’ is a disservice to the consumers and businesses in the state of California,” the letter said. “In addition, taking the power out of the hands of the public prosecutor to charge someone with a felony crime will have a serious impact on public safety.’”

BAM INTEL: A Growing Trend - Fusion Centers Connect Private and Public Sector Thinking

“The private sector owns about 80% of all critical infrastructure, and a communication disconnect could result catastrophically in a disaster scenario.”

Share This

“So the title ‘Catching Terrorists and Making the World a Safer Place’ certainly caught my eye! And the content of the post did not disappoint, as the author Chris Boorman of Informatica did a great job of crystallizing the issue that drove the creation of this blog over two years ago: ‘So how do we balance the freedom of movement we have come to expect as hard-working citizens with the need to spot terrorists?’”

[Post from Infoglide] Identity Resolution Daily on Twitter

“At Identity Resolution Daily, we often come across interesting tidbits about entity resolution, and now we can share them in real time. Just add our ID - @IDResolution - to your twitter sources. Happy tweeting!”

GreenvilleOnline.com: Consumers may see insurance rates rise

“According to Love, the average family spends about $1,000 more per year as a result of insurance fraud. That’s felt in higher insurance premiums, taxes, and the cost of goods and services, she said.”

Ezine: Fraud Alert - Lottery Retailers Win More Than Their Customers Do

“California Lottery did an undercover sting where they brought, what they knew to be, a winning lottery ticket to a retailer to have it verified. They caught many retailers on hidden camera telling them that the winning ticket was a loser and, subsequently, went on to claim the money themselves. On top of that, a statistician studied big wins of lottery retailers in Ontario, Canada and found that retailers won big jackpots a lot more than you would statistically expect them too.”

data quality pro: Rethinking Data Quality: The Need for a Data Quality Profession

“Processes, projects, products – each of these contributes to the efforts to improve data quality. But they haven’t solved the problems individually or collectively. To really make substantial and sustainable differences in the quality of data we need to take a different approach. We need to think of data quality as a profession.”

Share This

So the title “Catching Terrorists and Making the World a Safer Place” certainly caught my eye! And the content of the post did not disappoint, as the author Chris Boorman of Informatica did a great job of crystallizing the issue that drove the creation of this blog over two years ago: “So how do we balance the freedom of movement we have come to expect as hard-working citizens with the need to spot terrorists?” His answer is “technology” and of course we agree.

When Identity Resolution Daily first began in the summer of 2007, we pointed out the constant tension between freedom and privacy versus the need for security:

In the US, the debate between personal privacy (and perhaps liberties in general) versus security is a long-standing one with roots in the very founding of the nation itself. Folks interested in obtaining data often wonder how much people are willing to give up in the name of greater security or convenience. On the other hand, those more focused on privacy worry about how data is obtained, what it’s used for and where it ends up.

Infoglide CEO Mike Shultz also discussed the responsibility that comes with providing technology that deals with identity:

It was important to all of us here that we didn’t create some sort of Big-Brother-enabling technology. As a result, we designed software that can resolve identities across multiple sources while protecting data privacy and security.

The point he made about the design of the software being critical is vital, and The Center for Digital Government’s white paper entitled “Resolving Identity: The Importance of Who’s Who and the Search for the Perfect Engine” delves into what technology can do to answer questions like “who’s who” and “who’s related to whom.”

In a more recent post, we talked about the components needed for an effective identity resolution solution. It’s not enough to have great similarity matching algorithms, and it’s not even enough to be able to find hidden connections in real time across millions of rows of data, although both those capabilities are obviously required. The real test in catching terrorists and making the world a safer place using identity resolution is how decision-making is automated and integrated into existing business processes.

Share This

Share This

intelligent enterprise: They Better Get This MDM Program Right

“As reported in The New York Times and on the TSA Web site, the Secure Flight program will improve upon current practices in matching passenger identities to watch lists in many ways. At first glance, this appears to be a well thought-out program that conforms to several basic tenets of Master Data Management (in bold below), in this case for the ‘Customer’ entity.”

EHRWMS: Georgia’s Best EMR Used By Three of Top Ten Pediatricians

“Of approximately 100 respondents, 28 used an EMR, of which 40% used the EncounterPRO Pediatric EMR. There were only three other EMRs used more than once, and they were used by only 10%, 7%, and 7% of the survey respondents respectively.”

Government Executive: Enforcement agencies boost cooperation on drug investigations

“In addition, ICE agents for the first time will fully participate in the Organized Crime Drug Enforcement Task Force Fusion Center. The center allows participating federal, state and local law enforcement agencies, including DEA and the FBI, to share information and analytical resources to enhance their overall investigative capacity.”

SmartData Collective: The Data-Information Continuum

“Data could be considered a constant while information is a variable that redefines data for each specific use. Data is not truly a constant since it is constantly changing. However, information is still derived from data and many different derivations can be performed while data is in the same state (i.e. before it changes again).”

Share This