Information Quality Aggregator

The Once and Future Data Quality Expert

2009-11-08T05:11:00Z

Wednesday, November 11 is World Quality Day 2009.

World Quality Day was established by the United Nations in 1990 as a focal point for the quality management profession and as a celebration of the contribution that quality makes to the growth and prosperity of nations and organizations. The goal of World Quality Day is to raise awareness of how quality approaches (including data quality best practices) can have a tangible effect on business success, as well as contribute towards world-wide economic prosperity.

IAIDQ

The International Association for Information and Data Quality (IAIDQ) was chartered in January 2004 and is a not-for-profit, vendor-neutral professional association whose purpose is to create a world-wide community of people who desire to reduce the high costs of low quality information and data by applying sound quality management principles to the processes that create, maintain and deliver data and information.

Since 2007 the IAIDQ has celebrated World Quality Day as a springboard for improvement and a celebration of successes. Please join us to celebrate World Quality Day by participating in our interactive webinar in which the Board of Directors of the IAIDQ will share with you stories and experiences to promote data quality improvements within your organization.

In my recent Data Quality Pro article The Future of Information and Data Quality, I reported on the IAIDQ Ask The Expert Webinar with co-founders Larry English and Tom Redman, two of the industry pioneers for data quality and two of the most well-known data quality experts.

Data Quality Expert

As World Quality Day 2009 approaches, my personal reflections are focused on what the title data quality expert has meant in the past, what it means today, and most important, what it will mean in the future.

With over 15 years of professional services and application development experience, I consider myself to be a data quality expert. However, my experience is paltry by comparison to English, Redman, and other industry luminaries such as David Loshin, to use one additional example from many.

Experience is popularly believed to be the path that separates knowledge from wisdom, which is usually accepted as another way of defining expertise.

Oscar Wilde once wrote that “experience is simply the name we give our mistakes.” I agree. I have found that the sooner I can recognize my mistakes, the sooner I can learn from the lessons they provide, and hopefully prevent myself from making the same mistakes again.

The key is early detection. As I gain experience, I gain an improved ability to more quickly recognize my mistakes and thereby expedite the learning process.

James Joyce wrote that “mistakes are the portals of discovery” and T.S. Eliot wrote that “we must not cease from exploration and the end of all our exploring will be to arrive where we began and to know the place for the first time.”

What I find in the wisdom of these sages is the need to acknowledge the favor our faults do for us. Therefore, although experience is the path that separates knowledge from wisdom, the true wisdom of experience is the wisdom of failure.

As Jonah Lehrer explained: “Becoming an expert just takes time and practice. Once you have developed expertise in a particular area, you have made the requisite mistakes.”

But expertise in any discipline is more than simply an accumulation of mistakes and birthdays. And expertise is not a static state that once achieved, allows you to simply rest on your laurels.

In addition to my real-world experience working on data quality initiatives for my clients, I also read all of the latest books, articles, whitepapers, and blogs, as well as attend as many conferences as possible.

The Times They Are a-Changin'

Much of the discussion that I have heard regarding the future of the data quality profession has been focused on the need for the increased maturity of both practitioners and organizations. Although I do not dispute this need, I am concerned about the apparent lack of attention being paid to how fast the world around us is changing.

Rapid advancements in technology, coupled with the meteoric rise of the Internet and social media (blogs, wikis, Twitter, Facebook, LinkedIn, etc.) has created an amazing medium that is enabling people separated by vast distances and disparate cultures to come together, communicate, and collaborate in ways few would have thought possible just a few decades ago.

I don't believe that it is an exaggeration to state that we are now living in an age where the contrast between the recent past and the near future is greater than perhaps it has ever been in human history. This brave new world has such people and technology in it, that practically every new day brings the possibility of another quantum leap forward.

Although it has been argued by some that the core principles of data quality management are timeless, I must express my doubt. The daunting challenges of dramatically increasing data volumes and the unrelenting progress of cloud computing, software as a service (SaaS), and mobile computing architectures, would appear to be racing toward a high-speed collision with our time-tested (but time-consuming to implement properly) data quality management principles.

The times they are indeed changing and I believe we must stop using terms like Six Sigma and Kaizen as if they were a shibboleth. If these or any other disciplines are to remain relevant, then we must honestly assess them in the harsh and unforgiving light of our brave new world that is seemingly changing faster than the speed of light.

Expertise is not static. Wisdom is not timeless. The only constant is change. For the data quality profession to truly mature, our guiding principles must change with the times, or be relegated to a past that is all too quickly becoming distant.

Share Your Perspectives

In celebration of World Quality Day, please share your perspectives regarding the past, present, and most important, the future of the data quality profession. With apologies to T. H. White, I declare this debate to be about the difference between:

The Once and Future Data Quality Expert

Mistake Driven Learning

The Fragility of Knowledge

The Wisdom of Failure

A Portrait of the Data Quality Expert as a Young Idiot

The Nine Circles of Data Quality Hell

Additional IAIDQ Links

IAIDQ Ask The Expert Webinar: World Quality Day 2009

IAIDQ Ask The Expert Webinar with Larry English and Tom Redman

INTERVIEW: Larry English - IAIDQ Co-Founder

INTERVIEW: Tom Redman - IAIDQ Co-Founder

IAIDQ Publications Portal

Patent Reexamination Hurdle Remains For Data Integration Patent

2009-11-07T11:47:34Z

The JuxtaComm ETL patent will not be challenged in court after all defendants settled but it will face reexamination from the patent board.

Do You Have A Foundation For MDM Success?

2009-11-06T20:11:06Z

There is a new whitepaper from Evan Levy of Baseline Consulting available on our website, “Master Data Management – Building A Foundation For Success”. The theme of the paper is about what components make up the foundation of any MDM project – whether a customer hub, a product hub, an operational hub, an analytical hub, one that is bought from a vendor or one that you build yourself. In all cases, there is a common infrastructure that time and again, leading practitioners such as Mr. Levy find to be required.

To summarize, Mr. Levy breaks it down into five core capabilities:

· Data Cleansing and Correction

· Metadata Management

· Access Services

· Data Migration, and

· Identity Resolution

He goes on to provide a recipe for success on how these components are to be used. One of the more interesting points he makes is that many organizations already have most of these capabilities – they just aren’t being applied to their MDM initiatives.

So, the questions you should be asking yourself are, “How much of a foundation for MDM do we already have internally? and “What components do I need to add to give my organization that ability to scale its MDM initiative efficiently, economically, and in line with business requirements?”

It’s a good paper and I think you will enjoy it and get much value from Mr. Levy’s insights.

IBM Reveals the First Fourteen InfoSphere Information Champions

2009-11-06T13:14:25Z

I was fortunate to be included in the list of the first IBM Information Champions for contributions to the InfoSphere software community.

The Mullet Blogging Manifesto

2009-11-06T06:34:00Z

Blogging is more art than science. My personal blogging style can perhaps best be described as mullet blogging. No, not the “business in the front, party in the back” haircut that I tried to rock back in the '80s (I couldn't pull it off, had to settle for a “tail” and had to cut that off because it made me look like an idiot – OK, more idiotic than usual). By mullet blogging I mean:

“Take yourself and your blog seriously, but still have a sense of humor about both.”

As a mullet blogger, I hold the following truths to be self-evident, but I decided to write them down anyway.

Blogging is All about You

Not you meaning me, the blogger — you meaning you, the reader.

Blogging should always focus on the reader and provide them assistance with a specific problem, even if that problem is boredom or simply a need for entertainment. Don't worry about your readers agreeing with you. They will either thank you for your help or tell you that you're an idiot – either way, you have started a conversation, which should always be your blogging goal.

Brian Clark recently shared something to think about using the following quote from Robert McKee:

“When talented people write badly it’s generally for one of two reasons:

Either they’re blinded by an idea they feel compelled to prove,

Or they’re driven by an emotion they must express.

When talented people write well, it is generally for this reason:

They’re moved by a desire to touch the audience.”

B = U²C³

Blogging = Unique and Useful content that is Clear, Concise, and Consumable.

The conventional blogging wisdom is to be both Unique and Useful. Although I normally like to defy conventions, I have to agree with the wise ones on these fundamentals.

One of the most important aspects of being unique is writing effective titles. Most potential readers scan titles to determine whether or not they will click and read more. There is obviously a delicate balance between effective titles and “baiting,” which will only alienate potential readers.

If you write a compelling title that makes me click through to an interesting post, then “You Rock!” However, if you write a “Shock and Awe” title followed by “Aw Shucks” content, then “You Suck!”

Therefore, your content also has to be unique – your topic, position, voice, or a combination of all three.

One of the most important aspects of useful is “infotainment” – that combination of information and entertainment that, when done well, can turn potential readers into raving fans. Just don't forget about the previous section – your content has to be informative and entertaining to your readers.

The key to good blogging is to follow the Three C’s – Clear, Concise, Consumable.

The attention span of a blog reader is not the same as a reader of books, newspapers (they still exist, right?), magazine articles, or the audience for presentations. Most people only scan blogs, rarely read a full post and even more rarely leave a comment – regardless of how well the blog post is written.

Write blog posts that get to the point and stay on point (i.e., clear), are no longer than they need to be (i.e., concise), and are formatted to be easy to read on a computer screen (i.e., consumable).

Laugh, Think, Comment

The three things that you want your readers to do.

Although it is not as blatantly formulaic as the title of the previous section, here is another method to my blogging madness:

Open with a joke
Say something thought provoking
End with a call to action

It's as easy as 1-2-3! In my defense, I didn't say open with a good joke. But seriously, humor can be a great way to start a conversation and hold your readers' attention for those few precious additional seconds while you are getting to your point. Obviously, there will be times when the seriousness of your subject would make comedy inappropriate, and if you are not naturally inclined to use humor, then you shouldn't try to force it.

Thought provoking content doesn't have to mean deep thoughts. There is no need to channel Jean-Paul Sartre, for example. However, to paraphrase Sartre: “Hell is other people's boring blogs.”

Obviously, comments are not the only type of call to action. However, blogging is a conversation facilitated by the dialogue and discussion provided via comments from your readers. Without comments, the conversation is only one way.

I love the sound of my own voice and I talk to myself all the time (even in public). However, the two-way conversation provided via comments not only greatly improves the quality of my blog content — much more importantly, it helps me better appreciate the difference between what I know and what I only think I know.

As Darren Rowse and Chris Garrett explained in their highly recommended ProBlogger book: “even the most popular blogs tend to attract only about a 1 percent commenting rate.” Therefore, don't be too disappointed if you are not getting many comments. Take that statistic as a challenge to motivate you to write blog posts that your readers simply can not resist commenting on.

Respond to the comments you do receive. This continues the two-way conversation and encourages comments from other readers. Make sure to never talk down to your readers (either in your blog post or your comment responses). It is perfectly fine to disagree and debate, just don't denigrate.

Obviously, you should block all spam (leading argument for using comment moderation) and never feed the troll.

Stories and Metaphors and Analogies! Oh, my!

I've a feeling we're not in Kansas anymore. Especially me, since I live in Iowa.

Darren Rowse recently shared some great tips about why stories are an effective communication tool for your blog, including a list of some of the different types of stories you can tell.

My blog uses a lot of metaphors and analogies (and sometimes just plain silliness) in an attempt to make my posts more interesting. This is necessary because I write about a niche topic, which although important, is also rather dull.

James Chartrand uses the term Method Blogging as (yes, you guessed it) a metaphor for blogging by comparing it to method acting. Try experimenting with different styles like an actor experimenting with different types of roles and movie genres.

Oftentimes, using stories, metaphors, and analogies in my content works very well. But I admit, sometimes it simply sucks.

However, I have never been afraid to look like an idiot. After all, we idiots are important members of society – we make everyone else look smart by comparison.

The King, Queen, and Crown Prince of Blogging

Meet the Blogging Royal Family: Content, Marketing, and Context.

Content is King. The primary reason that people are (or aren't) reading your blog is because of your content.

Marketing is Queen. “If you blog it, they will read.” Ah, no they won't — this ain't Field of Dreams. Some of the best written blogs on the Series of Tubes get hardly any love because they get hardly any marketing. In addition to providing RSS and e-mail feeds, I use social media (e.g., Twitter, Facebook, LinkedIn) to promote my blog content.

However, too many bloggers have a selfish social media strategy. Don't use it exclusively for self-promotion. View social media as Social Karma. Focus on helping others and you will get much more back than just a blog reader, a LinkedIn connection, a Twitter follower, or a Facebook friend. In addition to blog promotion (which is important), I use social media to listen, to learn, and to help others when I can.

Larry Brooks recently explained that although content may still be king, at the very least, you must pay homage to the new Crown Prince — Context. To paraphrase Brooks, context comes from clarity about your blogging goals, juxtaposed against the expectations and tolerances of your readers. Basically, this above all: to thine own readers be true.

Emerson on Blogging

“Nothing can bring you peace but yourself.”

One of my favorite writers is Ralph Waldo Emerson. The quote that started this section was pure Emerson. What follows is a slight paraphrasing of one of my all-time favorite passages, which comes from his essay on Self-Reliance:

“What I must do is all that concerns me, not what the people think. This rule, equally arduous in real and in online life, may serve for the whole distinction between greatness and meanness. It is the harder because you will always find those who think they know what is your duty better than you know it. It is easy in the world to live after the world's opinion; it is easy in solitude to live after our own; but the great blogger is one who in the midst of the blogosphere, keeps with perfect sweetness the independence of solitude.”

Bottom line — BE YOURSELF — Let your own personality shine through. Make people feel like they are having a conversation with a real person and not just someone who is blogging what they think people want to read.

I hope that you found at least some of this manifesto helpful. I also hope to see more of you around the blogosphere.

I'll be the balding blogger who used to almost have a mullet...

Additional Resources

In other words, people way smarter than me, which you should read on a regular basis.

ProBlogger (The Book)

ProBlogger (The Website)

Chrisg.com

Copyblogger.com

Top 10 Blogs for Writers 2009

October El Festival del IDQ Bloggers on Name Cleansing and a Russian Pipeline

2009-11-05T09:18:12Z

A festival of data quality blog posts featuring a Russian pipeline, the longest name in the world, name splitting, name cleansing and plenty more tips.

Economic Uncertainty Takes Its Toll on Data Security

2009-11-04T15:04:34Z

A difficult economy creates plenty of fear, uncertainty, and frustration within enterprises. Budgets get slashed, projects get put on hold, and overworked employees get stretched to the limits. Another victim of the recent downturn was data security efforts, a new survey shows.

I recently had the opportunity to deliver a Webcast that looked at the challenges of data security in an age of economic uncertainty, as demonstrated in a survey I helped conduct with the Independent Oracle Users Group (IOUG). The survey was part of my work with Unisphere Research/Information Today Inc. I was joined by Roxana Bradescu, senior director of database security product marketing at Oracle, and Ian Abramson, president of IOUG.

In our survey of 316 IOUG members, nine percent reported experiencing a data breach over the past year. While this may seem relatively low, it is still a 50-percent jump over last year's level. In addition, 20% of respondents feared a data breach was all but inevitable over the coming year.

Even more troubling, we found that corporate management has taken its eye off the ball in terms of data security. This is understandable given the distractions of the recent economic tsunami, but by cutting back on data security efforts, companies leave themselves open to costly incidents.

In my presentation, I observed that there was a notable reduction in data security initiatives since the last survey was conducted about the same time last year. There is less monitoring, less encryption, and reduced budget growth. Again, this is attributable to tight IT budgets and spending as a result of the turbulent economy – and this has repercussions across organizations' efforts to lock down their data assets.

The survey uncovered the following issues:

Data security spending slowed dramatically over the past year. In the 2008 survey, 41% of respondents said their data security budgets were on the rise. This was down to 28% this year. About 13% said their companies outright cut security spending, a three-fold increase.
More data is being sent to off-site third parties. The survey found more companies are turning to specialized third-party vendors for data administration and application development – a direct result of cost-cutting initiatives. About 36% now outsource, up from 28% a year ago. However, this opens data to all kinds of new threats – well beyond the control of the original data owners. Many publicly revealed data theft incidents involve laptops that are stolen from the vehicles or premises of third-party contractors.
More production data is being sent to non-production sites. Close to half the respondents, 46%, say live production data – which could include credit card numbers and other sensitive data – is being used with non-production environments, such as development shops, testbeds, and back-up sites. This is up from 43% a year ago.
Organizations are not paying enough attention to what privileged users are doing when they working within databases. The elevated privileges enjoyed by "super users," in fact, may be the Achilles' heel of data security. Only 39% say they have mechanisms that can prevent super users from abusing data. This especially gets risky if data is sent out of the production environment and over to development sites, test beds, or backup sites. You may completely trust your DBAs, but how about DBAs or developers in other departments?

As the economy improves and more automation is introduced to data management, we may see these issues handled in a more comprehensive way. In the meantime, managers need to be more aware of the potential issues from within their firewalls.

Data Quality Blog Roundup - October 2009

2009-11-04T10:00:39Z

Welcome once again to our (mostly) regular roundup of posts across from across the data quality blogosphere.

October saw a huge amount of blogging activity across the full spectrum of data quality, data governance and master data management topics.

Data Quality Blog Roundup - October 2009

Is there a blogger I've missed this month? Contact us.

MDM - It's not about the technology (part1) | MDM - It's all about the process | MDM - It's all about the people : Charles Blyth kicks off our roundup with an excellent trio of posts looking at the technology, process and people elements of MDM.

Wild wild MDM...: Dalton Cervo (@dcervo) presents the first in an ongoing series of posts that presents the first (as far as I know) MDM related comic strip.

Data Sherpas Needed : Stefanos Damianakis presents an interesting post that explores the massive growth of enterprise data and its future impact on our profession.

September Festival del IDQ Bloggers : Daniel Gent presents the September IAIDQ "Festival del IDQ Bloggers" on his blog "The Data Quality Edge"

Adventures in Data Profiling (Part 7) : Jim Harris (@ocdqblog) continues his excellent data profiling series, exploring customer name profiling in this, the 7th post in a series of 8.

Data Quality Doesn’t Get Any Respect : Bryan Larkin (@bslarkin) kicks off his new "DIRTY DATA DONKEY BLOG" with an interesting look at his experiences of data quality in the retail industry, includes a poignant quote:

...a data quality program that fixes data on one end but not the other of a strategic relationship really is little better than having the same bad data on both ends...

Some Thoughts About Completeness – Part 1 : Part 2 : This series of posts by David Loshin examines one of the most important data quality rules, completeness, providing some interesting anecdotes from his own data profiling experiences.

The Problem With Quality : Expert panelist Mike Meier launches a thought-provoking series looking at the concept of quality as applied to data quality and where the modern programmer resides in the equation.

"Tipping Point" for Data Quality and Data Governance? : Rich Murnane (@murnane) debates what it takes to create a tipping point for the disciplines in our field.

A game changer – Ferguson v British Gas : Daragh O Brien reports on a landmark lawsuit that could have major implications for our profession.

Lego Blocks and Data Quality & Russian Gas Pipes and Data Governance : Ken O'Connor with two excellent posts that really bring home the need for professionals in our space to educate the common-sense, undeniable benefits of what we teach and preach in these blogs.

Data quality definitions: fit for purpose? : Graham Rhind, (@grahamrhind ) launches a really interesting debate, shouldn't high quality data be fit for all purposes? What's your view...

Data May Require Unique Data Quality Processes: Steve Sarsfield (@stevesarsfield) with a timely reminder that there is no one-size-fits-all approach to data quality.

Splitting names : Henrik Sørensen (@hlsdk) provides a great insight into the challenges of splitting identities within a single attribute.

Useful Resources

Find all posts in : DQ Blog Roundup

El Festival del IDQ Bloggers (promote your DQ posts or why not host it on your site one month?)
Data Quality Blog Roundup (July 2009 Edition)
Data Quality Blog Roundup (June 2009 Edition)
Data Quality Blog Roundup (May 2009 Edition)
Data Quality Blog Roundup (April 2009 Edition)
Data Quality Blog Roundup (March 2009 Edition)

Customer Incognita

2009-11-04T03:13:00Z

Many enterprise information initiatives are launched in order to unravel that riddle, wrapped in a mystery, inside an enigma, that great unknown, also known as...Customer.

Centuries ago, cartographers used the Latin phrase terra incognita (meaning “unknown land”) to mark regions on a map not yet fully explored. In this century, companies simply can not afford to use the phrase customer incognita to indicate what information about their existing (and prospective) customers they don't currently have or don't properly understand.

What is a Customer?

First things first, what exactly is a customer? Those happy people who give you money? Those angry people who yell at you on the phone or say really mean things about your company on Twitter and Facebook? Why do they have to be so mean?

Mean people suck. However, companies who don't understand their customers also suck. And surely you don't want to be one of those companies, do you? I didn't think so.

Getting back to the question, here are some insights from the Data Quality Pro discussion forum topic What is a customer?:

Someone who purchases products or services from you. The word “someone” is key because it’s not the role of a “customer” that forms the real problem, but the precision of the term “someone” that causes challenges when we try to link other and more specific roles to that “someone.” These other roles could be contract partner, payer, receiver, user, owner, etc.
Customer is a role assigned to a legal entity in a complete and precise picture of the real world. The role is established when the first purchase is accepted from this real-world entity. Of course, the main challenge is whether or not the company can establish and maintain a complete and precise picture of the real world.

These working definitions were provided by fellow blogger and data quality expert Henrik Liliendahl Sørensen, who recently posted 360° Business Partner View, which further examines the many different ways a real-world entity can be represented, including when, instead of a customer, the real-world entity represents a citizen, patient, member, etc.

A critical first step for your company is to develop your definition of a customer. Don't underestimate either the importance or the difficulty of this process. And don't assume it is simply a matter of semantics.

Some of my consulting clients have indignantly told me: “We don't need to define it, everyone in our company knows exactly what a customer is.” I usually respond: “I have no doubt that everyone in your company uses the word customer, however I will work for free if everyone defines the word customer in exactly the same way.” So far, I haven't had to work for free.

How Many Customers Do You Have?

You have done the due diligence and developed your definition of a customer. Excellent! Nice work. Your next challenge is determining how many customers you have. Hopefully, you are not going to try using any of these techniques:

SELECT COUNT(*) AS "We have this many customers" FROM Customers
SELECT COUNT(DISTINCT Name) AS "No wait, we really have this many customers" FROM Customers
Middle-Square or Blum Blum Shub methods (i.e. random number generation)
Magic 8-Ball says: “Ask again later”

One of the most common and challenging data quality problems is the identification of duplicate records, especially redundant representations of the same customer information within and across systems throughout the enterprise. The need for a solution to this specific problem is one of the primary reasons that companies invest in data quality software and services.

Earlier this year on Data Quality Pro, I published a five part series of articles on identifying duplicate customers, which focused on the methodology for defining your business rules and illustrated some of the common data matching challenges.

Topics covered in the series:

Why a symbiosis of technology and methodology is necessary when approaching this challenge
How performing a preliminary analysis on a representative sample of real data prepares effective examples for discussion
Why using a detailed, interrogative analysis of those examples is imperative for defining your business rules
How both false negatives and false positives illustrate the highly subjective nature of this problem
How to document your business rules for identifying duplicate customers
How to set realistic expectations about application development
How to foster a collaboration of the business and technical teams throughout the entire project
How to consolidate identified duplicates by creating a “best of breed” representative record

To read the series, please follow these links:

To download the associated presentation (no registration required), please follow this link: OCDQ Downloads

Conclusion

“Knowing the characteristics of your customers,” stated Jill Dyché and Evan Levy in the opening chapter of their excellent book, Customer Data Integration: Reaching a Single Version of the Truth, “who they are, where they are, how they interact with your company, and how to support them, can shape every aspect of your company's strategy and operations. In the information age, there are fewer excuses for ignorance.”

For companies of every size and within every industry, customer incognita is a crippling condition that must be replaced with customer cognizance in order for the company to continue to remain competitive in a rapidly changing marketplace.

Do you know your customers? If not, then they likely aren't your customers anymore.

Data Governance & Jitterbit

2009-11-03T12:05:21Z

From time to time we feature vendors or products that are relevant to the data management community. Today we have an article from Jitterbit, the makers of a data integration products that has both an open-source version as well as an enterprise edition. As you know, we are huge fans of open-source technologies (this site is built on an open-source platform), so we are happy to have the following article from Dan Oxenburgh of Jitterbit.

Without some form of data governance, an information-based business will be extremely inefficient. Consider a common scenario in which an organization has customer data in several of their enterprise systems. These various systems are touched by a number of different groups including sales, marketing and support, – all of whom are constantly updating this customer data. This situation leads to a long list of process issues, for example: If a contact is updated by two systems and two different users, which change takes precedence? If the customer then updates their information through a web portal, what happens? The number of combinations and questions is nearly endless.

Data Governance: The Building Blocks for a Well-Managed Organization

Sharing and reusing data across applications, systems, and processes within the enterprise is a critical step in cutting costs, boosting productivity and becoming a more efficient organization – and by extension – staying competitive and agile. However, you need to do more than simply move data between point A and point B. To achieve tangible results you need to establish a data governance program that outlines policies, processes, and standards for managing your data. Established correctly, a strong data governance program can dramatically improve your organization’s ability to manage risk, maximize operational efficiency, and comply with industry and governmental regulations.

Data governance is the foundation for effective management of not only data but your entire organization. As such there are two key factors in a successful data governance program: the people and the technology.

The Right People

Successful data governance requires strong communication between business and IT. The common problem is that a communication gap exists between these cross-functional teams. Business does not understand the intricacies of complicated integration solutions that are managed and, in many cases, custom coded by their IT departments. On the other hand, resource-strapped IT does not have the time to fully comprehend the goals of the business or to communicate the value of their systems. As daunting as it may seem to connect data between applications, systems, and processes, it can be even harder to bridge the gap between these two groups. As a result, there are two options:

IT teams need must comprehend business processes, budgets and corporate politics.
Business team members need the ability to manage metadata and data modeling.

The easiest (and painless) of these options is to give business analysts a data integration solution that is simple enough for them to use, but powerful enough to support a complex data governance program.

The Right Solution

Effective data governance makes your data a valued asset. To this end, you need an infrastructure that is robust, flexible, and guarantees the availability, quality, consistency, and security of your data. This solution must make data:

Available to users and applications, when, where, and however needed – regardless of source or format.
High Quality, accurate, and complete.
Consistent and reconciled across all applications, systems, processes.
Secure and standardized according to organizational regulations.

This functionality is available in a majority of today’s integration solutions – from custom coded in-house solutions to the complex “blackbox” systems provided by behemoth software vendors. The problem with these solutions is that they are only available to the users who understand the intricacies of these systems – the IT department. As previously mentioned, this leads to a communication gap that can prevent an otherwise successful data governance strategy from ever getting off the ground.

Instead, you need all of the above capabilities wrapped in a system designed with the business analyst in mind.

Enter Jitterbit – Data Integration Simplified

Jitterbit is the popular open source application and data integration suite that provides business users a quick, cost-effective and simple way to design, deploy and manage a broad range of integration solutions. Our focus is the simplification of even the most complex integration projects. To this end, Jitterbit provides a fully graphical UI that allows you to drag and drop your data sources, targets, and web services onto the Process Designer canvas, creating a visual workflow of your entire integration from start to finish.

There is no need to write custom code, all parts of an integration process are pre-defined in Jitterbit and require only that you fill them out through the wizard-driven interfaces.

Jitterbit includes a formula builder that allows you to add powerful logic and conditions, as well as normalize and de-normalize disparate data. The formula builder includes over 100 pre-built functions (e.g. Conversion, Cryptography, Instance, Logical, String, etc.) that allow a tech savvy business analyst to not only define data governance plans but actually create and deploy the rules themselves. Simply point-and-click to add functions – or if you prefer to write your own, the user experience is similar to writing Excel macros. The formula builder also lets you test every function against your data sources and highlights potential issues.

The result is a simple solution to the problem mentioned at the beginning of this post. Jitterbit will allow you to easily add business logic such as “update only data from System A” or data logic such as “sync only records with valid permissions”. Passed records can be immediately synchronized across systems, while failed records can be assigned automated rules such as “insert default values in blank fields” or sent to a reject file for human intervention.

These capabilities are the building blocks for a powerful data governance solution, and can be created and managed by the same people who are establishing the overarching goals for processes and standards within the organization.

Win-Win Governance

With the right people in control of both the project’s goals and execution, establishing a data governance program does not have to be a daunting challenge. By eliminating the communication gap between business and IT, business is able to focus entirely on data governance while IT is freed to expend their resources on other projects. If you’re looking to take advantage of this win-win situation, you should give Jitterbit a look. You can download the open source Community version or try the 30-day trial of Jitterbit Enterprise today. Jitterbit Enterprise is available to customers who purchase an annual support contract. Visit Jitterbit’s site at http://www.jitterbit.com

5th Normal Form: The Achilles Heel of ETL & Data Warehouse Projects

2009-11-03T11:38:56Z

The lack of awareness of what fifth normal form is (or that it actually exists) is a major threat to data quality and integrity in all organisations.

The highest threat exists in those organisations carrying out Data Warehousing (DW) or Extract, Transform, Load (ETL) Projects.

In this post, John Owens, creator of the Integrated Modelling Method and an expert in business systems and process analysis provides a detailed account of 5th Normal Form, how it is breached and how to ensure data quality and integrity is maintained.

5th Normal Form: The Achilles Heel of ETL & Data Warehouse Projects

What is 5^th Normal Form?

I first started talking to people about 5NF some years ago when lecturing for Oracle on relational systems design. The definition that was available at the time included phrases like "relations that obey no symmetric constraint", etc. and made very little sense.

I have searched the Internet and have been still been unable to find a good definition anywhere. On the contrary, many of the definitions I have found are completely wrong because, what they define as 5NF, is in fact a breach of it!

I find that Fifth Normal Form is best explained by an example of how it is breached.

An Example

Imagine the following is a table that lists manufacturers, the products they produce and the retailers that sell these products:

Over enthusiastic data analysts could look at this table and say: "There is redundancy here. We should normalise this table further.”

They would then break the table down into three separate tables like these.

Redundancy has been removed, so problem solved? Far from it!

If you now do a select * joining the three tables the query will return the rows below:

The row marked with the arrow is a spurious row. In the original table Debenhams did sell irons but only those made by Morphy Richards. In their drive to remove redundancy, the analysts have destroyed the integrity of the data.

This example shows that the original three-column table conforms to 5NF whereas the three normalised tables breach it.

How Do Breaches of 5NF Arise?

The most common ways in which 5NF is breached are:

Normalising tables in databases without first creating the normalised structures in an ERD
Mechanistic and simplistic modelling in the Corporate Data Model
Importing data from disparate databases into a single database without first modelling the required normalised structures in the Corporate Data Model

Normalising Tables in Databases

This error has been described above in the erroneous normalisation carried out on the table with the columns Manufacturer, Product and Retailer.

Simplistic Data Modelling

The most common cause of this is the mechanistic resolution of many-to-may relationships in a data model.

An example will once again help to explain this.

The following diagram shows three many-to-many relationships linking the Data Entities Manufacturer, Product and Retailer:

One rule of data modelling is that many-to-may relationships are resolved by introducing intersection entities. Applying this rule mechanistically and without real analysis of what the above data structure really represents would give us the following structure:

But does it breach 5NF?

One simple way to find out is to trace a route from any entity and see if you can traverse the other entities and get back to where you started. Doing this for the original structure and for the resolved structure would give us the following:

In both cases we can trace a route back to where we started. This is an indication that the structure is breaching 5NF.

But what have we done wrong? We have followed the basic rules for resolving many-to-may relationships. How can the solution be wrong?

We have made two basic errors:

The first of these is that we have mechanistically resolved the many-to-may relationship without asking, “Do we need to know about this relationship and, if so, what information do we need to hold about it?” All intersection entities represent information about a many-to-may relationship so, unless we need to hold information about a many-to-may relationship, we should not resolve it.
The second, yet more crucial error is that we have created data structures in isolation from the Function Model. The key rule for quality data modelling is that the only data that should be included in a Corporate Data Model (and in any resulting database) is that required to support the Business Functions of the enterprise.

From this it is obvious that, in order to know what data we need to hold and what the structure of that data should be, we must first model the Business Functions.

The Function Solves the Problem

The three original Data Entities of Supplier, Product and Retailer could be valid in an electrical goods distribution business.

So what Functions in such a business would create a data structure that would bring these three Data Entities together?

The Marketing Department might want to establish those products and manufacturers that would be preferred by retailers with whom the company would like to do business.

Let’s call this Business Function “Establish Potential Customer Preferences”.

The data structure that would support this Function would look like this.

If we convert this data structure into a database, then the Data Entity “Retailer Preferences” would be implemented as a table that had columns called Retailer, Product and Manufacturer.

This is exactly like the table we started with in our original example.

So by driving the data structures through the Business Functions we automatically arrive at a properly normalised structure.

Merging Data from Disparate Databases

We have seen from the above example how 5NF can be breached by modelling errors. In Data Warehousing and ETL projects it can inadvertently be breached by bringing together data structures that in their original environment complied with 5NF, but when brought together, do not.

It is quite possible that an enterprise has an application with a table in its database with the columns Manufacturer and Product, another application with a table that has Manufacturer and Retailer and a third application with a table with Retailer and Product.

These tables in their own separate worlds will cause no problems to the business and will return correct results query after query.

But bring them all together into a single database as three separate tables (as they existed in their original applications) and the business has a major data quality and integrity problem.

Summary

Fifth Normal Form is not a myth. It is essential to data integrity in all relational databases and in data warehouses that derive their data from relational databases.

Breaches are caused by:

Mechanistic modelling in the Corporate Data Model.
Fragmenting data tables under the guise of “normalising” them.
Importing tables from disparate databases into a single database without first modelling and normalising the required structures in the Corporate Data Model.

Breaches can be avoided by adhering to following fundamental rules:

This is the most essential rule: Only ever model data and data structures required to support the Business Functions of the enterprise.
Never do data normalisation in a database. Always model the normalised structure in the Corporate Data Model first.
Only merge data after you have modelled the required structure in the Corporate Data Model and have then created the appropriate structures in the database receiving the imported data.

Did you find this article useful? If so please share on twitter:

http://bit.ly/1lVjBw Read 5th Normal Form: The Achilles Heel of ETL & Data Warehouse Projects, by @JohnIMM on @dataqualitypro #dataquality

Useful Resources

Integrated Modelling Method: An Introduction

Business Systems Modelling: Function Modelling (Tutorial 1)

Business Systems Modelling: Data Structure Modelling (Tutorial 2)

IMM Website

IMM Approach

Business Function Modelling

Data Structure Modelling

Business Process Modelling

Information Flow Modelling

Author Profile

John Owens is passionate about bringing simplicity, power and elegance to the world of Business Systems Analysis, Business Process Modelling and BPM. He is an international consultant and mentor to a wide range of enterprises of all sizes in the UK, Ireland, Europe and New Zealand. He has put all of this knowledge into a set of books and the Integrated Modelling Method (IMM™) which is available at his website www.integrated-modeling-method.com. John is based in New Zealand and provides mentoring to enterprises of all sizes, from start-ups to large corporations, to aid them improve their business and increase their cash flow.

The Chaos Theory of Data Quality

2009-11-02T13:35:15Z

“One of those issues that is always a source of frustration in the enterprise,” explained Michael Vizard in his recent IT Business Edge blog post, The Never Ending War for Data Quality, “is the quality of the data we spend so much time and money processing. The quest to make sure we have high quality data is nothing short of a never-ending battle between the forces of order and the chaos that envelopes every attempt to organize anything.”

I have to admit that this is one of my pet peeves. A remarkably common misconception is that the only way to deal with the pervasive nature of “imperfect data” is to somehow magically keep all of the data “perfect” all of the time.

Data frequently contains numerous variations caused by different conventions, lack of standards, omissions, and other inconsistencies. The traditional approach to data quality is to heavily rely on standardization and other data cleansing efforts in order to prepare data before it can be effectively used for making business decisions. These preparation activities attempt to create a consistent format of parsed attributes with standardized values.

“Alas, the war over data quality can never really be won,” explains Vizard. “What can be done is that the number of instances where we have conflicting data and outright errors can be sharply reduced. There’s no shame in having bad data; everybody does. The only real sin is not trying to do anything about it.”

I agree with Vizard on the points that everybody has bad data and that we do need to do something about it.

However, the time is long overdue for us to stop depending on outdated approaches to data quality.

Perfection (especially in data) is impossible to achieve. Intelligent business decisions can be made using imperfect data – without extensive data cleansing. Instead of trying to make the data perfect, we need to focus on enabling enterprise applications to handle the unavoidable reality of imperfect data, which is something that humans do naturally.

Advancements in mathematics and machine learning algorithms provide the capability to adapt to (and overcome) data’s inherent chaos, and enable enterprises to make better data-driven business decisions.

I call this approach the Chaos Theory of Data Quality.

The Growing Importance of the Algorithm

The Growing Importance of Mathematics

Adaptive Software

Drowning in Imperfect Data

A Sisyphean Task…

No smoke without ire – Life Insurance Overcharging in Ireland

2009-10-30T09:27:00Z

RTE News in Ireland ran a story last night on overcharging by Irish Life Assurance companies arising from a mis-classification of customers as smokers. (link to the item is here, but you may not be able to access it if you are not in Ireland). On foot of two complaints, the Irish Financial Services Ombudsman investigated [...]

The Tell-Tale Data

2009-10-30T03:31:00Z

It is a dark and stormy night in the data center. The constant humming of hard drives is mimicking the sound of a hard rain falling in torrents, except at occasional intervals, when it is checked by a violent gust of conditioned air sweeping through the seemingly endless aisles of empty cubicles, rattling along desktops, fiercely agitating the flickering glow from flat panel monitors that are struggling against the darkness.

Tonight, amid this foreboding gloom with only my thoughts for company, I race to complete the production implementation of the Dystopian Automated Transactional Analysis (DATA) system. Nervous, very, very dreadfully nervous I have been, and am, but why will you say that I am mad? Observe how calmly I can tell you the whole story.

Eighteen months ago, I was ordered by executive management to implement the DATA system. The vendor's salesperson was an oddly charming fellow named Machiavelli, who had the eye of a vulture — a pale blue eye, with a film over it. Whenever this eye fell upon me, my blood ran cold.

Machiavelli assured us all that DATA's seamlessly integrated Magic Beans software would migrate and consolidate all of our organization's information, clairvoyantly detecting and correcting our existing data quality problems, and once DATA was implemented into production, Magic Beans would prevent all future data quality problems from happening.

As soon as a source was absorbed into DATA, Magic Beans automatically did us the favor of freeing up disk space by deleting all traces of the source, somehow even including our off-site archives. DATA would then become our only system of record, truly our Single Version of the Truth.

It is impossible to say when doubt first entered my brain, but once conceived, it haunted me day and night. Whenever I thought about it, my blood ran cold — as cold as when that vulture eye was gazing upon me — very gradually, I made up my mind to simply load DATA and rid myself of my doubt forever.

Now this is the point where you will fancy me quite mad. But madmen know nothing. You should have seen how wisely I proceeded — with what caution — with what foresight — with what Zen-like tranquility, I went to work!

I was never happier than I was these past eighteen months while I simply followed the vendor's instructions step by step and loaded DATA! Would a madman have been so wise as this? I think not.

Tomorrow morning, DATA goes live. I can imagine how wonderful that will be. I will be sitting at my desk, grinning wildly, deliriously happy with a job well done. DATA will be loaded, data quality will trouble me no more.

It is now four o'clock in the morning, but still it is as dark as midnight. But as bright as the coming dawn, I can now see three strange men as they gather around my desk.

Apparently, a shriek had been heard from the business analysts and subject matter experts as soon as they started using DATA. Suspicions had been aroused, complaints had been lodged, and they (now identifying themselves as auditors) had been called in by a regulatory agency to investigate.

I smile — for what have I to fear? I welcome these fine gentlemen. I give them a guided tour of DATA using its remarkably intuitive user interface. I urge them audit — audit well. They seemed satisfied. My manner has convinced them. I am singularly at ease. They sit, and while I answer cheerily, they chat away about trivial things. But before long, I feel myself growing pale and wish them gone.

My head aches and I hear a ringing in my ears, but still they sit and chat. The ringing becomes more distinct. I talk more freely, to get rid of the feeling, but it continues and gains volume — until I find that this noise is not within my ears.

No doubt I now grow very pale — but I talk more fluently, and with a heightened voice. Yet the sound increases — and what can I do? It is a low, dull, quick sound. I gasp for breath — and yet the auditors hear it not.

I talk more quickly — more vehemently — but the noise steadily increases. I arise, and argue about trifles, in a high key and with violent gesticulations — but the noise steadily increases. Why will they not be gone? I pace the floor back and forth, with heavy strides, as if excited to fury by the unrelenting observations of the auditors — but the noise steadily increases.

What could I do? I raved — I ranted — I raged! I swung my chair and smashed my computer with it — but the noise rises over all of my attempts to silence it. It grows louder — louder — louder! And still the auditors chat pleasantly, and smile. Is it really possible they can not hear it? Is it really possible they did not notice me smashing my computer?

They hear! — they suspect! — they know! — they are making a mockery of my horror! — this I thought, and this I think. But anything is better than this agony! Anything is more tolerable than this derision! I can not bear their hypocritical smiles any longer! I feel that I must scream or die! — and now — again! — the noise! Louder! Louder!! LOUDER!!!

“DATA!” I finally shriek. “DATA has no quality! NO DATA QUALITY!!! What have I done? What — Have — I — Done?!?”

With a sudden jolt, I awaken at my desk, with my old friend Edgar shaking me by the shoulders.

“Hey, wake up! Executive management wants us in the conference room in five minutes. Apparently, there is a vendor here today pitching a new system called DATA using software called Magic Beans...”

“...and the salesperson has this weird eye...”

The 2010 DataStage Roadmap Now with 100% Less Release Numbering

2009-10-29T20:36:41Z

The DataStage Roadmap news from the IBM IOD 2009 Global Conference from Las Vegas with the enhancements to the product following the DataStage 8.1 release.

Profit by Data Quality Best Practices

2009-10-29T11:01:19Z

In this post, Virginia Prevosto,FCAS and Peter Marotta, AIDM from ISO provide an account of how best-practice techniques ensures high levels of data quality across billions of insurance premium records.

Profit by Data Quality Best Practices

For Want of a Nail the Kingdom was Lost

Like the missing horseshoe nail that cost a kingdom in the old Mother Goose nursery rhyme, faulty data can produce devastating bottom line consequences in the property / casualty insurance business.

Consider how incomplete or miscoded data can produce cascading miscalculations in underwriting, risk selection, coverage and pricing. Data is at the heart of customer service and support functions. Claims data — from both internal and external sources — support decisions and planning for claims settlement. Past claims experience help companies identify emerging claim trends. The quality of data can lead to either profitably helpful or dangerously misleading predictions for future claims-adjustment and settlement costs.

Insurers use data to manage litigation, detect fraudulent claims and limit financial exposure to claims through reinsurance, but this practice works only when the data is credible. It is no overstatement that sound, profitable property / casualty operations begin – and end – with quality data.

But what is quality data and how can companies attain and sustain that quality?

Our firm, whose expertise in data aggregation and management is widely recognized by the insurance industry, has developed best practices and standards to ensure the quality of our statistical data. We maintain one of the largest private databases in the world, with 10.6 billion records at any given time. Although we have developed these principles for the insurance business, the elements and principles that follow are equally applicable to all organizations.

We define quality data as data fit for its intended use. The five key characteristics of quality data are:

Accuracy: information in the database represents exactly what it is supposed to
capture.
Validity: the value of a data element in the database is identified as an allowable
value.
Reasonability: data is consistent with prior data or other available information.
Completeness: every recorded transaction contains all necessary information, and all
pertinent transactions are being recorded and reported.
Timeliness: transactions are consistently recorded, processed and changed within
established and prescribed timeframes.

With the above characteristics of data quality in mind, here are the key principles we recommend as guidelines to organizations for managing data quality:

Data Stewardship

Maintain a corporate program with senior management oversight
Understand roles and responsibilities in data ownership, acquisition, quality assurance, storage and distribution
Make each functional area with data responsibility accountable for their own performance and data management

Data and Data Quality Standards

Develop internal standards and, where appropriate, seek useful external standards
Harmonize multiple standards and promote consistent operations across multiple systems and platforms

Organizational Issues

Establish an in-house unit to create and assess data
Aquisition across the organization
Tap into external resources where appropriate

Operations and Processes

Develop processes to maximize data quality and use new technologies to manage data
Monitor regulatory requirements that may affect data and data quality

Data Element Development and Specification

Design and maintain data, system, and reporting mechanisms for sound data management and quality for end-user service
Review the current level of data detail and assess whether or not historical or retrospective data are necessary for developing system or reporting specifications
Define data element and design data-reporting specifications to enable convenient modifications and updates

Data Management and Data Quality Tools

Develop tools such as a corporate data dictionary, edits and business rules, data-flow documentation, process model and mapping, and data-translation criteria by data source and recipient
Leverage technology resources like the Internet, predictive and data-visualization tools, and new data exchange standards for improved data management and quality

Measurement

Develop performance metrics to measure poor data quality, such as costs for crrecting errors and reports, investigating and preventing errors, fines and regulatory scrutiny.
Benchmark results for each data source

Individual support

Institute support for data management and data quality at both individual and organizational levels

Privacy

Educate users about privacy issues, policies and compliance with privacy regulations
Control access to, and use of, non-public data
Promote best practices in data privacy

In Summary

Just as organizations protect their financial, physical and intellectual property assets, so should organizations ensure that their data assets are accurate, reliable and protected from unauthorized access.

As businesses in various industries "go global", the value of data quality for informed business decisions, financial success, and market reputation will also grow exponentially.

Does your organization adopt any of the techniques in this article? What are your experiences? Please share your views below.

Did you find this article useful?

If so, please tweet it: "Profit by Data Quality Best Practices" http://bit.ly/2Hkd1w #dataquality #datagovernance

Useful Resources

See all posts in: Methodology

About the Authors

Virginia Prevosto and Peter Marotta are principals in the Consulting and Research Department of ISO (http://www.iso.com), based in Jersey City, N.J., maintains one of the largest private databases in the world by obtaining roughly 2 billion detailed records of insurance premiums collected and losses paid each year.

ISO’s professional staff members analyze insurer data and turn it into meaningful information. ISO provides data, analytics and decision-support solutions to professionals in many fields, including insurance, finance, real estate, health services, government and human resources.

About IAIDQ

Celebrating its 5th Anniversary this year, the IAIDQ is the only Professional Association focusing specifically on the needs and issues of information quality and data quality professionals. Lead and run by a team of volunteers (all of them practitioners or researchers in the Information Quality field), the IAIDQ runs on a not-for-profit basis to develop the profession and the skills and understanding of professionals. The Association was founded by Larry English and Tom Redman.

Among our key initiatives are the development of a vendor neutral certification for Information Quality professionals (the CIQP certification), the development of Communities of Practice in various areas of interest (geographic, industry based, knowledge-domain based), and generally working to raise awareness of the importance of good quality information as a critical business asset and the importance of managing it as such.

The Association has recently published the first ever Salary and Job Satisfaction study in the Information/Data Quality Profession, which examined the remuneration and satisfaction of IDQ professionals from around the world. That report can be downloaded from the IAIDQ Website (http://iaidq.org/publications/pierce-2009-07.shtml)

You can find us at: http://iaidq.org, or on Twitter at http://twitter.com/iaidq or on LinkedIn (where we are rolling out extensive supports for our Communities of Practice).

Building A Foundation For MDM

2009-10-28T22:55:09Z

Since recently attending the Gartner MDM Summit, reading the latest report on MDM Trends from Forrester, and speaking with several customers, a few trends are starting to emerge:

There is no one single technology that will help organizations solve all of their MDM challenges
While some MDM products are supporting multiple domains, they are still either customer-centric or product-centric
Analytical style of MDM is gaining in importance, and
The common challenges across all data domains and hub styles are data integration and data quality

At Informatica, we have been working with many customers that are seeing these same trends as well. They are selecting different approaches and tools for customer master and product master, sometimes even different tools for operational and analytical MDM applications. But the common threads are data integration and data quality:

Data Integration

Access to multiple systems inside and outside the firewall
Ability to handle multiple latencies from real-time to massive batches
Tracking the metadata history, lineage, and audit trails

Data Quality

Profiling source system to understand what you are starting with
Identity Matching to detect duplicates
Address Validation for all countries of operation and customer locations
Cleansing and Standardization to ensure data consistency and accuracy across all attributes
On-going monitoring and data quality scorecards to bring continual visibility to the program

When you look at the Informatica solution set you will see that we provide the capabilities that span these requirements. It’s an opportunity to help our customers be more successful by laying a foundation for MDM that provides a consistent integration layer across all data domains while applying data quality to existing data and having those same data quality rules applied at the point of entry.

What do you think? Is having a consistent foundation for all MDM projects important?

IBM Settles and Microsoft Bails and Teilhard Now Owns Data Integration

2009-10-28T19:00:32Z

Teilhard Technologies owns the rights to data integration software in North America and companies will have to pay up to sell competing software.

IBM is Stampeding Products onto the Cloud

2009-10-28T17:48:59Z

IBM is putting products from most of its Information on Demand product lines onto IBM private and public clouds including DataStage and QualityStage.

Bank of Ireland – again

2009-10-28T12:43:40Z

The Irish Times today reports that Bank of Ireland are again investigating incidents of double charging of customers who use LASER cards.

I wrote about this last month (see the archives here), picking up on a post from Tuppenceworth.ie earlier in the summer. I won’t be writing anything more about the issue (at least not for now).

Looking back through my archives I found the picture below in a post that I’d written back in May when Simon on Tuppenceworth first raised his issue with BOI’s Laser Cards.

Information Quality Aggregator

The Once and Future Data Quality Expert

IAIDQ

Data Quality Expert

The Times They Are a-Changin'

Share Your Perspectives

Related Posts

Additional IAIDQ Links

Patent Reexamination Hurdle Remains For Data Integration Patent

Do You Have A Foundation For MDM Success?

IBM Reveals the First Fourteen InfoSphere Information Champions

The Mullet Blogging Manifesto

Blogging is All about You

B = U2C3

Laugh, Think, Comment

Stories and Metaphors and Analogies! Oh, my!

The King, Queen, and Crown Prince of Blogging

Emerson on Blogging

Additional Resources

October El Festival del IDQ Bloggers on Name Cleansing and a Russian Pipeline

Economic Uncertainty Takes Its Toll on Data Security

Data Quality Blog Roundup - October 2009

Data Quality Blog Roundup - October 2009

Useful Resources

Customer Incognita

What is a Customer?

How Many Customers Do You Have?

Conclusion

Data Governance & Jitterbit

Data Governance: The Building Blocks for a Well-Managed Organization

The Right People

The Right Solution

Enter Jitterbit – Data Integration Simplified

Win-Win Governance

5th Normal Form: The Achilles Heel of ETL & Data Warehouse Projects

5th Normal Form: The Achilles Heel of ETL & Data Warehouse Projects

What is 5th Normal Form?

How Do Breaches of 5NF Arise?

Normalising Tables in Databases

Simplistic Data Modelling

The most common cause of this is the mechanistic resolution of many-to-may relationships in a data model.

The Function Solves the Problem

Merging Data from Disparate Databases

Summary

Useful Resources

The Chaos Theory of Data Quality

Related Posts

No smoke without ire – Life Insurance Overcharging in Ireland

The Tell-Tale Data

The 2010 DataStage Roadmap Now with 100% Less Release Numbering

Profit by Data Quality Best Practices

Profit by Data Quality Best Practices

For Want of a Nail the Kingdom was Lost

In Summary

Useful Resources

About the Authors

About IAIDQ

Building A Foundation For MDM

IBM Settles and Microsoft Bails and Teilhard Now Owns Data Integration

IBM is Stampeding Products onto the Cloud

Bank of Ireland – again

B = U²C³

What is 5^th Normal Form?