Thank you all for being regular visitors to this blog, liking my content and promoting them in social media. I really appreciate all the support I received.

I am looking forward to a great 2012 with lot of new things planned. And promise to have even better content. Also look forward to my renewed blog site which is under construction. I will keep you posted.

Have a wonderful 2012 and keep rocking!!

Click here to see the complete report.

Here’s an excerpt:

A San Francisco cable car holds 60 people. This blog was viewed about 1,600 times in 2011. If it were a cable car, it would take about 26 trips to carry that many people.

Duplicate processing starts as soon as customer records are consolidated in MDM from the different sources data, or at worst, duplicate records found in same system in absence of business rules to prevent it.

The process has the following two steps and can take significant effort depending on how distorted the data is.

Step 1: Identifying duplicates (Data Matching)

Data matching is a process of identifying duplicate records. There are 2 matching techniques which today’s MDM applications use to detect duplicates.

1. Deterministic – Exact compare of data elements to assign scores.
2. Probabilistic – Matching based on phonetic (sounds like), likelihood of occurrence, leveraging statistical theory, and pinpoint variation and nuances. This technique assigns a percentage indicating the probability of match.

Although there are certain advantages to both the matching techniques, probabilistic matching scores an upper hand due to high accuracy in matching records.

In a recent article, Scott Schumacher compares Probabilistic and Deterministic Data Matching which explains key differences between above two matching techniques. Albeit matching is a very interesting (and one of my favorite) topic, lets save this for another column and concentrate on step 2, the main objective of this post.

Step 2: Merging Duplicate Records (Surviving golden record)

Once the duplicate records are identified, they need to be merged to create what we call as “single version of truth” or “golden record”. This record by definition will be the best possible data representing current, complete and accurate view of customer information.

The main challenge we often face during merging process is determining what data elements and their values from duplicated customer data needs to be considered to create the golden record. The foundation stone is building “Intelligent” data survivorship rules during automated merging when system has to decide what data to pick from duplicated records. The controls built into MDM should be comprehensive so complete and accurate master information is realized.

Below are some of the rules which can be followed in an automated customer de duplication process.

1. Select the data element coming from a most trusted source. In other words, the rules should know which source data element has higher priority. We usually figure out which source gets more priority by employing a step called data profiling on the source data. This will help us get metrics on data quality. More on profiling can be found here.
2. Pick the data which is most recently changed. You would want to select a most recently updated address from source A over older address from a duplicate record coming from source B.
3. Choose the data elements which are populated with more details. For example,
1. For names, choosing Caroline over Carry, Robert over Bob, Michael Harris over M. Harris etc.
2. For address, choose1607 Chestnut Drive. over 1607 Chestnut Dr
3. Choose postal codes which are complete. For example 43219-1933 over 03219 (zip+4 is better than incomplete zip)
4. Ignore null and empty values of suspect records.
5. Ignore ambiguous values. Ex: 123-45-6789 for social security number. (Well, these values should not be here if only you were punctual and did good data governance and data quality control beforehand).

Above list provides some of the commonly used rules during data stewardship. However there may be some variations depending on quality of data and customer situation. Sometimes you may need to build these specialized rules based on how new customer records are created in the system over updating existing records. However, the above guidelines are good starting point.

Do share your thoughts.

To get money out of consumer’s pocket, businesses are constantly exploring new ways. They often ask questions such as, what’s the minimum my company can do to keep customer satisfied. How to win customers loyalty under tough competition from rivals? Who are our true customers and what do they really need from us?

If you observe closely, it all boils down to putting customers at center and designing the business to meet their requirements. The catch is, getting this done while making sure the business does not suffer in the process.

Customer Centricity, as Wikipedia puts it, is a strategy which companies employ to align the organization’s products and services with the wants and needs of their most valuable assets- the customer.

Looking from companies perspective, in my opinion, there are two important benefits with customer centric approach:

• Ensure a healthy relationship is achieved and maintained with customers
• Maximize profits for the organization in the long term.

In the past, organizations would find out what their customer’s necessities are by building loyal relationship.

Let’s take this real life example; I was going to a general store in a small town I grew up for getting the groceries for our family. The owner would simply provide me all the required items for a month and just note down the amount we were to pay by the end of the month in his register. Added to that he also knew- what are the items our family will need for the month and have them ready for me to pick up when I arrive at the store. Isn’t that just great?

Well, it will be somewhat inappropriate to expect such a relationship today between customers and organizations.  It may be the bank, the wireless line provider or the retail store where we shop regularly; each of them operate globally and have millions of customers. It’s hard to understand everyone’s needs. Then there is this changing requirement of customers. The availability of different brands, variety of items and bundled offers today makes it impossible for anyone to predict what product/service a customer needs on a particular day.

However, the good news is the advancement in technology today. We are equipped with tons of intelligent software which can gather the data and convert into useful information. Using these sophisticated technologies, companies have ways to track

• Who their customers are.
• And predict to a comfortable accuracy, what their needs are.

Customer data in an organization is often spread across different departments/lines of businesses. Converting this data into valuable information about customers is the first step towards achieving customer centricity. This customer information is often referred to as master data and will need to be managed in a central repository so as to address data quality issues and duplication. This helps in better analytics and to achieve a single, accurate and complete view of customer information.

Finding out the needs of customers has become easier in past few years due to advancement of social media. There has been an increasing link between MDM and social networks as Gartner puts it in their recent press release. Gartner highlights social network as one of the three trends that will shape the MDM market in coming years.

Organizations can derive crucial insights from social networks like- What are people saying about my company, product and brand? Who is that person? Whom do they know? How much influence does the person have on his network? This information can help in understanding current trends, needs of existing and potential customers thus taking business achieve greater customer centricity.

To conclude, customer centric businesses should simply orientate themselves to the needs and behaviors of the customers, rather than internal drivers such as the quest for short term profit. This helps them go long way in having loyal customers and do what they do best, get money out of consumers pocket, in a decent way this time & probably for the stuff their customers really need.

What do you think about this blog? Do share your opinion using the comments section below. I would love to hear from you.

I stumbled upon a new phrase recently, when a colleague of mine from sales used it to explain the benefits of MDM implementation as a customer data hub – the ‘Intimate View of Customer’. It means going a step further and helping organizations in understanding their customers a bit more.

Imagine this scenario – A rich client, who has been the premium customer for a really long time, walks into the bank for renting a safe deposit box. The usual process follows; a smiling bank staff takes charge and pulls customer details on bank’s internal application (which does a SOA call to backend MDM system as the bank now has a fully integrated master data hub for customer information). When customer’s details appear on screen, it shows the complete 360 view of the customer including his full name, his nick names, current address where he resides, multiple billing addresses, his privacy preferences, different accounts and products he has opted in.

Along with all the above information, the staff also gets an alert message which prompts him about an upcoming event related to this customer. For simplicity sake, let us assume the alert message says – This customer’s birthday is coming up on this weekend and he is going to be 60 years old.

Now that the bank employee is having a chance to see the complete view of customer’s details and the number of different accounts/products this customer is having with the bank, it allows him to decide how important this customer is for the bank. Without wasting much time, he talks to his staff (on the background) and arranges for a small birthday cake while he helps in getting necessary paperwork done for safe deposit box.

Later when many of the bank staff gather around the customer and surprise him with their wishes and a cake, the customer gets delighted by the treatment bank has given him.

This depiction is a simple example that shows recording interesting information about the customers is a very important aspect. Like the names of family members (usually stored in the form of relationship or household), birthdays and the attributes like retirement dates, help show your customers that you know them more intimately. When customers ask for a specific service, the organization gains their appreciation by having the answers they want. Also serving up additional information and offers that respond to their needs can deepen that appreciation.

Many organizations today attribute their growth to the culture they adopt while selling. In above scenario you can see how the information provided by MDM can help create a lively atmosphere for the bank’s staff to give a personal touch to the service provided. You can be sure by the time this transaction is over, the customer will truly and madly love the bank and will be so happy to invest more money leading to more profit for the bank.

Customer service is an important factor influencing success of organizations, and yet, the hardest thing to perfect. MDM technology can take your business to a new level of customer service by providing an “Intimate View of Customer”. Because, there is more to customer service than to just have smiling staff and tellers who give lollipops to kids.

What do you think? Share your opinion about this post using the comments section below. I would love to hear from you!!

I had chance to work with several financial customers over past 3 years. Many of these organizations start off their MDM journey with primary objective, and often considered the Holy Grail, to achieve ‘single unified view’ of their customers. They envision MDM system to provide a correct, complete and consistent access to customer data which is otherwise scattered across multiple business units. Banks tend to treat customers as their critical assets as opposed to retail scenario where products are the core assets. They want to reconcile data to identify their unique customers and move out of account-centric systems.

Once functional, MDM hub provides valuable customer information along with customer’s current holdings and positions on each of the financial instruments. This is to ensure that a much deeper insight about customer’s relationship with the bank is achieved. This information can then be used to allow intelligent cross-sell & up-sell. Hence the complete “360 view” is not just about customers but also about the services which the customer is using.

There are multiple ways in which the financial services can be modeled in MDM system. While the representation of services and their linkage to a customer purely depends on the data model offered by the MDM product, there are certain commonalities which we see in every master data expedition.

The above diagram shows how services offered by different business units of the bank to a customer, flow into MDM system. While the existing infrastructure (shown in grey) only show customer’s individual contracts with the company, MDM system (shown in aqua) brings in customer accounts across all business units to deliver full 360 degree view of customer. In this depiction, John Doe and Sarah Doe use multiple services offered by the bank. The bank account opened by John has a credit card service, trading account and insurance attached to it. He also has a home loan from bank where his wife Sarah is a co-applicant. They both have life insurance coverage from the bank where Sarah is beneficiary.

Since we would like to flow different types of services offered by the bank to the MDM system, service modeling in MDM can get really complex. However, there is some good news to it. Unlike customer data, we do not want each and every detail of service products to get reflected in MDM. These services are managed in their respective sources and continue to be owned by them. However, some of the basic information about these services needs to be accommodated so as to know customers better. We call this type of accounts in MDM as referenced accounts. Meaning, they are being referenced by MDM, but they continue to be managed and owned by respective source systems.

Below are some of the guidelines I follow while modeling services in MDM.

• It’s critical to determine which service attributes add value. While data mapping activity, scrutinize an attribute by asking the question – Will this attribute help me in recognizing how valuable this customer is for the bank?
• Attributes like Risk Score, Credit History of the customer, Total Yearly Revenue (for Organization type of customers), and Account Open Date are some of the important attributes which needs to be considered. These attributes allow banks to accurately measure risk exposure.
• Accommodate account to account relationships. Example- the insurance incentive provided along with gold card, mortgage and free checking bundle etc.
• The role played by the customer on an account, different billing addresses he is using for a given contract should be captured.
• An attribute which changes its value many times a day (Ex: Savings bank balance, Payments via Credit card) will not constitute master. These attribute are not only difficult to manage, but also add significant performance overhead to the system.

In a financial scenario, customer is core entity and needs to be managed efficiently. Centralization of customer information allows us to manage this customer efficiently in one place by taking control of data quality, standardization and duplicate removal. Also, MDM makes this data available seamlessly across the enterprise via a standard like SOA. I would try to keep only key information about services and contracts by following above guidelines. This allows us to concentrate on efficiently managing customer data which is the true asset of a financial organization.

What do you think about financial service modeling? What additional aspects can you think of? Do share your thoughts via comments.

Geocoding, as most of us know, is the process of finding geographic coordinates which are expressed as latitude and longitude. For example: [13.746076, 100.540073] represents an address – Thanon Ratchadamri, Pathum Wan, Bangkok 10110 in Thailand.

It’s a truly amazing process which serves as a great guide for people who want to get to their destinations faster. We are seeing geocoding being commonly used on web today for services like finding driving directions, locating geographically nearest gas stations, restaurants or service locations when on road. (Heck, there are 1000’s of location service based apps today which wouldn’t have existed if this process wasn’t there).

Extending David’s discussion further – In future, we would like to see employment of geocoding in the way we store the location information in our Master Data Management systems. Today, MDM products provide provision to store address attributes like – street name, street number, city, state, postal code, country etc. This will have to change eventually as we will be using only 2 attributes to represent the location of our customer with accuracy at its finest.

The biggest advantage which we can gain with geocoding is identifying duplicate addresses.

The missing zip codes and non-standard representation of address fields are some of the issues we have with address formats today. Added to this, each country has its own way of representing addresses. These aspects of address lead to a wrong matching no matter how powerful our algorithms are, causing significant number of false positive (different member – match) and false negative (same member – don’t match) scenarios.

Geocoding resolves these issues as it provides a consistent location format across geographies and is compact and accurate.

So, few decades down the line, if someone asks you your address, act like miss smarty pants and tell them the coordinates.

I can’t wait to get a rebirth & be in 22nd century! If only nature permits!! How about you!?

I often visit this department store which sells variety of merchandise. Obviously, I and my wife realize the fact that we don’t need to visit this shop every month as the prices here are comparatively higher than other similar stores. But I think their promotional coupon (guaranteed to reach our mail box every month) is doing the intended job just perfectly; attracting both of us to spend on unnecessary stuff.

When I visited this store couple of months back, I was asked by the checkout associate to sign up for email promotions to get another $5 waiver on my current purchase amount. Who doesn’t like waivers? I instantly signed up and wore my winning smile.

Not to my surprise, within few days the store started sending promotional emails. And as obvious as it can get, the promotions are daily emails which claim to have best prices in the market. But what nettled me was the fact that there were two copies of email dropping into my inbox every day!!!

Now, why would they be sending two emails with same content? I could only think of below 2 reasons.

• There is a bug in the application which sends these emails. (My geeky side thinks of this first).
• They have 2 copies of my email address stored in their database.

I have a strong reason to believe it’s the latter, since the name spelled in the emails was slightly different – One said Prashanta & the other Prashant; a clear indication that they have got 2 duplicate records of mine in their system.

Next thing I tried was to cancel their newsletter. When I unsubscribed for the first time, I got a notification saying this will take 7 business days (Isn’t that super slow in this fast-forward world? Are we in 21^st century yet?). And it sure did work; I started getting only one copy of the email after about a week. Well, the good thing about it was, I got to choose which name I want them to use.

Above instance might be a simple case of duplicate input of a customer data into the marketing system. My point is, many organizations today don’t have a standard process to identify who their customer is. As we all know, the data entry errors can get worse and can have catastrophic effects. The wrong input of customer data is a common phenomenon considering most of it is done manually. The good news is there are applications available in the market today that have quality control and powerful matching functionalities to help organizations reduce (if not fully remove) duplicates.

The marketing campaigns which are run should concentrate on actual number of “real” customers the store is getting instead of counting email addresses which were gathered. Otherwise, there isn’t an outcome which benefits both the parties. The customer gets pestered by insistent emails as in the case above and the organization ends up having wrong information about customer leading to higher data management cost. As long as there is no accurate and complete view of customer, significant amount of investment by these organizations keeps going down the drain.

There are several master data management products available in market. These products help customer consolidate and manage their master information in one single place. Once set up, the MDM hub contains a wealth of information about an organization’s customers, accounts, and products, all in a convenient, integrated place leading to various forms of information security attacks. As a repository maintaining an authoritative system of record for the enterprise, MDM hub would require extra levels of protection for the information it manages.

When we deploy an MDM system for an enterprise, we hope to wrap multiple security layers around it to protect the data from un-authorized access. Some of the master data is confidential; not only do you not want to lose it, you don’t want others to even view it without authorization. The users and applications which are interested in master data should be able to access this data based on their roles and responsibilities.

To achieve this objective, here are some of the key questions which should be answered before implementing MDM solution.

Who should be able to run transactions?

MDM system should be able to authenticate every incoming request. These transactions may be getting invoked either from a user or from a system which is interested in master record.

Who can see what?

System should allow plugging in data visibility rules. This is to ensure only certain portion of the data is visible to certain users.

Who can modify what?

There should be opportunities to plug in data entitlement rules which define which user/system is eligible to modify which portion of the data as I discussed in my earlier blog Key Master Data Management Functionalities

What was modified, When it was modified?

A comprehensive audit log facility should be provided by the system to capture the time and user details of the transactions performed in the hub. This feature should perform the task of capturing relevant information with acceptable performance overhead.

Along with above features, the product should have a flexible framework which allows it to be configured with existing security provider implementation of the organization. (Ex: IBM Tivoli Directory Server, Netscape Directory Server etc)

Enterprises must present a tight security around master data that meets very stringent data quality levels, especially in light of recent compliance regulations and demands. Increasing enforcement of data privacy regulations make it mandatory to safeguard master data more than ever before. Above mentioned security requirements for MDM system not only help enterprise meet these regulations, but also help in retaining their valuable customers.

I often get looked down upon, showered over by sarcastic smiles whenever I try to bring in performance related topics while designing the solution. I have also seen the same ridiculing faces turning grim when the master data hub fails to deliver the speed which is in agreeable standards. The truth is performance engineering is one of the big things to be considered during any software design and MDM is no exception to this. In fact performance and scalability are two very important aspects of MDM deliverables.

So, now that we know we need to give so much emphasis to performance, let’s see what stops us from getting the system deliver optimal performance.

The first important challenge is the quantity of data a MDM system needs to manage. Depending on the domain being addressed and the size of the organization, the data volume varies anywhere from few millions to few hundred millions. As the data is sourced from distributed applications, volume of data continues to grow over time hampering the performance of the system.

Next thing is the overhead involved in cleaning, grooming, validating and standardizing the data coming into MDM. Whenever a persistent transaction is carried out; MDM needs to compare, match and merge the incoming data. This is to ensure the consolidation of master data from different sources is effective so a single, true version of master data is delivered.

The third major contributing factor to performance is the heterogeneous client infrastructures and variety of hardware configurations. Believe me, there are as many varieties as the number of projects you have worked on. Simply put, each customer is different. There are different database management systems, different middleware, partitions, and hundreds of varying parameters.

Setting up MDM product and tuning it to achieve optimal performance under these diverse environments remains the biggest challenge today. The technical teams need to be more competent than ever before.

Some of the other aspects which impact performance are – poor data model design, inability of the architecture team to look at the big picture and absence of best practices.

In order to provide reliable, secure and scalable MDM solutions which meet customer’s evolving needs for master data across the enterprise, a lot of attention should be given to performance. The growth of data over a period of time should be given special attention while designing the data model. The data validation and matching algorithms need to be efficient. The current and future state of the data needs to be forethought. Engaging a performance evangelist at the beginning of the initiative and designing the system keeping performance in mind are going to take away lot of re-work and re-engineering at the later stages of the project.

I hope you liked this post. Please share your thoughts via comments. I would like to hear from you about your experience implementing MDM with a vision towards performance.

In this blog post, I am trying to list down some of the important features the Master Data Management product (or the MDM solution in general) should have so that the organizations implementing MDM can realize maximum benefit from the initiative in a short duration of time.

Here is my list of key MDM features.

DATA MODEL

• Out of the box support for multiple domains like Customer (including partners, suppliers), Product, Location and Account.
• Efficiently manage relationship between domains (Customer to Account, Customer to Product, and Customer to Location etc).
• Categorization, grouping and hierarchical management of master data entities.
• Easy to configure and administer reference master data.
• Support for localization and internationalization.

DATA QUALITY

• Configurable & easy to implement simple and complex data validations.
• Ability to integrate with data quality tools like IBM QualityStage, Trillium.
• Real time support for standardization of data (Person and Organization names, Addresses, Phone numbers, Email and Identification numbers).

DUPLICATE RECORD PROCESSING

• Powerful data de-duplication algorithm to remove duplicate records (Or identify duplicate records).
• MDM product should be able to collect key data elements of master record which is getting added and find duplicates in the repository. This can be achieved by having good fuzzy search algorithm built into the product (As pointed out by Henrik).
• Ability to easily configure critical data elements required for matching according to organizations requirements.
• Easy to setup survivorship rules to persist accurate, up to date and most recent information resulting in creation of single record.

INTEGRATION

• SOA enabled services as I discussed in my earlier post – MDM and SOA: The Perfect Marriage.
• Support collaborative, operational and analytical styles of implementation.
• Option to load data in batch mode. (Mergers and acquisitions and initial load of data need a faster, reliable and easy load option).
• Option to build intelligent rules to notify important events associated with master data.
• Ability to setup easier integration with 3^rd party applications and business processes that consume data. Provide connectors and adapters to propagate data to any application, database in real time.

CUSTOMIZATION OPTIONS

• Flexible data model and services layer.
• Extensible business services and ability to build composite business services using existing services to meet specific client requirements.
• Option to introduce new domains with minimum development effort.
• High performance, scalable and standard based architecture.

TRACKING CHANGES

• Audit functionality which capture and retrieve WHO – WHEN – WHY changed the data.
• Efficient framework to store and retrieve historical information about master data, support regulatory compliances.
• Track changes coming from source systems and build cross references to each connected system.

SECURITY

• Ability to integrate with existing active directory products of the organization.
• Easy configuration of data visibility rules (Who can see what? Who can change what?)

I know this may not be a complete list. So, I request you to provide your valuable comments about the features listed here and your inputs about any prominent characteristics which I have failed to recognize.