Master Data Management

NoSQL, NewSQL and MDM

noreply@blogger.com (Ash Srivastava) — Sat, 23 Jul 2011 10:31:00 +0000

Fixing poor data quality at its source and managing constant change is what Master
Data Management is all about. As new database technologies are evolving they will change the MDM solutions landscape improving performance and scalability of working with large datasets(billions of rows). Most MDM solutions use RDBMS(MSSQL, DB2, Oracle) for managing data and we all know performance tuning is a pain point in today’s MDM solutions, lets look at the exciting new things happening in the database world as these improvements will come to the MDM landscape sooner or later.

Evolving DB Landscape

For Matthew Aslett, senior analyst at the 451 group, there are currently three trends in the industry:

the NoSQL databases, designed to meet the scalability requirements of distributed architectures, and/or schemaless data management requirements,
the NewSQL databases designed to meet the requirements of distributed architectures or to improve performance such that horizontal scalability is no longer needed
the Data grid/cache products designed to store data in memory to increase application and database performance

What is NoSQL?

NOSQL is a sort of all-encompassing term which includes all kinds of databases which do not use SQL or use very little SQL. The following are the types of NoSQL Databases most popular:

Wide Column Store / Column Families - Hadoop / HBase, Cassandra,Cloudata, Cloudera, Amazon SimpleDB
Document Store – MongoDB, C ouchDB, Citrusleaf
Key Value / Tuple Store - Azure Table Storage, MEMBASE, GenieDB,
Tokyo Cabinet / Tyrant, MemcacheDB
Eventually Consistent Key Value Store - Amazon Dynamo, Voldemort
Graph Databases – Neo4J, Infinite Graph, Bigdata
XML Databases - Mark Logic Server, EMC Documentum, eXist

Other Technologies used with NoSQL:
1. Memcached - Memcached is a general-purpose distributed memory caching system that was originally developed by Danga Interactive for LiveJournal, but is now used by many other sites. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source (such as a database or API) must be read. Memcached runs on Unix, Windows and MacOS and is distributed under a permissive free software license.

2. MapReduce - MapReduce is a patented software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers. MapReduce has different flavors the most common being Hadoop MapReduce

Good:

1. Elastic scaling
2. Big data - the volumes of “big data” that can be handled by NoSQL systems, such as Hadoop, outstrip what can be handled by the biggest RDBMS.
3. Economics - NoSQL databases typically use clusters of cheap commodity servers 4. Flexible data models

Bad:

1. Support – Lack of Support is a turnoff for Enterprises
2. Analytics and business intelligence - NoSQL databases offer few facilities for ad-hoc query and analysis.
4. Administration - NoSQL today requires a lot of skill to install and a lot of effort to maintain.
5. Expertise - almost every NoSQL developer is in a learning mode.
6. Does not Support ACID(atomicity, consistency, isolation, durability) making it suitable for non-transactional purposes only

What is NewSQL ?

As per the 451 Group, “NewSQL” is shorthand for the various new scalable/high performance SQL database vendors. [...NewSQL vendors] have in common the development of new relational database products and services designed to bring the benefits of the relational model to distributed architectures, or to improve the performance of relational databases to the extent that horizontal scalability is no longer a necessity.

We would include (in no particular order) Clustrix, GenieDB, ScalArc, Schooner, VoltDB, RethinkDB, ScaleDB, Akiban, CodeFutures, ScaleBase, Translattice, and NimbusDB, as well as Drizzle, MySQL Cluster with NDB, and MySQL with HandlerSocket. The latter group includes Tokutek and JustOne DB. The associated “NewSQL-as-a-service” category includes Amazon Relational Database Service, Microsoft SQL Azure, Xeround, Database.com and FathomDB.

It is only a matter of time as these new technologies mature and will make their way into the MDM landscape. Exciting times ahead !

HTML5 and MDM – What you need to know?

noreply@blogger.com (Ash Srivastava) — Sat, 23 Jul 2011 03:18:00 +0000

There is nothing you need to know about HTML5 from an MDM perspective. In future maybe HTML 5 might have some graphing tags which could be useful for Business Intelligence but at this point it is a far fetched dream. Anyways it is good to be aware of new technologies as you never know what you can benefit from.

What is HTML5?

HTML5 is the much needed upgrade to HTML. HTML5 will be the new standard for HTML, XHTML, and the HTML DOM. The previous version of HTML came in 1999. The web has changed a lot since then.

HTML5 is still a work in progress. Some rules for HTML5 were established:

New features should be based on HTML, CSS, DOM, and JavaScript
Reduce the need for external plugins (like Flash)
Better error handling
More markup to replace scripting
HTML5 should be device independent
The development process should be visible to the public

New Features

Some of the most interesting new features in HTML5 are new tags for:

The canvas element for drawing
The video and audio elements for media playback
Better support for local offline storage
New content specific elements, like article, footer, header, nav, section
New form controls, like calendar, date, time, email, url, search

Browser Support

HTML5 is not yet an official standard, and no browsers have full HTML5 support. But all major browsers (Safari, Chrome, Firefox, Opera, Internet Explorer) continue to add new HTML5 features to their latest versions.

What does business need ?

noreply@blogger.com (Ash Srivastava) — Thu, 06 May 2010 17:16:00 +0000

An excerpt from Data Intelligence Gap

So, exactly what is it that the business needs to know that the data can’t provide? Here are some examples:

What the Business Wants to Know	Data needed	What’s inhibiting peak efficiency
Can I lower my inventory costs and purchase prices? Can I get discounts on high volume items purchased?	Reliable inventory data.	Multiple ERP and SCMsystems. Duplicate part numbers. Duplicate inventory items. No standardization on parts descriptions and numbers. Global data existing in different code pages and languages.
Are my marketing programs effective? Am I giving customers and prospects every opportunity to love our company?	Customer attrition rates. Results of marketing programs.	Typos. Lack of standardization of name and address. MultipleCRM systems. Many countries and systems.
Are any customers or prospects “bad guys”? Are we complying with all international laws?	Reliable customer data for comparison to “watch” lists.	Lack of standards. Ability to match names that may have slight variations against watch lists. Missing values.
Am I driving the company in the right direction?	Reliable business metrics. Financial trends.	Extra effort and time needed to compile sales and finance data – time to cross-check results.
Is the company we’re buying worth it?	Fast comprehension of the reliability of the information provided by the seller.	Ability to quickly check the accuracy of the data, especially the customer lists, inventory level accuracy, financial metrics, and the existence of “bad guys” in the data.

Again, these are some of the many reasons where data lacks intelligence and can’t provide for the needs of the corporation.

Suspect Duplicate Processing in IBM MDM Server

noreply@blogger.com (Ash Srivastava) — Wed, 05 May 2010 15:09:00 +0000

The task of evaluating data, finding suspects in the data and collapsing them based on rules is an exhaustive process. If the suspects do not have a high possibility of a match then what action should be taken? How can automated merge be leveraged so the manual process of collapsing data can be minimized?

There are a lot of questions which the business wants the answer for before it can make an informed decision. Lets talk about the basic terminology which business should know when talking about Suspect Duplicate Processing(SDP) or Duplicate Suspect Processing(DSP).

What is SDP ?
IBM MDM can identify the duplicate parties in real-time, as part of adding or updating the party data or offline as part of Evergreening. Suspect Duplicate Processing (SDP) feature provides mechanism to identify these duplicate parties. Terminology Business users should know:
- Critical Data
- Match/Non-Match Score
- Match Category
- Match Matrix

What is Critical Data ?
The term ‘Critical Data’ refers to data elements that are selected by business to be used for comparision in SDP. If all Critical Data fields match between two records then they are considered exact match. For Example: Last Name, SSN, Address Line One

What is Match/Non-Match Score?
Each critical data element is given a score
For example:

Critical Data

Match Relevancy Score

Non-Match Relevancy Score

Last Name

1

1

SSN

2

2

Address Line One
8
8

What is Match Category ?
Match category is based on the Match/Non-Match Score.Out of the Box(OOTB) there are 4 Match Categories:
A1 - Match/Non-Match score indicate that a definite duplicate party has been found.
A2 - Match/Non-Match score indicate that high probability that a duplicate party has been found.
B - Match/Non-Match score indicate that it is fairly unlikely that a duplicate party has been found.
C - Match/Non-Match score indicate that the suspect party is not a duplicate.
These categories can be customized.

What is the Match Matrix?
Match Matrix brings together Match/Non-Match Scores and Match Categories.
- 0 means data element not present in either or both the new and existing records
- Negative value means data element is present in both the new and existing record and it does not match
- Positive Value means data element is present in both the new and existing record and it matches

Last Name	SSN	Address Line One	Match Score	Non-Match Score	Category
1	2	8	11	0	A1
-1	-1	-8	0	11	C
1	2	0	3	0	A2

The categories in the match matrix are decided by business. Hopefully this provided a basic overview of the Suspect Duplicate Processing concept in IBM MDM. Feel free to leave comments or ask questions.

Five Pillars of MDM

noreply@blogger.com (Ash Srivastava) — Wed, 03 Mar 2010 23:38:00 +0000

The first pillar, content, is not only the backbone of MDM, but also the foundation of any
data model. Master data content is usually enterprise-oriented and highly specific to
your business, relying on terminology that is often unique to your company.
What separates MDM from other technologies is that, in mastering the subject area,
it goes beyond the mere content of your data and presents relationships between
different data elements, including hierarchies and groupings. For instance, a “household”
frequently refers to a group of people living in the same location. Such a grouping
requires its own set of rules and definitions, as established in a data model. While a retailer
may merely group all the related people living at the same physical address, a bank, for
example, may only consider wage earners living at a given address. Again, it depends on
your business. Establishing the parent/child hierarchy within a given household may be
crucial for some businesses, such as insurance companies, but not others.

Similarly, MDM is able to support access rules to deal with the details of data flowing
into and out of the MDM hub. The delivery (or provisioning) of the data content and
relationship details to other systems means enforcing the rigor associated with security
and access policy at an individual data-element level to ensure that all rules and details
associated with data access is maintained. While most believe that data access or
linkage is limited to in-house or corporate systems, MDM can support any data source,
so integrating with third-party or external data is no more complex than an internal
billing system. As more and more data is available and linked, tight access control is
essential to ensure that data doesn’t fall into the wrong hands.

Because corporate information is always changing, mastering data also involves
supporting a robust change control component within MDM. Business data changes
in real-time, and an MDM environment must recognize which changes are acceptable.
A mature MDM hub understands when a data element change can be supported
in an automated fashion, or when it requires intervention from an external process
(e.g. another system, a data steward, etc.). Unlike traditional IT-based change control
methods that focus on application changes or business process changes, MDM focuses
on data value changes.

MDM also involves the actual processing of data, from basic matching and identification
to data correction and CRUD (create, read, update, delete) processing. Processing rules
contain the details for determining whether two records are the same. For example,
are customers Robert Smith and Bob Smyth the same? If so, should one of the values
be updated? If Bob calls and updates his address, should that change be propagated
throughout the other databases in the system, including the master record? Does it
matter if the address change comes from a third-party data vendor instead? Which
update should the system trust? And if it turns out that the data was incorrectly entered,
the data model should support unwinding the match to set the record straight.
This discussion of the five pillars is helpful in understanding the differences between
MDM and other strategic technology solutions, and will help to put the data model in
better perspective.

in reference to: Data-Modeling-and-MDM.pdf (application/pdf Object)

Analytic Auteurs

noreply@blogger.com (Ash Srivastava) — Wed, 03 Feb 2010 22:12:00 +0000

This is an interesting excerpt from SmartDataCollective article by Steve Bennett calledAnalytic Auteurs. To read the complete article please visit SmartDataCollective

"Now I never thought of myself as an auteur, but my favourite analogy for 'doing analytics' was making a movie. The analogy runs something like this:

As the director (think of me as a cross between Tim Burton and Spike Jonze) I assemble a team for 1 - 2 years and together we create the analytic solution the organisation needs. The solution is generally made up of two components. The first is a suite of information products (data marts, cubes, reports, dashboards, etc.). The second is the skills for the organisation to 'self serve' and create new information products without us.

We then we disperse to different places. Only to meet again when the next 'movie' opportunity arises.

My trusted experts are:

* Business Analyst - Script Writer
* Data Integrator - Special Effects
* Solution Designer - Cinematographer
* Information Designer - Set Designer/Head of Props
* Quality Control - Post-Production
* Change Agent - PR and Marketing

Why do the same people agree to work with me more than once? Well, you'll have to ask them to really know. But I can tell you what I tell them when I'm trying to get them them for one more gig. In no particular order I promise them:

* A real intellectual challenge
* A core team they can trust
* That they will learn from both the challenge and their fellow team members
* Very good remuneration - at least as good as they will get elsewhere
* Fun!

Don't get me wrong. The core team does get new blood each time and sometimes people just muscle their way into the team by being damn good at what they do. Never underestimate serendipity and the pleasure at meeting another person of whom you think 'wow - they're good!'"

How Multidomain MDM Can Support Business Users

noreply@blogger.com (Ash Srivastava) — Wed, 03 Feb 2010 20:59:00 +0000

How Multidomain MDM Can Support Business Users

Multi-Domain MDM: Kid in a candy store ?

noreply@blogger.com (Ash Srivastava) — Tue, 22 Dec 2009 23:18:00 +0000

Market is maturing from Customer Data Integration(CDI) to more holistic view of Master Data Management. Multi-domain MDM seems to be the buzz of the moment, even though when companies do not seem to have mastered the art of single domain MDM.

It is one of those conformist truisms which people will never dare contradict to your face. Go to any Business Intelligence, Master Data Management or Data Governance conference, website, blog or individual and they will all sing from the same hymn sheet here. Multi-domain is the MDM panacea. - Duncan Slater, Accenture Information Management Services

I could not agree more with Duncan's article on Enterprise MDM and Apple Pie. Organizations need to think through the people, process and technology aspects of MDM. It goes back to the piece I wrote about Data Governance. It is an industry which is being driven by vendors rather than the companies which need the solution. Vendors are selling big dreams for big revenue, and clients are like kid in a candy store.

The client needs to step back and evaluate their needs before they go shopping for buy-one-solve-all problems system.

In Memory Computing

noreply@blogger.com (Ash Srivastava) — Sat, 19 Dec 2009 18:00:00 +0000

Gartner MDM Summit - Siperian Perspective

noreply@blogger.com (Ash Srivastava) — Thu, 17 Dec 2009 18:04:00 +0000

The hot topic at the summit was multi domain MDM.

Interesting notes from the conference:
• Gartner expects the market for MDM to grow from $1.1 billion (in license revenue) in 2008 to $3 billion in 2013.
• North America is still the largest market for MDM (over 50%) followed by Europe (35%).
• IBM, SAP and Oracle control 40% of the MDM market, while specialists like Siperian control 30%.
• Analysts are saying that there’s a “land grab” on for multidomain MDM, but most vendors have yet to span master data province.

The future looks exciting.

In reference to:
- Master Data Management (MDM) Solutions - Customer Data Integration (CDI) Hub by Siperian

Data Governance – The Rock on Which MDM is Built

noreply@blogger.com (Ash Srivastava) — Sat, 12 Dec 2009 02:27:00 +0000

Andy argues that data warehousing and ERP were failed attempts to get "one version of the truth". In MDM emphasis is on business ownership of data i.e. Data governance. Well written. Having dedicated resources to take ownership from business and clear definitions of "customer" and "product" are crucial to success.

"Building a proper business case is a key element of an MDM initiative. It is only by analysing the costs of the current state of duplicate data and estimating what that is costing the business in lost orders, incorrect deliveries or invoices, misplaced inventory, inability to cross-sell or lower customer satisfaction and retention, that executives will stand up and take notice.

By building a business case around such hard data, it should be possible to get senior management attention. In some cases where this has happened, it has also been possible to gain the necessary senior level support for true data governance initiatives. Such activities, business-led rather than IT-led, involve identifying the key data and processes of a company and agreeing who should be ultimately responsible for defining business terms. Such groups also take ownership for resolving disagreements when they arise, e.g., over whose definition of “gross margin” should actually be used. Significant resources need to be assigned to such activities, not just as a one-off project basis but on a permanent basis.

The best run companies are putting in place such data governance initiatives and giving budget and power to the data governance groups that are needed to resolve definitions and resolve data quality.

An MDM project may still struggle even after data governance has been established, but if a strong data governance initiative can be built and sustained, then MDM projects have a least a chance of success."

Once data governance is resolved then also the success of an MDM team is dependent on a highly competent technical team. Companies like Infotrellis should be invited at this point to ensure successful implementation.

Companies should read this before coming up with their MDM strategy.

in reference to: BeyeNETWORK: Data Governance – The Rock on Which MDM is Built (view on Google Sidewiki)

About :

Andy Hayler - founder and CEO of The Information Difference, a boutique analyst and market research firm, advising corporations, venture capital firms and software companies. He is a regular keynote speaker at international conferences on master data management, data governance and data quality. He is also a respected restaurant critic and author (www.andyhayler.com). Andy has an award-winning blog www.andyonsoftware.com. He can be contacted at Andy.hayler@informationdifference.com.

Infotrellis - InfoTrellis provides strategic consulting to help clients define and achieve their Customer Data Integration (CDI) and Master Data Management (MDM) vision. It was founded by the team which architected and developed WebSphere Customer Center (WCC), formerly known as DWL Customer, the market leading CDI/MDM solution by IBM/DWL and in the process created the CDI/MDM space. Services provided by InfoTrellis team cover all aspects of a CDI/MDM solution lifecycle ranging from evaluating client's CDI/MDM needs, capturing detail business requirements, vendor selection, design & implementation, testing, deployment and training. info@infotrellis.com

Data Federation Technology

noreply@blogger.com (Ash Srivastava) — Fri, 11 Dec 2009 01:53:00 +0000

This is actually one of the ways of implementing MDM(Master Data Management). The two ways:
i) Storing actual data
ii) Storing references to actual data(Data Federation)

There are pros/cons with both. So the implementation method chosen should be considered carefuly based on needs

in reference to:

"data federation technology"
- What is data federation technology? - Definition from Whatis.com (view on Google Sidewiki)

Outsourcing Your Data Warehouse

noreply@blogger.com (Ash Srivastava) — Fri, 11 Dec 2009 01:40:00 +0000

Steve makes a point. Keeping Data in house teaches a company about their data. The knowledge gained by their staff about the data while developing these applications is valuable and can assist business.

in reference to: Smart Data Collective | Outsourcing Your Data Warehouse

Master Data Management and Customer Data Integration for a Global Enterprise

noreply@blogger.com (Ash Srivastava) — Fri, 11 Dec 2009 01:04:00 +0000

I will add this to my list of reads.

Critical Data	Match Relevancy Score	Non-Match Relevancy Score
Last Name	1	1
SSN	2	2
Address Line One	8	8