Data Management Strategies

Reject the Suck of Enterprise Information Systems

2010-03-08T06:30:56-08:00

My nephew Dave is a major in the U.S. Army. Dave is risking life and limb serving in Iraq. I am intensely and justifiably proud of him. Everything that my nephew does in Iraq is difficult. The treatise “On War” rightly observes, “Everything in war is very simple, but the simplest thing is difficult. The difficulties accumulate and end by producing a kind of friction that is inconceivable unless one has experienced war.” The soldiers and marines in Iraq experience this “friction” daily. They address this friction with the phrase, “embrace the suck”. This phrase is indicative of their grim determination to succeed in a hostile environment where even the simplest tasks are difficult. “Embrace the suck” is both an implied order and wise advice. It means, “Face it, soldier. This ain't easy. Now let's deal with it.” With bravado and grit like that in the face of a daily and dangerous grind, you can see why I am proud of Dave.

Enterprise IT has its own kind of friction. In enterprise IT, a simple task, such as adding a data attribute to an information system, is difficult. The irony is that this friction is of our own making. Enterprise information systems do not have to suck. Instead of embracing the suck, we ought to make a deliberate effort to reject the suck of today’s enterprise information systems.

In two previous blog entries ("A Less-Stodgy Enterprise IT" and "Flexible Metadata for a Less-Stodgy Enterprise IT"), I talked about the stodginess of enterprise IT. Making even simple changes and improvements to enterprise IT systems is difficult. But it doesn’t have to be this way. To reject the suck and stodginess of enterprise information systems, you need just two things:

Front-end software that can understand new data and metadata without modification
Back-end software that that can help you create and recreate your metadata

The first thing you need is front-end software that can understand new data and metadata without modification. This is amazingly straightforward: web browsers and productivity applications such as Microsoft Office and Open Office perform this job very nicely.

When you add new data attributes to an enterprise information system, new data and metadata can be rendered simply as new sections in a Word document, new columns and rows in an Excel spreadsheet, new fields in an InfoPath form, and new hyperlinks in a browser. When you use web browsers and productivity applications as the UI for enterprise information systems, changing backend data and metadata requires few if any changes to the front-end software.

You might not have realized this, but Microsoft Word 2007 is a terrific XML editor. Try it. Click here to use Word 2007 to edit an XML file. (Word’s native file format is XML.) Actually, Word is more than an XML editor; it is a powerful UI into XML data. So is Excel. So is InfoPath. And web browsers are quite good at displaying XML data. When you have XML-savvy UI tools on the front-end of enterprise information systems, you are free to be flexible with XML data and metadata on the back-end.

The second thing you need to reject IT system stodginess is back-end software that that can help you create and recreate your metadata. This is also amazingly straightforward: XML databases perform this magic.

The data inside enterprise information systems is usually structured in a way that makes its metadata obvious. Metadata is usually defined via the native structure of the data itself (columns, rows, paragraphs, sections, etc.) However, it is vital to realize that metadata can also be defined through tags and annotations. If the data storage format supports it, existing data can be annotated with new tags. By using annotations, you can change the metadata of your data. In other words, you can use tags to add new metadata to your existing data.

Relational database technology is over 30 years old, and it shows. Relational databases do not readily enable annotations to existing data. The metadata in relational databases is defined exclusively through the native structure of the data – the columns and rows.

Unlike relational databases, XML is built around tags and annotations. XML data offers flexible metadata. XML is designed from the ground up to enable you to change the metadata. You are free to add and/or transform the annotations to XML data anytime you need to. A good XML database, such as MarkLogic Server, can automate the task of creating and recreating your metadata. In fact, a good XML database will do entity extraction, whereby you can tell the database to search through your existing data and insert new annotations around new entity types you define and want to find and process.

Relational databases can’t really do this. It is difficult to change the metadata of relational data. About the only thing you can do is change the structure and thereby the metadata of relational data (by using the DBMS’s DDL or by using ETL processes). And then you still have to change all of the software that sits in front of that relational data, because there are no nifty UI tools that render relational data the way that Office can render XML data. So information systems that have relational data on the backend tend to have lots of stodginess.

With XML, you can change the metadata on the backend without changing the software that implements the UI. Yes, you will likely be obligated to change the software that implements the business logic, but that is precisely where XQuery and keeping the data in XML all of the time makes things easier.

In a typical enterprise N-tier application, you have data in a relational database, with OO code (and probably some SQL stored procedures too) to implement the business logic, and more code of some kind to implement the UI. If you have adopted SOA, the app also has internal and external service interfaces. So a typical enterprise information system has a data model, an object model, a message model, and a UI model (which all must align with the business model). And then it has a bunch of code to deal with the impedance mismatches between all of those models. So, when you have to change the data model and its associated business logic, you are obligated to change lots of extraneous code to boot.

Now imagine an enterprise information system that has all of its data in XML and its business logic in XQuery. Adding a new data attribute is straightforward. Sure, you might have to change some XQuery code for the business logic, but the XQuery code is pure business logic, and is not encumbered by any of that extraneous code that translates the data between the various models in a traditional app. XML is used as the data model, the message model, and the UI model, and the data readily flows between the back-end and the front-end. I have defined a set of XML-based development tools that I call the "XQuery Development Stack". (The XQuery Dev Stack is ideally suited for MODS -- the Methodology for Overcoming Data Silos.) Information systems built with MODS and the XQuery Dev Stack are not stodgy. They are not stodgy because the metadata is easy to change and the data is not buried beneath a mountain of software.

You will notice that I did not mention implementing an object model in an XML-based information system. There should be no object model in an XML-based information system. Object-oriented code is largely responsible for the suck of enterprise information systems. Don’t embrace the suck of enterprise information systems, reject it. With OO development, the dose is the poison, and enterprise IT has overdosed on OO programming. My colleague Joe Maquire wrote about this in an earlier blog entry here. I will talk more about the poison that is OO development in a future blog entry.

Who's Managing the Data Assets?

2010-03-07T17:49:28-08:00

by Noreen Kendle

Most organizations are aware of the value of their information/data assets; information has become a core asset to many organizations. If data is lost or stolen, there is a definite cost; and in most cases, a total data loss will bring an organization to its knees. Data is a representation of the business. The business uses this data to operate, track, report, manage, and predict the business; data is clearly a business asset that needs to be managed. Business assets are traditionally managed as part of the business operations. So what is the role/responsibility of an IT department in the management of this business asset?

IT has traditionally been responsible for designing and applying technology solutions to business problems, as well as support of those solutions. The focus of IT has been the application and management of technology. The same approach has been used for the data assets. IT has traditionally approached data by applying technology solutions to store, move, protect, and process the data.

A database is typically used to store the data. A role of a database administrator is to manage the technology that is used to store the data assets. This is clearly a technology role. What about the data assets stored within the database? This is where the roles/ responsibilities are unclear. The hype cycle of data governance and ownership has shed light on this grey line between the business role and IT roles regarding the management of the data assets.

Is the role of an IT department to support and manage what the data assets within the technology or only to apply and manage the technology that supports the delivery of information to the business? It is clear that the data assets that fall outside of technology (i.e. paper records and files) are typically outside the responsibility of an IT department.

Prior to computerization, the information/data of an organization was managed directly by the business people who produced/used the data. The data was managed by the people who knew what the data represented. There was no disconnect between the business and the data as there is today. With computerization, the business organization’s data assets were encapsulated deep within technology. This made it nearly impossible for a businessperson to manage their data assets as they had when the assets were in a paper file. The management of the data assets fell into IT by default and not by design, as it became an accepted practice for the business to rely on IT for the management of the data assets.

Technology has evolved over the years, to a point where there is no longer a clear distinct between the business and its technology, in both directions. Many techno-savvy business folks can effectively perform tasks that previously fell under the responsibility of the IT department. In fact, there has been a mushrooming of what is known as “IT shadow groups” within many business organizations. These groups are not part of the IT department, but perform many traditional IT type functions and thus indicate a desire of the business to take back the control/management of their data assets. Should the business organization, rather than IT, manage the data assets as they did prior to computerization?

Interestingly, there is an “I” in “IT. Does this mean IT’s role is intended to be 50% in support of the business’s information assets and 50% the technology that supports the information assets? Should IT play a significant role in management of the business’s information asset? Since data is a representation of the business organization, a significant information management role in an IT department would require a more business-focused rather than a technology-only-focused for an IT department.

The best fit for the role/responsibilities of information/data management within an organization is an interesting challenge many organizations face, knowingly or unknowingly. However, more importantly is assuring that the responsibility and accountability for the data assets are assigned, regardless of placement within the organization. Sadly, these often fall between the cracks and the important data assets are undermanaged or managed in a reactive manner.

Catalyst Europe 2010 highlights and discount

2010-03-04T12:28:15-08:00

Join us in Prague for Burton Group's Catalyst Europe Conference 2010! We are offering a discounted price of €995 if you use the promo code “INSIDER” during registration.

Many organisations are preparing for economic recovery by rethinking assumptions, striving for regulatory compliance, charting alternative courses, and monitoring which business scenario might unfold. One thing is clear; navigating the new normal requires more than automation. It requires the ability to leverage data to develop insight that can inform and drive business decisions.

But knowing the path isn’t the same as walking the path. Business professionals are inundated with information, and it’s getting harder to know what is “true.” Business people struggle to share their knowledge and most organisations are mired in unreliable data. Business cannot thrive under these conditions. Simply put, IT must help the business determine how to make sense out of petabytes of information, and use it to make better decisions

Catalyst 2010 will help you create the information environment for the new normal. We’ll examine how to leverage data, content, social media, and analytics effectively and deliver business insight.

How enabling insight can build on and extend existing approaches to business intelligence (BI)

How organisations can address the eroded utility and quality of data

How attention management helps information overload

How organisations can effectively combine structured and unstructured data, and manage it as information

How to foster insight through the intersection of search, social media, information delivery, and visualisation

How to reveal patterns in data that go beyond operational questions

I am looking forward talking with you at Catalyst,

Chris Haddad

Vice President, Data Management Strategies

Flexible Metadata for a Less-Stodgy Enterprise IT

2010-03-03T06:27:43-08:00

Blogger: Lyn Robison

The standard recipe for creating enterprise information systems has been around for a long time: relational data behind application logic behind a user interface. This basic N-tier architecture has been and continues to be the de-facto standard for enterprise IT application development. Sure, service oriented architecture and Agile methods are intended to remedy several of the pesky problems that are inherent with N-tier development. But still, the fundamental architecture persists, and that architecture is data in a database behind software for application logic behind software for the UI. Unfortunately (and demonstrably), this is a recipe for stodginess.

In fact, the N-tier design paradigm just touches the surface of the stodginess. My colleague Jack Santos rightly pointed out to me that IT architecture today really is NxN-tiers. It is not only two dimensionally data-process-logic, but two dimensions multiplied by multiple different implementations, due to stove pipe approaches, varying off the shelf software implementations, or (now) external cloud solutions. Each have an N-tier approach, and each can stand on their own.

When enterprise IT groups are forced to spend multiple man-days changing code in several NxN-tier software layers just to add a data attribute, those enterprise information systems are patently stodgy.

Here is an example of enterprise stodginess that we have encountered right here at Burton Group. My colleague Joe Bugajski wrote a paper on governance that is quite popular with our clients. However, we have no way of knowing how many readers agree with Joe’s recommendations or how many readers have implemented or plan to implement Joe’s recommendations. We don’t even know how many client “key issues” or “problem statements” Joe addressed in that paper.

To find out how many readers agree with Joe and are implementing his recommendations, we will need to implement a way to better collaborate with our clients. We will also need to make sure that inside these collaboration system(s), we use the same customer identifiers that we use for customer data within our other information systems. IOW, it will not help us to have our collaboration systems tell us that “Customer 47” is implementing Joe’s recommendations and then have our CRM system tell us that it has no record of a “Customer 47”. Clearly, effective collaboration and proper data identifiers are a powerful combination.

To find out how many “key issues” or “problem statements” Joe addressed in his paper, we will need to go back and add that metadata to Joe’s document. And then we will need to change the code (in perhaps several layers of software) to make use of that new metadata.

We have to make those code changes because the metadata for our data is hard-coded and our data is trapped behind our software. These are problems with NxN-tier architecture that neither SOA nor Agile can remedy. Whenever you have to modify software in order to take advantage of new data and metadata, you have stodgy systems.

Eliminating this stodginess requires a different mindset. You have to stand enterprise IT’s traditional approach to creating enterprise information systems on its head. Basically, you have to use flexible metadata and you have to put the data in front of the software instead of behind it. You must not bury the data behind software that is hard-coded to only understand certain types of information. You have to use commodity software that can readily understand, process, and display lots of different information types.

In order to avoid in stodginess in enterprise information systems, you need:

Front-end software that can understand new data and metadata without modification
Back-end software that that can help you create and recreate your metadata

I don’t have room in this blog post to tell you what that software is -- I only have room to tell you what that software isn’t. That software isn’t a relational database trapped behind software for application logic behind yet more software for the UI. Avoiding stodginess requires something else entirely. I will describe this front-end and back-end software and its associated cures for enterprise IT stodginess in my next blog post.

A Less-Stodgy Enterprise IT

2010-02-25T16:00:39-08:00

Blogger: Lyn Robison

Enterprise IT isn’t as cool as the Web, and here are some reasons why. The Web allows information to flow, and the Web also allows people to collaborate and build relationships with one another. Enterprise IT generally doesn’t. Enterprise IT is stodgy because IT systems are most often implemented in nonflexible silos that were designed primarily to automate individual business processes. Silos in Enterprise IT are not designed to let information flow, and enterprise IT silos are most often not designed to let people collaborate and build relationships with one another. In short, enterprise IT is not cool because IT doesn’t let information flow or people collaborate.

Enterprise IT is obligated to automate business processes. We gotta do that. The business expects enterprise IT to automate their work. But business process automation should not be all that we do in enterprise IT. The business needs more from enterprise IT than process automation. In fact, if process automation is all that we do in enterprise IT, the business will think we are stodgy, because we are.

To find information about how to let people collaborate and build relationships with one another, see the CCS blog.

Letting information flow in enterprise IT requires two things:
1. Lowering the level of information model pre-agreement
2. Managing identifiers

The Web is a terrific platform for sharing information because the level of information model pre-agreement is low. A web client and a web server need only understand the HTTP protocol and a few basic document types. Typical data integration between enterprise IT systems requires a high degree of information model pre-agreement, and that makes data integration difficult, and that makes enterprise IT stodgy. What many IT professionals fail to realize is that when you are integrating data that is to be consumed by humans, you can use the Web model, and you don’t need to have this high degree of information model pre-agreement.

When identifiers are managed properly, enterprise IT gets less stodgy. For example, UPS and FedEx can easily track parcels, using tracking numbers. People with common names can be removed from terrorist “watch lists”, because of the RFID tags in their passports. Automobiles are easily distinguished using their license plate and/or their VIN. And airplanes can be readily identified using their tail number. When enterprise IT manages identifiers improperly, which sadly is quite common, the business can never get a complete view of their interactions with a particular customer, because that customer is identified differently in different systems. The business can never manage any long-lived and valuable assets through their useful life, because each long-lived asset is identified differently in different systems. And businesspeople cannot find the information they need on a wide variety of vital business topics, because information about those topics is buried deep within silos and is irretrievable because it is misidentified. And this makes enterprise IT stodgy.

Let's fix IT's stodginess by enabling people to collaborate and build relationships, by lowing the level of information model pre-agreement, and by managing identifiers properly.

COTS, SaaS, XML, and Pixie Dust

2010-02-16T18:48:19-08:00

Blogger: Lyn Robison

Back in 2007, I wrote a Burton Group overview entitled, “Build, Buy, or Borrow: Choosing Custom Development Software, Open Source Software, Commercial Off-the-Shelf Software, or Software as a Service”. In this blog post, I will build on the ideas I presented in that paper and offer some refinements to my original recommendations.

These newly refined recommendations provide more detail and emphasis on COTS and SaaS, and add two new elements. These newly refined recommendations offer a path to an enterprise IT infrastructure and application portfolio that can be both radically simple and more effective than the burdensome patchwork of clunky legacy systems that you find in enterprises today. Here are my updated recommendations. Enterprises should use:

COTS software for the front-end
SaaS for the back-end
XML for the data
Pixie dust (in the form of identifiers and data services) to make information float from system to system

1. COTS for the Frontend

COTS software offers flexibility with low cost, which is readily apparent when user interfaces are implemented using COTS software packages on PCs and mobile devices. Certain COTS packages (namely browsers, media players, word processing programs, and spreadsheet software) can become low-cost, flexible user interfaces for enterprise systems. They can display all kinds of information to businesspeople, including narrative text, graphics, photographs, video, audio, structured or tabular data, charts, highly structured documents, electronic forms, etc. And these off-the-shelf UIs can go beyond simply displaying data -- they can let businesspeople edit data and submit it to backend systems for data processing and storage.

COTS packages such as browsers and productivity applications are low cost UIs because IT developers don’t have to write much, if any, UI code. These COTS packages also sport interfaces with which businesspeople are already familiar. Browsers and productivity applications run on multiple clients, including PCs, Macs, and lots of mobile devices. Clearly, if the data is HTML, OOXML, ODF, or some other readily-consumable format, browsers and productivity application software can make highly flexible, low-cost UIs for enterprise information systems.

For a demonstration of how well browsers can display various kinds of information, use your favorite browser to look at the World Wide Web. For demonstrations of how well COTS software can provide highly-flexible UIs for enterprise systems, see the templates here and here. In the SLA example, IT and businesspeople could use this SLA form to create a new SLA or update an existing SLA, which could then be saved to a backend database that stores OOXML data, and that could be queried by IT professionals whenever they need to know what service levels are expected of them. The Expense Report example is a template that could be customized and used as the UI for an expense system where employees could fill out the spreadsheet that they then save to a backend database that can validate OOXML data and process the expense reports.

2. Saas for the Backend

Saas offers the fastest time to value for generic business processes. SaaS is most appropriate for those business areas where the processes are standardized. I’ve seen lots of enterprises where businesspeople believe their processes are “different” and cannot be automated using standardized business software. In many cases, however, these processes are “different” because they are actually substandard. IOW, if businesspeople would adopt standardized business processes and the pre-built SaaS packages that can automate those standardized processes, they would significantly improve their business. In those cases where the data is too sensitive to trust offsite with a SaaS vendor, then the SaaS application often can be hosted within the enterprise’s data center. SaaS is a part of cloud computing, and enterprises should consider internal, private, and public options for cloud computing, as their security requirements dictate.

3. XML for the Data

When data is in XML, thousands of existing software packages can handle that data. Data in XML is universally approachable, accessible, and usable. Many IT professionals think of XML only as a format for transferring data between applications, which is a counter-productive perspective. More than just data transfer, XML is a powerful format for data storage. XML is a useful compromise -- XML may not be perfect for structured data or for unstructured data, but it is a useful compromise that can handle both. The Microsoft Office suite and the Open Office suite store their data (their documents) in XML, which clearly demonstrates that XML works great for narrative text storage as well as for structured data storage. As further evidence of how far XML can go, check out a demo that drew gasps at TED2010. The file format for Photosynth and Deep Zoom is XML-based.

4. Pixie Dust (in the form of identifiers and data services) to make Data float from System to System

This is best illustrated by a couple of examples. Since Gartner’s acquisition of Burton Group, we have been getting up to speed on Gartner’s internal systems. I have learned that I have at least three IDs with which I must log in to various Gartner IT systems. If Gartner ever needs to combine the data about me that resides within these various systems, they would have a tough time doing it. How do you do cross-system joins of important data when the identifiers are not consistent between the systems? It ain’t easy.

Here is another example. Within Burton Group, we have multiple systems that contain customer data. Two examples are our CRM system and our Catalyst registration system. If I want to find out the customers who are attending Catalyst and who have had recent dialogs with my team, I immediately encounter the problem of customer identifiers that don’t match between those two systems. I can try to use email addresses as identifiers -– which is about my only option, but that makes my task far from easy. If we used uniform identifiers throughout our disparate customer systems, I could use simple UI tools such as my browser or even Microsoft Excel to find the customers who are attending Catalyst and with whom we have had dialogs recently, and I could do it without building an elaborate data warehouse and an expensive set of BI applications.

We would need a data service or two to be written that would enable me to use COTS software to query the CRM system and the Catalyst registration system based on customer ID. Managing IDs and writing data services that work with COTS UIs is what MODS and the XQuery development stack are all about.

Compared to the World Wide Web, enterprise information systems are clunky and kludgey. If I want to get information from CNN, I don’t have to know any details about CNN’s computer systems. I just point my browser to http://www.cnn.com/ and I get ready access to the info I need. It is ironic that every enterprise has complete and ultimate control over its own information systems, and yet, enterprise information systems are clunky and kludgey. For example, to get information about Burton Group clients, I have to know the intimate details of our CRM system, our Catalyst registration system, and each of our other systems that might contain customer data. I cannot easily combine the data between these systems and I cannot use my browser or productivity applications with which I am already familiar to do it.

Enterprise information systems are clunky and kludgey, but they do not need to be. Enterprise information systems can and should be better than what you can find on the Web. The approach I am suggesting is to use COTS software on the front-end, use SaaS applications on the backend, use XML to store the data, and have IT developers focus on bringing data together for businesspeople, instead of having IT developers continually build more clunky applications, which only add to the growing crowd of aging legacy systems that imprison valuable data within silos and that seem to remain forever on life support in the data center.

Must IT solutions always be based on software?

2010-02-11T15:50:13-08:00

Blogger: Lyn Robison

When businesspeople ask for an IT solution, the assumption is that they asking for a piece of software. When IT professionals gather the requirements for a new IT solution, they focus on gathering software requirements. I mean, IT professionals don’t start with electricity requirements; they start with the software requirements. Sure, there are process requirements and data requirements, but software will implement those processes and that data, so software requirements are the overarching concern. Clearly, in enterprise IT today, “requirements gathering” primarily involves gathering the requirements for a piece of software.

To design and build custom IT solutions, enterprise IT groups have user interface designers, software developers, and software testers. Software is the thing that the IT solution team implements in order to deliver the IT solutions that the businesspeople asked for.

Let’s not forget the computer hardware. The software requirements are what determine the hardware requirements. The software gets the hardware that it needs to satisfy its functional and nonfunctional requirements. Hardware requirements are determined by software requirements. (And the electricity requirements are then determined by the hardware requirements.)

Software does the data, software executes the processes, and software determines the hardware. It is apparent that today’s IT solutions are based on software. My question is: must they be?

Must IT solutions always be based on software? They didn’t use to be. Back in the 60’s and 70’s, IT solutions were based on hardware – mainframe hardware. You picked your mainframe, and the software and the data came along with it, almost as accessories. The focus of the solution was the mainframe hardware. Then with the advent of PCs, computer hardware became commoditized. With PCs, you could decide on your software, and then be confident that you could find whatever commodity hardware the software requires. IT solutions have been based on software ever since.

But here’s a question: what will happen when software becomes a commodity? What will IT solutions be based on then? Here’s my answer: information. When software becomes a commodity, IT solutions will be based on information requirements. When businesspeople ask for a new IT solution, IT solution teams will gather information requirements, and the information requirements will drive the software requirements, which will then drive the hardware requirements.

Some might think that software will never become a commodity. It has already. Here is a portion clipped from the Salesforce.com homepage.

It looks to me like the Salesforce folks are saying that software need no longer be the basis for IT solutions any more. Enterprises can get all of the software they need from the cloud. Software is a plentiful commodity. IT solutions can be based now on information, not on software. Gather your information requirements, then select your software, etc.

The only catch is the fact that your data is stuck within various data silos in the cloud. The real work is in figuring out how to migrate your data, integrate your data, protect your data, and manage your data. Data management has a bright future.

Can data be merely the servant of the software?

2010-02-01T12:28:54-08:00

Blogger: Lyn Robison

Whenever a business needs to manage something, they usually need an information system or two to help them keep track of it. The IT professionals who implement these systems need to ask the businesspeople a fundamental question: do you need to know the quantity of this thing or the quality of this thing?

If the business is tracking 3/8 inch sheet metal screws, the answer is probably “quantity”. All sheet metal screws of the same grade, type, and class have the same attributes, so quantity is the primary considerations. When mere quantity is the consideration, you can get away with the data merely being the servant of the software. The IT people gather the software requirements, and off they go to build the system.

On the other hand, if the business is tracking customers, the answer is probably “quality”, meaning they want to track the qualities of each individual customer. Each customer is distinct and not interchangeable, and they each have unique attributes that must be tracked for business reasons. These attributes will need to be assembled from multiple information systems, which means that the data cannot be the servant of any one particular piece of software. So the IT people must gather information requirements, in place of or in addition to software requirements.

Building an information system that tracks quantity is very different from building an information system that tracks quality. If you are tracking quantity, you are no doubt tracking a fungible commodity, and your information can safely treat the data itself as a fungible commodity that can be easily aggregated. OTOH, if you are tracking quality, you must not treat the data as a fungible commodity. You must manage identities beyond the boundaries of any particular data silo.

Speaking of sheet metal screws, I was asked an interesting question by an astute reader of this blog. He asked, “let's use your 3/8 inch screws...the billions of 3/8 inch screws worldwide are certainly fungible, but if I put them in a blisterpack of twenty, give them a UPC and place them in inventory at a Home Hardware, are they non-fungible assets worth tracking, comparing, etc?”

My answer is, one blisterpack of screws is indeed the same as the one on the shelf next to it, but is different from the one in another store from a different supplier only if the screws are of a different grade, type, or class. For example, stainless steel 18-8 Phillips pan head sheet metal screws are all the same no matter the supplier, the retailer, or the packaging. Even then, all sheet metal screws are fungible (interchangeable) within their class. Likewise, one gallon of gasoline is the same as another, no matter what oil company it comes from. “Chevron with Techron” is an effort to differentiate the grade, type, or class. But one gallon of Chevron with Techron is the same as another gallon of Chevron with Techron, so Chevron gasoline is still fungible.

With fungible commodities, your IT systems only have to track the quantity within each grade, type, or class. With non-fungible assets, your IT systems must track identity, and must do so beyond the bounds of any particular application or data silo. This means that data cannot be the mere servant of a software package. With non-fungible assets, the software must bow to the needs of the data.

Spinning your Wheels with BI

2010-01-26T16:29:22-08:00

Blogger: Lyn Robison

BI system implementations follow a cycle. The cycle starts when businesspeople begin seeing the need for better intelligence. The decision is eventually made to implement a new BI system. The enterprise spends hundreds of thousands of dollars on BI, ETL, and DW tooling. They spin up a project or series of projects to load the data into the DWs and to create the queries and reports that deliver the intelligence to the businesspeople. With the cost of time and labor, development tools, software licenses, and computer hardware, it is quite easy for an enterprise to spend a million dollars or more on a new BI system.

The sophisticated tooling, the complex BI systems, and the cutting edge technology are impressive. It is easy to think of the BI system as a large, highly developed, powerful engine. All that is needed to fire this engine up is some fuel. That fuel, the data, comes from the enterprise’s operational systems. Powerful ETL tools extract the data from the operational systems and transform the data so that its shape conforms to the mold in the DW. The ETL processes load the data the way that barrels of oil or stacks of lumber or bushels of wheat are loaded into warehouses (after all, data is nothing but a fungible commodity that can be easily replicated, aggregated, summarized, sliced and diced, right?). Once the data is loaded, the BI engine does its work beautifully. The engine plows through huge quantities of data, aggregating it, summarizing it, analyzing it, and allowing businesspeople to mine it for gems. Sure, the data is not perfectly accurate, but everyone assumes that the numbers will be close enough. After all, smart people are accustomed to working with uncertainty. And look at that BI system do its work! The complex, powerful BI system is truly a sight to behold, a thing of beauty, and the IT people are justifiably proud of their hard work.

Then the questions start. A few businesspeople ask to see where the BI system’s numbers are coming from. Are the operational systems from which the BI data is pulled the authoritative sources for information on the topic at hand? Did data from all of the relevant data sources get included? Is there any documentation on the BI system’s data handling, such as data models or data flow diagrams that show the data’s lineage? What calculations and processes does the BI system use to produce this information? Do those calculations and processes reflect the corporation’s policies and business rules? Does the information reflect the changes that occurred when that new manufacturing plant came online or when we switched to that other supplier? Were the identifiers reconciled properly between systems? Can anyone say for sure whether or not customer XYZ and customer 47 are actually the same customer? If not, how do we know that the BI system is summarizing the data about each of these customers correctly? If the numbers from the BI system are off, how far off are they? How can we know?

These questions about the reliability of the BI information eventually become a problem. No one can give a satisfactory ROI calculation for any data quality efforts. So data quality issues are never addressed in a systematic way, and as a result, the data quality is consistently neglected and continually deteriorates. Eventually, businesspeople aren’t sure how much they can trust the numbers coming from the new BI system. Businesspeople begin seeing the need for better intelligence. The decision is made to implement a new BI system. Here we go again. Let’s build another BI system!

In this cycle, the enterprise tries in vain to address problems of information reliability and usefulness by creating yet another system. Unfortunately, that does not solve the enterprise’s basic problem. The enterprise does not lack systems. The enterprise lacks reliable, useful business information.

Eventually, the business leadership sees that they are spending millions of dollars on IT, and they begin to look for cheaper ways to get business intelligence, through approaches like IT outsourcing, cloud, and SaaS. If they could peer into the future, they would see that IT externalization will at best give the business the same mess for less.

The fundamental problems that make BI systems ineffective are the mistaken perceptions about data that are harbored by most IT professionals. In truth, data is not merely a fungible commodity that can be moved around and aggregated and summarized to fill up data warehouses the way that crude oil can fill up oil tanks or steel barrels. To be useful for any purpose, data must be an accurate picture of reality.

To be an accurate picture of reality, data must accurately represent individual, non-fungible assets. Barrels of oil might be fungible (interchangeable) – one barrel of oil might be just as good as another, but one customer is not just as good as another. Customers are not interchangeable. Neither are buildings, factories, documents, contracts, decisions, or any number of vital business assets. If the data representing those non-fungible assets is treated merely as a fungible commodity, the data will not accurately represent reality, and the enterprise’s data will be expensive but unreliable for business decisions.

Leverage Data from disparate Silos for Configuration and Change Mgmt

2010-01-19T10:21:28-08:00

Blogger: Lyn Robison

I recently presented a case study in how a large enterprise is successfully leveraging data from disparate silos for CMBD and change management.

You can view the presentation here: