How to avoid the 7 deadly sins of data integration:

• Thou shall not give into data "gluttony" and instead live the lean consumption data integration lifestyle.
• Thou shall not show "greed" over the consumption of more and more and more data, but instead embrace data quality as nirvana.
• Thou shall not show "envy" if you notice your competiors have embraced integration on-demand. Don't worry, Informatica can guide you along your journey.
• Thou shall not show "wrath" when your hand coding projects come in over budget because you will see there is a better solution.
• Thou shall not exhibit "sloth" at having data to manage in multiple applications and databases, Informatica is the light that will show companies the way.
• Thou shall not exhibit "lust" when you complete your data integration merger milestones with Informatica in record time, under budget and with a single view of your customers.
• Thou shall not show "pride" when your company reaches the data integration pinnacle with Informatica, just beat the pants off your competitors and wink!

�

The answer is yes; this is an event, but only one of hundreds of thousands — or even millions — that may typically occur across an enterprise every day. The trick is being able to determine what particular events make a material difference to the business, and then applying analytics against those more important events.

I recently had the opportunity to join two experts in the complex event-processing arena — Brenda Michelson and David Olson – in a thoughtful online roundtable discussion on how business events need to be processed, and the role of event processing in what has been a very eventful year for businesses.

As the world progresses toward complex event processing – and it will take time – the role of enterprise data integration will grow even more critical. The ability to sense and respond in near real-time fashion to events across the organization will require a robust and infallible data integration infrastructure, that delivers quality data to any part of the enterprise that needs it.

A soda can shifting off a pallet obviously is not a big event that needs to be monitored and analyzed. But something like the collapse of credit markets is a significant event that has huge ramifications for a business. Recognizing important events from the “event cloud” that rolls through every day is an artful science known as “complex event processing,” or CEP.

CEP is not a black-and-white exercise, as there is a wide spectrum of events that may or may not have an impact. Some events that may even be hidden or innocuous may have long-term impacts. The trick is identifying those events that need to be tracked and understood. For example, sensors in a production facility may sense errant patterns in output that have matched the lead-up to shutdowns in the past. The CEP engine may proactively notify facility managers via alerts, as well as sales managers downstream that delays could be imminent.

A little while back at a previous session, Dr. David Luckham, considered the father of complex event processing, responded to a question on what an example of a “complex event” may be. His answer: World War I.  The war was a complex event, with many different and unseen components that converged, and had a lasting impact to this very day.

Organizations may not be dealing with World Wars, but still are nonetheless complex organisms attempting to survive and thrive in complex environments. There are many things going on at once, with many different relationships, with many different impacts – again, coming out of the event cloud.

From a management perspective, the most important part of the CEP exercise is developing a capability to identify both threats and opportunities among the voluminous streams of data coming in from internal and external sources. This is where business intelligence comes into the picture. Brenda observed that data flowing in from across the enterprise – from sensors, from applications, and from external sources – is part of “one large flow,” of which essential pieces need to be captured and subjected to analysis. The business needs, through rules development, to determine what events are and aren’t important – and when an insignificant event suddenly becomes a significant one.

“At the edge, where the events are being generated, and specifically, with RFID, if you think every time the can of soda moves on the pallet in the warehouse, events are generated,” she states. “So you do need to filter some of those at the edge. And there will be business rules as well applied at the edge. So there is some type of analytic processing in there. Maybe it says, ‘I only really care when a high-volume item leaves the warehouse. I want to be notified when the pallet of TVs goes out, make sure I know about it. The pallet of soda I don't care about as much.’”

Why does business intelligence need complex event processing behind it? CEP provides a real-time snapshot of developments as they occur across the enterprise, and elevates these views to decision makers. Event processing, in conjunction with business intelligence, enables decision makers to expand their knowledge relating to hundreds of thousands of events that are flowing in and out of most businesses on a daily basis. This means more than implementing a dashboard or decision algorithm, Brenda said.

“Think about it in the context of information flow, and the patterns that emerge from this flow. Analytics and business intelligence is often about making informed decisions for your business tomorrow. What you're really doing is you're trying to make informed actions for your business right now.”

So how can we catch the tax dodgers?  If we can catch them, then maybe our state and local government offices can continue to provide the level of service we’ve expected of them rather than having to cut-back in light of the recession we find ourselves in. 

I was reading about a number of examples where state authorities have actually been raising taxes … without raising taxes.  Sounds implausible doesn’t it? 

For example, the State of Texas uses data integration technology to recover $70 million per year in uncollected sales and use taxes.  How do they do this?  They use Identity Resolution, technology which enables companies and government organizations to search and match identity data in batch and real-time.  Have a look at our website to understand how Identity Resolution is helping organizations find criminals, uncover tax fraud and help ‘raise’ taxes without really raising taxes.

The speed of innovation is accelerating as the state of mind and culture of technology users and vendors are shifting toward openness and collaboration. Dr. Eric Von Hippel from Massachusetts Institute of Technology put it this way in “Democratizing Innovation”:

“When I say that innovation is being democratized, I mean that users of products and services—both firms and individual consumers—are increasingly able to innovate for themselves. User-centered innovation processes offer great advantages over the manufacturer-centric innovation development systems that have been the mainstay of commerce for hundreds of years. Users that innovate can develop exactly what they want, rather than relying on manufacturers to act as their (often very imperfect) agents.”

Informatica has a 45,000+ member-strong developer network called Informatica TechNet. We also have a vibrant local user group community around the world. Both online and through local live events, the Informatica developer community actively exchanges best practices, knowledge and technical tips. We are also in touch with our customers who enlighten us regarding their innovative use of our products and offer us suggestions on product directions that they’d like to see. So, it was welcome news for us to hear about Teradata Developer Exchange launch. After all, Teradata and Informatica have a large installed base of joint customers across Enterprise Data Warehousing and Data Integration that support successful data warehousing. When I meet one of these joint customers, in many cases, the first thing that comes out of his or her mouth is, “can you introduce me to other customers using Informatica and Teradata who are doing something amazing?”

Our innovative communities are becoming more and more used to sharing their foundational knowledge with others. Across industries, our users discuss connecting data warehouses with portals and web sites, leveraging Web services by campaign management, optimizing real-time performance including real-time data loading, and managing data quality. We encourage you to check out both communities, Informatica TechNet and Teradata Developer Exchange and find out for yourself how you can benefit from networking and learning from peers, partners and colleagues. Better yet, you can participate in the discussion groups and/or launch a new community of interest - you are the force of democratizing innovation.

Say you can replace a $60/hour IT worker in the U.S. with a $30/hour worker overseas (the difference was greater 5 years ago, but the gap is closing).  A 50% cost reduction looks quite enticing, especially when multiplied by 100’s of staff.  The real benefit is less since there are some extra costs that erode the savings.  Costs such as network and systems infrastructure, diseconomies such as mis-understandings and rework resulting from imperfect communications, and operational risks from security issues or political unrest. If you add it all up, maybe the savings are just 40%. Fine – it’s still a big number. 

So here’s the problem – what does the CIO do in year two when the pressure comes to reduce the IT budget again?  Does (s)he offshore more staff?   Or find yet another corner of the world where the labor costs are even lower? Labor arbitrage (replacing expensive labor with cheap labor) is a short-term strategy and is not sustainable.  In fact it makes things worse.  As mentioned earlier, the gap between U.S. and overseas salaries is closing; primarily because overseas salaries are increasing faster than U.S. salaries.  So while the CIO might see a one-time decrease in costs, (s)he may get a surprise a few years later when the new baseline costs start to increase faster than the old baseline.

Let’s get back to the title of the article – how can you stay employed by eliminating work?  If you are doing routine repetitive work such as software development, testing, or application maintenance, then to senior management it looks like a simple solution to use cheaper labor to do the work.  But if instead you work to eliminate or minimize routine work by developing re-useable components, automated testing procedures, leverage metadata to reduce maintenance efforts, and other similar productivity improvements, then this drives two benefits.  First, you end up doing more value-added work that is hard to send off-shore. It may seem counter-intuitive that you can keep your job by automating routine work, but the reality is that every enterprise has more work than available resources.  So if you streamline the routine work, you then have more time for high-value activities.  It is much less likely that your organization will send creative and innovative work to off-shore teams.

Second, automation is more sustainable for achieving continuous improvement.  Organizations that have implemented Lean management practices for example (as per my blog 10 Weeks to Lean Integration) have demonstrated year-over-year improvements in productivity by continuously finding ways to eliminate wasteful activities. There are tremendous opportunities for improvement in organizations that are working in functional silos - you can actually reduce costs more by optimizing the end-to-end processes than by hiring cheaper labor.

In short, by automating routine activities, you end up making yourself more valuable to the organization and you establish practices that support continuous improvement.  There are of course no guarantees of employment in the modern workplace, but doing value-added rather than routine activities is one of the best ways to maximize the odds.

We were talking about a variety of examples of where a better understanding of data helps improve lives and save lives.  I want to talk about how and why data is critical.  Data allows us to understand what is happening, who is affected, where disease is spreading and how to combat it.  It’s not applications, or business process or people. Yes, all of these help – but at the heart of the ability to understand health is “data”.  We live in a real-time world where health issues are reported at the speed of light across the internet before we even wake up. We all watch, we all listen and we all observe.  Yet, how do we act?  How do we take in the data and act upon it?

Data integration is a critical enabler in this process.  It provides the ability to access data from any system, from anywhere, pull it together, deliver it correctly, accurately and in a timely manner to those that can act.  This is important.  I wanted to share a few examples of such cases in order to help people understand the need for data, and the analysis of that data, that helps save lives. If we can analyze more data, track more cases, share health records effectively, then we can continue to improve lives and save lives.

Consider then:

• The Ohio Department of Job and Family Services uses data integration to streamline case management and get assistance to families quickly
• The U.S. Health Resources and Services Administration uses data integration as the platform for a geospatial warehouse that enables citizens to locate health services near where they live
• The U.S. CMS Office of Clinical Standards and Quality (HHS) uses data integration to build and maintain a central data store of quality of care information called the Quality Improvement Organization Warehouse
• The U.S. Department of Defense uses data integration as a critical component of the Armed Forces Health Longitudinal Technology Application (AHLTA), its system for monitoring the overall health of military personnel
• The U. S. Department of Defense and the Veterans Administration use data integration to make sure timely, accurate data on medical and surgical products is fully synchronized within their Product Data Bank
• The Linked Person Record for Cancer (LPRC) links data from disparate sources to create a person-centred view of screening, diagnosis and treatment across the cancer continuum. This linkage of information will support improved identification of new cases of cancer in Ontario and allow analysts and researchers to measure the quality of cancer treatment, plan for future resource needs, and gain understanding of outcomes.
• In Australia Identity Resolution is being used by the Cancer Council of New South Wales to help in the fight against cervical cancer.  This organization created an award-winning register and Identity Resolution is a critical element of this system.  It provides a way of matching a new test against the existing database which is important because pathology laboratories rely upon a patient’s previous test history when screening their current test. It is also critical in ensuring that each patient on the Register is reminded to attend for subsequent cancer screening.
• The Hong Kong hospital authority is responsible for all hospitals in Hong Kong.  They adopted a data integration platform in 2004 following the outbreak of SARS (Severe Acute Respiratory System) and use it to enable their doctors and medical staff to have better access to relevant information to help improve patient care across the region.

When I consider these examples, I am humbled.  Data integration is the enabling technology and a critical element for doctors and the healthcare profession to help improve and save lives.

Another audience in attendance, namely members of the QA and testing teams responsible for ensuring the quality of production applications, building and protecting realistic testing environments for their internal applications is another huge challenge. These team members need to sub-set and create test environments without impacting production systems performance or requiring a duplicate hardware footprint. Masking and protecting the data once it’s pulled from production is also a necessary step of ensuring control of the information housed within these critical systems.

It was refreshing to hear these challenges from real practitioners trying to solve problems for some of the largest organization in the world. Their pain validated the need for Application ILM solutions. These are real production-impacting issues that if not addressed will have huge cost and productivity impacts. If you’re an Informatica partner or practitioner, expanding your knowledge of these new offerings might make you a hero! I urge you to take a look.

Now, baby boomers are the establishment, and their offspring, another large generation born between 1977 and 2001, is setting a new agenda for business — with profound implications for enterprise data management.In an article in the latest edition of Teradata Magazine, I described the changes organizations should expect from this generation, alternately referred to as "Generation Y" or the "Millennials." The needs and demands of this generation are likely to be felt up and down the hierarchy, and enterprise data managers will be among the first to feel the effects.

First of all, some of the stereotypes about the Millennials, most of who are in their teens and 20s, are true. These are young people who spend a great deal of their time online, Twittering from cell phones and posting messages on Facebook. In essence, they are almost always connected.

What does this mean for businesses? For starters, Millennials are accustomed to real-time information anytime and anywhere. They were raised on the Internet, and see global networking as part of their daily lives and don't pay much attention to formal hierarchies or communication channels.

They will impact businesses in at least two ways: as consumers and as employees. I spoke with Ari Banerjee, director of enabling technologies for Yankee Group, who pointed out that meeting this new generation's expectations as consumers will require more emphasis on enabling back-end systems to deliver analytics that can better meet their demands. Organizations must analyze and respond to transactions in near real-time, as customers browse and make decisions‚ for example, while shopping for a new cell phone on a Web site. "Most of the time, they make impulse buys. You need the mechanism or the medium to act on and monetize that impulse." And because there are no "normal business hours" anymore, companies need to react whenever such opportunities arise.

As employees, this generation may be an enterprise's greatest asset as it cultivates the younger market. "Organizations that enable digital natives with technology will achieve a competitive advantage in some sectors," says Gartner's David Gootzit. In my chat with Gootzit, he observed that the democratic construct of social networking technologies is influencing management styles as well. "Across the board, they view organizations in general as needing to be more flat. If you think about social networking, there's no hierarchy. They connect with whomever they can. And if they accept the connection, you're peers." At one marketing firm, the CEO met with some new hires to welcome them to the organization, Gootzit recounts. Within hours, he had received messages from some of the younger employees offering suggestions and complaints. "They thought nothing of bucking the traditional corporate chain of command," he says.

Millennials are also accustomed to having information instantaneously, or quickly accessible via online search. When they enter the workplace, they are often frustrated with slower legacy systems. They have to construct queries‚ or worse, wait days for the IT department to build reports, for the information they need.

Do companies have the technology and wherewithal to satisfy the demand for instantaneous data and information? A recent Accenture study, "Millennials at the Gates," observes that Millennials "are now bringing those expectations into the workplace‚ and turning IT policies upside down in the process." The Accenture study finds a disconnect between enterprise technology and how young workers want to use technology and collaborate in the workplace. "The implications for enterprise IT organizations are potentially significant," the report concludes. "As Millennials begin to exert more influence in the workplace, IT leadership teams will have to find a way to adapt their policies and procedures to accommodate the new technology these younger workers are demanding. And they must do so without putting the enterprise at risk."

Undoubtedly, the BAs are mission-critical to organizations. However, the "identity crisis" is common among BAs, as concluded by Forrester Research analysts Carey Schwaber and Rob Karel. They found it difficult to define the duties of a "typical" BA even after reviewing 29,000 job listings and interviewing more than 300 of them. One thing clear is for BAs, they need to develop and tap their skills in addition to their chosen areas of expertise in business and technical fields. BAs’ communication and relationship-building skills help bridge the gap in IT and business. Forrester Research mentions that this extends to the communication with external partners and customers, so that BAs can help provide context for understanding the end-to-end solution requirements.

Informatica delivers a capability called Business Glossary to help provide the right context for business and IT. It enables data analysts, business analysts, and data stewards to collaborate; create, manage, and share a common vocabulary for business terms and technical definitions. It fosters cross-functional alignment and helps all parts of the business better understand the context and usage of data. One customer I recently talked to said that 10-15% of the effort of their data quality and governance is spent on defining and verifying terms. By providing business context to technical artifacts, the business glossary makes it possible to catalog, govern, and use data consistently while increasing productivity within and across IT and business.

An organization’s impetus to ‘operationalize’ decision making through Pervasive BI has never been stronger. BAs can use their Business-IT savvy to help push further integration of IT into business and promote greater business focus. In the era of Pervasive BI, a successful Business Analyst not only plays a greater role in refining and translating the business requirements into solutions but also closes the gaps and areas of inefficiency in the inner workings of business and IT.

We again sponsored a CRM Integration Roundtable breakfast with local prospects, and our local UK customer sportingbet presented a case study describing how they have significantly improved their overall marketing automation efforts by integrating their operational platform with their Salesforce CRM and Eloqua marketing automation systems.  Linking the three platforms together enabled them to leverage information on their customers’ service usage and behavior to more effectively target marketing programs and promotions to sustain and increase their overall business.  Essentially achieving every marketers dream (including the author’s).

Sportingbet.com is a major brand in the online retail gaming sector providing a wide range of products including online sports, casino, poker and virtual games across Europe, Australia and the Emerging Markets.  With over 2,000,000 customers serviced by 27 separate web-sites in different languages and currencies, sportingbet had a major challenge in dealing with data integration and data conversion.  Additionally the Marketing group was looking to improve returning subscriber traffic to their sites through automated email marketing programs to sustain and grow customer usage.  The challenge was that over 200 terabytes of subscriber usage information was locked up in an on premise data warehouse.  And to complicate things further, the data is in a multitude of languages and currency formats.   And to maintain an extremely high level of data security sportingbet employs two separate firewall ‘rings’, presenting a further challenge to integrating their on premise (the operational systems) and cloud computing applications (Salesforce CRM and Eloqua).

Working with the local Fujitsu Consulting office, they went looking for a way to integrate the data between their operational systems and their Salesforce CRM and Eloqua Marketing Automation platforms.  Fujitsu recommended and selected the Informatica On Demand Data Synchronization Service to accomplish the import and synchronization tasks.  The Informatica architecture was ideally suited to support their security needs and the SQL database connectors helped to simplify the systems integration process by eliminating the need for additional custom database coding.

Integrating daily, weekly and monthly subscriber usage data from the data warehouse into Salesforce, the Eloqua system is then able to send timely, relevant email marketing campaigns to active sportingbet subscribers to encourage return visits to the sportingbet portals.  Streamlining the extract and upload processes using the automated scheduling feature built into the Data Synchronization Service meant that the integration jobs run automatically eliminating the need for manual intervention.  The powerful data transformation features provided through the web-based mapping and expression editor enables transformation of data between the various systems to assure language and currency accuracy.

The sportingbet Marketing team now has timely, accurate subscriber activity information available within Salesforce and to their Eloqua system.  With this tightly integrated solution they are able to spot trends and changes in behavior at the macro and the subscriber level to improve business efficiencies without needing to hire additional staff or invest in costly on-premise equipment and software.

You’ll find a brief overview of the sportingbet case study on the Informatica On Demand website.

And if you’re interested in learning more about the Informatica architecture and the process of working with firewalls and cloud-based applications, you’ll find an informative white paper on the topic on the Informatica On Demand website.

Next stop on the tour is Chicago.  Please check back to learn how a leading financial services information provider is leveraging  both on premise and on demand data integration solutions to improve their Sales efforts.