Swati explains how test data management helps deliver quality applications.

Organizations continue to be challenged with building and delivering quality applications. They face increasing risks associated with protecting data and complying with regulations.

Recent research from the Ponemon Institute indicated that 62% of companies use actual customer data to test applications, exposing sensitive information to testers and developers. Time-to-market of applications is critical to business success, yet long testing cycles, inadequate test environments, and lack of realistic test data contribute to this challenge.

According to the National Institute of Standards and Technology, testing teams spend 30% to 50% of their time setting up test environments instead of actually conducting testing. The cost spirals as defects are caught late in the cycle when they are expensive to correct. Software downtime costs companies billions of dollars.

By creating realistic test data, organizations can build and deliver quality applications.

So what is test data management? Test data management is creating targeted, right-sized test environments instead of cloning entire production environments. Development and test environments are then more manageable, speeding up testing and delivering quality applications.

We’ve developed several best practices in test data management:

Step One: Discover test data

In test data management where you may have several production databases, you need to first find the right test data for your test cases, including any sensitive information. You need a 360-degree view of test data assets through understanding the data relationships across the complex, heterogeneous environment for test data management.

This could be across multiple related applications and databases. For example, it could be a “customer” record from Siebel CRM, together with related details on purchased items from a legacy DB2 inventory management system. You want to capture the end-to-end business process and associated test data that will enable you to subset the appropriate data needed for the test case.

Step Two: Automate creation of realistic “right sized” test data through subsetting

How are our enterprises creating test data today? They are either creating it manually or just cloning their entire production system to obtain a test database.

For example, a healthcare company could take several months to manually create test data for customer accounts, delaying their testing efforts. This also causes their applications to be error-prone due to unrealistic test data.

Some companies are also cloning their entire production system. The downside of cloning your entire production instance is that you now have a data growth problem that will use significant storage. In addition, you have privacy issues because you have now exposed sensitive data to developers and testers using production data for testing. It’s critical to subset based off your production environment, from which you can then create “right size” realistic test databases for various types of application testing.

Read my next post for best practices associated with masking, refreshing and analyzing test data results.

Deanna examines how to revitalize your healthcare data infrastructure to prepare for the future

Driven by competition and regulations, healthcare leaders are under tremendous pressure to control operational cost, improve service delivery quality and provide a more coordinated and personalized healthcare system.

To bring transformational change in their business, healthcare leaders need reliable and accessible information to prioritize and allocate funding, resources and technology to remain competitive and ensure compliance with regulatory quality measurements.

It is a daunting task to glean the right information from a complex IT environment littered with information silos. Healthcare providers cannot calculate the true cost of care if they cannot correlate data that describes the resources and cost of care provided with patient outcomes and post-discharge patient satisfaction. If healthcare leaders don’t pay enough attention, the silos will multiply and make IT more complex with every changing business need, competition and regulatory requirement.

So, how can we build healthcare data infrastructure that meets the future needs?

Well, it's impossible to design for all future needs out of the gate. Flexibility and adaptability are the keys to a sustainable data infrastructure and remaining innovative and agile to meet your analytics needs. You need to enable resources with tools that allow you to spend more time leveraging insights from the data, rather than spending time managing the data.

To really enable your organization with the right data infrastructure, consider these factors:

• Align business and technical resources with a common target and vocabulary to accelerate progress on your initiatives
• Increase agility and decrease time to deliver new reports to your decision makers with a design optimized for analytics
• Enable department heads with the tools they need to be innovative and collaborative
• Adapt to evolving regulatory requirements to maximize reimbursements and compliance
• Expand analytical dashboards and reports to include emerging clinical areas without re-implementing an entire platform

"All-in-one" data warehouse solutions and Electronic Medical Records (EMRs) may deliver the capabilities needed to support near term, tactical objectives for regulatory compliance reporting for CMS and other government agencies. But they may not support the innovation and collaboration for a healthcare organization to transform care delivery, remain competitive, and optimize quality of care. Packaged business intelligence solutions may not support existing technology design investments, and may require compliance with an application-specific and rigid data model.

Achieving rapid and successful analytical value requires the proper balance of a comprehensive data schema design across clinical, financial, and operational data elements paired with the ability to support existing models and technologies. Only a flexible model structure developed specifically for the healthcare industry can support this goal.

Stay tuned for future posts on how IBM healthcare solutions can help revitalize your healthcare data infrastructure.

Despite differences, China and the US face some similar ehealth challenges

As I prepare for my trip to China where I’ll share global healthcare perspectives with customers and colleagues, I’m pondering the similar challenges facing the United States and China. In both countries, individual organizations, regions, and states/provinces are at different stages of technology adoption. Yet they share a common denominator: both countries need a solid information governance framework and a consistent vision of what healthcare needs to be.

In both places, three key themes are emerging:

Organizations, both private and public, need to institute a solid information governance framework that addresses the lifecycle of data. This requires careful, proactive discussions about how to manage data as it is created, how to integrate the data, and how to wisely analyze the data for current and future business processes and goals. Yes, there are differences in governmental regulations and philosophies, but common challenges and approaches abound. Consider:

• How will electronic data be protected against privacy breaches?
• How do you ensure that only authorized users have access to data?
• How do you monitor compliance to policies and regulations without over burdening your technology infrastructure?
• How will data be masked or encrypted for research and other secondary uses?
• How will a single view of the citizen or patient data be created and the underlying data linked across organizations and geographies?
• How do you make patient-centric or citizen-centric data available for care coordination, including data that might be deemed as social or human services as opposed to pure healthcare?

Paper silos should not be replaced with electronic silos. Both countries and their respective healthcare institutions are working hard to address information frameworks for data sharing, as care coordination for chronic disease is a priority. Plus, the healthcare delivery systems recognize the value of using electronic data for analytics and research to establish best practices and treatment protocols.

But simply replacing paper files with electronic files will do little to change healthcare practices and costs. We must unleash the power of the information we are creating to benefit individuals, as well as entire populations.

Technology is but one component of the ehealth transformation journey. Ehealth adoption and healthcare transformation are complex, multi-year journeys. And perhaps the biggest challenge in these journeys is changing business practices and processes, while retraining a workforce to embrace the changes. Education and learning styles differ, yet the human factor of responding to change is universal. Thus it is essential that we think about the human element as we design systems and processes.

I’m excited to be visiting China again, and hearing firsthand how their healthcare journey is evolving. I’m always fascinated by how western medicine and traditional Chinese medicine intersect, making the journey even more complex. I’ll share an update in my June blog post.

Stop by booth 101 at TDWI Chicago this week.

TDWI is this week in Chicago, and we’re excited about the chance to share some of what we’re doing. Stop by booth #101 to say hello and learn more about InfoSphere Warehouse 10, big data, Cognos, InfoServer and MDM, Netezza, SPSS, Systems Z and IBM Watson.  We’ll have plenty of experts on hand to answer your questions and get your feedback. And since the conference is in my backyard, I get to attend, too.

We’ll be giving away hard copies of Understanding Big Data for your reading pleasure – and we just might have a few Watson hats tucked away for people up to the Jeopardy challenge.

To dig a bit deeper into data warehousing and analytics, check out one of these sessions from my colleagues. All three will be held Wednesday in Riverside Center West at the Hyatt.

A New Era in Data Warehousing & Analytics
Wednesday, May 9, 12:00 PM – 12:20 PM
Larry Heathcote will talk about the shift to analytics-driven organizations, as companies start to manage “geysers” of structured and unstructured data. He’ll discuss how the data warehousing ecosystem is evolving to deliver big data, streaming data, operational and deep analytics straight to the front-line decision makers – in real time. He’ll also have a great overview of the real time operational insight capabilities of new InfoSphere Warehouse v10.

Deliver the Analytical Freedom Users Want, and the Management Control IT Needs
Wednesday, May 9, 12:25 PM – 12:45 PM
Kathy Konkel will explain how IBM business analytics can help resolve the tension between business and IT users. IT can retain the necessary governance and management while empowering business users to make decisions based on real-time analytics.

Accessing and Leveraging Predictive Analytics
Wednesday, May 9, 12:50 PM – 1:10 PM
Vince Polito will provide an overview of predictive analytics as an extension of BI. Explore the differences between BI, statistics and predictive analytics, and hear how predictive analytics can be leveraged in a variety of ways.

Follow @IBM_InfoSphere for more on our #TDWI presence, and be sure to follow me (@CrystaAnderson) and @LarryHeathcote for more from the show.  We hope to see you there! But if you can't make it in person, check in online for videos and more.

IBM Champion for DB2 Kim May helps build safety nets through training

With an English degree and a background in technical training, Kim May has taken a different route than many of our IBM Champions. As vice president of business development for the Fillmore Group, Kim embraces the “continuous need to learn,” finding new ways to train and update people on IBM technology. “I’m one of the few non-technical people on the planet who reads every DB2 LUW posting,” Kim laughed, noting that she belongs to dozens of LinkedIn groups.

In the 1980s and 90s, training was considered far more important in many companies. But now, with better telecommunications and tight travel budgets, companies expect to do more with less. Hence, training is often cut to save costs, with shorter classes and more reliance on virtual sessions and manuals. Kim admits that reality has changed, and training usually isn’t sold as part of a technology solution: “They throw the manuals at the team and say, ‘Figure it out.’”

But manuals don’t deliver the same experience as live, in-person learning. Kim explains, “The sit-down, classroom, hands-on training is so critical to getting people the baseline skills, and then the webinars, user groups, the mentoring, through the community…those are for the changes and new features.”

As a result, training has become more reactive. People don’t make time for self-paced, online training until something goes wrong. Then they start grabbing the relevant Redbooks and signing up for sessions.

Virtual groups are helping to fill this gap, though, building a “safety net” that enables users to keep learning and sharing. Kim is looking forward to launching The Fillmore Group's new website in June, featuring four virtual user groups to support ongoing education around DB2 Connect, Q Replication, DB2 Server for VSE & VM and DB2 Use in the Mid-Atlantic Region. The early response to the virtual groups has been very positive, as the initial DB2 Connect conference call, presented by Curt Cotner, attracted 350 people. “When we got the conference call bill,” Kim said, “we knew we were on to something.”

Kim shared a recent scenario for helping a customer migrate from Oracle to DB2. “In a perfect world, I would have sent every single one of their DBAs to a week of training before the migration and a week of training within a month after the migration, when they had their hands in and saw what happened. Then I would do three days of training every six months for the next two years.” Even when working with a really sharp group of DBAs, they can’t learn everything overnight. “You want their skills to mature,” Kim continued, noting that ongoing training through user group meetings should blend learning new things while reinforcing old skills.

Kim urges companies making the switch to DB2 – many of whom do so for financial reasons – to set aside some of the savings for training to enable DBAs for success. Doing so will reap rich rewards in productivity.

Listen to our podcast for more of our interview with Kim, visit her blog with the Fillmore Group and follow her on Twitter @KimMayTFG.

IBM Champion Kim May

When bad data goes viral, even cats get job offers

Financial analysts have been saying for months that the economy is rebounding after our multi-year recession. New jobs are being created, especially in the technology sector, and job opportunities for those who are qualified are plentiful. In fact, the market is doing so well, my cat was recently sent a job opportunity by way of email.

You read that right. My cat. By email.

How is this possible, you might ask? As I’ve alluded in previous blogs, it’s all about the dissemination of bad data.

Back in the late 1990s, just prior to the Dot-com crash, I bought some luggage. At the time, I entered my cat’s name as the recipient. I can’t remember why I did this. It was either a joke, or maybe I was afraid of sending my full name to what may have been a shady online operation, or maybe something in-between. In any case, my cat—Slate—became the proud owner of a rolling duffel bag.

We still have the luggage. It’s a bit worn, but still operational. Slate, however, did not fare so well, and passed away in 2001.

Sort of.

While Slate is gone, his name apparently lives on in email.

About once every few weeks, I receive a letter from one organization or another offering Slate some kind of bargain, or incentive, or—in the most recent case—a potential job interview. The majority of these letters are spam, but some of them do walk the line of legitimacy. Most recently, a recruiting firm reached out and asked Slate if he was interested in working from home. I did not even attempt to reply.

Not that it would really matter. At some point, that online luggage company sent Slate’s name along with my email address to some other company. For the past decade, this information has been swirling around the Internet along with millions of other names and emails, becoming part of bulk mailing lists and subsequently bringing my cat back to life.

I like the concept, but I am concerned with the implications.

In the olden days, when we encountered bad data, that data stayed in a system and — whether we acted upon it or not — was available for clean-up. Methods of data quality have been the subject of many white papers and have formed the basis of entire software packages.

But now with big data, that concept of keeping bad data in one spot has changed completely. Now, we have big bad data that is stored not just on one server, but duplicated exponentially across locations, spreading inaccuracy throughout, and leaving no single source of truth where a correction can be made. That means that no matter what I do or to whom I complain, I will never fully stop the flow of email addressed to Slate.

Once the big bad data valve is opened, it becomes impossible to fix and we must then just grin and bear it, or in the extreme cases, change our email address.

I won’t go that far yet. As with other technological breakthroughs, I’m willing to accept the negatives with the positives and I’ll let Slate’s name live on a bit longer. But I can’t promise I won’t be tempted to reply the next time Slate is offered another job interview. I just hope the recruiter has a good sense of humor.

Download the free new ebook and learn more about big data, from both business and technology perspectives

At the Gartner Business Intelligence Summit in Los Angeles held on April 2-4, big data was a big topic of discussion. Many people were looking for insight into the substance behind the buzz.

In a nutshell, big data means three things:

1) Large data volumes

2) Multiple data formats (e.g., unstructured and structured

3) Variable data latency

Did you realize that approximately 80% of all data in an organization is unstructured? Any data warehouse, business analytic, MDM, or other data integration solutions may need to be designed to handle big data, and that means you need tools to identify, extract, analyze, and integrate unstructured data.

For organizations trying to make sense of the emerging Big Data story, we recommend taking a look at this recent IBM ebook, Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data. The book does an excellent job of laying out the new integration and analytical challenges posed by big data, from both the business and IT perspectives. Let us know what you think!

From the Book: Table of Contents

Part 2: Big Data from the Technology Perspective
All about Hadoop: The big data lingo chapter
InfoSphere BigInsights: Analytics for big data at rest
IBM InfoSphere Streams: Analytics for big data in motion

Chief Privacy Officers can use MDM to see business benefits while protecting privacy

As a consumer, are you worried about how your personal and financial information is used? As a patient, are you concerned about how your medical information is shared?

These concerns have led to legislation that protects individuals – prompting organizations to create positions such as Chief Privacy Officers (CPOs). CPOs are typically responsible for ensuring an organization adheres to privacy regulations and policies – and managing the associated risks and impact to the business

Let’s take a look at this through the lens of a healthcare organization. Since the Health Insurance Portability and Accountability Act (HIPAA) was enacted in 1996, numerous regulations and technologies have been introduced that impact the privacy of a patient’s health information. The HITECH portion of the American Recovery and Reinvestment Act (ARRA) introduced new privacy concerns by encouraging the meaningful use of electronic health records.

With so much electronic protected health information (PHI) in transit from multiple sources, CPOs must establish health systems that ensure the right information is going to the right recipient. This is true for both patient information and provider information – which are equally important for protecting a patient’s identity.

Here is an eye opening statistic from our recent video white paper, Implementing Provider Registries: In the U.S. alone, incorrect provider data costs the healthcare system $26 billion each year. The costs are a result of:

• Returned mail
• Delayed payments from incomplete or inaccurate information
• Incorrect and duplicate records
• Fines due to lack of compliance
• Redundant testing resulting from limited data sharing

Master Data Management (MDM) technologies help boost CPO confidence in securely transmitting PHI with two types of provider registries– individual and organizational. An individual-level registry includes information about physicians, nurses and other providers. An organizational-level registry contains entity-level information about clinics, hospitals, practices, etc., and technical information about consuming and contributing systems. These two registry types work together to give the best information about how to interact with individual providers by associating them with the practices, clinics and health systems where they are employed, affiliated or refer patients.

Provider registries can be used in a number of ways, including to exchange information between clinics or between hospitals or labs and clinics; to issue public health alerts; and to supply information about an individual provider’s credentials.

Not only is it important to know that information about the correct patient is being sent to the correct provider, but health systems must now worry about delivering that PHI to the correct location. The rapid adoption of electronic technologies, as well as the distributed nature of modern care delivery, results in the need to communicate with providers via multiple methods and locations.

For example, a physician may want to receive patient updates or lab results at their EMR account at the hospital, and referral requests from other physicians through their office email account. On weekends or days off, they may want to be reached by a third method.

Provider registries improve interactions and enable provider communications to be easily personalized by including details about the locations of a provider, where and when they treat patients, and what types of information they want to receive for both clinical and business purposes. Healthcare CPOs can be confident about the security of their PHI while gaining other business benefits that systems administrators throughout the enterprise will relish:

• Automatic provider information updates ensure data in clinical and billing systems is reliable
• Provider data contained within a single system is easier to validate
• More accurate provider records
• Reduced billing time and cost because missing information is automatically populated

A little MDM technology – such as patient and provider registries - can go a long way towards ensuring privacy – and can help Chief Privacy Officers recognize business benefits at the same time.

This is part of the MDM for CxOs series. See which other C-level offices will be discussed, or propose an addition.

With 37 different definitions of "employee ID," one company struggled to differentiate employees

In my previous post, Defining Terms: Agree, or Doom Your Project, I suggested that getting business and IT to agree on definitions of critical business terms is a good start to an information-intensive project. That may not sound like too big of a challenge…unless you’ve walked in the shoes of a company like one I describe below.

A large manufacturer had 37 different definitions of the term “Employee ID” across multiple divisions. Using spreadsheets to record and compare the representations, the company discovered at least 15 different data types and formats, with different characteristics and usages — a result of multiple legacy systems, acquired companies with their own systems, and home-grown applications.

So what difference did it make? People in IT — the ones who needed to deliver the reports — required a tremendous amount of experience and undocumented knowledge to determine which “Employee ID” data to use in which circumstances.

The data quality issues became unmanageable, with the wrong data sources being used in various applications and reports. Developers and analysts spent enormous amounts of time trying to determine what data to use where. Of course, the data quality issues then affected the business people trying to make decisions based on questionable data.

IBM is delivering tools that facilitate the critical collaboration that’s needed. But more on that another time. In the meantime, if you’ve had a similar experience, I’d love to hear about it.

IBM Champion for Information Management Marc Hebert battles data cholesterol

How’s your data cholesterol?

Marc Hebert, IBM Champion for Information Management, has spent 35 years working with enterprise applications, and cites data cholesterol as a byproduct of concerns over Y2K. To prepare, organizations invested heavily in mission-critical ERP systems, the majority of which were built on modern relational databases. The goal was to enable organizations to easily load lots of data.

But no one really thought about getting that data back out.

More than a decade later, companies have accumulated tons of data, just as planned. Proportionately, less and less of it is active, so it clogs things up – much like cholesterol in arteries.

As Marc explained, “Traditionally, when all this data was accumulating over the last decade, the normal way to deal with it was to throw more hardware at it, which was mostly storage and CPU. A few years ago, with the latest downturn, IT departments really began to hit the wall and they no longer had the freedom to add hardware to the solution. Even though, according to Moore’s Law, hardware and storage are getting cheaper and cheaper at a dramatic pace, it still adds up to real money.”

In his role as chief operating officer of Estuate, a Silicon Valley-based IT services firm, Marc has watched this trend evolve. Today, he notes, “Customers now have to pay more attention to this accumulation of data. Products like IBM Optim – really the only true enterprise data management platform in the market – really enable customers to get control over unbridled data growth, and thereby drive economies out of the system.”

As technology grows more important to business transformation, IT has become a “fundamental enabler” of productivity improvements and efficiency, especially in large companies.

Indeed, as Marc described, companies are seeing big pay-offs in productivity from their data. Many have gone beyond accounting and the manufacturing floor, as marketing, sales and service are now finding benefits from actually using their growing stores of information. Really leveraging those benefits requires still more data, though. In fact, Marc noted, “Interconnectedness of information across the enterprise, touching all departments, is almost by definition becoming big data.”

That’s leading some companies to develop a new type of asset. Marc calls this a corporate archive repository, which allows companies to put all their end-of-life data into one place, with a set of repeatable processes that follow corporate data retention policies while allowing users to easily access the data as needed. By formalizing their data’s end-of-life process, companies can more easily comply with compliance and audit requirements.

Marc contrasts the corporate archive repository with the traditional approach that involved tapes stashed in a vault. With that approach, “the institutional memory of that tape, the half-life of that memory, is quite short. Sooner or later, people forget where it is and how to access it in case it’s needed.”

Instead, an enterprise data management platform like Optim can help companies use a corporate archive repository to access the data they need, even when it’s been retired. So perhaps such a repository acts like a sort of statin for data management, helping organizations manage the data cholesterol that can threaten their IT health.

As databases grow even larger, such repositories will become more important. Marc noted that a “big database” used to be less than 100 GB, but now 5 TB isn’t unusual – for each system. Some more data-intensive industries – think telco or internet-based – string together several even larger databases.

Marc looks at the writings of futurists like Ray Kurzweil to consider where we may be headed, looking at sensors and other big data contributors that are creating a “neural skin” around the earth.

Listen to our full podcast interview with Marc for more insights about data cholesterol, enterprise data management and the future of big data.

IBM Champion Marc Hebert