Let me quote from John’s concluding paragraph to set the stage for this post:

“Ultimately, the Web as a software stack is robust enough to deliver networked apps and messaging, and consumer-level hardware is robust enough to handle the consequences of working remotely (encryption) and using your own hardware (virtualization), so both of these factors will make consumerized IT and BYOH models increasingly compelling for users—not necessarily IT departments, but end users themselves. And it’s the growing pool of technically savvy, mobile, job-hopping end users who will increasingly demand that IT departments adapt to the way that they work, and not vice versa.”

There are good reasons (oh… like security) why IT can’t simply open the ol’ server ports to any networked app that happens to be popular with the “mobile, job-hopping” end user community. But this is IT history repeating itself. What we’re witnessing today – i.e. the push by users to drive innovation into the data center - has happened over and over. Do any of you remember the early 80’s ‘collapse’ of the glass house, when central IT was forced by the business unit to extend or even replace big iron in favor of client/server deployments… and the advent of the PowerBuilder™ GUI set who RADed their way right past those lovely dumb-terminal “green-screens”?

Once again, the user community is forcing IT to adapt, this time to the technology of the social network. Is this a flaw of IT service management (ITSM)? Not when ITSM helps align IT with business needs. Do IT personnel lack an understanding new technology? Nope. IT personnel are very often technology power users outside of the data center. Well, what is the cause of this gap between the cutting edge end-user and the IT department?

In my humble opinion, it’s two-fold.

The sheer complexity of technology implementation is a large anchor suppressing IT-driven innovation, and the lack of innovative ITSM tools greatly contributes to IT’s pragmatism.

Any user community pushing for new technology would certainly be more cautious if their job was on the line in regard to successful technology implementation. It’s a wee bit less complex to “introduce” a consumer-level product to your cubicle, section, or even department than it is to understand, adopt, deploy, maintain, change, train people to use, and support that same new technology for thousands of internal and external users across a mix of platforms on a regional and global scale – not to mention making new stuff work with existing stuff – all the while keeping the current production systems humming right along.

Consider the challenge of change management – IT routinely must respond to volumes of requests for change (RFC) from a broad and diverse user base. Our research (see past StackSafe blogs) reports that IT often must resort to smoke-tests, component testing, and “patch-and-pray” to process the volume of RFCs.

As a result, 25% of all changes made to production cause problems, and IT must roll back 10% of all changes because of unresolved problems. This can be directly mapped to the complexity of the technology infrastructure, not to mention all that wonderful new stuff that IT is pushed to add to production. IT organizations can improve their testing maturity through process improvements. For example, they can adopt change management best practices, build and maintain representative staging environments, test all changes, test changes end-to-end, and schedule changes.

IT process improvements are always beneficial in theory, but IT rarely has innovative IT tools available to streamline more rigorous process execution. As a result, IT expends extra time and effort improving their control over current technology that could be used to introduce the very cutting-edge technologies for which their user community is clamoring.

Because there are many views and statistics being thrown around about the CMDB these days, it was interesting to see Patricia and Ronni use a couple of survey questions to get a sense of where things stand among conference attendees.

First off, it appears that most companies at the conference are going down the CMDB path – eventually, at least. The session attendees were asked where they are in the CMDB process, here is a break out of the answers:

• 33% are currently in progress with a CMDB
• 19% will start a CMDB efforts in 6 months
• 12% will start a CMDB in 6-12 months
• 20% will start a CMDB by the end of 2009
• 9% are not planning a CMDB

There seems to be quite a bit of traction (both now and in the future), that also maps to the results of the IT Skeptic’s survey. A number of companies are already working on a CMDB and good number plan to continue work.

Another survey question showed additional interesting results. About half of the attendees were implementing ITIL v2 and about ten percent were implementing v3. This would seem to indicate that the deployment of a CMDB is actually outpacing ITIL adoption. This is an ironic occurrence since the CMDB term actually stems from ITIL in the first place.

Regardless, Ronni also had some points of consideration for companies looking to implement a CMDB:

1. Ensure that CMDB data is accurate. Whatever data you include in the CMDB to be reliable to be effective.
2. Test to make sure CMDB is trusted. Is IT running the data regularly with proper effect?
3. Agree on why you are implementing CMDB. It is important to quantify the value of the effort given the length of time that implementation may take.
4. Choose your CMDB vendor wisely. There are no standards for CMDB today, so different vendors may be using different methodologies.

One goal of the CMDB is to improve visibility into the impact of a change. To the extent that this will reduce downtime, we think this is a good thing. However, as Patricia and Ronni pointed out, CMDB projects are very complicated and require considerable effort to provide value. It seems the work will continue for some time.

All these advantages are true. There are also hidden costs associated with this approach.

In our recent study, IT Operations Research Report: Testing Maturity Part II Applications and Operating Systems, we discovered several costs associated with heterogeneous environments. They included:

Increased Total Cost of Downtime

Heterogeneous environments show a greater total cost of unplanned downtime that organizations that used one operating system for all tiers of the stack.

Increased IT Labor Hours Due to Unplanned Downtime

Companies with multiple operating system environments devote over 60% more in IT staff time to address unplanned downtime emergencies.

Increased Cost of Changes Due to Production Problems

Companies with multiple operating systems show a greater cost of changes due to production problems compared to single operating system environments.

Greater Total Number of Changes

Companies make 26% more changes to heterogeneous operating system environments than to single operating system environments.

How Does This Impact the Data Center?

We therefore shouldn’t be surprised that so many organizations go down a single operating system path. A significant number of companies choose to standardize across individual stack tiers. A majority of companies also choose one operating system, such as Windows, for all tiers of their software infrastructure stacks as we discussed in depth in a previous blog post on Windows dominance in the data center.

Download the Report

A full copy of the IT Operations Research Report: Testing Maturity Part II – Applications and Operating Systems can be downloaded here.

Mike has over 22 years of experience in applications development and has worked in the health, retail, manufacturing, and loyalty marketing industries. He has worked on some of the largest databases in the world and has a broad range of experience from mainframes, client server, embedded systems, linux clusters, data warehouse, business intelligence, enterprise portals, financial applications, BPM, and SOA to name a few.

StackSafe: Tell us about yourself. How did you become focused on IT Operations?

Mike Kavis: I am the chief architect at Catalina Marketing. My focus is primarily on enterprise architecture. From the operations standpoint, I am focused on the run-time governance aspects of SOA. It is critical to monitor the health of our customer facing systems and monitor and measure our service policies and SLAs.

StackSafe: What is the biggest challenge to SOA success today?

Mike Kavis: The biggest challenge today is not the technology, it is the people. In our case, we are implementing both BPM and SOA. This brings a large amount of culture change both on the business and technology side. Organizational change management is the number one challenge that I am faced with on a daily basis. Technology problems can always be solved by leveraging the right technical resources, but getting people everyone on board with why change is needed, what the change means to them, and how the company will benefit if they change can be challenging. Don’t underestimate the impacts of resistance to change. It can undermine any project.

StackSafe: How well do you think virtualization is being employed by companies today? Do you see any problems with virtualization ahead?

Mike Kavis: Virtualization is being deployed with the goal of reducing power and simplifying server management. Most companies that I am familiar with are doing a decent job of implementing virtualization. My company has consolidated well over 100 servers onto a clustered VM environment which has helped reduced costs and made life much easier for the administrators.

One issue I see with virtualization is many applications like our BPM tool and our ESB, require very specific VM settings. Vendors do not seem to be publishing the optimal settings and companies are left performing trial and error exercises to optimally configure each virtual machine. When issues occur, there can be a tendency for vendors to start pointing fingers at each other.

StackSafe: You have mentioned that underestimating or ignoring impact of change is a leading reason why enterprise initiatives fail. How can companies better understand the impacts of change?

Mike Kavis: I highly recommend assigning a full time person to the role of organizational change management. The cost of this resource will pay for itself over and over. We tried to tackle it from a steering committee approach, but we all had our plates full of various other critical tasks. The first step towards understanding the impacts of change is to perform a readiness assessment. This needs to be done from a business, technology, customer viewpoint. Once this assessment has been performed, a strategy must be put in place to deal with the areas where the impacts of change are the greatest. The tasks associated with change must be managed in the project plan along with the tasks for developing and deploying the solution.

StackSafe: What do you view as the biggest threat to availability for IT operations teams?

Mike Kavis: Complexity. As more companies attempt to implement SOA, the complexity of today’s production environments increases substantially. This is a tradeoff. With SOA, we are giving the business the ultimate flexibility and increased speed to market. By making things simpler for our customers, we create a huge management challenge on the backend for IT. SOA gives us a distributed, heterogeneous environment that can quickly get out of control if it is not properly governed. This requires new skills in operations and many new tools required to monitor and manage this complex environment. There are so many points of failure in this time of architecture that availability can be extremely challenging if you don’t establish a robust run-time governance strategy.

StackSafe: What is your opinion of the alignment between developers and IT operations with regards to improvements to business applications?

Mike Kavis: In my 22+ years in IT, I have always seen these two groups act as two separate silos with different priorities. Operations tend to rank manageability as the highest priority which can lead to the wrong solution for the business applications team. In my world, operations tends to shy aware from tools such as wireless access, instant messaging, and many other tools that their customers are screaming for. At the end of the day operations needs to balance standards and security with customer needs and productivity. I believe the answer is to embed operations people in business application projects from the start so that the operations people can get a better understanding of the business requirements and form better relationships with the development staff.

StackSafe: If a company could focus on one area to improve uptime and availability, where would you recommend they begin?

Mike Kavis: I believe it starts with architecture. Too often solutions are not architected properly or are built with no regards to the existing infrastructure. I feel that IT spends a lot of time addressing the symptoms but ignoring the root cause. This is especially true as we move into an era where integration and service orientation is becoming critical to the business. As our environments become more complex, we should invest more time and effort up front in our projects so we spend less money in the long run keeping the wheels on the bus.

Today’s blog offers technical virtualization hints and tips. It’s intended for those of you leaning toward investing in the technology and effort to build a virtualized staging server for IT testing. If you are brand new to the concept of virtualization as a technology, there is a wealth of information on the Internet, such as a primer by Datamation, and broader coverage available from SearchServerVirtualization.com. I also recommend you review an article by Rob Latimer, Principal Consultant for Glasshouse Technologies, entitled Guidelines for virtualization preparedness to help you evaluate virtualization from a clear-headed perspective.

A number of the organizations with whom we speak have already invested in server virtualization for consolidation reasons. Many of these organizations are looking to utilize virtualization for change and release management, as well as other second-generation virtualization uses. However, fewer organizations have deployed virtual staging environments for IT testing – in large part because they are still deciding how and where virtualization will benefit their staging and testing process.

That is why we have developed a ‘starter’ set of technical guidelines to consider when planning to use full virtualization to stage, change, test and analyze the impact of change to your software infrastructure. Full virtualization in this case is characterized by one or more virtual machines (VM) running on a single hypervisor on a physical machine.

First, you will want to set expectations with management. They will likely ask “why should we utilize virtualization for your staging platform for IT testing?”  Look back at numerous StackSafe blogs for help in identifying how virtual staging and testing can improve your testing maturity.

In general, a virtualized staging server can help you address some common problems faced by IT Operations teams:

• Production faults, performance problems and/or downtime caused by change
• Production changes that result in problems
• Changes that require roll back from production
• Testing at least 90% of the changes that are deployed to production
• Testing changes across the entire infrastructure from end-to-end
• Excessive time and/or resources to build, maintain, and synchronize staging and testing environments to represent the production environment
• Painful and costly business application downtime
• The need for quicker turnaround on changes to production in order to maintain competitiveness

Next, consider the basic ‘pieces’ of a virtualized staging server, which are offered by multiple vendors.

• The virtualization platform. This platform features a hypervisor that hosts ‘guest’ VM operating systems. The hypervisor lets VM OSes share the physical machine memory, CPU, and other bare metal services. This virtualization platform can be used to host virtual staging and testing, virtual production, virtual disaster recovery backups, or any other virtual environment.
• P2V (physical-to-virtual) import technology.  This is required to copy physical machines onto the virtual platform as VMs (think server consolidation). By capturing the software image of the production server, P2V creates VMs that more closely represent the physical environment than  building representative VMs.

Finally, you will want to follow certain planning steps when building a virtualized staging server:

1. Locate a server with CPU that supports virtualization for your virtualized staging server. Nearly all CPUs support virtualization these days, but make sure.
2. Determine where you want the virtualized staging server to reside. If you have a test lab, you will likely want to supplement the existing staging and testing environment. This placement provides you with access to testing tools and existing staging servers that you may want to virtualize.
3. Decide on how the virtualized staging serverwill be networked. This will require answering questions like: where will itreside or which networks should it be able to access? P2V may require network access between the physical machinesand the virtualized staging server. Security and system ownership issues will also dictate some of these decisions.
4. Determine what type of VM you expect to run on the virtualized staging server. This includes the base OS and hypervisor, and all of the VMs that are expected to be running simultaneously on the system. For example, if you plan to make and test change to five 2 CPU/core VMs w/ 1GB of memory, the system will require approximately 12 CPU/cores and 6GB of memory.
5. Scope the virtualized staging server to run multiple VMs, bearing in mind that each VM can require different levels of system resources.

One other suggestion: Manage your import process according to your testing requirements.  If you perform redundant testing of the same system components (for example quarterly failover testing), it should be relatively easy to import your baseline configuration and maintain this ‘gold master’.  If on the other hand you expect to perform a variety of tests against a variety of system configurations, pay close attention to the virtualization staging server resource load to ensure that you aren’t introducing stress to the system that may be incorrectly blamed on the change/s made to the imported VMs.

There are other virtualized staging server-related concepts to consider that we will cover in future blogs, including the types of tests and analysis IT may want to perform, and a review of the change and release management activities that don’t adapt well to a virtualized staging server.

In a previous blog post, we discussed the challenges of the unfortunate middle.

Today, we want to focus on the “Test Leaders” and how improved testing maturity leads to clear business advantages.

Testing maturity in the report is measured according to the following criteria:

• Does the organization maintain a staging/testing environment representative of production?
• How thorough is the organization at testing changes?
• What % of changes to the production environment are testing prior to rollout?

So what does being a “Test Leader” mean? The business benefits of improved testing maturity include:

Reduction in Number of Unplanned Downtime Incidents

Test leaders exhibit significant less unplanned downtime incidents per year than those in the unfortunate middle including:

• 30% less downtime incidents on an annual basis than the “Unfortunate Middle”.
• 33% lower rollback rate, (changes that have to be rolled back due to problems that can’t be resolved in production)

Reduction in Percentage of Changes that Cause Problems in Production

“Test Leaders” are much better at identifying problems during testing and minimizing the delays in releasing changes to software infrastructure including:

• 22% lower problem rate (the percentage of changes that cause problems in production).
• 21% lower delay rate (the percentage of changes causing significant delay is 21% lower for Test Leaders).

Improved Testing Efficiency

Test Leaders are more efficient testers.

• On a per test basis, Test Leaders exhibit a 17% reduction in IT Hours spent per test than respondents in the unfortunate middle.

Improved Smoothness in Change Management Process

Test leaders see significant improvements in the numbers of respondents who view their change management process as “smooth all the time”.

• 100% increase in reported change management process smoothness compared to the unfortunate middle.

Business Benefits of Improved Testing Maturity

The business benefits of improved testing maturity by IT operations teams seem clear. By investing in building representative staging and testing environments for the multi-tier business applications and testing the impact of as close to 100% of changes as possible across the entire software infrastructure stack, IT operations teams can:

• Reduce downtime
• Reduce number of problems in production caused by changes
• Improve testing efficiency
• Improve smoothness in the change management process

Downloading the Report

You can download a full copy of the IT Operations Research Report: Testing Maturity here.

A recent study on testing maturity for applications and operating systems conducted by StackSafe and Research Edge reveals that applications heavily influence the approach companies adopt for change management and testing. This is in part due to differences in application complexity, change lead time, and downtime costs. It is also determined by the unique nature of the application and the manner of organizational operation.

From our earlier blog post three application profiles have emerged from the manner in which applications influence change management and testing.

One profile, the High Stress Environment, is characterized in part by companies who’s IT operations departments must support E-commerce applications.

E-commerce Applications and Downtime

Let’s take a closer look at the impact of E-commerce applications on change management and testing. One important characteristic from the study jumps out: companies that rely on E-commerce typically experience a higher percentage of emergency changes than companies running other application types.

As we’ve discussed previously in regard to testing maturity, companies experience more problems in production when they require IT operations to address a higher number of unscheduled (e.g. emergency) changes. True to form, our most recent study shows that over 25 percent of companies supporting E-commerce applications test less than 50 percent of all changes. As web applications continue to grow in both users and complexity, more outages (a la Yahoo’s Cyber Monday snafu) may become commonplace. Don’t forget that:

“With an ecommerce website or an advertisement supported website, this downtime means lost revenue and worse yet, lost customers.”

Because of the nature of E-commerce applications, IT operations appears to face more pressure in making changes to multi-tier applications. More than half (52 percent) of all E-commerce companies support in-house software development, so expectations are likely higher for short development-to-deployment cycles. The customer-facing nature of E-commerce and the constant change associated with web-based strategies and technology combine to create the emergency change scenario discussed above. The study shows that E-commerce changes are introduced without sufficient lead time or planning when compared to other application types.

Companies report that 29 percent of E-commerce changes are considered emergencies, which naturally causes delays and increases tensions within the IT department. As the preceding graphic demonstrates, IT is not at all confident of changes to E-commerce production 16 percent of the time. Small wonder, seeing that 28 percent of changes cause problems to E-commerce production systems.

At StackSafe, we believe that it doesn’t matter if an application exists in a e-commerce environment. IT requires the capability to establish representative staging environments for IT testing, to test every change, and to test changes against the entire infrastructure, no matter how tight the timeframe or how high the change priority.

When IT cuts corners with testing, particularly with E-commerce application testing, “always on” may end up being “sometimes on” or “maybe on in a few hours.” In business-critical situations, such as preparing for high spikes in traffic around peak online shopping times the diligence given to testing changes will make or break the online shopping experience.

Download the Report:
You can download a full copy of the IT Operations Research Report: Testing Maturity Part II – Applications and Operating Systems here.

Past IT Operations research reports have focused on downtime, change management and testing maturity.

Key Findings on Testing Differences in Multi-Tiered Applications:

• Applications heavily influence the approach companies adopt for change management and testing. This is in part due to differences in application complexity, change lead time and downtime costs. Applications driving change management and testing fall into three general camps:
  • Prudent Planners – companies supporting Enterprise Resource Planning (ERP), Production/Supply Chain Management and Transaction Processing
  • High Stress Environments – companies supporting Customer Relationship Management and E-commerce
  • Laissez-faire Planners – companies supporting Website Hosting

• “Prudent Planners” are generally very large companies, where downtime caused by production problems can be quite expensive. Not surprisingly, most have implemented automated change management and testing procedures. As a result, even though change volume is high, these companies generally have proportionally fewer problems in production stemming from change management.

• “High Stress Environments” such as E-commerce and CRM seem to face more pressure in making changes to multi-tier applications. IT departments supporting E-commerce are often faced with emergency changes that can cause delays and increase tensions within the department. Companies supporting CRM, on the other hand, face more pressure to reduce the cost of testing compared to the average company.

• “Laissez-faire” Planners are less oriented toward automated change management, even though they are most likely to have staging/testing environments in place. It appears automating change management is of less value when the focus is on Website Hosting. These companies may benefit from greater emphasis on scope or frequency of testing.

Key Findings in Testing Differences by Operating Systems

• Increased downtime costs are associated with use of multiple operating systems (heterogeneous environments) and the Unix operating system in particular.
• Companies supporting multiple operating systems have nearly double the cost of unplanned downtime than companies utilizing a single OS to support all system tiers (web, data base, application)

Downloading the Report

A full complimentary copy of the IT Operations Research Report: Testing Maturity Part II Applications and Operating Systems is available for download.

We will be returning in future blog posts to explore these topics in more detail.

Nonetheless, there are real implications that must be understood when shortcuts are taken. Below is a list of the “Dirty Half-Dozen” testing shortcuts with more description about the problems they cause and why they so often lead to downtime for IT Operations.

Shortcut #1 – Patch and Pray

• What it is: Bypassing testing altogether and deploy changes directly into production.
• Rationale: “This has probably been tested by the vendor or by QA, so most of the big problems have most likely been fixed. Besides, we’ll clean up any mess after implementation.”

• Implication: Obviously, patch-and-pray approaches increase the risk that the change will result in production downtime. Further, no testing whatsoever makes finding the cause of a problem much more difficult. If you are implementing the patch on a Friday, don’t make any weekend plans.

Shortcut #2 – “Non-Customer-Facing” Tests

• What it is: Deploying new configurations and patches on live systems that are low-priority or non-customer facing applications.

• Rationale: “I need to test this change quickly, but I don’t have a test environment that looks like full production. I’ll use the group that is least likely to complain as my guinea pigs and roll the change out to full production if my blackberry doesn’t explode with angry users.”

• Implication: Two problems here. First, the change is still getting into the live environment, so there is some level of risk being assumed. Second, limiting testing to one group might not be representative of the entire environment. Naturally, incomplete testing methods like this limit the ability to understand the full impact of the changes.

Shortcut #3 – Scheduled Downtime

• What it is: Testing the change in production during scheduled downtime.

• Rationale: “I am not sure what this change is going to do when my environment is live. I’ll wait until the system is down to perform my test.”

• Implication: Do you have the luxury of scheduled downtime? It is increasingly hard to come by in this era of high availability. Also, any changes made to the production environment introduce risk of data and process corruption, which may prove unacceptable to more sensitive systems.

Shortcut #4 – Borrowing Redundant Resources

• What it is: Taking redundant disaster recovery (DR) systems offline for testing purposes.

• Rationale: “My DR systems look a lot like the live environment. I’ll use them for a quick test.”

• Implication: If you are using disaster recovery systems for testing of this sort, you’d better not have an actual disaster…or there could be trouble.

Shortcut #5 – Component Testing

• What it is: Limiting testing to an individual software component or configuration change rather than testing the change as part of a full end-to-end distributed stack test.

• Rationale: “I have a portion of my IT service that I think is fairly representative of the entire environment. I’ll test the change on this component first and roll the change out to full production if nothing goes wrong.”

• Implication: Similar downsides to the “Non-customer” facing tests. It is difficult to gauge how well representative any segment is of the entire end-to-end application. Also, putting a change into any live environment – no matter how well segmented – assumes some level of risk.

Shortcut #6 – Tabletop Testing

• What it is: – Evaluating changes using documents, spreadsheets and diagrams on a conference table.

• Rationale: “My team has seen changes like this before. They’ll be able to give me a general sense of what I can expect if we hold a meeting along the lines of a miniature change advisory board.”

• Implication: Table-top testing becomes the only practical method when the changes and/or the environment to be tested is too large or complex to duplicate, or when there is a lack of resources and time. However, this approach provides no actual insight into the true impact of the change on production environments.

All of the above approaches have significant limitations that make them impractical or inefficient for understanding the business impact of changes to the software infrastructure. As a result, in far too many cases, IT Operations is forced to make changes and adjustments in non-representative staging environments, and to live production software infrastructure, without any testing whatsoever. This practice leads to the serious failures and downtime that organizations try so desperately to avoid.

Inadequate IT operations testing is a key reason for changes going awry

• Four out of ten companies consider insufficient pre-production testing a key reason for the downstream production problems faced in implementing changes to multi-tier applications. The other reasons identified for changes going awry are almost all IT operations testing related, including interaction between systems, vendor patches/upgrades, lack of visibility into new apps before going live and in-house patches/upgrades.
• The more changes that are pushed through an organization, the more likely pre-production testing is considered a problem

Delays and rollbacks are common

• Nearly a quarter (22%) of the changes made to applications and the software infrastructure cause significant delays
• More than ten percent of changes are rolled back and never implemented - problems are encountered in production that they simply cannot resolve

Software Infrastructure changes tend to be either not tested at all or not tested thoroughly

• Most companies test changes to the critical applications, but leave many changes to chance.
• Even when production changes are tested, testing is often less than comprehensive, limited to components of the stack. Only 42% testing the entire multi-tier stack

A third of all enterprises lack the capacity to conduct serious pre-release testing

• While most IT department conduct tests, 30% leverage servers or backup data centers, but lack an infrastructure specifically designed for testing and staging
• While the majority (65%) maintain testing or staging environments, utilization is highly variable

If you are interested in the rest of the findings from the study, feel free to download the report here. We will continue to post information from these report that we think are of interest from time to time.