Server virtualization is becoming widespread, bringing multiples benefits to infrastructure management: reduction in the number of physical servers, simplified provisioning and rationalization of technological architectures.

However, with virtualization come new practices that do not match with old infrastructures habits, especially when it involves new server allocation.

The challenge does not reside in how simple is the creation of a new server in “three-clicks” - generating virtual server sprawl - but rather the difficulties of properly configuring the characteristics of a virtual machine.

The old adage “Small is beautiful” is rarely applied to server configuration. In the past when buying physical machines, the amount of server capacity required would have been overestimated in order to avoid potential performance problems.

There are challenges finding the ‘sweet spot’ when applying this approach when configuring server capacity in a virtualized environment:

-       An undersized virtual machine will obviously lead to performance problems

-       An oversized virtual machine will also lead to performance problems

When planning processor management in a virtual environment, it is important to consider that to allow virtual machine execution the hypervisor should be able to schedule every virtual processor to a physical processor.

Extra considerations need to be factored into capacity planning, such as within a 4-core server, setting up multiple virtual machines configured each with 4 virtual processors will produce significant contention on physical processors, making the simultaneous execution of multiple virtual machines more difficult and generating latency. The waiting time resulting of this practice will therefore impact the overall performance of every virtual machine running in the server.

On the memory management side, the hypervisor distributes the physical memory pages to the virtual machines according to their needs, and manages memory shortage through a global paging strategy that impacts performance of every virtual machine on the server.

Oversizing virtual machine memory configuration can lead to issues when operating applications embark on opportunistic behavior such as reserving all available memory for their own use (eg: Oracle, SAP, SQL Server, Exchange …).  In this situation, both guest operating system and hypervisor cannot rely on a strategy to determine efficiently which pages should be pinned into physical memory and which one should be swapped. As a consequence, excessive global paging activity will generate memory contention in every virtual machine degrading overall server performance.

Already the apparent simplicity provided by virtual infrastructures is being questioned, and the potential risks created by configuration side effects between virtual machines are exposed. Whatever the virtualization platform (Xen, VMware, POWER…), the implementation of virtual infrastructures does not preclude of fine tuning the performances of the systems to find the right balance point. After all, haven’t we been warned about the risk of putting all one’s eggs in one basket?

This month, ORSYP is sponsoring HP Discover 2011 and Gartner IT Infrastructure, Operations & Management Summit.

In Las Vegas, we will be a silver sponsor at this year’s HP Discover from June 6-10. Last fall, we enhanced our industry presence by joining HP’s marketplace referral program. Through sponsorship of this show, we are continuing our support of HP and enjoy attending their leading industry events.

After our time in Vegas, it’s off to Orlando where we’ll be a silver sponsor at the Gartner IT Infrastructure, Operations & Management Summit from June 13-15. Last year at the show, attendance was phenomenal and demonstrated to us that there is a genuine need for information and education in IT Operations Management (ITOM.) We look forward to meeting even more people this year and hearing the buzz!

As always, follow along as we Tweet from each show and stop by the booth to say hello.

So, now that we’ve outlined what issues produce hidden costs, let’s look at what this means when it comes to dollars and cents.  Assuming that a company has 50 UNIX servers, each running 30 CRON jobs per day, this equals 1,500 CRON jobs running per day.  For this example, let’s assume a common industry rate of failure of five percent, meaning that 75 of this company’s CRON jobs per day fail to execute as planned.  If it takes an IT manager roughly 10 minutes to find the failure, fix it and then restart the program that totals an average of 750 minutes or 12.5 hours per day.  Finally, assuming an average labor rate of $130 per hour, the day’s job failures have cost the company $1,625.

This number does not take into account the business costs associated with a job not being executed on time or lost productivity associated with downtime.  Although this amount may seem small at first glance for an enterprise company with massive IT budgets, when multiplied across the number of working days in a year, factoring in time for vacations and holidays, it still comes to 230 days and a cost of $373,750.  If a company were able to reduce its failure rate modestly from five percent to two percent, that would represent a savings of $149,500 on an annual basis.

In addition to the financial costs, there is an impact on human capital resources as well. Today IT departments are juggling priorities trying to make do with a minimal number of assets.  Without a unified 360° view of all the non-critical jobs and tasks running, IT managers are left to monitor the jobs on each server as best they can.

If the average job requires 10 seconds of monitoring , referring back to the example above, a network environment running 50 UNIX servers with a total of 30 CRON jobs each per day would equal a significant number of hours from an IT manager.  How many man-hours are required?  The numbers may surprise you.  The environment in this example would run 345,000 jobs per year, requiring 958 hours of monitoring from IT.  Translated into work weeks, this task would consume 24 weeks, or approximately half a year of IT management hours that could have been re-allocated to other important IT activities.

An investment in the critical inspection and measurement of a company’s embedded infrastructure can result in significant levels of cost savings and IT department efficiency.  A thorough review of how job processing is managed - including number of jobs running, automation analysis, job reviews and status reporting - can provide valuable insight into the level of proficiency at which a company is managing its daily job processing.  Perhaps more importantly, it will highlight the areas in which a company can reallocate valuable dollars and resources to other critical projects.

As most IT managers know, even the smallest jobs can add up and produce big costs. Most companies today have IT environments with an interwoven mix of hardware platforms, operating systems, applications and toolsets.  Embedded deep in this mix are thousands of jobs and tasks running both critical and non-critical processes. In most cases, the IT manager uses an enterprise job scheduler to manage the critical processes that keep the business operational, but what about the non-critical processes?

There are convenient, inexpensive options available that leverage operating system utilities, but as environments expand to support business growth, so does the number of jobs required to maintain continuity within business operations.  This phenomenon makes it extremely difficult for IT managers to continue to monitor and support them without allocating additional IT resources to the project.

Another layer of difficulty is added to this issue when these jobs fail. IT managers aren’t notified until it it’s too late – meaning they discover such failures only after the job or task fails to run as scheduled.  At this point, the IT manager is required to go searching through the infrastructure to locate the failure, fix it and then restart the job; knowing this could have all been prevented.

The amount of time lost correcting a job failure, not to mention the cost of jobs not being completed on time, is an expense that managers often overlook.  While each instance may not be reflected as a line item, over time they accrue and represent a larger-than-anticipated expense in terms of both IT resources and lost production.  So what on the surface seems to be an inexpensive option actually increases the total cost of ownership.

What would seem to be a rather fundamental function of the IT department can have a profound impact on both budget and asset allocation within a company’s IT structure.  Even non-critical job failures have the ability to detract from higher priority business objectives requiring IT support.  Next week, we’ll break these issues down and put our theory into numbers to outline just what these hidden costs can add up to.