VMTurbo Monitor

When we interview VMware administrators and architects, hands down, they tell us that their biggest challenge is determining and ensuring virtual infrastructure health.  This challenge led us to develop the VMTurbo Monitor.

The VMTurbo Monitor provides the following features, packaged in an easy-to-use virtual appliance:

• ‘Red-Yellow-Green’ heatmap of VM and host health
• Real-time display of key virtual infrastructure performance metrics
• Detection of CPU, memory, and I/O bottlenecks

Best of all, we’re offering VMTurbo Monitor completely free of charge.  Download, start monitoring, and let us know what you think!

VMTurbo Host Reporter

VMTurbo Host Reporter provides offline charts and graphs that display virtual infrastructure trends over time.   System administrators use it to:

• predict the capacity remaining in their environment
• identify hosts and virtual machines that may be experiencing performance problems
• eliminate wasted or idle resources

VMTurbo Host Reporter is available now for a free 30 day trial.  It is completely integrated with VMTurbo Monitor, and both products are delivered in a single virtual appliance.  Click here if you’d like to download the virtual appliance.

This is just the start of what we’re bringing to the virtualization market this year.  Stay tuned!

In the talk, our CEO and cofounder Shmuel Kliger will discuss the 5 key management gaps that exist in today’s virtual and cloud environments.

If you are thinking of building your own cloud, or considering how to leverage third party infrastructure-as-a-service, you won’t want to miss this talk.

You can vote by visiting this SXSW PanelPicker link directly and clicking the ‘Thumbs Up’ button (registration required).

Anyone looking for a virtualization moment of zen today needed to look no farther than Cody Bunch of ProfessionalVMware.com, when he opined on Twitter:

%RDY %RDY %RDY – It’s like having customers waiting for donuts. When you have more customers than donuts, you have a problem. #donutzen

This is a great way to describe the CPU co-scheduling problems that can crop up when using VMware, and based on what we have seen in the field, these problems are quite common.  The co-scheduling problem arises when ESX Servers need to schedule multiple processors to service a virtual symmetric multiprocessor (vSMP). In order to emulate the semantics of an SMP, these processors  must be co-scheduled concurrently to service the vSMP.

So, if your SQL Server VM requires a vSMP with 4 vCPUs, then it will need to grab 4 physical processor cores in order to execute.  The ESX co-scheduling mechanisms will first try to run it, even if it does not have 4 available vCPUs. However, as soon as it hits an event requiring all 4 vCPUs, it will place the vSMP into the CPU ready queue until 4 cores become available to service all 4 vCPUs.   It is possible that the vSMP will remain waiting in the ready queue for a long time.  Other VMs requiring only 1 vCPU may grab cores as soon as they become available, starving the vSMP VM.

Of course we can think about this in terms of Cody’s donuts. Suppose the donut store keeps you waiting for an order of 4 donuts, until it services all customers requiring only 1 donut.  During the morning rush hour when the stream of customers requiring 1 donut seems never ending, you will wait forever until your order of 4 donuts may be satisfied.

OK, OK, I think you get the point…so let me continue…
This problem can be diagnosed by examining the %rdy values on the ESX Server, but solving it is another matter entirely. If you are willing to sacrifice all of your vSMP virtual machines then you could make this problem disappear instantly. However, many mission critical applications require the performance benefits of SMP architectures. Forcing them to avoid virtual infrastructures would significantly limit the value of virtualization.

Worse. Often, as traffic demands increase, one would like to allocate more resources to their vSMP virtual machines. Consider an application using that is running on a 2 vCPU VM. Suppose one wishes to accelerate the processing speed of peak traffic by doubling the allocation of vCPUs to 4.  What is one to do if instead of improving processing speeds, they witness a dramatic decline? Such decline is due to increasing waiting time in the CPU ready queue; grabbing 4 vCPUs may take substantially longer than getting 2 vCPUs.

Most customers we speak with are running some fairly beefy virtual machines that require more CPU horsepower that a single vCPU virtual machine can provide. So, they try to strike a balance and periodically examine vSMP virtual machines to see if the %rdy values on a given ESX Server are high. If these values are high, they usually try to VMotion the VMs to different hosts to address the problem.

Unfortunately, solving this issue is a real time problem. CPU ready will fluctuate as demand on the virtual infrastructure changes, and as demands on the applications running on that infrastructure change. Taking a point-in-time snapshot of the environment may solve the problem right now, but it won’t cure it for good.

Now is probably a good time to mention that one of the most popular features in VMTurbo’s virtual appliance is real-time virtual machine rightsizing. Our virtual appliance ensures that all of your virtual machines are running at the right size, at the right time, even as demands on your applications and infrastructure change. If you’d like to rightsize your environment, let us know and we’ll give you a link to download our virtual appliance. Download and be rightsized in just minutes!

Virtualization management technologies provide many controls to move workloads, allocate resources, and change configurations settings at the physical and virtual layer. It is also possible to collect performance metrics on key resources such as CPU, memory, network I/O, storage I/O, CPU wait times, CPU ready queues, ballooning, swapping and the like.

However, what is missing from this picture is an intelligent guidance capability that can determine what acitons should be taken in the context of these controls to ensure each application gets the resources that it needs to perform effectively.  Furthermore, in periods when resources are constrained (eg infrastructure capacity is in high demand) there are no controls to determine which applications get priority based on their business importance or service level.

This service assurance capability is critical in ensuring that organizations can successfully continue on the journey of virtualizing a greater percentage of their business applications.

(Note from John: Andrew has been working in the IT management space for the last 20 years and has some great perspectives on the operational and financial benefits of virtualization.  This is his first blog post.  Welcome to Turbo Talk, Andrew!)

Virtualization presents great opportunities to exploit the unused compute capacity that exist in the datacenter.  While most organizations have collapsed their infrastructure using virtualization, they have designed their deployments to accommodate the peak workload of all applications that reside on a given physical platform. This is very wasteful and costly because there are significant periods/points in time when system resources are completely underutilized.  Traditional capacity management techniques do not provide the answers of how to exploit this unused capacity because they were developed in the physical world. In the physical world there was no notion of being able to dynamically reconfigure the environment to move workloads around to exploit free resource capacity.

In order to realise the true potential of virtualization, an overall virtual infrastructure service assurance capability is needed.  This capability must have the intelligence to determine how to best rightsize and place workloads so that infrastructure utilisation can be driven to higher levels without causing interference between workloads.  This capability should also prevent utilization of resources (eg CPU, memory, Network i/O, Storage I/O, Storage IOPS) from reaching levels which could impact application performance, and should provide the ability to recommend when additional physical capacity is required in order to meet the growth in demand caused by higher levels of application usage or provisioning of new applications. In periods of low demand, the ability to consolidate workloads onto a fewer number of physical machines should also be possible, so that power consumption can be reduced without compromising application performance.

This capability should not just exist in the production environment.  It should be possible to perform “what if” scenario planning in order to play out new situations so that the impact can be understood.  Examples of “what if” scenarios might include provisioning new VM‘s/applications, increasing workload demand at key times of the year, or understanding the implications of hardware replacement and refresh.

If the virtualization vendors or independent vendors can deliver these capabilities then IT organizations will be in a much better position to improve their ROI on their investment in the virtualized stack.

What are your thoughts about capacity planning and service assurance?  We would be interested to know if these are the challenges you are facing within your virtualized environment.

We coin new terms in the technology industry all the time, so with an ode to tech pundit Geoffrey Moore and our friends at Merriam-Webster, let me add a new one to the mix:

capacity planning chasm: n.  The gap that exists between capacity planning and real time operations.

This gap has always existed within traditional IT, but in today’s virtualized, dynamic datacenters, the gap has widened and consequently the chasm is more treacherous than ever.

Organizations who have fallen into the chasm typically…

…use a wide variety of performance and capacity management tools, VCenter, spreadsheets, and rules-of-thumb to predict future demand on the virtual infrastructure.

…focus on a small number of metrics (usually CPU and memory) due to the challenges in analyzing the shear volume of data coming out of various virtualization management systems.

…operate with very conservative consolidation ratios to assure service levels.

…can’t assure the performance and availability of business-critical virtualized applications because they are operating to a static plan that does not account for the dynamic nature of the virtual infrastructure.

…isolate workload-intensive virtual machines to dedicated hosts or clusters to reduce the risk of introducing bottlenecks to the virtual infrastructure.

…are forced into a reactive, fight-fighting mode, rather than a proactive mode of virtualization management.

If your organization has fallen into the virtual infrastructure capacity planning chasm, here are some ways to help it climb out:

1)  Don’t just focus on “the usual suspects” when conducting your capacity planning analysis. Most customers we speak with are focused on CPU and memory utilization as the focal point of their capacity planning.  Don’t ignore some of the other key statistics like CPU ready queues and storage I/O.  Often these statistics can reveal potential capacity bottlenecks within the virtual infrastructure.

2)  Conduct joint capacity planning and management exercises as frequently as possible. Clearly operations teams and capacity planners or architects are not going to be poring over the utilization patterns of the virtual infrastructure together on a daily or even weekly basis.  However, standard capacity planning cycles (e.g. once per quarter) do not provide the responsiveness that many organizations need when it comes to answering the tough questions like: “How many more VMs of type XYZ can I add to the infrastructure without introducing performance issues?” and “When will I run out of capacity based on predicted growth rates?”   This is a discussion that needs to be a collaboration between the two teams if the desired result (a healthy, rightsized environment) is to be achieved.

3) Ensure your capacity planning toolkit and process is QoS-aware. Certain VMs and applications are business critical and should be treated as such during the capacity planning process.  Make sure that your proposed distribution of VMs takes this variable into account, along with the more traditional utilization metrics.

Do you have capacity planning best practices you’d care to share?  Let us know in the comments section below!

As we wind down for the weekend, just wanted to share an interesting post related to cloud computing adoption.  In the article, Bernard Golden compares the predicted death of local television to the evolution currently taking place in the cloud computing space, and discusses the gaps that must be filled before cloud computing can become the dominant method of IT service delivery.

Thanks for reading! See you next week and enjoy the long weekend!

Follow VMTurbo on Twitter

• In terms of the health and potential growth of the virtualization market, it’s good to see that there are still some well-known organizations are just getting started with virtualization.   At this week’s Uptime Institute symposium, Jack Nichols from the U.S. House of Representatives presented a case study about how his organization massively cut electricity costs by consolidating underutilized servers.  Over a two year period they consolidated over 100 systems, generating power savings of $1000 per day.
• With cloud computing budgets on the rise, hosting providers are gearing up to meet the demand.  Randy Bias of Cloudscaling wrote an excellent article about the labor side of the cloud equation.  The bottom line?  Even with greater automation capabilities, process and culture are still going to play a huge role in operations, maybe even moreso in the cloud than in traditional IT environments.

Thanks for reading, and see you next week!

Over the last few weeks the blogosphere has been a-Twitter(!) with talk of VPLEX, EMC’s new virtual machine “teleportation” solution.

VPLEX is enabling IT operations to more easily move virtual machines back and forth between datacenters. It opens up some great new options as to how companies manage disaster recovery and better balance workloads between datacenters.   However, it also will force IT operations teams to answer some seemingly innocuous, but very difficult questions…

When do I migrate my virtual machines?

What is the right trigger or event that indicates that the operations team should move virtual machines from one datacenter or another?  We’ve seen that even in the most simplistic multi-datacenter scenario (2 datacenters), this decision is very challenging.  Why?  Because (usually) one datacenter is viewed as the disaster recovery site and is running few (or no) live VMs.  Organizations we speak with want to take advantage of the large amount of idle capacity available there, but don’t have the ability to ensure that they can do so without compromising their ability to maintain service levels in the event of a disaster.

Where do I migrate my virtual machines?

If I have multiple datacenters or clouds at my disposal, which one(s) should be the destination for the migrated virtual machines?  How do I decide the best location for these VMs to run?  There could be substantial benefits gained by running the VMs in one location versus another (e.g. lower cost of operations, lower latency, closer proximity to end users).  Unfortunately, many of the organizations we speak with don’t have the visibility into these variables or have tools to analyze that data.

Which virtual machines do I migrate?

Many of our users have deployed applications comprised of multiple virtual machines (e.g. 3 tier applications).  If the virtual machines associated with a single application are not running in the same datacenter, performance may suffer.  The logical response is to keep these virtual machines together in the same datacenter so that latency between components is kept to a minimum.  However, the decision is not so simple.  What if the application is non-critical and very lightly used?  Maybe it would make sense to break up the virtual machines that constitute the application across multiple datacenters, allowing business critical virtual machines unrelated to that application to get prioritized access to resources.  This kind of decision requires real-time analysis to ensure that a tradeoff like what is proposed here can be accurately and quickly evaluated.

As we continue to pile more and more questions like these onto operations teams, quality of service and efficiency suffer.  This is no fault of the operations teams, it’s simply a fact of life in a world where there is a flood of data coming from a variety of monitoring and management tools and not enough people or time to analyze it all.  If you feel like you are stuck in this IT operations data deluge, please let us know in the comments or directly via this link.  We can help.

Related links:

• Steve Todd of EMC dives deep on some of the innards of the VPLEX solution.
• VPLEX Undressed (a more skeptical viewpoint)
• VPLEX unleashes federation on the globe

Many of the organizations we’re speaking with have reached out to us because their virtualization initiatives have slowed or grinded to a halt after some initial success. The slowdown in virtualization deployment usually happens once the low hanging fruit (test, development, departmental applications, etc) has been addressed. Here are a couple of reasons we’ve heard from our customers about why this is happening:

1. Major concerns about potential performance degradation from application owners – Even though virtual infrastructure technology has been mainstream for several years, many times business and application owners will resist the move to a virtualized platform. Eventually, they will make the move, but often insist that the virtual machines or hosts supporting their applications be oversized. This is counterproductive because it reduces available capacity for other virtual machines that may need the resources, and it can in some cases cause poor performance for the VM itself or other VMs.

2. Actual issues with application performance – The organizations that dip their toe into the water of virtualizing production workloads with intense resource requirements (usually I/O) find that existing tools don’t give them the control they require to balance the demands of these workloads with the capacity available, while ensuring that other workloads are prioritized (or de-prioritized) properly. This results in the implementation of more complex processes and constraints to try and reduce the risk of virtualizing these workloads. This is also a case where overprovisioning virtual machines and hosts is applied as a “band-aid” to the problem.

The end result is that the business case for virtualization ends up being revised, as the potential to reduce capital expenditure is constrained and expected savings in operational costs are more difficult to realize because of the added complexity (additional support processes, constraints, etc).

So how can one jumpstart stalled virtualization initiatives? Here are a couple of things that can help based on what we’ve seen in the field:

1. Implement Quality of Service policies – Tiered virtual machine service offerings (e.g. gold, silver, bronze) are a mechanism to ensure that the business and application owners understand the relationship between the cost of the service and the performance and availability attainable. By having these two levers, virtual infrastructure architects can work with the business to properly size virtual machines based on the expected workload, while ensuring that mission critical VMs can be prioritized over those that are less important to the business.

2. Integrate capacity planning, demand management, and performance management – Virtual infrastructure capacity planning and performance management, especially in larger organizations, are usually handled by separate teams. Because virtualized workloads are dynamic and can be brought online and offline very quickly, slight changes in virtual infrastructure capacity, workload balance, or demand can quickly have a negative effect on performance and availability. All three of these functions need to be managed in lockstep and in real time to ensure that the virtual infrastructure is healthy and cost effective to operate.

What other techniques have you used to get your virtualization initiatives restarted? Are you seeing any other challenges that have slowed down your progress along the path to a dynamic datacenter? Please let us know in the comments.

Related articles by Zemanta

• When It Comes to Virtualization, Are We There Yet? (gigaom.com)
• Death of the Sysadmin and 3 reasons why today’s virtualization management tools are failing (vmturbo.com)