VCPgeeks

Troubleshooting Performance Related Problems in vSphere 4.1 Environments

2011-02-24T07:21:00.000-08:00

Source : communities.vmware.com

The hugely popular Performance Troubleshooting for VMware vSphere 4 guide is now updated for vSphere 4.1 . This document provides step-by-step approach for troubleshooting most common performance problems in vSphere-based virtual environments. The steps discussed in the document use performance data and charts readily available in the vSphere Client and esxtop to aid the troubleshooting flows. Each performance troubleshooting flow has two parts:

How to identify the problem using specific performance counters.
Possible causes of the problem and solutions to solve it.

New sections that were added to the document include troubleshooting performance problems in resource pools on standalone hosts and DRS clusters, additional troubleshooting steps for environments experiencing memory pressure (hosts with compressed and swapped memory), high CPU ready time in hosts that are not CPU saturated, environments sharing resources such as storage and network, and environments using snapshots.

This document by no means covers the entire breadth of performance-related problems. We request the readers of this document, including VMware performance community members and vSphere administrators, to help us enhance this document by letting us know about all the performance problems they encounter in their vSphere-based virtual environments, including those that require elaborate troubleshooting steps. We hope that the community will actively contribute by engaging in live discussions, providing feedback, and asking questions. All this input will serve as the topics for future updates.

vsphere41-performance-troubleshooting.pdf (755.0 K)

Due to a possible dead lock on rpmdb, upgrading ESX 4.0 to 4.0 Update 1 can fail or time out and leave the host in an unusable state

2011-01-24T07:21:00.001-08:00

Due to a possible dead lock on rpmdb, upgrading ESX 4.0 to 4.0 Update 1 can fail or time out and leave the host in an unusable state

Symptoms

When attempting to upgrade ESX 4.0 to ESX 4.0 Update 1 (U1), you may experience these symptoms:

Upgrade operation may fail or hang and can result in an incomplete installation
Upon reboot, the host that was being upgraded may be left in an inconsistent state and may display a purple diagnostic screen with the following error:

COS Panic: Int3 @ mp_register_ioapic

Purpose

ESX 4.0 U1 includes an upgrade to glibc version 5.3 which implements a change in locking mechanism compared to glibc version 5.2 already installed with ESX 4.0. If rpm command is run during the installation of ESX 4.0 U1, a dead lock may be placed on rpmdb. For more information, see RedHat PR 463921.

As a result, upgrading ESX 4.0 to 4.0 U1 can fail or time out and leave the host in an unusable state.

While this issue is not hardware vendor specific, this has been reported to occur on HP Proliant systems if Insight Management Agents are already installed and running on the host being upgraded. Investigations into this issue revealed that Insight Management Agents run rpm commands on a regular basis which triggers the deadlock during the U1 installation. This can also occur on any system from other vendors that has a process or an application running rpm, or if you happen to manually run the rpm command, like rpm -qa, while Update 1 installation is in progress.

Note: VMware esxupdate tool can be used standalone and is also used by VMware Update Manager and VMware Host Update Utility.

Resolution

Who is affected

Customers using VMware vSphere 4 upgrading to ESX 4.0 U1 on HP Proliant systems with a supported version of HP Insight Management Agents running.
Customers running rpm commands on systems from any vendor while upgrading to ESX 4.0 U1.

This affects any of the following upgrading scenarios:

Upgrade using Update Manager
Upgrade using esxupdate
Upgrade using vSphere Host Update Utility

Note: ESXi is not affected.

Solution

ESX 4.0 Update 1 has been re-released with changes to avoid this issue. The installation process checks for running agents and stops them before proceeding.

The re-released ESX 4.0 Update1 is referred to as ESX 4.0 Update 1a and is available via vSphere Update Manager (VUM) and the VMware Downloads site.

Note: The changes in ESX 4.0 Update 1a do not address the issue with glibc locking mechanism. It is critical that you do not run rpm commands on any host while the ESX 4.0 Update 1a installation is in progress.

If you meet one or both of the conditions of Who is Affected and you already ran the original ESX 4.0 Update 1 installation but have not rebooted the host, do not reboot the ESX host. Contact VMware Technical Support for assistance. For more information, see How to Submit a Support Request.

WARNING: Rebooting the host means the host may need to be reinstalled because it is not recoverable after a reboot.

WARNING: If you have virtual machines running on local storage, they may not be retained if you reinstall ESX 4.0 as a result of this issue. Contact VMware Support for assistance before reinstalling.

Restarting hostd (mgmt-vmware) on ESX hosts restarts hosted virtual machines where virtual machine Startup/Shutdown is enabled

2011-01-24T07:09:00.000-08:00

Details
This is an issue with virtual machines that are set to automatically start or stop and that are hosted on ESX 3.x. Manually shutting down, starting up, or restarting hostd through the service console causes hosted virtual machines that are set to automatically change power states to stop, start, or restart, respectively.

Disable Virtual Machine Startup/Shutdown for the ESX host through VirtualCenter or a VMware Infrastructure (VI) Client that is directly connected to the host.

GUI Method

To disable Virtual Machine Startup/Shutdown:

Log in to VirtualCenter.
Select the ESX Server host where you want restart hostd.
Select the Configuration tab.
Select Virtual Machine Startup/Shutdown.
Select Properties.
Deselect Allow Virtual machines to start and stop automatically with the system.

CLI Method

If the host is not reachable through VirtualCenter or the VI Client:

Log in to the ESX Server service console as root.
At the command line run vimsh.
At the [/] prompt, type:
hostsvc/autostartmanager/enable_autostart 0
Type exit. You can now safely restart mgmt-vmware (hostd).

How to Divide & Combine vSphere 4.x license keys

2011-01-24T06:48:00.000-08:00

Dividing vSphere 4.x license keys

To divide vSphere 4.x license keys:

Go to http://www.vmware.com/account/login.do and log in to the license portal.
Expand the product edition (e.g vSphere 4 Standard) under Your VMware Product License Keys to view the available license keys.
Click Divide.
Select the license you wish to Divide by clicking on the associated radio button.
Click Continue.

You can review the order information for the license you wish to split and decide how many new licenses you want to generate.
Enter the count for each of the new license keys.
Click Continue.

On the confirmation page, you can review the split operation. A warning message appears.
Click Confirm.

A dialog is displayed while the operation is in progress. When the Split Operation is complete, you return to the Licensing page. The original license key is no longer visible in the portal and you see the newly generated license keys indicated by New.

Combining vSphere 4.x license keys

To combine the vSphere 4.x license keys:

Go to http://www.vmware.com/account/login.do and log in to the license portal.
Expand the product edition (e.g vSphere 4 Standard) under Your VMware Product License Keys to view the available license keys.

Note: You cannot combine license keys that belong to different editions. For example you cannot combine a vSphere Standard License key with a vSphere Enterprise License Key.
Click Combine.
Select the licenses you wish to combine by clicking on the associated check boxes
Click Continue.

On the confirmation page, you have a chance to review the combine operation. A warning message appears.
Click Confirm to proceed with the combine operation.

A dialog is displayed while the operation is in progress. When the Combine Operation completes, you return to the Licensing page. The original license keys are no longer visible in the portal. You see the newly generated license keys indicated by New.

ESXTOP - Deep Dive

2011-01-14T10:29:00.000-08:00

Source - www.yellow-bricks.com

ESXTOP

Intro
Thresholds
Howto – Run
Howto – Capture
Howto – Analyze
Howto – Limit esxtop to specific VMs
References
Changelog
This page is solely dedicated to one of the best tools in the world for ESX; esxtop.

Intro

I am a huge fan of esxtop! I read a couple of pages of the esxtop bible every day before I go to bed. Something I however am always struggling with is the “thresholds” of specific metrics. I fully understand that it is not black/white, performance is the perception of a user in the end.
There must be a certain threshold however. For instance it must be safe to say that when %RDY constantly exceeds the value of 20 it is very likely that the VM responds sluggish. I want to use this article to “define” these thresholds, but I need your help. There are many people reading these articles, together we must know at least a dozen metrics lets collect and document them with possible causes if known.
Please keep in mind that these should only be used as a guideline when doing performance troubleshooting! Also be aware that some metrics are not part of the default view. You can add fields to an esxtop view by clicking “f” on followed by the corresponding character.
I used VMworld presentations, VMware whitepapers, VMware documentation, VMTN Topics and of course my own experience as a source and these are the metrics and thresholds I came up with so far. Please comment and help build the main source for esxtop thresholds.

Metrics and Thresholds

Display	Metric	Threshold	Explanation
CPU	%RDY	10	Overprovisioning of vCPUs, excessive usage of vSMP or a limit(check %MLMTD) has been set. See Jason’s explanation for vSMP VMs
CPU	%CSTP	3	Excessive usage of vSMP. Decrease amount of vCPUs for this particular VM. This should lead to increased scheduling opportunities.
CPU	%SYS	20	The percentage of time spent by system services on behalf of the world. Most likely caused by high IO VM. Check other metrics and VM for possible root cause
CPU	%MLMTD	0	The percentage of time the vCPU was ready to run but deliberately wasn’t scheduled because that would violate the “CPU limit” settings. If larger than 0 the world is being throttled due to the limit on CPU.
CPU	%SWPWT	5	VM waiting on swapped pages to be read from disk. Possible cause: Memory overcommitment.
MEM	MCTLSZ	1	If larger than 0 host is forcing VMs to inflate balloon driver to reclaim memory as host is overcommited.
MEM	SWCUR	1	If larger than 0 host has swapped memory pages in the past. Possible cause: Overcommitment.
MEM	SWR/s	1	If larger than 0 host is actively reading from swap(vswp). Possible cause: Excessive memory overcommitment.
MEM	SWW/s	1	If larger than 0 host is actively writing to swap(vswp). Possible cause: Excessive memory overcommitment.
MEM	CACHEUSD	0	If larger than 0 host has compressed memory. Possible cause: Memory overcommitment.
MEM	ZIP/s	0	If larger than 0 host is actively compressing memory. Possible cause: Memory overcommitment.
MEM	UNZIP/s	0	If larger than 0 host has accessing compressed memory. Possible cause: Previously host was overcommited on memory.
MEM	N%L	80	If less than 80 VM experiences poor NUMA locality. If a VM has a memory size greater than the amount of memory local to each processor, the ESX scheduler does not attempt to use NUMA optimizations for that VM and “remotely” uses memory via “interconnect”.
NETWORK	%DRPTX	1	Dropped packets transmitted, hardware overworked. Possible cause: very high network utilization
NETWORK	%DRPRX	1	Dropped packets received, hardware overworked. Possible cause: very high network utilization
DISK	GAVG	25	Look at “DAVG” and “KAVG” as the sum of both is GAVG.
DISK	DAVG	25	Disk latency most likely to be caused by array.
DISK	KAVG	2	Disk latency caused by the VMkernel, high KAVG usually means queuing. Check “QUED”.
DISK	QUED	1	Queue maxed out. Possibly queue depth set to low. Check with array vendor for optimal queue depth value.
DISK	ABRTS/s	1	Aborts issued by guest(VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason.
DISK	RESETS/s	1	The number of commands reset per second.
DISK	CONS/s	20	SCSI Reservation Conflicts per second. If many SCSI Reservation Conflicts occur performance could be degraded due to the lock on the VMFS.

Running esxtop

Although understanding all the metrics esxtop provides seem to be impossible using esxtop is fairly simple. When you get the hang of it you will notice yourself staring at the metrics/thresholds more often than ever. The following keys are the ones I use the most.
Open console session or ssh to ESX(i) and type:

esxtop

By default the screen will be refreshed every 5 seconds, change this by typing:

s 2

Changing views is easy type the following keys for the associated views:

c = cpu
m = memory
n = network
i = interrupts
d = disk adapter
u = disk device (includes NFS as of 4.0 Update 2)
v = disk VM
p = power states

V = only show virtual machine worlds
e = Expand/Rollup CPU statistics, show details of all worlds associated with group (GID)
k = kill world, for tech support purposes only!
l  = limit display to a single group (GID), enables you to focus on one VM
# = limiting the number of entitites, for instance the top 5

2 = highlight a row, moving down
8 = highlight a row, moving up
4 = remove selected row from view
e = statistics broken down per world
6 = statistics broken down per world

Add/Remove fields:

Changing the order:

Saving all the settings you’ve changed:

Keep in mind that when you don’t change the file-name it will be saved and used as default settings.
Help:

In very large environments esxtop can high CPU utilization due to the amount of data that will need to be gathered and calculations that will need to be done. If CPU appears to highly utilized due to the amount of entities (VMs / LUNs etc) a command line option can be used which locks specific entities and keeps esxtop from gathering specific info to limit the amount of CPU power needed:

esxtop -l

More info about this command line option can be found here.

Capturing esxtop results

First things first. Make sure you only capture relevant info. Ditch the metrics you don’t need. In other words run esxtop and remove/add(f) the fields you don’t actually need or do need! When you are finished make sure to write(W) the configuration to disk. You can either write it to the default config file(esxtop4rc) or write the configuration to a new file.
Now that you have configured esxtop as needed run it in batch mode and save the results to a .csv file:

esxtop -b -d 2 -n 100 > esxtopcapture.csv

Where “-b” stands for batch mode, “-d 2″ is a delay of 2 seconds and “-n 100″ are 100 iterations. In this specific case esxtop will log all metrics for 200 seconds. If you want to record all metrics make sure to add “-a” to your string.
Or what about directly zipping the output as well? These .csv can grow fast and by zipping it a lot of precious diskspace can be saved!

esxtop -b -a -d 2 -n 100 | gzip -9c > esxtopoutput.csv.gz

Analyzing results

You can use multiple tools to analyze the captured data.

perfmon
excel
esxplot

Let’s start with perfmon as I’ve used perfmon(part of Windows also know as “Performance Monitor”) multiple times and it’s probably the easiest as many people are already familiar with it. You can import a CSV as follows:

Run: perfmon
Right click on the graph and select “Properties”.
Select the “Source” tab.
Select the “Log files:” radio button from the “Data source” section.
Click the “Add” button.
Select the CSV file created by esxtop and click “OK”.
Click the “Apply” button.
Optionally: reduce the range of time over which the data will be displayed by using the sliders under the “Time Range” button.
Select the “Data” tab.
Remove all Counters.
Click “Add” and select appropriate counters.
Click “OK”.
Click “OK”.

The result of the above would be:

With MS Excel it is also possible to import the data as a CSV. Keep in mind though that the amount of captured data is insane so you might want to limit it by first importing it into perfmon and then select the correct timeframe and counters and export this to a CSV. When you have done so you can import the CSV as follows:

Run: excel
Click on “Data”
Click “Import External Data” and click “Import Data”
Select “Text files” as “Files of Type”
Select file and click “Open”
Make sure “Delimited” is selected and click “Next”
Deselect “Tab” and select “Comma”
Click “Next” and “Finish”

All data should be imported and can be shaped / modelled / diagrammed as needed.
Another option is to use a tool called esxplot”. You can download the latest version here.

Run: esxplot
Click File -> Import -> Dataset
Select file and click “Open”
Double click host name and click on metric

As you can clearly see in the screenshot above the legend(right of the graph) is too long. You can modify that as follows:

Click on “File” -> preferences
Select “Abbreviated legends”
Enter appropriate value

For those using a Mac, esxplot uses specific libraries which are only available on the 32Bit version of Python. In order for esxplot to function correctly set the following environment variable:

export VERSIONER_PYTHON_PREFER_32_BIT=yes

Limiting your view

In environments with a very high consolidation ratio (high number of VMs per host) it could occur that the VM you need to have performance counters for isn’t shown on your screen. This happens purely due to the fact that height of the screen is limited in what it can display. Unfortunately there is currently no command line option for esxtop to specify specific VMs that need to be displayed. However you can export the current list of worlds and import it again to limit the amount of VMs shown.

esxtop -export-entity filename

Now you should be able to edit your file and comment out specific worlds that are not needed to be displayed.

esxtop -import-entity filename

I figured that there should be a way to get the info through the command line as and this is what I came up with. Please note that needs to be replaced with the name of the virtual machine that you need the GID for.

VMWID=`vm-support -x | grep  |awk '{gsub("wid=", "");print $1}'`
VMXCARTEL=`vsish -e cat /vm/$VMWID/vmxCartelID`
vsish -e cat /sched/memClients/$VMXCARTEL/SchedGroupID

Now you can use the outcome within esxtop to limit(l) your view to that single GID. William Lam has written an article a couple of days after I added the GID section. The following is a lot simpler than what I came up with, thanks William!

VM_NAME=STA202G ;grep "${VM_NAME}" /proc/vmware/sched/drm-stats  | awk '{print $

IOBlazer - storage micro-benchmark tool - run in VM - brand new baby from VMware Labs

2011-01-14T10:09:00.001-08:00

IOBlazer is a multi-platform storage stack micro-benchmark. IOBlazer runs on Linux, Windows and OSX and it is capable of generating a highly customizable workload. Parameters like IO size and pattern, burstiness (number of outstanding IOs), burst interarrival time, read vs. write mix, buffered vs. direct IO, etc., can be configured independently. IOBlazer is also capable of playing back VSCSI traces captured using vscsiStats. The performance metrics reported are throughput (in terms of both IOPS and bytes/s) and IO latency.
IOBlazer evolved from a minimalist MS SQL Server emulator which focused solely on the IO component of said workload. The original tool had limited capabilities as it was able to generate a very specific workload based on the MS SQL Server IO model (Asynchronous, Un-buffered, Gather/Scatter). IOBlazer has now a far more generic IO model, but two limitations still remain:

The alignment of memory accesses on 4 KB boundaries (i.e., a memory page)
The alignment of disk accesses on 512 B boundaries (i.e., a disk sector).

Both limitations are required by the gather/scatter and un-buffered IO models.
A very useful new feature is the capability to playback VSCSI traces captured on VMware ESX through the vscsiStats utility. This allows IOBlazer to generate a synthetic workload absolutely identical to the disk activity of a Virtual Machine, ensuring 100% experiment repeatability.

http://labs.vmware.com/flings/ioblazer

Power CLI (Part 2) - for Power users how to work with VC over CLI

2010-10-28T09:12:00.001-07:00

http://www.virtualizationadmin.com/articles-tutorials/general-virtualization-articles/use-powercli-quick-stats-part2.html

Power CLI (Part 1) - for Power users how to work with VC over CLI

2010-10-27T07:39:00.000-07:00

www.virtualizationadmin.com - Author: Scott D. Lowe

http://www.virtualizationadmin.com/articles-tutorials/general-virtualization-articles/use-powercli-quick-stats-part1.html

Train Signal VMware vSphere Video Training

2010-10-27T07:34:00.000-07:00

Train Signal VMware vSphere Video Training

NEW! As a follow up to David Davis's best selling video series on VMware ESX Server, David has released an entirely new video training course covering VMware vSphere. The course is available from www.TrainSignal.com! This video series is over 17 hours and provides hands-on demonstration of VMware vSphere, from installation to advanced features. In the video series, David covers awesome new vSphere features like Fault Tolerance (FT), Host Profiles, Hot Add of CPU/RAM, and Data Recovery.

To watch a free demo of the course, visit the Train Signal.com VMware vSphere video training website! You'll be glad you did.

What is MultiCore Virtual CPU?

2010-08-02T03:05:00.000-07:00

VMware multicore virtual CPU support lets you control the number of cores per virtual CPU in a virtual machine. This capability lets operating systems with socket restrictions use more of the host CPU's cores, which increases overall performance.

You can configure how the virtual CPUs are assigned in terms of sockets and cores. For example, you can configure a virtual machine with four virtual CPUs in the following ways:

* Four sockets with one core per socket
* Two sockets with two cores per socket
* One socket with four cores per socket

Using multicore virtual CPUs can be useful when you run operating systems or applications that can take
advantage of only a limited number of CPU sockets. Previously, each virtual CPU was, by default, assigned
to a single-core socket, so that the virtual machine would have as many sockets as virtual CPUs.

When you configure multicore virtual CPUs for a virtual machine, CPU hot Add/remove is disabled.
For more information about multicore CPUs, see the vSphere Resource Management Guide. You can also search the VMware KNOVA database for articles about multicore CPUs.

CAUTION You must assign a value to configuration parameter keywords. If you don't assign a value, the
keyword can return a value of 0, false, or disable, which can result in a virtual machine that cannot power on.

Prerequisites

IMPORTANT To use the VMware multicore virtual CPU feature, you must be in compliance with the
requirements of the operating system EULA.

* Verify that the virtual machine is powered off.
* Verify that you have virtual machine hardware version 7 or later.
* Verify that the total number of virtual CPUs for the virtual machine divided by the number of cores per
socket is a positive integer.

VMware Log Locations & Descriptions

2010-08-02T02:36:00.000-07:00

The vCenter Server logs can be viewed from:

The vSphere Client connected to vCenter Server (click Home > Administration > System Logs)
The vSphere Client connected to VirtualCenter Server (click Administration > System Logs).

The logs are located in %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs, which translates to C:\Documents and Settings\All Users\Application Data\VMware\VirtualCenter\logs in Windows 2003 and C:\ProgramData\VMware\VMware VirtualCenter\Logs in Windows 2008.

The vCenter Server logs appear as vpxd-xx.log.

The vpxd-profiler-xx.log is used for the VPX Operational Dashboard (VOD),which can be accessed via https:///vod/index.html.

• Vmkernel – /var/log/vmkernel – records activities related to the virtual machines and ESX server.

• Vmkernel Warnings – /var/log/vmkwarning – This log is a copy of everything marked as a warning or higher severity from vmkernel log. It is much easier to look through this for warnings and errors, instead of filtering through the full information in the vmkernel logs.

• Vmkernel Summary – /var/log/vmksummary – Used to determine uptime and availability statistics for ESX Server; human-readable summary found in /var/log/vmksummary.txt

• ESX Server host agent log – /var/log/vmware/hostd.log – Contains information on the agent that manages and configures the ESX Server host and its virtual machines (Search the file date/time stamps to find the log file it is currently outputting to).

• Service Console – /var/log/messages – This log is the log from the Linux kernel (service console), which is generally only potentially useful in the case of a host hang, crash, authentication issue, or 3rd party app acting up. This log has NOTHING to do with virtual machines. The SERVICE CONSOLE (red hat kernel) has NO awareness of the VMs (worlds) running on the VMKERNEL.

Location of Logs & Brief note on that

• Web Access – /var/log/vmware/webAccess – Records information on Web-based access to ESX Server.

• Authentication log – /var/log/secure – Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.

• VirtualCenter agent – /var/log/vmware/vpx – Contains information on the agent that communicates with VirtualCenter.

• Virtual Machines – The same directory as the affected virtual machine’s configuration files; named vmware.log – Contain information when a virtual machine crashes or ends abnormally.

/var/log/vmkernel Vmkernel Records activities related to the virtual machines and ESX host

/var/log/vmkwarning Vmkernel Warnings A copy of everything marked as a warning or higher severity from vmkernel log. Easier to look through than vmkernel log

/var/log/vmksummary Vmkernel Summary Used for avaialability and uptime statistics. Human-readable summary in vmksummary.txt

/var/log/vmware/hostd.log Host Agent Log Contains information on the agent that manages and configures the ESX host and its virtual machines

/var/log/vmware/vpx VirtualCenter Agent Contains information on the agent that communicates with VirtualCenter

/var/log/messages Service Console Log from the Linux kernel. Useful for underlying Linux issues. The kernel has no awareness of VMs running on the VMkernel

/var/log/vmware/esxcfg-boot.log ESX Boot Log ESX Boot log, logs all ESX boot events

/var/log/vmware/webAccess Web Access Records information on Web-based access to ESX Server

/var/log/secure Authentication Log Contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon

/var/log/vmware/esxcfg-firewall.log ESX Firewall Log Contains all firewall rule events

/var/log/vmware/aam High Availability Log Contains information related to the High Availability (HA) service

/var/log/vmware/esxupdate.log ESX Update Log Logs all updates completed using the esxupdate tool

There’s a new Knowledgebase article on this here: VMware KB: http://kb.vmware.com/kb/1021806 Location of log files for VMware products

Troubleshooting Virtual Machine snapshot problems

2010-06-14T04:13:00.000-07:00

A nice guide on Troubleshooting VM snapshot problems --> http://is.gd/ckwDc

This troubleshooting guide explains basic concepts about Virtual Machine snapshots and different troubleshooting paths depending on the problem. This guide was designed for ESX3.5 and extra considerations have to be taken if working with ESX3.5i or ESX4(i). The formulas and most of the procedures described in this document were created by Ruben as part of a continuous troubleshooting improvement process.

Ruben is also creator of SnapVMX utility.

While troubleshooting Virtual Machine (VM) snapshot problems sometimes it is important to retrieve a lot of information in order to take the most appropriate decision in accordance with the situation. That collection and arrangement of information may take a long time especially if the VM has many snapshots.

SnapVMX was created to speed up the troubleshooting process bringing you instantaneously all the information that you need to evaluate the situation and take the correct decision, reducing the downtime to the bare minimum needed to solve the problem,

Source : Eric Blog

Also Try below VMware KB TV - Consolidating snapshots (VMware KB 1007849) - video below

Using MOB- (Managed Object Browser) --> Ops Panel for ESX

2010-06-14T03:49:00.000-07:00

I was recently reading Eric Sloof article,
Where you can go and edit (index.html - which opens as ESX server home page) customize by using MOB (Via Java Scripting).. here is the cool video by Eric.

The Operations Panel is a script tool, which runs on the client browser and extends the default ESX server web page with a short list of all available virtual machines. It gives the user the ability to perform simple power operations (start, stop, suspend, resume). Easily accessible user interface for some of the most common operations on an ESX host, available directly from the ESX home page.

Ops Panel for ESX from Eric Sloof NTPRO.NL on Vimeo.

Top 10 Free vSphere ESX Tools and Utilities

2010-06-14T03:31:00.000-07:00

Top 10 Free vSphere ESX Tools and Utilities

http://www.kendrickcoleman.com/index.php?/Tech-Blog/top-10-free-vsphere-esx-tools-and-utilities.html

64GB Addressable Memory Limit on ESX 3.x Host

2010-06-14T03:21:00.000-07:00

Prior to ESX 3.5 Update 3, the ability to address more than 64GB of memory on ESX Server 3.5 is suppressed by default. In a standard installation, a 36bit MTRR mask is forced, even though the machine may support 40bit mask values. This means that the ESX Server may see any memory above 64GB as memory that is in use. For example, if an ESX server has 256GB of RAM, the Memory Usage counter displays 192GB in use and only 64GB free. If you attempt to create a virtual machine using memory exceeding the available 64GB of memory, you see an Insufficient Memory error. This condition is documented in the following location: http://www.vmware.com/support/vi3/doc/vi3_esx35u3_rel_notes.html

For more details & Resolution refer Jason Boche Article below..

http://www.boche.net/blog/index.php/2010/05/24/esx-3-x-host-64gb-addressable-memory-limit/

What is Cloud Computing?

2010-06-11T11:17:00.001-07:00

Cloud Computing explained - Click here for More on Cloud

How to power off an unresponsive VMware ESX virtual machine

2010-05-24T09:13:00.000-07:00

Sometimes Guest OS will go stale / hang due to various reasons..
follow the below VMware KBTV Article to handle the situation..

VM World 2009 - Glimpse of EMC Ionix Data Center Insight (DCI)

2010-05-21T13:06:00.000-07:00

My take away on IONIX DCI is - Ionix DCI will understand your IT infrastructure better than you and show case and index for blink away search any equipment & it's span across your Data Center.. I would recommend you to watch the below video (Full Screen) to get to know more..

EMC Unveils Ionix Data Center Insight Laying Foundation for Unified IT Configuration Visibility Across the Data Center.

nix Data Center Insight is a key piece of EMC's newly announced Ionix IT management suite — powerful solutions for automating IT management across a unified infrastructure of storage, server, network and virtualization resources. With Ionix, customers can confidently move from physical to virtual to cloud computing — maximizing the value of their new enterprise IT architecture.

With Ionix Data Center Insight, customers can quickly and easily:

Automatically populate both EMC and third-party CMDBs with best-practices configuration items (CIs) and allows customization to define customer-specific CIs

Build a single, reconciled view of the truth about the IT environment

Visualize application and service dependencies across the data center — including applications, servers, networks, storage — both physical and virtual

Leverage the solution as a critical component of a federated and modular CMS to achieve a single, accurate and current view of the IT environment, so those responsible for configuration management can understand — across multiple domains — the resources underpinning business services

Architected to support heterogeneous, third party data sources, Ionix Data Center Insight initially supports discovery from a wide variety of EMC Ionix data sources. Ionix Data Center Insight automates the integration of these discovery sources and populates best-practices-based configuration information into the leading CMDBs such as EMC Ionix Service Manager CMDB. Additionally, Data Center Insight provides a web services visualization layer that enables cross-domain dependency mapping with search functions — spanning applications, servers, networks and storage — physical and virtual

How to connect to an ESX host using a SSH Client

2010-05-08T05:54:00.001-07:00

How to connect to an ESX host using a SSH Client

This article details steps for connecting to an ESX host using an SSH Client. This video uses the PuTTY SSH Client, but a number of different clients work similarly. The steps in the video are the same for different versions of ESX, but for the video, we used ESX 4.
For more information and context, continue reading Connecting to an ESX host using a SSH client

How to connect to an ESX host using a SSH Client

Hitachi Data Systems has released a free eBook on Storage Virtualization..

2010-04-23T08:26:00.000-07:00

Hitachi Data Systems has released a free eBook on Storage Virtualization..
Click on below link to download eBook..

http://www.hds.com/go/dummies/index.html?WT.ac=us_hp_sp1r1

VMware has released the vSphere 4.0 Hardening Guide

2010-04-23T08:05:00.000-07:00

VMware would like to announce the availability of the final release of the vSphere 4.0 Security Hardening Guide. This version incorporates the extensive feedback from the VMware community on the previous draft release, which was published in January. We would like to thank all the people who took the time to go through the draft release and provide their comments.

vSphere Hardening Guide provides guidance on how to securely deploy vSphere 4.0 in a production environment. The focus is on initial configuration of the virtualization infrastructure layer, which covers the following:

•The virtualization hosts (both ESX and ESXi)
•Configuration of the virtual machine container (NOT hardening of the guest OS or any applications running within)
•Configuration of the virtual networking infrastructure, including the management and storage networks as well as the virtual switch (but NOT security of the virtual machine’s network)
•vCenter Server, its database, and client components
•VMware Update Manager (included because the regular update and patching of the ESX/ESXi hosts and the virtual machine containers is essential to maintaining the security of the environment)

http://communities.vmware.com/docs/DOC-12306

VMware KB video showing patching of ESX Host. vSphere 4 Update Manager in action.

2010-04-15T12:12:00.000-07:00

If you never used VMware vSphere 4 update manager before, this video shows you how to get started and how to successfully apply a necessary security patches to your ESX host.
The GUI interface seems to be difficult to understand, but hopefully with this video you’ll be able to find your way around. A terms like schedule patches definitions download, baselines, remediation, ESX hosts, scan of ESX host, maintenance mode will be shown. It’s an easy step-by-step video you can follow if you’re new to VMware and managing Virtual Infrastructure with VMware vSphere 4.
Popout

This video details installing patches or updates to your VMware ESX host using VMware vCenter Update Manager. This video was created using ESX 4.0 and Update Manager 4.0. The same basic steps apply to other versions of ESX. This video also shows you how to determine how often your host checks for patches or updates, where to schedule the task, and how to run the task manually.

Source: VMware KB

How to maintain a the same MAC address all the time within your vSphere 4 installation.

2010-04-15T12:07:00.000-07:00

In VMware vSphere 4 environement, when you move your VM to another host or the VM has different path on the same host, the MAC adress of the VM is changed.
If you want to guarantee that the same MAC address is assigned to a given virtual machine every time, even if the virtual machine is moved, or if you want to guarantee a unique MAC address for each virtual machine within a networked environment you have 2 choices. You can do it on the VM configuration level or you can do it on the guest level.
On the VM level you can do it via GUI or by editing your VMX file (just make sure that you unregister the VM from vCenter first, otherwise vCenter will overwrite the value) you go to Menu VM > Edit Settings > Network Adapter and you set the radio button on Manual. Then you can assign a static MAC address there between the authorised range by VMware. The MAC address range is:
00:50:56:00:00:00-00:50:56:3F:FF:FF

If in some rare cases you need to keep the same MAC as for exaple on your Physical server (for some software license files tightened to a MAC address for exemple), and the MAC adress is outside of the range, you have the possibility to do it inside of your Windows (linux) VM.
On the Windows VM you do it in the properties of your NIC:
Start the VM and in the Windows system go to Control panel > network connections > Properties

Click the Configure button and go to Advanced Tab where on selecting Locally administered value you are able to enter your own value.

And for linux VM you can try this:
ifconfig eth0 down
ifconfig eth0 hw ether xx:xx:xx:xx:xx:xx
ifconfig eth0 up
There is reference webpage I found about MAC addresses ranges on VMware’s Website here. The page is is for VMware ESX Server 2.1, but it’s still valuable… There might be other resources on that, feel free to post a comment…

Source : www.vladan.fr Feeds

VMware Update manager does not scan hosts??? Disable Internet Explorer Enhanced Security Configuration on Win2k8

2010-04-15T11:42:00.000-07:00

VMware update manager hated me and wouldn’t scan any of my hosts, one of the many fixes I was trying was making sure that my VMware Update Manager server could reach the patch stores to download updates and do patch checks and so decided to make sure of this I wanted to remove the painfully irritating Internet Explorer Enhanced Security Configuration. Now before I get loads of comments saying this is dangerous etc etc and that it’s a security risk,I know it is but for testing purposes I wanted to do it so I did. Now the point of this blog is that to disable it isn’t as easy as it used to be in 2003 as you used to just:

Go to Control Panel
Add/Remove Programs,
Add/Remove Windows Components on the left hand side

Untick it from the drop down list

But for Windows Server 2008 I couldn’t find it anywhere in the features drop down lists which has replaced the Windows Components in Server 2008. Fortunately while on another Server 2008 server I noticed the place to disable it. In the Server Manager root page under the Security Information section is Configure IE ESC

All you need to do is click on this link and disable it for either Administrators and/or for Users. I only needed for Administrators so this is what I selected for the testing.

Source : EMC Feeds

Rename VMDK files – else you may have to experience Backup Problems

2010-04-12T13:48:00.001-07:00

What if you end up with the same vmdk file name on different datastores?

A simple case scenario that can happens:

A single VM has several hard disk attached to it. The hard disk VMDK files are located on different datastores. When creating a new hard disk which location is on different datastore to a VM, the name given to this VMDK is the same as the first VMDK.

For exemple, you VM has a disk called TS01.vmdk which is located on DATASTORE1, and you create a second disk located on DATASTORE2. This disk takes the same name as the first one…. This is not a problem for VMware, but it is for third party backp products like Symantec Backup Exec 2010. In fact what happens to the backup job wanting to backup a VM like this? The VMDK is marked as a corrupted and the job fails, so it's impossible to restore either.

What's the solution? VMware KB had give me an simple answer what to do in the case like this. You'll have to use vmkfstools via command line to do the job. Don't worry, it's not difficult.

How to:

01. First, make sure that the VM has NO Snapshots. Then power down your VM go to menu Edit Settings.

On the next screen you just go to the hard disk you want to rename. Select the hard disk and click the remove button. The screen change and you have the possibility to check a radio button to delete the disk from datastore. Don't check this radio button. Otherwise your disk gets deleted..

02. Validate your choice on clicking OK.

03. Then you must connect to the console of the ESX server to which the datastore2 is attached to. Use putty or go directly to the server room.

04. Via putty navigate to the folder where the hard disk is located and execute following command:
vmkfstools -E <old_name> <new_name>
So in our case the command looks like this (I'm renaming from ts01.vmdk to ts01_2.vmdk):

vmkfstools -E ts01.vmdk ts01_2.vmdk

05. Then go to your VI client and add the renamed disk back to the virtual machine. In the Inventory browser by right clicking the datastore and where the VM is stored and select "Browse Datastore", select the folder of the VM than right click the vmx file and select "Add to Inventory".

This works fine for ESX 4, but to be able to use it with ESXi 4 you have several choices:

01. You can use the unsupported (Tech Support Mode) to get to the console of ESXi 4 since the CLI tools won't work for writing data in the free version of ESXi 4.

02. If your ESXi 4 is licensed with paid VMware License you can use PowerCLI or VMA to do the job.

Source : www.vladan.fr