If you open a datastore, the Provisioned amount is displayed:

In my (humble) opinion, besides knowing how much Free space is left on the volume, Provisioned is important too so you know just how far in the hole you’re digging yourself by over-provisioning datastores, and it would be nice to see this in the list view of all Datastores as a way of comparison.

Since we don’t have that column available to us (VMware, pretty please?), a bit of Powershell can give us what we need.

connect-viserver your_vcenter_server
$datastores = get-datastore | where-object {$_.name -match "Servers"} | get-view
$datastores | select -expandproperty summary | select name, @{N="Capacity (GB)"; E={[math]::round($_.Capacity/1GB,2)}}, @{N="FreeSpace (GB)"; E={[math]::round($_.FreeSpace/1GB,2)}}, @{N="Provisioned (GB)"; E={[math]::round(($_.Capacity - $_.FreeSpace + $_.Uncommitted)/1GB,2) }}| sort -Property Name

In my example above, I am using where-object to filter only for datastores that have “Servers” in the name. Remove it or customize it as needed. The snippet above produces the following output:

.. you could even append “Export-CSV c:\path\to\output.csv -NoTypeInformation” to the end to write it to a CSV file, useful for Excel or other things.

Based on the following pages:

• http://communities.vmware.com/thread/260454
• http://www.daniweb.com/software-development/vbnet/threads/84742

The first step was to grow each VM’s VMDK from 10GB to 15GB. This was easily accomplished with Powershell:

Get-Folder Desktops | Get-VM | Get-HardDisk | Where {$_.CapacityKB -eq 10485760} | Set-HardDisk -CapacityKB 15728640 -Confirm:$false

Simply put, this one-liner acts against all VMs in the “Desktops” folder, and if it currently has a 10GB drive (10,485,760 KB), it uses the Set-HardDisk command to expand it to 15GB (15,728,640 KB). This worked incredibly well. I did run in to one or two VMs that had trouble expanding because they could not be “stunned” while currently in the process of VMotioning to another host. They weren’t actually migrating, so I quickly powercycled them and ran the one-liner again, with success.

The VMDKs are thin-provisioned so no additional space was consumed on the datastores, but now the disks can grow beyond 10GB, up to 15GB.

The next step which fortunately is “not my problem” is to expand the C: partition from 10GB to 15GB. I advised our desktop manager to try Dell’s “extpart” utility which can expand the C: partition while the VM is booted. I have used it successfully with Windows Server 2003, but not tried it with XP. I believe it will work though. Our desktop manager will probably make use of Altiris and some batch scripts to run extpart on all the desktops.

Using iostat -x 1 I could see that the utilization of the disks was always around 100% and the average wait could explode up to 3000+msec which is not that great. I assumed it was syslog causing the problem but I’m not quite sure. Either way, I don’t need all the verbose data from the ESXi hosts in syslog and clogging up the I/O subsystem so here is a way to change verbose logging to something else (none, error, warning, information) (based on http://communities.vmware.com/thread/285254)

Here is what I did:

1. Enable Remote Tech Support Mode (SSH) on all hosts seen by the vCenter server “serverName” using Powershell:

   connect-viserver serverName
   get-vmhost | foreach-object { get-vmhostservice -vmhost $_ | where {$_.Key -eq 'TSM-SSH'} | start-vmhostservice -confirm:$false }
   

2. Create a script that SSH’s to a given server to run the commands:

   #!/bin/bash
   
   ssh -l root $1 "mv /etc/vmware/hostd/config.xml /etc/vmware/hostd/config.xml.orig && sed -e 's/<level>verbose<\/level>/<level>warning<\/level>/' /etc/vmware/hostd/config.xml.orig > /etc/vmware/hostd/config.xml && mv /etc/opt/vmware/vpxa/vpxa.cfg /etc/opt/vmware/vpxa/vpxa.cfg.orig && sed -e 's/<level>verbose<\/level>/<level>warning<\/level>/' /etc/opt/vmware/vpxa/vpxa.cfg.orig > /etc/opt/vmware/vpxa/vpxa.cfg && services.sh restart hostd && /sbin/auto-backup.sh"
   

   Using sed, it updates /etc/vmware/hostd/config.xml and /etc/opt/vmware/vpxa/vxpa.cfg to replace the default “verbose” logging levels with “warning” (which could be any of the levels mentioned earlier)

3. Run this script on a separate linux host using the ESXi hostname as the first and only argument and then accept the root key and provide the root password. The script will update the files, restart all the services, and then backup the changes so they are saved.
4. Once all the hosts have been changed, run this powershell code to stop the SSH service:

   connect-viserver serverName
   get-vmhost | foreach-object { get-vmhostservice -vmhost $_ | where {$_.Key -eq 'TSM-SSH'} | stop-vmhostservice -confirm:$false }
   

After some poking around, the vCenter log (C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs\vpxd-*.log) contained these telling lines:

[2010-09-15 09:14:55.354 03728 info 'App'] [Vpxd::ServerApp::Init:759] Calling: VpxdVdb::Init(Vdb::GetInstance(), false, false)
[2010-09-15 09:14:55.354 03728 error 'App'] ODBC error: (IM002) – [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
[2010-09-15 09:14:55.354 03728 error 'App'] Error getting configuration info from the database
[2010-09-15 09:14:55.354 03728 error 'App'] [Vpxd::ServerApp::Init] Init failed: VpxdVdb::Init(Vdb::GetInstance(), false, false)
[2010-09-15 09:14:55.354 03728 error 'App'] Failed to intialize VMware VirtualCenter. Shutting down…
[2010-09-15 09:14:55.354 03728 info 'App'] Forcing shutdown of VMware VirtualCenter now

Poking around the VMware Knowledge Base, I found the article 1003928: Troubleshooting the database data source used by VirtualCenter Server which mentions the registry key “HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB”.  The value for the key “1″ was blank, meaning no DSN was specified. I changed the value to our System DSN “vcenter”.

Next I opened up odbcad32 to actually look at the “vcenter” System DSN but could not open it because the SQL Native Client 10.0 driver was missing! I downloaded it from the Microsoft SQL Server 2008 Feature Pack, August 2008 and installed it and then was able to open the DSN. I found that the “Server” field was simply set to “localhost” and when I tried to test the connection, it timed out. I looked at the old ViewComposer DSN and it had a server and instance specified – “VIEW45-VC\SQLEXP_VIM”. I put this in the “vcenter” DSN and then with integrated Windows authentication, I was able to change the default database to “VIM_VCDB” and test the connection and it succeeded. I saved the DSN and attempted to start the vCenter service. This time it started and stayed running and the logs looked good.

VMware KB wins again!

All our servers including our VMs are backed up using Tivoli Storage Manager (TSM) which does an incremental backup every night to a disk pool and then is offloaded to a tape pool during the day.

We can use TSM to restore individual files inside a guest VM but we can’t easily (or at least have never tried) to restore an entire VM if it were to be lost.

A few months ago during a fresh installation of ESXi 4.0 (prior to the release of 4.1), I accidentally selected the wrong SCSI disk to install ESXi on and ended up nuking an entire VMFS datastore. Luckily it was only hosting about 25 virtual desktops and those were easily recreated. But it got me thinking about what would happen if we accidentally lost an entire VMFS datastore hosting server VMs, or even just lost one server VM for any reason.

Therefore I decided to look in to backing up the entire VM and not just the files inside it. I asked around and unequivocally heard that the best solution was the Veeam Backup & Replication suite. My only experience so far with Veeam has been their FastSCP product and that was a nice piece of software, though I did not need to use it that much. I don’t think I even have it installed.

I plan to ask Veeam for a 16-socket trial license so I can get an idea of what backing up my entire environment will be like. I will use changed-block tracking to enable quick incremental backups after the large initial one.

My backup target will be an old EVA8000 array that we purchased and has about 15TB of FC and FATA disk. I only need about 6TB so I plan to carve out a large Vdisk and present it to a Windows Server 2008 Standard R2 64-bit HP blade server where I will install Veeam Backup & Replication. I will also assign the existing VMFS datastores to the Veeam Backup & Replication server in order to do LAN-free (SAN only) backups going from the VMFS datastores, through the backup server, to the destination Vdisk.

I am very excited about this and hope the license-purchase quote that I also asked for will not be prohibitively expensive. There are several local resellers as well as CDW that I could purchase it from, so I will have to shop around and see who can offer the best deal.

I think it’s great what Veeam has put together and I look forward to checking it out.

Note: most things to take effect after the update, such as newer services, and drivers that can be safely unloaded/loaded without requiring a reboot.

Note: the NIC driver may be updated causing a brief drop in traffic.

To upgrade the VMware Tools on a Windows guest without rebooting, use this bit of PowerShell code with the vSphere PowerCLI.

First run this to set things up (connects to the specific vCenter server using your current Windows credentials and sets the installation parameters to pass to the installer):

Connect-VIServer
$insParm = '/S /v"/qn REBOOT=ReallySuppress"'

Then run this for each VM, changing with the name of the VM to update tools on, without rebooting:

$vmView = Get-VM | Get-View; $vmView.UpgradeTools_Task($insParm)

Example:

$vmView = Get-VM Babyruth | Get-View; $vmView.UpgradeTools_Task($insParm)

A task will start in vCenter and the tools should be updated on the VM. It takes about 2-3 minutes. If it seems to be taking longer than that to complete, check the VM console to make sure no errors have popped up that require attention, and also check if setup.exe and one or more msiexec.exe processes are running.

The task should finish successfully and the VMware Tools Status for the VM should change to “OK”

There is plenty of discussion online about various ways to accomplish this (“How to install VMware tools without a reboot?“), but what I described above is what works for me. It is definitely possible to write a PowerShell script to traverse through a folder or cluster and upgrade Tools on all the VMs contained within, but I prefer to take it one at a time so the process doesn’t get out of hand. Tools upgardes are generally painless but sometimes there are issues.

Every new release of vCenter/ESX is like Christmas morning as there are plenty of new features to be discovered and bugs fixed that streamline previous operations.

I was dismayed at first to read that vCenter required a 64-bit OS as not too long ago I had built up a nice pair of vCenter 4.0 servers on Windows Server 2003 Standard (32-bit) and liked the way they were working. I was running SRM 4.0 along with Update Manager and everything was working great.

I knew that with the 64-bit requirement, I might as well go to Windows Server 2008 R2 64-bit for my two vCenter servers, but I had some hurdles to jump over. Specifically, we aren’t really running any Server 2008 machines in-house, save for those that we have to experiment and kick the tires with. So I decided to build a Server 2008 template and then deploy a pair of new vCenter servers from that.

The process of building a new template VM has been documented widely across the virtualization blogosphere, but I shall throw my two cents in the ring with a post describing my own win2k8 template. But as usual, that is for a later time.

The other hurdle to leap over, in addition to getting Server 2008 set up for production use, was the fact that I was going to have to relocate my vCenter servers to brand new machines. I tried this once awhile ago with VirtualCenter 2.5 and the results were diastrous. So I was not encouraged with the fact that I was going to have to try to move again.

Luckily, the VMware team understood that a lot of people would be in the same boat due to their 64-bit-only requirement so they put together a set of python scripts to aid in the moving of vCenter to a new host.

The guides that I referred to during this upgrade are:

• vSphere
  • vSphere 4.1 Release Notes
  • vSphere Compatibility Matrix
  • vSphere Upgrade Guide
• SRM
  • SRM Admin
  • SRM 4.1 release notes

I started out by stopping my vCenter services, backing up the Microsoft SQL 2008 Standard databases for VC and SRM, and then detaching and copying them to the new vCenter servers where I already had Microsoft SQL 2008 Standard  64-bit installed. I attached the DBs and made sure my SQL “vpxuser” account was set to owner.

Then I created a 64-bit System DSN for the vCenter database by running “odbcad32″ at the Run command. I also created a 32-bit System DSN for SRM by running c:\windows\sysWOW64\odbcad32 at the Run command (thanks to boche.net for that tip)

A side note: my VUM database is part of my vCenter database, so I didn’t have to create a DSN for that – it will just piggyback on top of the vCenter DSN as it did on my Server 2003 vCenter servers.

After my DSNs were created, I followed the instructions in the Upgrade Guide to copy the datamigration folder off the vCenter 4.1 ISO on to my 32-bit server and ran the “backup.bat” to backup my vCenter certificates and other configuration that lives outside the database. It also backed up my VUM and Orchestrator configuration (even though I don’t use Orchestrator).

I copied the resulting datamigration folder to the new host. It now has “data” and “log” folders that hold the backed up configuration. I ran “install.bat” on the new host and pointed it to my vCenter 4.1 media. One thing that got me at first was it prompted for the Update Manager media path and I gave it what I thought was the path – the Update Manager folder inside the vCenter installer, but that was incorrect and I had to end up giving it the same path that I gave for vCenter. It would be nice if it would just check there first for the Update Manager installer files and then only prompt if it can’t find them.

Then the datamigration install.bat file launched the vCenter installer and I stepped through that, changing the install path to the E:\ drive that I had set up for it and pointing it to the DSN that I had configured. I also allowed it to automatically update the ESXi/vCenter host agents. Soon the installer finished and then also launched the VUM one. I stepped through that as well, again changing the path to E: and pointing it to the DSN that it needed.

The VUM install was also successful and then datamigration install.bat utility ended. I didn’t even get to see any of the host agents being updated because when I got the client installed and logged in, everything looked great. I had a few servers that needed to be reconfigured for HA but other than that, the install went great.

I repeated this process on my 2nd vCenter host and it went well too.

I did have to break up my linked-mode config as it sort of stopped working – I could log in to one server and see both, but if I logged in to the other one, it would only show that one and an authentication failed message for the other one. I had to “force” it to isolate itself from linked-mode, but I think that was because the vCenter service hadn’t stopped in time. But 2nd time was a charm and after joining them back together, all was well.

I then installed SRM 4.1, paying careful attention to the release notes (linked earlier) describing how to relocate SRM to a new server with a new name. I installed the HP EVA Storage adapter and visited my SRM config to make sure it was OK. It was but I did have to reconfigure the EVA credentials. I don’t know if they were lost or what but it was no big deal. I ran a test recovery plan and that worked perfectly.

I then reconfigured VUM to not download VM patches for Windows or Linux and none for ESX3. I also added the ESX4.0 to 4.1 upgrade zip and created a baseline for upgrading all my ESXi 4.0 hosts to 4.1.

Finally I remembered to copy the sysprep files off my 2k3 vcenter servers so that we can continue deploying/customizing 2k3 and XP server VMs.

At this point, everything looks great and I’m heading home. ESXi upgrade next week!

I had never actually computed these since we haven’t had any storage issues lately and the storage that we’re using is more than adequate (actually overkill).

Anyway, since vSphere Client doesn’t exactly have a line on the VM Resources area indicating current IOPS (would be nice), we have to figure them out. If we look at the Performance tab for a VM, we can look at the disk rate usage but at least for me, choosing any of the “Summation” counters such as “Disk Read Requests”, “Disk Write Requests”, or “Disk Commands Issued” just resulted in a performance graph that never actually loaded. But that’s okay because we don’t want to take one VM’s storage profile and assume all the other VMs are behaving the same way. Maybe the user assigned to this VM is a power user who is always installing apps, or maybe they never log in to their desktop at all.

Therefore we have to go up to the host level. I have between 9 and 10 hosts across my VDI clusters so I picked one of my clusters and went to the Hosts tab and sorted by % Memory to find the host that is using the most memory. This  indicates to me that it is hosting the most virtual machines in the cluster. All our VDI VMs have the same amount of memory assigned (512mb) so I knew this would be the easiest way to find out which host in the cluster had the most VMs (would be nice to have a column in the Hosts tab that indicated how many VMs are on the host!)

Now that I know which host has the most desktops, I can look at the host’s disk performance and divide that by the number of running VMs. With a lot of VMs on this host, I can be confident that the numbers I calculate will be a truer average than choosing a lightly loaded host.

So on my heavily loaded host, I went to the Performance tab and clicked on “Chart Options”. In there I chose Disk real-time stats  (since the counters I’m looking for are only recorded in real-time and not kept longer than an hour). I chose all my naa.* objects which are my datastore LUNs (one LUN per datastore, and I ignored the mpx.* object which is the local SCSI controller and the host object itself).

For Counters, I chose “Disk Commands Issued”. When the graph rendered, I saved it as an Excel spreadsheet. I repeated this for “Disk Read Requests” and then also “Disk Write Requests”. I could have had all 3 counters in one graph/spreadsheet but I wanted to make things easy on myself when calculating the averages.

I should point out that it would be a good idea to collect this data during an hour where normal activity is taking place – not first thing in the morning when people are logging in and firing up various applications, and not in the evening when fewer people are logged in. Late in the morning (but not too close to lunch) would be a good time. But to play devil’s advocate, it may be a good idea to do this during the busiest time for the environment because the future storage needs to be able to handle that load. Look at the host graph for Disk Usage over a week and try to spot a trend for the busiest day/time. Possibly export to Excel to get a more granular look at the data.

Now in the spreadsheets, each row is a collection of performance data. The first column is a timestamp for the collection date/time which are 20 seconds apart (the collection interval). The next columns are for the counter in question for each LUN. So to get the average number of commands over the past hour for the LUN, I used AVERAGE for each column. This gave me the average number of commands issued over the last hour for each collection period. Dividing this number by 20 resulted in the average commands per second for each LUN. Adding up the average commands per second for each LUN gives me the average commands per second for the server, over the past hour.

With the average commands per second for the server over the hour, I then divide by the number of VMs on the server, 43, and I get the following results:

• average read/write requests per vm per second: 6.69
• average read requests per vm per second: 5.30
• average write requests per vm per second: 1.39

Taking the individual read or write request numbers and dividing by read/write sum, I find the follow percentages:

• read: 79.23%
• write: 20.76%

This ratio is pretty much identical to the common assumption that Windows XP virtual desktops have an 80:20 read:write ratio.

Armed with this information, I’m sure the vendor can put together a storage solution which will adequately host our new virtual desktop infrastructure.

It would be nice if VMware could add these types of calculations in somewhere so that this manual math is not needed, but I’m glad I could at least export the chart data and manipulate it in Excel.

The problem was whenever viewing a server profile or a defined network, the Flex UI would just sit there with a box entitled “Loading” that said “Please wait…” with a neverending progress bar. One of those progress bars that just is infinite and pretty worthless. If you’re going to include a progress bar in your display, it should actually indicate the progress of the current action and stop at some point. But this one just sat there all animated and infinite.

I power cycled one and then both VC Ethernet modules. I tried restoring a backup of the config that I just made. I power cycled the entire enclosure. Nothing would solve this problem for me. Right before I was about to call HP, I visited the HP 1/10GB Virtual Connect Ethernet Module download page and was pleasantly surprised (ecstatic really) to see that version 3.01 had been released (with no sign of 3.00 even having ever existed) and one of the purported fixes was for the very issue I was experiencing!

I quickly downloaded the firmware and applied it to the two enclosures where I had previously applied the 3.00 version. Problem solved!

Moral of the story: check for newer firmware before trying anything else. My problem was that 3.00 had come out not too long ago and I knew that VirtualConnect firmware is not updated too often so I subconsciously assumed there wouldn’t be anything new.

But when connecting with the client to the 2nd vCenter server, it would show both vCenter servers and their hosts/VMs.

So obviously something is up with the first vCenter server. I ran the Linked Mode setup again and it acted like it wasn’t linked, which was how it appeared. So I went through the prompts to set up the linked mode and when it came down to actually setting it up, it would fail. It did this twice. Looking at the log that it mentioned, it was complaining that the vCenter Server service was not stopped. It had successfully stopped the Web Access service, but not vCenter Server.

I loaded up services.msc and went to stop the vCenter server service but was prompted that our Site Recovery Manager service would also need to be stopped. After agreeing and stopping both services, the Linked Mode configuration completed successfully and started the vCenter Server and Web Access services. It did not however start the Site Recovery Server service.

At this point it looks like the Linked Mode configurator needs to understand about other services that depend on vCenter Server and track them to stop/start before/after the linked mode is configured.

I don’t know why the vCenter server became unlinked from the 2nd one, but it works good now!