I have had an issue with my vSphere Web Client when powering on any VM:

The “Power on virtual machine” operation failed for the entity with the following error message.

Permission to perform this operation was denied

In the logs, I found the following:

inherited from com.vmware.vim.binding.vim.fault.NoPermission: Permission to perform this operation was denied.
[ERROR] http-9443-5 com.vmware.vsphere.client.vm.powerops.impl.VmPowerOpManager com.vmware.vim.binding.vim.fault.NoPermission:
object = com.vmware.vim.binding.vmodl.ManagedObjectReference@83c3e21c,
privilegeId = System.Read
inherited from com.vmware.vim.binding.vmodl.fault.SecurityError:
inherited from com.vmware.vim.binding.vim.fault.NoPermission: Permission to perform this operation was denied.

The VM actually does power on, which is visible in the vSphere Web Client after a GUI refresh.

After some Googling and reaching out on Twitter, I learned that VMware has a Knowledgebase article on this: KB2018590. Thanks, Jehad Affoneh! Jehad is an UI Engineer Platform User Interface at VMware. He’s currently working on the vSphere Web Client. Check out his blog: mynameisjehad.com.

One last tidbit I want to share on the subject, from the KB-article:

VMware is aware of this issue and is working to resolve this in a future release.

There’s currently no solution and/or workaround available for this bug, which means we need to muster up the patience for this ‘future release’… Hold on to your hats, folks.

I have implemented a vSphere Web Client solution recently for my own (small and non-profit) hosting environment.

I wanted to give users access to their vSphere VMs without requiring the full-blown vSphere Client. I’ve been trying to get WSX working as a stand-alone Web App (using William’s fantastic how-to), but it seems WSX still contains some bugs on vCenter communication. For now, I chose to use the vSphere Web Client for my users.

Design

Although the physical servers are situated in the datacenter (primary site), the vCenter Server is placed on a machine in my home network (secondary site), because the physical hardware is rather restricted (only two boxes with 16GB RAM each). I run Vyatta site-to-site OpenVPN between sites.

For argument’s sake, I want the Web Client to be load balanced and highly available. This way, I get to play around with KEMP’s technology and create a geo-redundant load balanced vSphere Web Client. I understand that there are lots of SPOFs and supportability issues in this scenario, but that’s beyond the scope of this discussion.
After evaluating both pfSense (Viktor van den Berg wrote a nice how-to here) and the KEMP Virtual LoadMaster (nice tutorial by Eric Sloof here), I decided to use the KEMP VLM for (geo-)redundancy and load balancing.

Implementation

vSphere Web Client

First, I installed two Windows boxes with the vSphere Web Client. These machines (vwc01 and vwc02) are placed on two separate sites but are both connected to a single vCenter instance, vc01.

vCenter Server and Database

Again, I am creating a load balanced vSphere Web Client for fun, so I realize I have a couple of SPOFs left. Primarily, the vCenter Server and Database are a SPOF, especially since it runs on a separate site. This makes the site-to-site OpenVPN a major SPOF as well. If the link between sites fails, all components are quite useless. The LoadMaster and vSphere Web Client in the datacenter can’t connect to the vCenter Server, and while the ones in my home network will still function, there wouldn’t be any ESXi-hosts for vCenter to connect to. But again, I do realize this and really just want to create some KEMP setups for me to play around with.

KEMP Virtual LoadMaster HA

I configured the KEMP VMs with the following interface parameters:

I’m using 10.10.30.0/24 for heartbeat only, it is a separated and isolated network. 10.10.20.0/24 is the network where all services reside. For simplicity, I used a stretched VLAN between sites. I could’ve created two separate networks, but as this is just an exercise in ‘getting it to work’ more than anything else.

And HA parameters:

This results in a fully functioning HA pair:

 on vlm01

 on vlm02

KEMP Virtual LoadMaster Virtual Services

Next, I created a ‘virtual service’ to enable vSphere Web Client load balancing.

• Virtual Address: 10.10.20.20
• Port: 9443
• Name: vSphere Web Client
• Protocol: tcp

I then modified the service settings:

• Service Type: HTTP/HTTPS
• Persistence Mode: Source IP Address
• Persistence Timeout: 1 Day
• Scheduling Method: weighted response time
• SSL Acceleration: Enabled (incl. Reencrypt)

Finally, I added the ‘Real Servers’ and their ‘Check Parameters’:

• Check HTTPS Protocol on Port 9443
• Check URL: /vsphere-client
• Real Server Address: 10.10.20.21 and 10.10.20.22, the IP-addresses for either vSphere Web Client VM.
• Real Server Port: 9443

Client Access

I registered the virtual service IP address (10.10.20.20) in my local DNS (client.virtuallifestyle.nl) in order for my users to easily connect to the vSphere Web Client.

To check if the KEMP VLM really is load balancing:

Breaking stuff

Now, this blogpost wouldn’t be as fun if I just left it at this. I obviously need to break stuff:
Here’s what happens when I disable one of the vSphere Web Client nodes:

Likewise, here’s what happens when I disable one of the KEMP Virtual LoadMaster nodes:

Guess what happens if the primary site fails completely:

Lastly, here’s what happens when I disable one of the vSphere Web Client nodes and one of the KEMP Virtual LoadMaster nodes across sites (i.e. vWC01 and VLM02):

Concluding

The point of this exercise was to find out if I could create a geo-redundant configuration of a stateless application like the vSphere Web Client using KEMP’s Virtual LoadMaster. Even though this implementation does have a bunch of rough edges (razor-sharp ones at that), but it definitely works. The VLM’s installation and configuration, even in a CARP HA scenario is a laugh and can be completed in under 5 minutes while delivering a powerful set of load balancing and redundancy functionality.

Dave Stork has published a couple of posts (‘Differences in Exchange Load Balancing recommendations by Microsoft and vendors‘ and ‘Exchange, Load balancers and recommendations‘) on using LoadMaster in Microsoft Exchange environments. Please read up on his experiences, tips and tricks, it’s well worth the read!

I got an e-mail from VMware Press notifying me of this sweepstake:

VMware Press, the official publisher of VMware books and training materials, has launched a 60 day Facebook sweepstakes beginning May 1 and running through June 30^th. Prize offerings include a $100 Amazon gift card and three VMware Press books of the winner’s choice; nine second prize winners will win an eBook of their choice. Good luck – enter now!

Seems like a good way to fill up your (digital) bookshelf!

I finally managed to find a reader interested in ‘USENIX Short Topics in System Administration Series #24, ‘Cloud Computing with VMware vCloud Director‘:

Happy reading, @eroozen

I am deeply honoured and very proud to have been awarded the vExpert Award for 2012!

This year, there are a staggering number of new vExperts! Congratulations to all 150 of you:

• Adam Sekora
• Akio Shimizu
• Alastair Cooke
• Alex Musicante
• Alexander Walden
• Alfred Hollingsworth
• Andi Lesslhumer
• Andreas Neufert
• Andreas Peetz
• Andrew Miller
• Angelo Luciani
• Anthony spiteri
• Arjan Timmerman
• Artur Krzywdzinski
• Askar Kopbayev
• Brandon Hahn
• Brandon Riley
• Brandon Willmott
• Brian Tobia
• Brian White
• Bryan Kuhn
• Bryan O’Connor
• Chris Kranz
• Chris Sterner
• Christopher Wahl
• Craig Waters
• Craig Waters
• Daichi Ogawa
• Dan Brinkmann
• Darryl Miles
• Dave Henry
• Denny Cherry
• Derek Seaman
• Don Jones
• Doug Baer
• Earl Gay
• Edward D Czerwin
• Edward J Grigson
• Enrico Signoretti
• Eric Monjoin
• Erik Bussink
• ERKAL ASLANKARA
• Frederic Martin
• Gabriel Chapman
• george L. Reed II
• Gunnar Berger
• Gurusimran Khalsa
• Hany Michael
• Hersey Cartwright
• Howard Marks
• Ichiro Shibutani
• Itzik Reich
• Jason Langer
• Javier Larrea
• Jeff Drury
• Jeremy Bowman
• John Walsh, Jr
• Jose Maria Gris
• Josh Coen
• Joshua Andrews
• Joshua Atwell
• Joshua Williams
• Karol Boguniewicz
• Keith Norbie
• Kenneth Hess
• Kent Altena
• Kiran Ranabhor
• Kirill Menshikov
• Kyle Ruddy
• Larus Hjartarson
• Lauren Malhoit
• Lee Caswell
• Link Alander
• Lorenzo Galelli
• Lucas Carlson
• Luigi Danakos
• Marc Huppert
• Marcel Brunner
• Marcelo Tavares Soares
• Mark Iveli
• Martijn Baecke
• Masaomi Kudo
• Massimiliano Moschini
• Matt Boren
• Matt Davis
• Matt Jones
• Matthew James Popovich
• Matthew Vogt
• Michael Ellis
• Michael Preston
• Michael Requeny
• Michael Webster
• Miho Yamamoto
• Mike Schubert
• Nicholas Marshall
• Niels Engelen
• Niklas Akerlund
• Pas Apicella
• Paul Gifford
• Paul McSharry
• Paul O’Doherty
• Paul Schaapman
• Paul Valentino
• Paul Wood
• Paula Neil
• Peter Del Rey
• Peter van den Bosch
• Philip Ditzel
• Piotr Masztafiak
• Randy Stanley
• Raymond van’t Hag
• Remon Lam
• Rick Schlander
• Rickard Nobel
• Robert van den Nieuwendijk
• Rodrigo Gazzaneo
• Ron Singler
• Ryan Birk
• Samuele Cerutti
• Scott David Lowe
• Scott Sauer
• Sean Duffy
• Shigeru Hihara
• Simon Davies
• Stephane Grimbuhler
• Steve Jin
• Stuart McHugh
• Tadao Tsuchimura
• Takayuki Tanaka
• Tan Wee Kiong
• Thomas Brent Quick
• Tim Antonowicz
• Todd Knapp
• Tomoya Goto
• Toni Maatta
• Tony Lux
• TORU IIJIMA
• Toru Miyashita
• Tsuneyuki Mitsugi
• Tyler Britten
• Vincent Wu
• Vladimir Eskin
• Vladimir Nazarov
• Warren Olivier
• Wei-Ren Wang
• William Huber
• Yasumasa Mita
• Yoshimasa Yamamoto
• Yuri Semenikhin

Check out the official announcement and the complete list here: VMTN Blog

Just a little post to let you know that I just noticed that VMware’s released a ‘My VMware’ app in the iOS App Store. Go get it in the App Store.

It looks informationsavvy, and that’s just what I want; more details about my account, support contracts and licenses.

So a while ago I promised to give away some books to my readers to enjoy instead of having the books collect dust on my analog book shelf.

I found new homes for the following books:

• ‘Administering VMware Site Recovery Manager 5.0‘ by Mike Laverick
• ‘Automating vSphere with VMware vCenter Orchestrator‘ by Cody Bunch
• ‘vSphere 5 Clustering Technical Deepdive‘ by Duncan Epping and Frank Denneman
• ‘VMware vSphere Design‘ by Forbes Guthrie, Scott Lowe and Maish Saidel-Keesing

• USENIX Short Topics in System Administration Series #24, ‘Cloud Computing with VMware vCloud Director‘ by John Y. Arrasjid, Ben Lin, Raman Veeramraju, Steve Kaplan, Duncan Epping, and Michael Haines

And the winners are…

All winners have been notified and have received their books already!

• Adam Robinson from the United States
• Kees Koekkoek from the Netherlands
• Sjors Robroek from the Netherlands
• Igor Wormsbecher from the Netherlands

Igor was so kind to e-mail me this picture:

Sjors left a pic in the comments below: “directly on top of the “to read” pile, hopefully in preparation of vcap5”

I hope the others will e-mail (or post a pic in the comments), too!

I’d like to wish these guys a lot of fun reading these books and learning from them, just as I did.

Thanks!

I also want to thank Mike Laverick, Cody Bunch, Duncan Epping, Frank Denneman, Forbes Guthrie, Scott Lowe and Maish Saidel-Keesing for writing these awesome books. Thanks guys!

I was cleaning out my bookshelf the other day, and noticed I have a few hard copies of books for which I also own the digital version. I usually read books on my iPad, so it really is a shame that all these nice and shiny books remain unused.

I figured, why not just give them to you?

Prizes

• ‘Administering VMware Site Recovery Manager 5.0‘ by Mike Laverick
• ‘Automating vSphere with VMware vCenter Orchestrator‘ by Cody Bunch
• ‘vSphere 5 Clustering Technical Deepdive‘ by Duncan Epping and Frank Denneman
• ‘VMware vSphere Design‘ by Forbes Guthrie, Scott Lowe and Maish Saidel-Keesing
• USENIX Short Topics in System Administration Series #24, ‘Cloud Computing with VMware vCloud Director‘ by John Y. Arrasjid, Ben Lin, Raman Veeramraju, Steve Kaplan, Duncan Epping, and Michael Haines

#Winning!

The giveaway is really simple. Just subscribe to my RSS-feed (use this button: ) and leave a comment below. Make sure the comment contains the name of the book you’d like to win and your name. That’s it. Nothing else. You could buy my a ‘new iPad’ or the new Apple TV, obviously, but that won’t increase your chances of winning.

Everyone can enter the giveaway (starting now!), just make sure you enter a valid e-mail address in the comments field. I will of course never sell, give away or do anything else with your contact details other than publish the names of those who won and contact you to get the details needed to ship the books. I solemnly promise to shift-delete your records after you’ve received your prize. Scouts’ Honor.

The contest will close March 20th, 20:00 CET (GMT+1). I will contact the winners at that time; Winners must respond by March 22th, 20:00 CET (GMT+1) or the prize is forfeited.

The winners will be chosen at random using the random.org ‘True Random Number Service‘. The numbers entered are the internal comment identifiers automatically assigned to your comment by the WordPress framework.

The rest of the week I’ll be attending the ‘VS-5.0-ACMT‘ training at XTG in Gouda, The Netherlands. XTG has gratuitously offered me a spot on the training! I’m looking forward to four days with Gabrie and Marcel van Os :-)

The training course is actually an XTG in-house developed four-day hands-on deep dive into VMware vSphere 4 and 5, focussing on configuration, management and troubleshooting. It’s aimed at VMware VCP-certified professionals looking for more in-depth knowledge on vSphere. The training covers a lot of command line tools for logging, network and storage configuration and vCenter Server management for monitoring and troubleshooting purposes.

Course Modules

• Module 1: Introduction
• Module 2: Logging and monitoring
• Module 3: Networking
• Module 4: Storage
• Module 5: VMs
• Module 6: vCenter Server and the VCSA
• Module 7: HA & DRS
• Module 8: Deployment (optional)

Review

I will review the training on my blog during or shortly after the weekend when the experience is still fresh in my memory.

The initial version was released in December 2010. Since then, version 1.0.1.72 and 1.0.1.175 (update 1) have been released, adding a couple of new features:

1.0.1.72, June 2011

• Integrate remote support and diagnostic operations from vCenter in conjunction with Dell’s Proactive Systems Management
• Direct link to dell.com for warranty renewal
• Updated connectivity to M1000e CMC
• Future updates will not require total re-installation of the virtual appliance
• Trial license available
• Support for LC 1.5, OpenManage 6.5, and ESX 4.1 U1

1.0.1.175 u1, October 2011

• Support for vSphere (ESXi) 5
• Bug Fixes
• Enhancements to deployment and usability functionality
• Support for 12G servers
• Delivered as a patch (so no full OVF)

1.5, March 2012?

Dell is expected to release version 1.5 of the product at the end of March 2012. Expected new features are:

• Released as both an update within the appliance and as a full OVF
• New license option for 25 and 100 hosts.
• Support for the 12th generation (12G) of Dell PowerEdge Servers
• 12G servers do not require OMSA, creating an ‘Agent Free’ option utilizing the iDRAC and Lifecycle Controller
• Deploying ESXi to (dual) SD-card Modules
• Lockdown mode support
• New Compliance Wizard (to check if ‘Collect System Inventory on Restart’ is enabled, to install and configure OMSA and to fix the Connection Profile)
• Improvements to the initial firmware installation process
• Various bug fixes
• Option to exit maintenance mode automatically of a host after firmware update complete

Check out two screenshots on the Compliance Wizard and exit the maintenance mode after firmware update:

 Other News

I noticed there’s a new free one host version is located. Get it here: http://marketing.dell.com/software-download.

I’ll keep you posted if version 1.5 is actually released!

I admit, I’ve always been a fan of Mike’s books. I bought every book he has written, right from the first Administering VMware Site Recovery Manager, version 1.0 (which you can get for free, along with the version 4.0 version of the book here: 1.0, 4.0).

I’ve used his books to write up my ‘SRM in a box’ guide to evaluate the product in a home lab or on a laptop without going through the hassle of a physical (storage-) environment. In November 2008, I used a 4GB laptop for this purpose, only four months after the initial product release. I read Mike’s book back-to-back about five times to be able to squeeze every megabyte of vRAM out of the set up. I learned a lot about SRM, to say the least.

Today, I rebuilt the lab based on SRM and vSphere v5. Like in 2008, the lab gobbles up all the physical RAM I have at my disposal (16GB in a Dell Precision M4600 laptop). It’s still a tight fit, but it runs smooth enough to act as a companion of Mike’s new book:

I was surprised to find a complimentary copy of Mike’s new book Administering VMware Site Recovery Manager 5.0 in the mail a couple of weeks ago. I had just bought the eBook version for the iPad. I’ve had the pleasure to read both the digital and the paper version of the book, and I am very pleased. Mike is a real expert on the subject, and together with the new VMware Press style, finish and overall quality, it’s a very pleasant experience. Also, the book reads almost like a the Tech Novel (currently a work-in-progress by Luke Maverick). The flow of the book made me read it like wasn’t a tech book at all. Well done, Luke and/or Mike!

I really like the chapters on vSphere Replication and bi-directional protection, as this is a new feature I hadn’t working with earlier. I’ll probably end up re-reading these chapters a couple of times in the near future, as I have a project involving SRM coming up.

Not only is it the first books VMware Press has published, it is also the only book I know on this version of Site Recovery Manager. It’s a must read for anyone interested in the subject, and I do recommend is whole-heartedly!

Have you encountered the SRA Error “Failed to load SRA from ‘C:/Program Files (x86)/VMware/VMware vCenter Site Recovery Manager/storage/sra/HP P4000 SRA’. SRA command ‘queryInfo’ didn’t return a response.“?

I have battled this error for about a week now, and only just found a solution.


I’ve tried a couple of things to fix this problem:

• Reinstalled the HP P4000 SRA like five times…
• Messed around with the settings for the SRM database; finally found “Install and Configure SQL Express 2005 for use with Site Recovery Manager V4” (it still works for SRM v5 and SQL Server 2008 R2 Express)
• Tried changing the account under which the SRM service runs (as suggested here)
• Tried forcibly setting the path for the temporary directory (VMware KB 1027218)

But none of these actually helped. I let the issue rest for a bit to clear my head. Today, I picked it up again. I dove right back in to the logs and noticed something odd. The SRA was outputting the following into SRM’s log files:

2012-02-21T15:17:44.631+01:00 [00508 error 'Storage'] queryInfo's stderr:
-->
--> Unhandled Exception: System.IO.FileNotFoundException: Could not load
file or assembly 'System.Core, Version=3.5.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089' or one of its dependencies.
The system cannot find the file specified.
--> File name: 'System.Core, Version=3.5.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089'
--> at HP.SRA.SRAProcessor.ProcessRequest(String[] args)
--> at HP.SRA.SRAProcessor.Main(String[] args)
-->

Somehow, the format and message seemed to indicate some sort of .NET error to me. I googled, and in fact discovered a lot of totally unrelated information about not having .NET installed.

So I went back to the two servers (both Microsoft Windows 2008 R2 SP1 VMs) and noticed that indeed .NET wasn’t installed. I quickly installed .NET, and hey presto, the SRA started working!

Well done HP for failing to add this pre-requisite to your documentation.

Update: HP does mention .NET 3.5 as a prerequisite in the document “SRAforVMwareSRMDeploymentGuide.pdf“, but in a somewhat confusing manner: “.Net 3.5 will be installed if not present“. Obviously, it doesn’t get installed automatically if not present.

A lot of my day-to-day work involves building and tearing down various configurations containing ESXi-hosts. For this, I use my laptop; a Dell Precision M4600. This means that the ESXi-hosts are virtual machines. I use vESXi templates and linked clones inside VMware Workstation to speed things up and save on disk space, instead of rebuilding each host anew.

I use the concept of templates and linked clones for my Windows virtual machines as well. With Windows, we get the advantage of sysprep to generalize the Guest OS, but with ESXi, such a tool doesn’t exist. We need some workarounds to make sure the Guest OS installation is generalized so each linked clone has its own unique identity.

A friend and co-worker (thanks Sjors!) pointed out that the ‘Reset System Configuration’ option in the DCUI might offer the required sysprep-like functionality. This option executes ‘/sbin/firmwareConfig.sh’, which basically removes the state (state.tgz and boot.cfg files in /bootbank) of an ESXi-host and reboots the machine. By executing this script before we shutdown the virtual machine and marking it as a template, we follow the same basic workflow as compared to sysprep (where you run sysprep and choose generalized OOBE).

Build the Virtual Machine

Make sure the virtual machine is created in the correct fashion, for either VMware Workstation 8 or vSphere 5 as the host platform. The document ‘Running Nested VMs‘ on VMTN explains it pretty well. Eric Gray does an even better job explaining how to nest ESXi on a physical ESXi 5 host.

For Workstation, the process is pretty easy. Create a VM, Guest OS type ‘VMware ESXi 5′. Make sure you enable the virtualized hardware-assisted virtualization in the VM’s settings (‘Virtualize Intel VT-x/EPT or AMD-V/RVI’).

Install ESXi

Installing ESXi isn’t a hard thing to do. Just randomly hit [enter] and [f11], and you’ll have ESXi installed. I used vSphere Update Manager to apply the latest firmware to the host (to save time on deployment later on).

Generalize ESXi

I actually took two separate actions to generalize ESXi:

• From the DCUI, I deselected vmnic0 as a network adapter to be used for the management network and removed the DNS suffix. I then proceeded to restart the management network.
• I then executed ‘Reset System Configuration’. Power the virtual machine off as soon as it reboots, though!

The script behind ‘Reset System Configuration’ does re-generate SSL thumbprints (certificates) and SSH fingerprints. Along with new MAC adresses (generated when you deploy linked from the template) and a different name and network identity (which you configure the newly deployed host with), each linked clone you deploy based on the template is unique.

Caveat: VMFS UUID

I used a small SCSI-disk of 5 GB, since I rarely use local storage for virtual machines. But when I used a 40 GB disk (I think the New Virtual Machine Wizard defaults to this size), the ESXi installer automatically creates a local VMFS-datastore called datastore1. The script behind ‘Reset System Configuration’ does not change or erase the VMFS-datastore. It doesn’t touch the VMFS-datastore at all. When deploying multiple vESXi VMs based on a template that contains a VMFS-datastore, you probably will encounter all kind of funky stuff as the VMFS UUID is not re-generated. Please make sure your template vESXi does not contain a local datastore!

Mark as template

The last step in creating the template (on VMware Workstation) is to mark the virtual machine as a template (‘Enable Template Mode (to be used for cloning)’) and create a snapshot (I named it ‘Reset System Configuration’).

Do you know this page on vmware.com all too well?

Not being able to access the bits and bytes you need to evaluate a product and install it in your lab is just the worst. Us geeks can’t handle this kind of rejection, VMware!

Please bring back the VMTN Subscription! Josh Atwell has posted a perfectly viable proposal on how this could work. Please show your support here.

VMware Labs presents its latest fling: VDSPowerCli.
PowerShell is a scripting language Microsoft developed to help administrators manage the Windows environment. Third parties can write their own snap-ins (dynamic linked libraries) to implement new commands, which are called cmdlets. With VDSPowerCli, users can use the cmdlets provided by PowerCLI to manage vSphere Distributed Switch(VDS).

Features

VDSPowerCli gives you the ability to manage:

• VMware vSphere Distributed Switch
• Distributed Port Group
• Distributed Port

Download and try out VDSPowerCli!

I am digging deeper into HP P4000 (LeftHand) SANs again. This has been a consistent side-project of mine for over two years now. I started using the VSA as a simple, fast and reliable appliance for a shared storage and SAN workshop I’ve delivered at my employers training center, and did a real life implementation of a production environment a while back.

I noticed that when designing a P4000 solution with a number of factors like Multi-Site, stretched HA/DRS clustering, MPIO combined with vSphere, you need to be extra carefull and really kick the tyres of the set-up in a lab to get to know all the gory details. Today, I tested a couple of those factors in the design.

Routed iSCSI

The first pitfall is an obvious one. VMware’s software iSCSI Initiator cannot do routed iSCSI. Multiple paths cannot be routed across those subnets by the ESXi 5 initiator. This effectively means that you can only do a P4000 Multi-Site SAN with a single Virtual IP.

With VMware ESX, the preferred configuration is a single subnet and with ESX servers assigned to sites in SAN/iQ. (source)

This actually has an advantage or two, too. Failover between sites is transparent; the ESXi-hosts do not notice site failure. Assuming your hosts didn’t go down as well during the failure, obviously.

Multipath I/O with vSphere

One of the things that caught my eye is a quirk with round-robin MPIO in a vSphere environment.

A P4000 stripes data in a volume across storage nodes in a site in a RAID0-fashion: each storage node hosts an equal part of the volume’s data. I tested this today by creating a simple three node cluster, creating a volume and writing data to that volume. Before the test, each of the three nodes had 1,37 GB provisioned. After I had written some data, all three nodes had 1,65 GB provisioned. This proves data is written in equal sizes chunks to each of the nodes.

When using MPIO in a vSphere environment, each volume only has one Gateway Connection:

But when using the P4000 DSM for MPIO on a Windows machine, each volume has a Gateway Connection to each of the storage nodes (and an additional ‘control’ session). If the host is assigned to a ‘site’ in the CMC, it’ll only connect to storage nodes in the local site.

This difference in how the connections between a host and the storage nodes for each volume is made dictates how I/O a given volume flows to the SAN:

With vSphere, all I/O goes through the node hosting the Gateway Connection. ‘s1v3′ (third VSA in the first site) is hosting the Gateway Connection and sends back-end I/O to other (local) nodes to fetch or write data. This makes a single storage node the bottleneck for I/O for a specific volume; and makes balancing which storage node hosts the Gateway Connection for a given volume very important: you’d want every storage node to hosts an equally spread number of Gateway Connections. I would recommend creating one or more volumes for each storage node. In a six-node storage cluster, create six (or twelve) volumes.

When using VIP and load balancing, one iSCSI session acts as the gateway session. All I/O goes through this iSCSI session. (source)

While with Windows and the P4000 DSM for MPIO, I/O flows from the host to each storage node. In this case, there’s no problem with a single storage node hosting a Gateway Connection like with vSphere.

In the graphs, you can clearly see a different I/O pattern for a vSphere (ESXi 5) host and a Windows machine using the HP P4000 DSM for MPIO. There is a down-side to using the DSM: the number of iSCSI sessions from a host is much higher, possibly up to the maximum number of iSCSI sessions supported by the OS and in fact limiting the number of servers or volumes.

Note: The difference in speed is irrelevant, but probably caused by the fact that I’m running six VSA’s and a Windows VM on a single laptop. I wanted to show how ESXi only utilizes a single path while Windows uses all paths; the goal of these measurements where not to compare speed (MB/s).

Which node will host the Gateway Connection for a volume?

With a Windows server with the P4000 DSM for MPIO, this is superfluous. It simply doesn’t matter which host will host the Gateway Connection for a host. But for an ESXi-host, it does.
This brings me to one of the worst bits of the HP P4000 SAN (and especially its documentation!): with VMware vSphere, there’s no real way of influencing SAN/iQ’s decision which storage node will host the Gateway Connection for a volume. In fact, creating ‘sites’ in the Centralized Management Console (CMC) and placing both ESXi-hosts and storage nodes in a specific site will limit SAN/iQ’s choice of storage node to host the Gateway Connection: only storage nodes local to the ESXi-host first connecting to the volume can host the Gateway Connection.

This is a very bad thing. Using two scenario’s, I’ll explain why:

Site failure

Imagine all the Gateway Connections roles balanced evenly over all the storage nodes. Your environment is functioning perfectly. Suddenly, the primary site fails. Gateway Connections for all volumes that were bound to a storage node in this site are transparently failed over to the secondary site. The VMs that went offline during the power outage (since the ESXi-hosts went down as well) will be restarted using VMware HA. Everything works as expected, and your design has saved the day.
That is, until you restore the primary site. Since there’s no way for the administrator to manually failback the Gateway Connections to the primary site to restore the evenly balanced pre-failure situation, the volumes are stuck on the secondary site. In my experience, there is simply no way to recover from this, other than messing about with reboots of the storage nodes hosting the Gateway Connections. Obviously, that’s not ideal.  Only when the storage node hosting the Gateway Connection goes offline, the Gateway Connection is moved.

There’s no other manual way a Gateway Connection can be moved, as far as I know. The ‘RebalanceVIP’ doesn’t rebalance Gateway Connections, although a lot of people claim rebalancing should happen automatically (in SAN/iQ 9 and up, anyway), I haven’t seen this work for a vSphere environment. Stopping managers on the storage nodes might actually do the trick, but I have not seen consistent failover of Gateway Connections. Lastly, you could run more managers (and designate your primary site as ‘primary’). This might make SAN/iQ prefer to host a Gateway Connection on a storage node on the primary site. If you have more information on how designating a site as primary will affect Gateway Connections, please reach out to me. Leave a comment below or ping me on Twitter.

First boot

The other scenario is just as bad. Imagine you are booting your environment for the first time, or you are recovering from a total shutdown (all storage nodes went offline). The first ESXi-host to connect to a volume will bind the Gateway Connection to a storage node in the local site. And since an ESXi-host usually accesses all datastores, all volumes are bound to a storage node in the local site, leaving no volumes left for the ESXi-hosts on the other site to connect to locally. Also, this limits the performance capacity of the SAN: only storage nodes on a single site will handle active I/O. The storage nodes on the other site will only do Network RAID-10 I/O replicated to them.

To avoid this problem you can do a little magic trick. Before you boot up the ESXi-hosts, remove them from both sites in the P4000 CMC. This will ensure an even spread of Gateway Connections between storage nodes on both sites, because even though the first booted ESXi-hosts connects to all volumes, SAN/iQ will balance Gateway Connections evenly across all storage nodes, regardless of their site affinity. Afterwards you can restore site affinity for the ESXi-hosts, but make sure to doublecheck the Gateway Connections first.

Site-affinity for Virtual Machines

So, we’ve now established that only a single storage node will handle I/O for a given volume. That means that all virtual machines’ I/O to that volume is handled by that storage node. If you’re running a Multi-Site SAN and a stretched vSphere HA/DRS cluster, you might run into trouble: an ESXi-host on site 1 can use a storage node on site 2 for virtual machine I/O. And that I/O gets replicated (via the Network RAID-10 synchronous replication in a Multi-Site SAN) back to a storage node in site 1. That means double the I/O is flowing over the intersite link, and potentially leading to high latency.

The solution is rather simple and  just a tiny bit elegant:

DRS Groups and Rules

By grouping all the virtual machines in a single volume into a DRS Virtual Machine Group, grouping the ESXi-hosts on a site and creating a vm-to-host DRS ‘should’ rule, you can enforce virtual machines to run locally. There’s one big catch though: since we cannot control which storage node hosts the Gateway Connection for a volume, so we have to manually find out which storage node (and thus which site) is hosting the Gateway Connection for a volume. We cannot guarantee that this mapping of a group of virtual machines to a site will match real life. I’m sure @PowerCLIMan can create a script that runs as a scheduled task inside vCenter to correct for this, though.

How to prevent all this Gateway Connection stuff

Simple. HP needs to create a VMware PSA multipathing plugin (MPP) for the P4000. As seen with the DSM for MPIO on Windows, the P4000 is working fine with a special plugin from HP. We just need this plugin on VMware.

On the other hand, you could do the following:

Leave the storage nodes assigned to sites; this removes the ‘random’ Network RAID-10 issue where a volume might be synchronously replicated within a single site, leaving the other site without a replica. Instead of placing the ESXi-hosts in sites, just leave them unassigned. This way, SAN/iQ does not know where the ESXi-hosts reside, and will happily allow all hosts to connect to all storage nodes. It’s still a limitation of one storage node handling all active I/O, and you’ll increase the amount of I/O travelling between sites, because you now will almost surely have ESXi-hosts on site 1 connecting to storage nodes on site 2 randomly. This could increase latency, too, because an I/O between a host in site 1 travels to the storage node in site 2. This storage node replicates this I/O back to a peer storage node in site 1.

I’ve migrated my RSS feed over to FeedBurner.
FeedBurner should automatically redirect the current feed to the new feed.

It shouldn’t matter which feed URL you’re using, but if you’re experiencing any problems, please use the new feed URL and tell me.

You just don’t talk about the family. Respect the omerta, Monster VM.

My first try to publish the video wasn’t all that successful; but I’m using the iPad and only just discovered iMovie. That makes editting the videoclip much easier. I’ve now cut out the unneeded bits and uploaded the marketing video and actual product demonstration again.

Please visit the VMware booth at VMworld 2011 Copenhagen to see a demonstration of the device itself (running the MVP) and the management software (Horizon Mobile). It’s well worth your time!