vCops 5.0 is installed quite easily by importing an OVA directly into vCenter. The result is a two VM vApp that is comprised of an Analytics VM and a UI VM. You can connect to the UI VM two distinct ways. The first is to use the https://<UI VM IP>/vcops-vsphere  URL or via the https://<UI VM IP>/vcops-custom URL. The vcops-vsphere URL is used by vCops Standard and Advanced, while the vcops-custom gives you access to the Enterprise and Enterprise+ features.

Once you connect vCops to vCenter (there is a first-run wizard for this), data is immediately available, but it will take at least 1 hour for 100VMs to retrieve capacity planning data, and at least two weeks (preferably 1 business cycle) to compute normalization values. The hardest part of fresh installs of vCops is waiting for the juicy data to become available so you can act upon it.

On a side note, unless you have LOTS of disk space, it may be better to install vCops as a thin provisioned set of virtual disks. Granted, for large deployments, where there is a huge amount of data, I would install this as a thick provisioned drive as the virtual disk utilization will grow quickly.

vCenter Infrastructure Navigator

VIN is a installed quite easily by importing an OVA directly into vCenter, however you must configure vCenter properly. First, ensure you have the vCenter 5 Web Client installed and connected to your vCenter server. Second, ensure you have set the vCenter Run Time Settings->Managed IP Address appropriately.  VIN integrates as a plugin into the vCenter Web Client and not the standard vCenter Client. In addition, it binds to the vCenter that imports it, not to any arbitrary vCenter. Which implies, that if you have a multi-vCenter environment, VIN must reside within each vCenter, unless those vCenters are in link-mode. This could be a drawback when consider in virtual-in-virtual cloud deployments.

vCenter Configuration Manager

vCM has the most requirements for installation. In fact, it installs as a series of applications and requires Windows 2008 as well as MSSQL 2008 R2 SP1 with SQL XML 3.0 SP3. So what does this mean? For me, it required me to upgrade my MSSQL 2008 installation to MSSQL 2008 R2 SP1 as well as burning yet another Windows 2008 license specifically for vCM as it requires IIS w/ASP which often implies use of ports already in use by other tools so a shared W2K8 R2 VM was not an option. This has not yet been implemented yet but I am moving in that direction. The first attempt to install failed, and I was told to wait for a new release, which has arrived. Now I will try once more and report back.

vCenter Operations Upgrades

Once vCops was installed installed everything worked well until I went to change my licenses. Then I hit a snag. If you change from a vCops Standard to a vCops Enterprise license, you are required to reboot the vCops vApp to effect the change, however going back from Enterprise to Standard requires you to unregister and register vCops. But this is not the real oddity. The true snag came when I was upgrading from v5.0.0 of vCops to v5.0.1. I had everything in place, vCops Enterprise licenses, the pak file, etc. The upgrade went well, but my screens were still showing no useful data (in other words I had grey icons everywhere on my VC Relationship on my Operations Dashboard (one I added as a customization) which shows the same information as vCops Standard. Even the vCops Standard view showed grey everywhere for icons.

The solution was to unregister vCops completely from vCenter including ALL license keys. Once this was completed, I reregistered vCops with vCenter THEN added back in the valid license keys. Now everything works as expected. What led me to what I consider a bizarre solution? I reinstalled vCops while the licenses were assigned and I was not able to login. So I assume, the license was locked to UUID and version of vCops. Simple enough solution. Could it have worked for the original issue, I think so.

Conclusion

I use vCops quite a bit as it has a displayed density of data that meets my needs. However, I also run other tools for other reasons, which is another discussion. vCops Enterprise adds quite a bit of extremely useful information. I just wish there was one central place to define what VMs an application consists and that central place could be used within vCops, APM, etc. Perhaps this is the direction VIN is going.

• Failure to “Disable any removable media devices connected to the virtual machines on the host” even though that options was unchecked during the upgrade
• Failure to properly shutdown vShield Firewalls

The details of the problems are:

When running VMware vCenter Operations Suite 5.0, the UI VM could not be moved to another host automatically by VUM due to a CDROM device that the UI VM uses. Why it uses this device and requires it to be always connected is unknown by me. But VUM did not like this behavior. VUM failed to put the host into maintenance mode.

Solution: vMotion the UI VM of VMware vCenter Operations 5.0 before patch remediation.

When running VUM, one host could not shutdown the local vShield Edge and Endpoint instances residing on that host.  The errors were not forthcoming so entry into maintenance mode was once more delayed.

Solution: Shutdown these VMs by hand.

None of these problems were disastrous but hopefully this will help others when they upgrade their vSphere 5 environment. Since, the work arounds are to by hand actions, it seems reasonable to expect the automated tools to perform the actions satisfactorily as well.

The first thing to realize, is that vShield Edge is written for security folks to use and not for virtualization administrators. The rules you apply are very similar to those found with in the O’Reilly book Building Internet Firewalls. Which is, actually, one of the better books out there on how to properly create the rules for your firewalls. My copy is well used and marked as I used to build my own firewalls.

I had, what I expected was a simple use for vShield Edge. I wanted to use Edge to protect a View environment while using my existing virtual firewall for my standard DMZ and network. However, I also wanted the View virtual desktops to talk to my new vCloud environment. But the first task was to add in View so that it could talk to my existing environment then add in the vCloud mechanisms. Figure 1 depicts what I wanted to happen, in this case, all I am adding is View into the mix with vShield Edge.

Figure 1: Simple Network w/vShield Protecting View

So why use vShield Edge and not my existing virtual firewall? Mainly because, I already serve on port 80 and 443  various existing web services and for View to work not only do I need to serve up port 80 but 443 for View traffic amongst others. So instead of adding in reverse proxies and other complicated features to give me access to more services hanging on port 80 and 443, I assigned a new IP to a vShield Edge Firewall.

vShield Edge, unlike VMsafe type firewalls (vShield App, Trend Micro Deep Security, Reflex Systems vTrust, Juniper vGW, etc.) sits between two vSwitch portgroups (whether on the same vSwitch or different vSwitches). In my case, different portgroups on two vSwitches for the external, and two portgroups on the same vSwitch for the internal side of the traffic.

However, there needs to be a few more things added to the diagram to make View work in a secure fashion. That is the View Connection Broker, and the View Security Server. These two would fall into different aspects of my network to protect them. Figure 2, shows their placement. But the key is why I placed these two services where I did.

Figure 2: vShield w/View Components

The View Security Server acts as a gateway between the desktops and the connection broker and the actual client. In essence, you get a SSL Tunnel from the View Client to the Security Server, and from there communication happens to broker, which hands off to the security server a desktop to which it can connect.  If the Security Server was not there, there would not be an SSL tunnel and the hand off would be between the client and the connection broker. Which means you would need to open quite a few more holes in your firewall. The connection broker could live within the View environment, but you really want to protect this server from the hostile environment of the desktops, so in essence the View trust zone (orange in figure 1 and 2) should only contain the desktops.

In our diagram are two vShield Edge (vSE) components, each with their own specific rules. The first is truly a bastion firewall, while the right most (in figure 2) is a way to keep the hostile desktop zone from reaching the internal VMs except on well known ports. But first, how to configure each Edge firewall so that you can actually use them. Just installing and turning on Edge is not enough.

Step 1:

Remember vShield is designed for security folks who understand firewalls, so a bit of information first:

Figure 3: Edge Internals

There are two sides and really two sets of tables within an Edge firewall. The first is the External Interface, with its associated firewall. The second is an Internal Interface with its own associated firewall. You can go in and out of each of these interfaces. So for example, if you want to have traffic go from the external to the internal interfaces. You need to allow, Incoming on the External (into the external interface into the middle area of the Edge firewall), and Outgoing on the Internal interface (or into the internal network). There exists a middle area, which is between these two interfaces which is covered by Network Address Translation (NAT) and routing rules.

Step 2:

Set up the NAT, to allow all those IPs on the Internal interface to masquerade as the external IP address. This we would think would be setup by default, but that is not the case. You need to setup the source NAT your self. In other words, you need to modify the source address of the VMs inside the vSE to masquerade as the external IP address. The actual, external interface is blocked out in Figure 4, however lets assume it is W.X.Y.Z and we want ANY internal IP to map to W.X.Y.Z.

Figure 4: vSE SNAT

Step 3:

Setup external and internal firewall rules to allow the following services through:

• ICMP (to allow people to ping your external addresses to see if they are alive)
• UDP based DNS (53/udp)
• HTTP (80/tcp)
• HTTPS (443/tcp)
• PCoIP (4172/tcp and 4172/udp)
• RDP (3389/tcp)

You also want the same rules for outgoing so that you can respond to the requests appropriately.We will assume the internal IP address is A.B.C.D and the External is W.X.Y.Z. So what does this mean in vSE speak? Enter the following rules:

So why 4 rules for every port we want opened? Remember Step 1, where we have two distinct interfaces? We need to deal with each interface separately. We allow these ports In if we also need to allow them Out, so that we have continued communication.  We could also try to set up the rules like we do for icmp and DNS-UDP, but  I found all 4 are needed for each port.

Step 4:

Use the default ‘block all’ Traffic Police to anything not already listed as open. During testing you may have to set the Traffic Policy to allow, but we really want this to be Blocked per Figure 5.

Figure 5: Traffic Policy

Step 5:

Allow Communication to the View Desktops. We do this via a route. The View desktops sit on the network I.J.K.0/24, and the Internal side of the vSE is A.B.C.254. So the static route looks like Figure 6. The route says to take all traffic destined for I.J.K.0/24 and pass it through the A.B.C.254 address. Which happens to be the

Figure 6: View Static Route

Step 6:

Using vShield App or vShield Edge, allow the following to communicate from the View Desktops and Security Server:

• Active Directory
• Internal DNS
• View Connection Broker
• and Ultimately my internal vCloud Director

Is that All?

But is this everything? Actually it could be but if you look closely at Figure’s 1 and 2, you will notice that that there is no firewall between the View Zone  and the Internal Zone, well not one you can see. The first approach to this was to use vShield App to keep the View Zone from the Internal Zone, but you can also use a vShield Edge appliance to achieve the same behavior. So that you end up with the following overall design of an existing environment (Figure 7).

Figure 7: Internal Protected from View Desktops using vShield Edge

What is very interesting here, is that vShield Edge can be used to protect bastions, but also to create separation between various trust zones while limiting access between the view desktops and the real internal aspects of your network.

Thanks to @vTexan and his VMware View 5 Install Guide. Without this helpful page, I would not have been able to proceed with my own implementation. Please be aware, however, that the steps @vTexan gives are not necessarily the most secure. Adding in the properly placed firewalls is just one aspect of a secure view implementation, but there is also the proper choice of role based access controls (RBAC) for active directory. The View Connection Broker, requires Active Directory access, but should be limited to a non-domain admin for a newly created View only OU into which the desktops should go. Why, because View needs to be able to modify all aspects of Active Directory for the desktops, so the permissions should be such that the Active Directory user used by View should be limited in scope on what it can modify.

An Aside

I tried to use vShield App to protect the View trustzone from the Internal trust zone but vShield App fails badly if the vShield App appliances suddenly become unavailable. Given that I had 3 such failures, in 8 hours, that were fairly catastrophic, I chose to use on vSE to protect my trust zones.

Conclusion:

So now, we have a fully protected View environment using several vSE’s to provide adequate protection. In addition, we now have a cook book of steps necessary to get vShield Edge up and running (Steps 1-4). Lastly, we have expanded those steps to those required to allow View to communicate properly from the Security Server to the VMs in question.

• Automated backup during a predefined backup window
• Notification of ANY issues immediately
• Automated backup testing

So what was the problem with VDR? The integrity check continually failed and therefore the backups continually failed. Yet, it worked for a while! Which was the frustrating part. The solution was the following:

• Unmount the Destination (which is a Windows VM that talks to my DISC GmbH Bluray backup library for long term storage)
• Remount using the Share name, I previously created instead of the F$ mechanism available to windows
• Edit all my backup jobs to point to the new Destination
• Test by backing up critical components

I believe that VDR does not understand mount points ending in $, which could be given that it is a Linux VM and $ is a special character in nearly every shell and 4th generation language.

So now, I wait until I know for certain this will work, in the meantime I will be testing other tools. Backup failures are NOT something I like to see. Ever.

UPDATE: This was definitely the solution, been running strong for quite a while.

UPDATE 2: If there is an Integrity Check issue, first shutdown datarecovery service on the VDR VM. Then go to the backup store and remove all the *.lck directories and files. Once that is done, you can restart the datarecovery service. Go to the Destination aspect of the VDR configuration tab within the vSphere Client plugin and re-run an integrity check. Once that succeeds you will than have to perform one more action. Go to each of your backup definitions and reconfigure them, selecting the backup store once more. This last element is necessary as the recatalog that happens apparently destroys this information.

VMware VDR fails its integrity check quite often which makes this a repetitive task.

The setup of the scsi-target-utils was pretty straight forward, I pointed the /etc/tgt/targets.conf to my 7.8GB LVM LUN for use for iSCSI. Which meant I had to use VMFS v5 to access it all. Which is my intention. But then that is where several issues occured:

• vSphere 5 requires the iSCSI vmkernel port to hang off a dedicated vSwitch and not a vNetwork Distributed Switch (vDS). This is a change from vSphere 4.

This predicated a slight change to my networking which led to the creation of a dedicated network just for iSCSI and the use of  one of my free Flex-10 NICs just for iSCSI. Unfortunately, in order to change the networking within the Server Profile within Virtual Connect for Flex-10, you first have to shutdown the node. Since my server upgrade, I have more than enough memory to run everything my nodes so that powering off one will not adversely effect anything. So, I went through the process of upgrading server profiles, putting nodes into maintenance mode, shutting them down, saving the virtual connect profile, rebooting the node and viola it was done. If I had more than 4 nodes, this would have been a week end nightmare. How bigger shops manage this is with quite a bit of scripting.

• vSphere 5 would not allow me to migrate my vmkernel device from a vDS back to a vSwitch.

The second problem I faced was that I had to recreate all my vmkernel ports for iSCSI instead of migrating them off the vDS on which they were set. Because there was no migration path available. You can migrate from a vSwitch but apparently not back to one.

• vSphere 5 required me to bind a vmkernel device for iSCSI utilization.

Not a huge issue, but it is a definite change from setting up iSCSI with older versions of vSphere, where all you needed to do was have the proper IP address and it would route appropriately.

• vSphere 5 good not see the iSCSI Server yet, the iSCSI Server could ping the vmkernel devices.

This is where the mojo comes in. The first problem I had was iptables, it was set to deny iSCSI traffic as well as iSNS traffic. That was fixed temporarily by disabling iptables. However, it still would not connect. The solution ended up being a change to how iSNS traffic worked. I disabled the iSNSAccessControl parameter from On to Off, restarted iSNS and suddenly things started to work.

My final /etc/tgt/targets.conf file consists of the following:

default-driver iscsi
iSNSServerIP W.X.Y.Z
iSNSServerPort 3205
iSNSAccessControl Off
iSNS On
<target iqn.2008-09.com.example:server.target1>
        backing-store /dev/vg_iscsi/lv_iscsi
        lun 1
</target>

And for my security settings I ended up adding these rules to my /etc/sysconfig/iptables file:

-A INPUT -m state --state NEW -m tcp -p tcp -s W.X.Y.0/24 --dport 3260 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp -s W.X.Y.0/24 --dport 3205 -j ACCEPT

Now I have a functioning iSCSI server with close to 8TBs available for use by vSphere 5. It will come in handy for some work I have to do on my SAN. Oh, and the hardware upgrade for this all purpose node? 32GBs of Memory plus a Sandybridge quad core running very fast. The 1GB dedicated link I am using is through a switch that will only be used for iSCSI, so all my iSCSI accesses are segregated from the rest of my network. That should help with security as well as performance!

• Install vSphere 5
• Join to vCenter
• Apply Host Profile

Yet it was not that simple.  I found that the Host Profile was asking for a Mac Address which I did not know, nor could I discover as it was not just asking for any Mac Address but one produced by ESX for a VMware vmknic. The solution was not obvious but something easily handled.

• Using the dvSwitch Switch Editor add the hosts in by hand.
• Using the Manage Virtual Adapters Editor go to each dvSwitch and add in the appropriate vmkernel devices (I had 4)

Now I thought I was ready to apply the profile, but not yet!

• Present any Storage Devices to the new Node
• Rescan the Storage for LUNs and Datastores
• Change the name of Datastore1 to be hostname:Datastore1 to be in keeping with my naming convention

Now, apply the profile.

There were still a few errors with the profile about a missing LUN, but no LUN is actually missing. Editing the profile to remove the offending LOCAL LUN did not change the behavior. Nor can I use Host Profiles to add my local administrative user anymore.

Following along the next steps are:

• Attach a VMware Upgrade Manager Profile
• Scan for Updates
• Remediate any Updates

So how do you get 100% Host Profile Compliance? I am not sure you can. All the local drives within HP blades actually show up as remote drives, and since they are not the same across the board, the profile can never be compliant. However, the host profile can be adjusted by using the steps in VMware KB 2002488. With this change, my hosts are now compliance once more. However, what this does, is disable both the Pluggable Storage Architecture (PSA) configuration and the Path Selection Policy (PSP) Configuration for all your local and remote devices. Any PSA and PSP changes will now need to either be done by hand or scripted in some fashion.

So why not upgrade all 4 nodes to vSphere 5 at this time? Support and Licensing reasons. I still have customers using vSphere 4, and having at least one node with that level will be helpful. The other is for Licensing reasons, I would need at least two more licenses for the 4th node. But this does lead to other issues. I cannot yet upgrade to VMFS-5, VM Hardware v8, or upgrade VMware Tools yet.

Which was that the database server I was using had a full drive.

None of my current management tools actually told me this, instead they went on running like everything was fine, when clearly it was not. So which tools did not show this information? VMware vCops Advanced, Solarwinds VMAN, and vKernel to mention just a few. But they do show issues with other nodes that had full disks, so I wonder what is so special about a Windows 2008 R2 Server running MSSQL?

This is the type of Alert I would expect to be emailed to me from all these tools. Yet, no email was sent.

The solution was simple, free up and add disk space to the VM. I removed from MSSQL a few old and unused databases, which gave me just enough breathing room to maintain my systems. The next step was to add more disk space. I did this by expanding the virtual disk and adding space to it using the build in Windows 2008 disk extend capability.

Now to find out why the tools did not report this accurately or even email me when this happened. Perhaps because the server was full and no writes could take place? Which means the tools need to change for such critical issues?

• Place node in Maintenance mode, evacuating all running VMs
• Power down node
• Remove node, add in/exchange memory modules, replace node
• Power on node, using ILO watch to be sure new memory was found properly
• Exit Maintenance mode

It was seamless, and once more done with Zero downtime.

However, adding in my new BL460c G7 into my HP C3000 enclosure was not that simple. Those steps started with:

• Place BL460c in Slot 4
• Power On
• Error ensued about placement of the riser card.
• Moved Riser card, Power on
• All was great

Okay so far it is pretty standard.

First Gotcha: The G7 blades include an ILO 3, which has several new security controls, all goodness, but to apply the ILO licenses to use the advanced features (like virtual media), I had to apply the license. To do this, you needed to Trust All to apply the license. So we moved on from here:

• Change ILO Trust mode to Trust by Certificate and import the HP SIM Server certificate.
• Mount the FW 9.30 ISO and upgrade the blade firmware using virtual media
• On Reboot set the Dynamic Power Mode per the the article vSphere Upgrade Saga: Staging Upgrade to vSphere 5.

Second Gotcha: To configure the P410i in the box, be sure to press any key to see extra Boot messages, else you will NOT be able to see the time to press F8 (Fn-Command-F8 if trying to use a Mac Book Pro via the Remote Desktop Client). It took installing vSphere 5 on this node to realize I had no disk on which to install.

Third Gotcha: Once everything was installed, I attempted to apply a Virtual Connect Server Profile so that the blade can properly access my networking. Even though the blade was found within the Enclosure within HP SIM, virtual connect could NOT see the enclosure. Actually, the Onboard Administrator said it could not participate in Virtual Connect.

Fourth Gotcha: The device would fail to reboot due to a power issue, which also affected the blade in Slot 3.

A bit of searching and I hit upon a possible solution: Move the blades in Slot 3 and 4 to Slots 5 and 6. Which means all my blades are now in the same cooling Zone within the chassis. Even with 6 Fans, the c3000 has two Zones that encompass more than just cooling.

Now, continuing with my new blade install:

• Join new blade to vCenter Cluster
• Apply vSphere 5 Profile
• …

Viola that worked. Virtual Connect Manager could see the blade, the server profile could be applied. Now, I begin to wonder what will happen when I put in another blade, but I will cross that bridge when I get there.

My issue with EVC stemmed from trying to enable it on an existing set of nodes so that I can easily transfer VMs between them. Exactly how EVC was planned to be used. However, in my environment somewhere along the line EVC was disabled. To enable it, I had to find the setting that would enable. I tried to first enable it for the chipset I had, that did not work. Then I enabled it for the previous chipset, of course that did not work. The solution was to enable it for the chipset I did not already have. In other words, I am running the chipset before Westmere, so in order for EVC to enable on my cluster, I had to set EVC mode to be Westmere chips. In other words, I am masking off all Westmere special functions.

This is what confused me, we all know we use EVC to mask off what we do not want, but if you have an existing cluster, you need to max off the chipset ABOVE the one you are on. That is, unless you desire to shutdown all your existing VMs, set EVC to the mode you desire and then power them al back on.

That was not something I was willing to do.

So, now I add to my pre-stage steps, setting EVC to mask off chipset functionality above what I already have. This will become a fairly major requirement when I add a Westmere based blade into my cluster.

Here are the steps I followed:

Step 0: Review any updates to firmware for my HP BladeSystem c3000 just in case something new came out since the last time I looked. There was a new upgrade for the Onboard Administrator.

Step 1: Given this upgrade was available the first step was to upgrade the Onboard Administrator firmware, which I could do directly talking to the Onboard Administrator web interface. Quite simple, and easy. Unlike previous Onboard Administrator ungrades there was no issues with loosing power or connectivity to any of the blades. While doing this, I made out a shopping list to ensure even more redundancy: Another Onboard Administrator as well as a replacement Insight Display Module.

Step 2: Upgrade the Flex10 Virtual Connect firmware. This is required to use vSphere 5. You must upgrade any Flex10/Virtual connect firmware to version 3.30 in order to use vSphere 5. Step 2 was to upgrade the secondary Flex10 first using the command line Virtual Connect Support Utility (VCSU) using the update command.

Step 3: Upgrade the primary Flex10 Virtual Connect firmware. There was an oddness with this upgrade using the VCSU, and that was the VM running the VCSU lost connectivity which seemed to cause a reboot. So something inside the VCSU failed and rebooted, which is really a horrible failure mechanism. However, I only lost network connectivity for a few seconds. Which was also much better than the last Flex10 Upgrade I did.

Step 4: Upgrade the firmware within my spare blade using the HP Firmware DVD v9.30. There was an ILO update, so to be sure all firmware was upgraded, I booted from the DVD twice just to be sure I caught everything.

Step 5: Install vSphere 5 ESXi onto the spare blade just upgraded. The trick is to ensure you pick the proper disk, in this case local disk instead of a SAN drive. I double checked this before proceeding. The install is so much easier than that of ESX.

Step 6: Configure the Management Network for the spare blade to not use DHCP but to use the provided IP Address, Netmask, and hostname.

Step 7: Decided to upgrade my existing vCenter instead of using the Linux appliance, mainly because I already have a Windows configuration for vCenter and the Database server.

Step 8: Backup your vCenter Database and SSL certificates. vCenter 5 will remind you of this as well and nicely ask you to ensure you are happy with your backup.

Step 9: Perform vCenter Upgrade

Step 10: Perform vCenter Client Upgrade

Step 11: Install vSphere Web Client for new License Reporting functionality. This required me to find out which process held port 9443 open, which turned out to be something running Java. However, what I do not know. There was a Java update to be made, once I did that, this error went a way. So I went through and removed all my third party tools from the vCenter server which eventually fixed the issue. A part of this step was to register your vCenter Server with the Web Client Server so that it can communicate.

Step 12: Upgrade vCenter Update Manager. Once more this is a polite reminder to make a backup first of your VUM database which requires you to select the I am happy with my backup checkbox. I find that a great improvement. While VUM upgraded fine, it did say it wanted to reboot my host, eventually that will happen as a matter of course. Need to test if everything comes back up on boot.

Step 13: Install all the vCenter Support Tools, Dump Collector, Syslog Collector, and Auto Deploy but not the Authentication Proxy. These are all new for vCenter 5. The key to these installs, is that if it asks, yes you want the Integrated VMware vCenter Server Installation. The Authentication proxy requires IIS which could cause issues so we hold off on that for now. Now reboot your vCenter server and make sure everything runs.

Step 14: Reconnect the ESX hosts to vCenter which deployed the new vCenter Agents onto the hosts.

Step 15: Create a new cluster so that I can enable EVC mode for Nahalem or better processors. This will help with new blades once they arrive. Somehow EVC was disabled on my other nodes.

Step 16: Configure VUM and import the vSphere 5 ESXi image for later use. The VUM the client for some reason did not enable by default which took going to VMware’s KB Article 1015223 to solve.

Step 17: Configure the newly installed spare node using the old nodes Host Profile. If you have Enterprise Plus licenses the use of Host Profiles for this type of staging is incredibly useful. Even though I am moving from ESX to ESXi, use of Host Profiles to at least get the network configuration correct is very important.  If you do not have Enterprise Plus, you can use an Evaluation Mode to do this. The gotcha here is that I am moving from ESX to ESXi so some of the Host Profile elements will not apply. So not only do I do a clone of the Host Profile, apply that profile, but I need to delete and recreate the Host Profile once the spare node is fully configured else there will be compliance issues.

Step 18: Reboot Node and enter the BIOS to set HP Power Profile to Custom and HP Power Regulator to OS Control Mode per VMware KB Article 1018206. Save and reboot. You can do this as a part of any of the node upgrade steps previously mentioned, but without this you will not be able to take advantage of the power controls within vSphere 5.

Step 19: Upgrade vSphere 5. This is a good test of your VUM install from Step 16. As of this writing there are at least 2 patches for vSphere 5. One requires a reboot.

At the moment, the system is staged for migration of VMs and upgrade of the other nodes. Staging such an upgrade is important as now we know that the firmware upgrades work, I can enable the proper EVC mode, and the other systems are still manageable.