I mentioned in my previous post that I’m presenting at the inaugural PING Philly meetup and my topic is on sizing and configuring backend storage. One of the main points I want to stress in my discussion is the importance that IOPS have when planning your storage configuration. To help illustrate this importance, I created a simple RAID group calculator that shows an estimate of the total backend IOPS required to satisfy your application’s performance needs. Here’s the link to the calculator: http://www.eprich.com/tools/simple-raid-group-calculator

Another thing to note is that this calculator is my first javascript project. Thanks to the incredible teaching skills (and infinite patience) of @jpirrello, I was able to complete this calculator in about a day’s worth of time. The calculator has a slight bias towards EMC hardware – specifically the disks. (Sorry NetApp, I still think you’re cool though!) As of now, the disk sizes don’t play any role in the calculator. That will be in the next version  I also made some assumptions on the total IOPS per disk. Here’s the break down:

SSD = ~5250 IOPS
15K = ~175 IOPS
10K = ~135 IOPS
7.2K = ~75 IOPS

These estimates are the average of a variety of drives I’ve found on the interent. I also compared them to this: http://blog.aarondelp.com/2009/10/its-now-all-about-iops.html.

I’d like to add more functionality to the calculator, so feel free to add comments below with your thoughts and suggestions for how I can improve it!

The last few months have been insane. First, I rejoined SunGard Availability Services where I am leading an awesome technical team with the ongoing development of our Enterprise Cloud Service. Then I spent most of October studying for the EMC Information Storage and Management Exam. After I achieved the EMC Proven Professional certification, I decided to learn a little programming with Python. And now I’m co-organizing a technical meetup group in Philadelphia called PING Philly.

PING (platform infrastructure networking group) Philly is a new meetup group that covers a broad range of IT topics including: systems, storage, cloud, virtualization, storage, networking, programming, etc. You name it, we’ll talk about it. One of our objectives with this group is to meet new local people in tech. It doesn’t matter if you are a network guy (or gal) or if you write code all day long – we want you there. If you are local to the South NJ/Philadelphia area, I highly encourage you to sign up and attend our inaugural meeting on November 30th, 2011 in Center City Philadelphia.

Meetings will be held once a month and will (hopefully) be sponsored. This typically translates to there will be free pizza The meetings are scheduled to run about 2 hours, starting at 7PM. The format will include presentations, Q&A, general discussions, and lightning talks. After the meeting, we will most likely congregate at a local bar for some delicious beer and we’ll continue to network.

Since I’m a co-organizer, I’ll be giving a brief presentation on storage: choosing and sizing the right RAID group for your virtual workloads. I’ll cover topics such as RAID levels, write penalties, calculating the number of disks needed to build a RAID group based on IO requirements, and how to get your IO requirements…things like the. A lot of that stuff is still fresh in my head from all of the studying. I’ll also post the presentation to my site via slideshare after the 30th.

Again, if you are local to the Philadelphia area, please join the meetup and make an effort to come out. Oh and if you are interested in sponsoring a meeting, please drop me a line. We’d love to hear from vendors. Thanks!

This post focuses on design considerations for using Auto Deploy Server to provision stateless ESXi hosts. Here are the three topics that I’ll address:

• Placement of New Services (Auto Deploy Server, DHCP, TFTP, Syslog, ESXi Core Dump Collector)
• DHCP Reservations
• Dedicated NICs for ESXi Management

Placement of New Services

Auto Deploy has a few prerequisites:

• PXE boot environment (DHCP and TFTP)
• Auto Deploy Server
• Image Builder (included with PowerCLI)

Some other nice to haves are Syslog and ESXi Core Dump Collector – both installable services that can reside on the same Windows host.

Note: Auto Deploy Server shares the same system requires as vCenter Server. Here are VMware’s suggested configurations:

• Medium Deployments – up to 50 hosts and 500 powered on VMs: 2 cores and 4GB memory
• Large Deployments – up to 300 hosts and 3,000 powered on VMs: 4 cores and 8GB memory
• Extra-large Deployments – up to 1,000 hosts and 10,000 powered on VMs: 8 cores and 16GB memory

Here’s my current lab configuration:

• Server 1 – vCenter Server and vCenter Orchestrator
• Server 2 – Auto Deploy Server, Microsoft DHCP Server, and TFTPD32
• Server 3 – VUM, Syslog Collector, ESXi Core Dump Collector

All of these servers reside on management VLAN.

During the installation, you’ll need to specify the size of the image profile depot. The default is 2GB but I changed mine to 5GB. You should consider the number of different server vendors, and the number and type of 3rd party host extension providers when planning the Auto Deploy installation.

Note: VMware includes Auto Deploy Server, a DHCP server, and a TFTP server with the vCenter Server Appliance. If you already have a functioning PXE boot environment, then you can use that provided the ESXi hosts can access it.

DHCP Reservations

Most normal management environments use static IP addressing. However, that won’t work with Auto Deploy because of the PXE boot requirement so the next best thing is to create DHCP reservations. This will ensure consistency and won’t affect my management tools. You can use a simple script to add the reservations. Below is the command line syntax and an example:

*Update: I forgot to mention that you need to right-click on cmd.exe and select Run as administrator for this to work. Also specifying the server name/IP is not required if you are running this on the DHCP server.*

netsh.exe dhcp server scope <scope>  add reservedip <IP_address> <mac_address> <description>

netsh.exe dhcp server scope 192.168.1.0 add reservedip 192.168.1.100 0025b503e01a esx1-mgmt

Dedicated NICs

All of the NICs in my lab hosts are connected to trunk ports and I have several VLANs already defined for things like Management, IP storage, vMotion, etc. This also means that I have a default VLAN (which is what I use for PXE booting). Since I do not want my Management traffic to be on the default VLAN, I need a new work around for PXE booting on the Management VLAN.

I decided to dedicate a pair of NICs for ESXi management traffic. This allows me to keep my management traffic isolated from everything else AND it gives me the ability to PXE boot on the Management VLAN. I’m sure there are other methods but I think this is more simplistic and it gets the job done.

If anyone has any other suggestions or would like to comment on what I’ve done, please feel free to comment. I’d love to hear how you addressed your design challenges.

I’ve mentioned before about how I really do not like repeating the same task over and over and over – ESPECIALLY when working in a GUI. One of those tasks is manually mounting a NFS datastore to dozens of ESX hosts. I wrote a powershell script (with some assistance from my friend Jason) to do all of the work – fast-like.

You’ll need to connect to a VI server instance before you run the script. I’m usually already connected to a vCenter instance which is why I left out connect-viserver. The script will prompt you for the NFS server info (FQDN or IP address PLUS the full path to the share), the new datastore name, and all of the hosts to where the new datastore will be mounted. The script will continue to prompt you for ESX hosts; when you are done just hit enter again.

This process does take a bit of time to complete because mounting the datastore is a serial process. One of the things on my to-do list is to see if I can mount the datastores in parallel.

Feel free to leave a comment if you have suggestions for improving the script.

# Enter the name or IP of the NFS server
$nfssrv = Read-Host "Enter the name or IP of the NFS server"

#Enter the full path to the share
$path = Read-host "Enter the full path to the share"

#Give a name to the new NFS Datastore
$nfsds = Read-Host "Enter a name for the new NFS datastore"

#Create an Empty array for $esx
$esx = @()

#Will continue to prompt you for hosts until you stop entering them.
do {
    $input =  Read-Host "Add an ESX host by FQDN or IP"
    if ($input -ne ""){
        $esx += $input
    }
   }
until ($input -eq "")

#Mounts datastore
foreach ($name in $esx)
        {
         New-Datastore -VMHost $name -Name $nfsds -Path $path -NfsHost $nfssrv -Nfs
        }

I spent a few hours this week assisting my team with some Iometer tests. After repeating the same steps over and over on the first two VMs (and looking at the other dozen or so VMs waiting in the queue), I realized that this is an ideal scenarion for some scripting. My goal was to create a repeatable process for running Iometer on multiple VMs from a single management box.

Getting Started

To demonstrate, I’ll be using three Windows Server 2008 R2 VMs where one will be the master server and the other two will be remote workers. Each of my VMs will have a second virtual disk added for the Iometer test. Make sure to leave the second disk as a raw disk – meaning it is online but not initialized or formatted by the guest OS.

First thing is to get your environment ready. Build one VM and configure it as needed. Make sure to install Iometer and VMware tools. You can download Iometer from here. After you build your first VM, you can either create a template from it or clone it as many times as you need. I opted for the template because I can run through the guest customization wizard and create one specific to Iometer.

Note: If you are testing against a machine with multiple raw disks, it’s very important that you verify that each disk is online. When adding multiple disks at once to a Windows system, sometimes the OS only brings the first physical disk online and the others are left offline. This will cause tons of IO errors in Iometer because the offline disk isn’t accessible. Save yourself the time and aggravation by verifying that all disks are online prior to running the tests.

Configuring Iometer

This step should be performed from the master Iometer VM. I created a new test using one worker testing the blue physical disk with 8 outstanding IO per target.

On the Access Specifications tab, I selected 512B 50% Read, 4K 25% Read, and 32K 25% Read as a random workload.

I changed the update frequency to 1 second so I can watch the results in near real-time. On the last tab, I’m going to run this test for 2 hours.

When you are done, save the configuration and leave Iomter open. Now to the VIX part.

The VIX Part

To me, this is the coolest part of the entire thing. I’m going to create a small batch file that will use VIX to kick off a dynamo session on each worker and have it connect to the master Iometer VM. Now to do this part, you’ll need to download and install VIX API, which you can get from here.

Creating the batch file is pretty easy, I simply copied this command for each VM that I want to test:

start /b vmrun -T vc -h https://vcenter_FQDN/sdk -u user.name -p password -gu administrator -gp  '
password runProgramInGuest "[Datastore_Name] VM_Name/VM_Name.vmx" "c:program files '
(x86)Iometer.orgIometer 2006.07.27dynamo.exe -i 192.168.1.13 -m 192.168.1.14"

-T specifies the type of server, in this case it’s vc for vCenter. You can also use esx.
-h specifies the host or vCenter IP/FQDN.
-u and -p are the credentials for vCenter.
-gu and -gp are the credentials for the guest OS.
runProgramInGuest is the VIX command.
“[Datastore_Name] VM_Name/VM_Name.vmx” is the path to the VM. Keep the quotes and brackets.
“c:program files (x86)Iometer.orgIometer 2006.07.27dynamo.exe” This is the full path to the executable in the guest OS.

Note: If you copy and paste this, remove the tics in between -gp and password and in between files and (x86).

***UPDATE***

In my haste to get this post published, I totally overlooked the fact that you need to initiate the commands in an asynchronous fashion. This is because the vmrun command will wait until the dynamo.exe process stops running – per VM. Well if we are running an iometer test for several hours and we have a dozen or so VMs to test…you see where I’m going with this, right? To fix this problem, use the start /b to call the vmrun command. I updated the command structure throughout the post. Start /b will run each command as a separate background process without opening new windows. If you want a window per command, remove the /b.

******

The last line of the script calls dynamo.exe with two switches: -i and -m. The -i switch is for specifying the machine name/IP where Iomter is running. The -m switch is for the name of the manager connecting to the master Iometer server. To summarize, .13 is my master Iometer server and .14 and .15 are my remote workers.

My final batch file will look something like this:

start /b vmrun -T vc -h https://vcenter_FQDN/sdk -u user.name -p password -gu administrator -gp  '
password runProgramInGuest "[Datastore_Name] VM_Name/VM_Name.vmx" "c:program files '
(x86)Iometer.orgIometer 2006.07.27dynamo.exe -i 192.168.1.13 -m 192.168.1.14"

start /b vmrun -T vc -h https://vcenter_FQDN/sdk -u user.name -p password -gu administrator -gp  '
password runProgramInGuest "[Datastore_Name] VM_Name/VM_Name.vmx" "c:program files '
(x86)Iometer.orgIometer 2006.07.27dynamo.exe -i 192.168.1.13 -m 192.168.1.15"

Pretty easy, right? Now you can just cut-n-paste for each VM. Make sure to update the IP for the manager. After you run the VIX batch file, you’ll see the following:

All of my workers are now reporting into my master Iometer server and I can now kick off the test that I configured earlier.

I’m definitely not a network guru but there are many occasions when I need to configure or troubleshoot something. I like to learn as much as I can so I do not need to impose on others…repeatedly. From time to time, I’ll come across a helpful tip or trick which makes my life less stressful. In this case, it’s knowing how to escape from a multi-hop reverse telnet session. My coworker Ramki taught me this one and I wanted to write a quick post so I don’t forget.

We were using a Cisco 2900 router as a terminal server to connect to console ports on remote switches and a remote router. The remote router was also was being used as a terminal server to connect to an entirely different set of remote switches.

So here goes – let’s telnet from router 1 to router 2.

router1#telnet 1.1.1.1 2003
Trying 1.1.1.1, 2003 ...
% Connection refused by remote host

Sometimes you need to clear the line.

router1#clear lin 3
[confirm]y [OK]
router1#telnet 1.1.1.1 2003
Trying 1.1.1.1, 2003 ...
% Connection refused by remote host
router1#clear lin 3
[confirm]y [OK]

Sometimes you need to clear the line…a lot.

router1#telnet 1.1.1.1 2003
Trying 1.1.1.1, 2003 ... Open
router2>

Success! Let’s go to another device.

router2>en
Password:
router2#telnet 1.1.1.1 2005
Trying 1.1.1.1, 2004 ... Open

switch3#

Success again!

Now let’s go back to router2. Press ctrl+shift+6 then ctrl+shift+6 then x. This will bring you back to router2’s prompt.

router2#

From here you can telnet to another line or you can exit this router. To exit from here, use the ctrl+shift+6 then x sequence again.

router1#

Definitely not rocket surgery but good stuff to know.

The other day I needed to migrate a Nexus 1000v from a server running vCenter Server 4.0 to vCenter Server 4.1. I spent some time searching the internets to see if anyone had some tips for how this should be done. I found a KB article on VMware’s support site that explains how to migrate the Nexus 1000v to a different instance of vCenter but it does not explain how to migrate any of the connected ESX hosts. This post will guide you through both portions of the migration.

Stuff in the KB Article – Migrating the Nexus 1000v

The first step is to disconnect the VSM from vCenter. As you can see in the picture below, the VSM’s status is Connected. To get this information, type show svs connections.

Disconnect the VSM from vCenter by entering config mode and typing:

svs connection vcenter
no connect

You can do show svs connections again to verify that the VSM is in a disconnected state.

The next step is to download the Nexus 1000v extension XML file. Point your browser to the IP of the VSM and save the file called cisco_nexus_1000v_extension.xml.

Next, log into the new vCenter using the vSphere Client and select Plug-ins -> Manage Plug-ins. Right click on any available white space below Available Plug-ins and select New Plug-in. Browse to the XML file and then click Register Plug-in. Do not enable this plug-in.

After you register the plug-in, you need to connect the VSM to the new vCenter Server. Log into the VSM and enter the following commands from the config menu:

svs connection vcenter
protocol vmware-vim
remote ip address
vmware dvs datacenter-name
connect

Now your VSM is reconnected to your vCenter Server. You can see the VSM registering with vCenter and the creation of all of the port-profiles in the tasks window of the vSphere Client.

Stuff NOT in the KB Article – Migrating the ESX Hosts

Here is where the fun begins! After you migrate your Nexus 1000v to a new vCenter server, you may be confronted with the following warning:

At this point, you’ll need to remove the VEM from the affected ESX hosts.

Note: To prevent losing network connectivity to your VMs, VMotion your VMs to another host and perform the following steps one host at a time.

Here’s the process at a high level:

• Create a new standard vswitch.
• Migrate at least one physical NIC to the vswitch.
• Migrate all of your virtual adapters to the vswitch.
• Remove the DVS.
• Add the host to the Nexus 1000v.
• Rinse and repeat.

Create the vSwitch however you wish; via the CLI or from the GUI. I personally prefer the CLI because I can type pretty quick. Here are the commands for the CLI:

esxcfg-vswitch -a vSwitch0                             <--Creates a standard vswitch called vSwitch0.
esxcfg-vswitch -l                                      <--You'll need to get the port number of the vmnic you are removing.
esxcfg-vswitch -Q vmnic# -V <portnumber> <dvSwitch>    <--This removes vmnic# fromport # from dvSwitchX
esxcfg-vswitch -L vmnic# vSwitch0                      <--Adds vmnic# to vSwitch0

Log into the vSphere Client and refresh the network information. You should see the new vswitch and vmnic appear under the Virtual Switch view.

Click on the vNetwork Distributed Switch view and select the Manage Virtual Adapters… link. Select vswif0 and Migrate to Virtual Switch. Click Yes in the confirmation window, select vSwitch0, and enter a Network Label and a VLAN ID (if applicable.) If your host’s physical NICs are 802.1q trunks and you do not have the proper VLAN ID specified, you will lose network connectivity.

Repeat the above steps for the remaining VMKernel interfaces although the VLAN ID is not as critical for them.

Hint: If you have multiple VMK interfaces, you’ll want to identify their network labels properly so you can easily migrate them back to the Nexus 1000v.

After you migrate all of the interfaces to the standard vswitch, click on the Remove link in the vNetwork Distributed Switch view.

The final step in this process is adding the host back to the Nexus 1000v.

Click on Home > Inventory > Networking in the vSphere Client. Expand the folder containing the Nexus 1000v. Right-click on the switch name and select Add Host to vNetwork Distributed Switch. Choose the proper host and adapters, select the DVUplink port group from the drop down list, and click Next. Match the vswitch port groups with the proper Nexus 1000v port-profiles (this is why I suggested labeling the VMK interfaces – it makes this part so much easier). Click next twice and then finish. At this point VUM will attempt to install the vDS module on the host. If it is already present, it will skip the installation. This step may take a few minutes to complete. After the process completes, you can safely delete the standard vswitch from the host.

Repeat this process for each affected host connected to the Nexus 1000v. Feel free to leave feedback with any changes, hints, or suggestions.

One of the objectives of the VCAP-DCA exam is knowing how to orchestrate and automate tasks in your virtual environment. The primary tool for the job is vCenter Orchestrator (vCO). In this how-to post, I’ll uncover some of the gotchas and provide tips for the following areas:

• Installing vCenter Orchestrator
• Configuring the vCenter Orchestrator LDAP connection
• Configuring the vCenter Orchestrator database
• Configuring connectivity to vCenter Server

Running workflows and administering vCenter Orchestrator will be covered in a future post.

Installing vCenter Orchestrator

Before installing, you need to know what the requirements are for running Orchestrator. You can view a comprehensive list of supported system requirements, LDAP services, browsers, and databases in the Installation and Configuration Guide on pages 15 – 17. This is on the exam blueprint.

Note: The link to the Installation and Configuration Guide is straight from version 1.6 of the VCAP-DCA exam blueprint. The version of vCenter Orchestrator called out in the link is version 4.1. This lab and demo uses version 4.0.

There is not much to the actual installation as it is installed with vCenter. The VMware best practice is to install vCO on its own machine. For demonstration and testing purposes, vCO is running on my vCenter 4.0 VM (in Fusion) and the database is on a separate VM (ESXi) running MS SQL 2008.

Getting Familiar

Start the VMware vCenter Orchestrator Configuration service and then point your browser to the host’s IP on port 8282.

vCO Login Page

The default username and password to log in is vmware. The first recommendation is to change the password after you log in. You should familiarize yourself with the vCO navigation menu on the left side of the screen. This is where you will spend the next 30 minutes or so configuring vCenter Orchestrator.

vCO Navigation Menu

Note: You will see green dots or red triangles in the left nav menu. The green dot indicates a successfully configured component where as the red triangle indicates an error or an item that has not yet been configured.

Status Indicators

After you change the password, you’ll need to specify the network settings. In the drop down list, select the appropriate IP address for use with vCO. Next, enter the FQDN of the machine where vCO is installed and accept the specified default ports and click apply changes.

Configuring LDAP

The next step is to configure LDAP. You should first verify your directory services information before continuing with the LDAP configuration. You should also create a new AD group specific for vCO Administrators. In my lab, I created one called vcoadmins. You will need to specify a vCO Admin group during the LDAP configuration.

Navigate to the LDAP section and select your LDAP client from the drop down list. Select the appropriate LDAP client (Active Directory here) and enter the name of your Primary LDAP host (domain controller). If you have another domain controller to use, you can add that to the Secondary LDAP host field. Leave the default LDAP port unless you have something different going on in your environment. You then need to specify the root element of your LDAP service. As you can see, mine is dc=homelab,dc=local. I’m not using SSL or a GC so those options stay unchecked. Enter the user name and password of a domain account that has browsing privileges in the directory. This account will be used by vCO to connect to the LDAP service.

vCO LDAP Config - Part 1

The next to lookup base boxes are for users and groups. These are the containers that vCO will use to look up users and groups in the directory. The next field is where you will specify the vCO Admin group that you created earlier. Members of this group will have administrative privileges to vCO. Accept the defaults for the remaining options.

vCO LDAP Config - Part 2

Configuring the Database

The next section is where you configure the database. If you have not done so already, you will need to create a new database for use with vCO. I used MS SQL Server 2008 and created a new database called vco.

Note: Remember which databases are supported for use with vCO. VMware provides support for MS SQL and Oracle, as well as experimental support for MySQL and Postgres.

You need to specify the connection details for the database. For this you will need the user name and password for a SQL account that has dbo privileges. You will also need the hostname or IP address of the database server and the port. Lastly, you will need to enter the name of the database and if there is an instance name and a domain. After you specify this information, you will see a confirmation status in red in the upper left. If the database configuration was successful, you will need to click on the Install database link.

vCO Database

After you configure the database, you’ll need to install the Server Certificate. Click on the Server Certificate link and then select Create Certificate database and self-signed server certificate link. Enter data in all of the fields and then click on the Create link. If the certificate create is successful, you will see the confirmation status in green in the upper left.

Connecting to vCenter Server

Click on the License link and select the option to Use vCenter Server license. Enter the name of your vCenter Server host, port 443 for SSL, and select the Secure channel box. Use /sdk for the path if it’s not already entered. Then enter the username and password of an account that can connect to vCenter. It should look like this when configured:

vCO Licensing

Navigate to the Plug-ins section. This section is where you will go to add future plug-ins to vCO. Enter the username and password of a user in the vCO Admins group and then click apply changes.

vCO Plug-ins

After configuring the plug-ins, click on the vCenter 4.X link to configure the connection to vCenter Server. Select Available in the drop down list. Then enter the FQDN of the vCenter host and leave port at the default of 443. Select the Secure channel box and leave the path with the default of /sdk. Specify the credentials for the administrator session and select the Session per user: option.

vCO vCenter Config

Test the connectivity to the vCenter Server by clicking on the Connection test link.

Enter the username and password and click OK to test the connection.

vCO vCenter Verification

The last step in the configuration process is to install the vCO service. Click on the Startup options link and then select the Install vCO server as a service link. After the service is installed, you will see a confirmation status in the upper left at which point you can click Start service.

vCO Service

This concludes the installation and configuration of vCenter Orchestrator.

Helpful Resources

The vCO team site is a great resource for all things related to vCenter Orchestrator. There are how-to articles for a number of tasks: configuring workflows and configuring custom libraries.

VMware’s vCenter Orchestrator documentation page.

A while back I wrote a blog post on how to cluster vCenter Server 4.0, you can read all about it here. Needless to say (yet I’m saying it anyway) that is NOT the supported method for providing high availability for one of the most critical components in the virtual environment. The supported method, as you may have guessed by the title of this post, is to use vCenter Server Hearbeat.In this post, I’ll highlight some of the not so obvious installation items and configuration tweaks. I’ll also review how vCenter reacted to my test scenarios.

Disclaimer: This post is not meant to take the place of the vCenter Server Heartbeat admin guide. You should follow the installation instructions per the admin guide and use this post to help with configuring and testing.

Installation and Configuration

Before jumping into this section, I want to point out some of the prerequisites:

• Install Windows Backup and Command Line Tools – If you are adventurous and want to go with this more painful, less fun option of installing vCenter Server Heartbeat, then make sure these are installed ahead of time.
• Firewall Ports – Either completely disable the Windows firewalls or allow ports 52267 (client) and 57348 (channel). This one hurt a little…because I thought the Windows firewall was already disabled. It was a real “duh” moment.
• Two Separate NICs – For obvious reasons, vCenter Server Heartbeat requires the use of two separate network cards. One is for the Channel communication (think private or heartbeat traffic) and the other is the Principal NIC (think public network traffic).

These are not all of the prerequisites and you should definitely read through the administration PDF before installing this software. Onto the notes…

There are two methods for installing vCenter Server Heartbeat or vCSHB: cloning the primary vCenter Server or backing up the primary vCenter Server and restoring that backup to the secondary vCenter Server. This post will focus on the first option – cloning the primary. This is the faster and easier way to install the software.

Installation – Cloning Option

The first thing you’ll want to do is clone the primary vCenter Server. Perform a simple clone – DO NOT make any modifications to the virtual hardware OR use any of the customization wizards. Once cloned, disconnect the Principal NIC and power on the VM. Change the IP address so that it is unique to the VM. If your primary server uses 192.168.1.1, then make the secondary server 192.168.1.2. Test network connectivity to the primary server by pinging its Channel IP address. Assuming that is successful, the next thing you’ll want to do is create a folder and a network share so you can copy some configuration files to it after the software is installed on the primary.

The installation of vCSHB is pretty straightforward. The installer instructs you to connect to a network share and copy a couple of small configuration files. These files are required to install the secondary node. Continue with the installation on the secondary server. When prompted, make sure to select Secondary as the role for this server.

Configuration Notes

Note 1: To help prevent split-brain scenarios, you can configure each server’s Principal NIC with a management IP. Make sure this management IP appears first in the order. Once the management IPs are applied, you can configure a second set of heartbeat pings under the Server Monitoring section of the vCSHB Console. The second set of heartbeat pings should be configured as such:

• ping from the primary to the secondary
• ping from the secondary to the primary

This option is chattier on the network but is helpful if the Channel network becomes unavailable for whatever reason.

Note 2: Enable and configure the auto-switchover option under the Network tab in the vCSHB Console. In my testing, I found that the default of 10 pings is an adequate amount of time to initiate an auto-switchover. You can tune this to suit your needs. If you followed Note 1, then keep in mind that a failover will not occur if  the Channel NIC is disconnected.

Note 3: Modify the time out value for disconnecting the vCenter Server from an external database. This one drove me a little crazy because the auto-switchovers were not working properly. When an auto-switchover event occurred, vCSHB would eventually timeout because the vCenter Server service on the active server would not stop. Once the VPXD.exe process was killed, the service would switchover. This happens because vCenter disconnects from the external database after 30 minutes by default. You can change this by editing the VPXD.cfg file and restarting vCenter Server service. VMware KB article 1027289 explains how this is done. After you update the VPXD.cfg file, you must configure the timeout values in vCSHB as follows:

Launch the vCenter Server Heartbeat Console.
Navigate to Applications: Summary.
Select VMware VirtualCenter in the Application pane.
Click Edit.
Modify the Stop Timeout value to reflect 180 seconds.
Click OK.

Note 4: vCSHB periodically checks the status of the vCenter Server by sending a quick API call. To do this, it needs to be configured with an account that has at least read-only permissions to vCenter. Configure this account in vCSHB by selecting Applications – Plugins – VirtualCenterNFPlugin.dll and click on Edit. Enter the username and password then click OK. To verify settings, click on Rules and scroll to the bottom. Select Check Connection to vCenter and click on Check Now.

Testing Scenarios

The objective of each of these test scenarios is to witness vCenter Server Heartbeat perform an automatic failover of the vCenter Server service. A test is successful if the protected service fails over from the active server to the standby server within a timely manner (less than 5 minutes) and does not require assistance or user input.

A manual switchover took one minute and thirty seconds to complete.

Test 1: Application failure – kill the NF service on active server.
I ran a continuous ping from a client machine to the shared IP and connected with the vSphere Client to the vCenter instance. Logged into the primary vCenter server and stopped the Neverfail Server R2 service. The service did not failover and my session and pings did not drop. The reason the vCSHB session remained active is because the protected application does not stop if the vCSHB services are stopped.

Test 2: Application failure – kill the vCenter Server service on the active server.
My vSphere Client connection immediately dropped and the server group in the vCSHB Console went red with an error. While waiting to see how the secondary server would react, my vSphere Client session reconnected to the primary vCenter server. I checked the status of the vCenter Server services and both of them were restarted by vCSHB.

Test 3: Network failure – disconnect the Channel NIC.
Disconnected the Channel network connection in vCenter. Within 20 seconds, the vCSHB Console posted a warning stating that the channel communication stopped. vCenter stayed up and running and the primary remained the active server. The secondary server remained as Standby and in passive. After reconnecting the channel network on the Primary server, vCSHB established channel communication and all was normal. After the channel communication was established, vCSHB performed all integrity checks and recovered properly.

Test 4: Network failure – disconnect the Principal NIC.
As I stated in Note 3, the vCenter Server service failed to stop on the primary server. After about 10 minutes of the service stopping, I manually killed the vpxd.exe process. Once the secondary server detected that the process stopped on the primary server, the switchover resumed and vCenter was restarted on the secondary server.

Test 5: Power failure – shutdown the primary server.
After about a minute, the secondary vCenter instance assumed the identity of the primary. I was able to refresh the vSphere Client and log back into vCenter.

Test 6: Controlled switchover – primary to secondary.
While connected to vCenter with the vSphere Client, I initiated a manual switchover by opening the vCSHB console – Server – Summary, select the Secondary Server in the Server Summary screen and click Make Active. The vCenter service then stopped on the primary, and the secondary assumed the role of active server. My vSphere Client remained connected during the switchover.

Test 7: Controlled switchover – secondary back to primary.
I initiated a switchover from the secondary server to the primary server using the same method as in Test 6. The failover process took slightly longer this time but was within the 5 minute requirement. My vSphere Client was was also disconnected during this test, mainly due to the longer switchover time. The switchover completed and I was able to reconnect to the vCenter instance with the vSphere Client.

Whether you are testing it for deployment in your environment or just learning it for the VCAP-DCA exam, I would like to hear your experience with vCenter Server Heartbeat. Feel free to leave some comments.

Useful Links

VMware vCenter Server Heartbeat Documentation
Recovering from a failed VMware vCenter Server Heartbeat switchover
Enabling fast auto-switchover on vCenter Servers with remote databases