I read a blog post over at Windows IT Pro recently that prompted this post. Mr. Greg Shields asks the question “When was the last time you VMotion-ed?”

To answer your question Mr. Shields, I did it just this morning and once this afternoon. Nobody even noticed.

Mr. Shields goes on to quote one of his clients:

Well, we thought about that.  But we found that we really don’t use VMotion pretty much ever.  We know that we can use VMotion, and sometimes we do.  But, our performance is acceptable so we don’t need DRM, and we find that that we’re really never doing activities that require us to relocate or re-balance our virtual machines.

With a followup to this:

That response really got me thinking about the uses of VMotion in contrast with all the publicity its feature gets in the IT press:

• You use VMotion prior to rebooting a host.
• You use VMotion to re-balance load (often through DRM’s automated processes)
• You use VMotion’s HA after an unexpected failure.

Since Mr. Shields didn’t mention a product that does not have a feature like Distributed Resource Scheduler (hence, the argument against it), I won’t either. However, considering the source, I can tell you that it is probably a Redmond-based product.

So, here is my response to Mr. Shields, since he asked:

Just a few small points to clarify for those new to VMware that may be reading this.

First, VMotion and HA are two different things. You don’t use “VMotion’s HA”.

HA or High Availability is when you have a host failure (for unplanned downtime).

VMotion is a live migration (for planned downtime such as maintenance). I find that those new to VMware get these two confused all the time.

DRS (Distributed Resource Scheduler), not DRM (Digital Rights Management?) is a feature that VMware uses to balance the load across the cluster of available host resources utilizing the VMotion functionality.

Having cleared up those few items, here is my perspective.

My clients like being able to use VMotion to do maintenance during normal business hours on their hosts. I see it all the time.

My clients also enjoy not worrying about performance at the host level (like you would do in the physical world) by utilizing DRS. As long as they have enough resources in the cluster and their virtual infrastructure architecture is properly designed, they don’t have to worry about manually placing or load balancing their VM’s on certain hosts.

In summary, I’m all for VMotion and DRS. You can leave DRM in iTunes. I buy all my music DRM free from Amazon anyway .

This VCDX exam note will be short and sweet. The networking section has a lot of repeated lessons (which is good ). The main takeaways for me in this section are:

Configure Advanced Service Console Networking: Redundant HA Heartbeat

• There are two heartbeats that occur. Inter-node and network connectivity (Service Console default gateway by default).
• Inter-node heartbeat occurs every 5 seconds by default
• Network connectivity (read isolation response) heartbeat occurs every 15 seconds by default.
• Redundant networking for HA can occur with NIC teams, a second service console, or both.
• Advanced HA options for redundancy include:
  • das.isolationaddress (Used to set another isolation heartbeat address. You can have up to 10 using das.isolationaddressX- where X is a number 1-10)
  • das.usedefaultisolationaddress (Value is true or false. Determines if the default isolation address of the Service Console’s default gateway should be used.)
  • das.failuredetectiontime (The timeout for the host (15000 milliseconds by default) to declare another host dead when receiving no heartbeats).
  • das.failuredetectioninterval (Changes the inter-node heartbeat interval)

Configure Hostname Resolution

• Three files
  • /etc/hosts
  • /etc/nsswitch.conf (Look for the “hosts:     files dns” entry. This determines the order of name resolution. Files then DNS).
  • /etc/resolv.conf (Nameservers and search suffix are listed in this file).

This VCDX exam note covers Objective 2.3 . This is a relatively short objective and much of the material has been covered in previous posts. So let’s dive in.

Define Configuration Options for VMKernel Ports: Peer DNS

While I have never used this option, it is my understanding that peer dns is used when you are using DHCP (which should not happen in production) to give the IP information to a vmkernel NIC. It is used to gather the DNS from the DHCP server. What I do not know is what parameters the -P or --peerdns options takes. Does anyone out there know this?

• esxcfg-vmknic -P
• esxcfg-vmknic --peerdns

MTU and TSO have already been covered in a previous post.

Understand VMKernel Routing

Well, the VMKernel doesn’t actually do any routing. I believe the objective is to understand how to set up the routing table (default gateway, etc.) in your VMKernel. The command that you need to use is esxcfg-route.

• esxcfg-route -l (list the current routing table)
• esxcfg-route -a 192.168.1.0 255.255.255.0 192.168.0.1 (add a specific route to the 192.168.1.0 network via 192.168.0.1)
• esxcfg-route 192.168.0.1 (sets 192.168.0.1 as the default gateway)

Troubleshoot VMKernel Configuration Issues

• /var/log/vmkernel (vmkernel log file)
• /var/log/vmkwarning (a subset of the vmkernel log file that just lists warnings)

This VCDX exam note will wrap up objective 2.1 with the skills and abilities section. So, let’s jump right in.

Configure Service Console Network Using CLI

To configure a service console network you need to create a vswitch and port group first (let’s say vSwitch0 and “Service Console”)

• esxcfg-vswitch -a vSwitch0
• esxcfg-vswitch -A "Service Console" vSwitch0

Then you need to link one or more physical nics to the vswitch (let’s say vmnic0 and vmnic2)

• esxcfg-vswitch -L vmnic0 vSwitch0
• esxcfg-vswitch -L vmnic2 vSwitch0

Finally, you need to create the vswif interface on the vSwitch and attach it to the “Service Console” portgroup. (let’s say the IP is 192.168.0.1/24)

• esxcfg-vswif -a -i 192.168.0.1 -n 255.255.255.0 -p "Service Console"

Configure VLANs

To add a VLAN to a portgroup you must create the portgroup, then you must assign a VLAN ID to the portgroup (let’s say “VMNet1″ for the portgroup, vSwitch1 for the virtual switch and 100 for the VLAN ID.

• esxcfg-vswitch -A "VMNet1" vSwitch1
• esxcfg-vswitch -v 100 -p "VMNet1"

Configure TSO and Jumbo Frames has already been covered.

Enable Cisco Discovery Protocol

Use the -B or --set-cdp option (let’s say the virtual switch is vSwitch0)

• esxcfg-vswitch -B down|listen|advertise|both vSwitch0

You will choose one of the four: down, listen, advertise, or both.

This VCDX note will cover Objective 2.1.K.3 “Enable Advanced Networking Capabilities”. Let’s dive right in.

TCP Segmentation Offload (TSO)

TSO is enabled in the VMKernel by default. If you want to disable TSO on a vmkernel interface when you create it you must use the -t or --tso option. This disables TSO on the vmkernel interface that is being created.

• e.g. esxcfg-vmknic -a -i 192.168.0.1 -n 255.255.255.0 -t vmkernel1

TSO must be enabled on Virtual Machines. The supported VM’s are
-Windows Server 2003 Enterprise and Datacenter with SP2 (32 and 64 bit)
-Red Hat Enterprise Linux 4 (64-bit)
-Red Hat Enterprise Linux 5 (32 and 64 bit)
-SUSE Linux Enterprise Server 10 (32 and 64 bit)

The virtual machine must use the Enhanced VMXNet driver for the virtual NIC. It is possible to use TSO in a Windows Server 2003 Standard Edition VM, for example, using this workaround posted by Scott Lowe. The bottom line is that the driver must be Enhanced VMXNet for TSO to work inside the guest VM.

Jumbo Frames

To enable Jumbo Frames for an entire vSwitch, the MTU must be adjusted to the max of 9000. You can also set the MTU when you create the switch. Adjusting the MTU is done with the -m or –mtu=MTU option.

• e.g. esxcfg-vswitch vSwitch3 -m 9000 or esxcfg-vswitch -a vSwitch 3 -m 9000

To enable Jumbo Frames on a single vmkernel interface, it must be done at creation time.

• e.g. esxcfg-vmknic -i 192.168.0.1 -n 255.255.255.0 -m 9000

NetQueue

NetQueue allows each virtual NIC to have a network queue instead of one common queue. This can increase the throughput capability of 10Gbe adapters. As of ESX 3.5 U2 the following network adapters support NetQueue:

• Intel 82598 10 Gigabit Ethernet Controller
• Neterion

NetQueue is disabled by default. To enable NetQueue at the command line, add the following line to the /etc/vmware/esx.conf file:

• /vmkernel/netNetqueueEnabled = "TRUE"

You can also enable NetQueue using the VI Client by highlighting the host and going to the Configuration tab. From there, navigate to “Advanced Settings -> VMKernel” Select the checkbox for VMkernel.Boot.netNetqueueEnabled . Then you must enable NetQueue on the adapter module using the following commands:

• esxcfg-module -s "intr_type=2 rx_ring_num=8" s2io (for the Neterion)
• esxcfg-module -s "InterruptType=2,2 VMDQ=16,16" ixgbe (for the Intel)

You must reboot the ESX server for the changes to take effect. To disable NetQueue, remove the line you added to the esx.conf or deselect the VMkernel.Boot.netNetqueueEnabled checkbox. You must also disable NetQueue on the adapter modules by using the following commands:

• esxcfg-module -s "" s2io (for the Neterion)
• esxcfg-module -s "" ixgbe (for the Intel)

I was recently setting up a Capacity Assessment for a client when I ran into an issue with perfmon. Perfmon is used on the physical windows servers to collect performance information. This information is used to help determine the best route (design, hardware specs., etc.) to virtualization for the physical servers. However, upon trying to collect the information on some systems I received an error that the performance counter “object was not found.” The VMware Capacity Planner data manager will tell you that this error is usually due to traffic not getting through a firewall or anti-virus product on the server.

I did some digging and found that the firewall was not the problem. I opened up perfmon directly on the problem servers and found numbers for counters instead of the names. This told me that I had corrupt counters. The fix in this case was really simple. I copied the counters (*.dat files) from a server that the performance metrics were working on (same OS) to the problem servers.

The two files in question are %systemroot%\system32\perfc009.dat and %systemroot%\system32\perfh009.dat.

Once these files were replaced with the ones from the good server, I restarted perfmon and the counters were there. The data collection could continue. There is a chance that you may need to dig a little deeper to restore the counters in your specific scenario. The following Microsoft KB article should help: KB300956

I’m up to section 2 on my VCDX Enterprise Administration exam review. This VCDX exam note deals specifically with Objective 2.1.K.2 (Create and modify virtual switches and virtual switch policies). You can easily create vSwitches and Portgroups using the VI Client. However, you may be in a situation (like on the Enterprise Administration Exam ) that you need to use the command line interface to create vSwitches and Portgroups. That is what this note will cover.

Notes for Objective 2.1.K.2: Create and modify virtual switches and virtual switch policies.

• List information about the pNICs on your ESX host (useful in helping to identify certain pNICs):
  • esxcfg-nics -l
• Create a new vswitch:
  • esxcfg-vswitch -a "vswitch_name" (where vswitch_name is the name of your new vswitch. Usually in the form of vSwitch#).
• Add a pNIC as an uplink to a vSwitch:
  • esxcfg-vswitch -L vmnic# vswitch# (where vmnic# is the chosen vmnic (pNIC) and vswitch# is the chosen vswitch).
• Remove a pNIC from a vSwitch:
  • esxcfg-vswitch -U vmnic# vswitch# (where vmnic# is the chosen vmnic (pNIC) and vswitch# is the chosen vswitch).
• Create a portgroup on a vswitch:
  • esxcfg-vswitch -A "portgroup_name" vswitch# (where portgroup_name is the name of your new portgroup and vswitch# is the chosen vswitch).
• Add a vmkernel NIC to a portgroup:
  • esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x "portgroup_name" (where the first x.x.x.x is the IP Address and the second x.x.x.x is the subnet mask. Also where portgroup_name is the name of your new portgroup.)
• Enable a vmkernel nic:
  • esxcfg-vmknic -e "portgroup_name" (where portgroup_name is the name of your chosen vmkernel portgroup).
• Set the CDP properties of a vswitch:
  • esxcfg-vswitch -B listen|advertise|both (Choose “listen” , “advertise”, or “both” for CDP configuration).

The load-balancing options and other virtual switch policies can be configured through the VI Client. You can also configure them through the command-line (like enabling VMotion on a vmkernel port) using the unsupported Virtual Infrastructure metashell (vimsh), which is largely undocumented by VMware. A good place to start  learning about vimsh is at VI-Toolkit.com.

Today’s VCDX note is just a general note about VMFS volume SCSI-2 Reservations. As you may know, VMFS is a clustered file system. It allows multiple hosts to read and write to the same VMFS volume simultaneously. This is accomplished through a mechanism called distributed lock handling. Distributed lock handling locks individual files instead of an entire VMFS volume.

For example, file locks are put on files associated with a VM so that it can only be powered on by one host at a time. However, there are occasions that the entire volume must be reserved using a SCSI-2 Reservation (e.g. a VMFS volume Metadata update). The short video below shows how you can observe this behavior in your own testing by administratively locking the VMFS volume with the vmkfstools command.

This exam note covers Objective 1.1.S.5 and 1.1.S.6. You need to know how to configure NFS datastores and iSCSI initiators using command-line tools. So, if you have your vmkernel and service console ports already set up for IP Storage, you can configure each (NFS and iSCSI) from the command-line.

Notes for Objective 1.1.S.5

Dealing with NFS from the command-line is actually quite easy.

• esxcfg-nas -l (Lists all of the NFS Datastores that the vmkernel has mounted.)
• esxcfg-nas -a  -o HostName or IPAddress -s ShareName Label (Where HostName is your NFS server’s host name, IPAddress is your NFS server’s IP Address, ShareName is the name of your exported directory (share), and Label is the label name you would like to give the NFS mount.
  • esxcfg-nas -a -o 192.168.1.100 -s /MyNFS NFS01
• esxcfg-nas -d Label (Where Label is the label name of your NFS mount. Deletes an NFS mount from the ESX host.)
  • esxcfg-nas -d NFS01

Notes for Objective 1.1.S.6

• vmkiscsi-tool -L -l HBA (Where HBA is the iSCSI HBA. The software initiator is vmhba32. Lists the iSCSI LUNs available to the ESX server.)
• esxcfg-firewall -e swISCSIClient (Opens the Service Console firewall for the iSCSI initiator.)
• esxcfg-swiscsi -e (Enable the software iSCSI initiator.)
• vmkiscsi-tool -D -a IPAddress vmhba32 (Where IPAddress is the IP address of your iSCSI target device. Adds a discovery target to your iSCSI software initiator.)
  • vmkiscsi-tool -D -a 192.168.1.100 vmhba32
• vmkiscsi-tool -T -l vmhba32 (Lists the targets that were discovered.)
• esxcfg-rescan vmhba32 (Rescan the vmhba32 adapter.)
• service mgmt-vmware restart (Restarts the ESX host agent so that vCenter reflects the changes.)

This VCDX Enterprise Admin exam note deals with managing VMFS volumes from the command-line.  While vmkfstools is the main utility used to manage VMFS volumes, there are several other commands that supplement vmkfstools to help identify and change vmfs volume characteristics.

Exam Notes for Objective 1.1.S.4

• Querying for information:
  • ls -la /vmfs/volumes (shows all the VMFS volumes available to the ESX server. This includes UUID and Volume Label, which is just a symbolic link to the UUID directory.)
  • esxcfg-vmhbadevs -m (shows a mapping of VMFS volumes to their Service Console partitions and vmhba names. The volume is identified by the UUID).
  • vdf -h (show some quick size statistics- Size, Used, Avail, %Used- on your VMFS volumes. The volume is identified by the UUID and the volume label).
  • vmkfstools -P -h /vmfs/volumes/VolumeLabel or UUID (Where VolumeLabel is the volume label and UUID is the UUID of the volume. Shows metadata information about the VMFS volume.)
• If you need to change the volume label on a particular VMFS volume use the following commands on each ESX server that has access to the VMFS volume.
  • ln -sf /vmfs/volumes/UUID /vmfs/volumes/NewVolumeLabel (Where UUID is the UUID of the volume and NewVolumeLabel is your new volume label. This command simply changes the symbolic link to the UUID volume directory.)
  • service mgmt-vmware restart (This restarts the ESX host agent on the host server.)
• Creating and extending VMFS volumes:
  • vmkfstools -C vmfs3 -b BlockSize -S VolumeLabel vmhba#:#:#:# (Where BlockSize is 1,2,4,or 8. The default block size is 1MB if left alone. Where VolumeLabel is the volume label for your new VMFS volume. Finally, your vmhba path in adapter:target:lun:partition e.g. vmhba32:0:0:1. This creates a VMFS volume.)
  • vmkfstools -Z VMFS_Extent Original_VMFS (Where VMFS_Extent is the extent to add to the original VMFS and Original_VMFS is the original VMFS. E.G. vmkfstools -Z vmhba32:0:1:1 vmhba32:0:0:1 . This will extend a VMFS volume. THE ORDER OF THIS COMMAND IS VERY IMPORTANT AS ANY DATA ON THE EXTENT LUN WILL BE DESTROYED!!! )