So I am at VMworld this week and I will attempt to cover session highlights that I feel will be useful to our existing and potential clients. The first session is all about the vSphere Management Assistant (vMA). If you haven’t started using the vMA, I suggest you get familiar with it.

After vSphere 4.1 there will be no more Service Console (COS). VMware will be transitioning to ESXi and I am encouraging all of my clients to get a head start on migrating to the ESXi platform. The time to plan for this is now.

Session MA6580: Bridge the ESX/ESXi Management Gap Using the vSphere Management Assistant Highlights

-vMA is a 64 bit …
(more…)

When implementing the Cisco Nexus 1000v, High Availability (HA) Slot Sizes can be affected on your vSphere Cluster. HA slot sizes are used to calculate failover capacity for HA when the “Host Failures” HA setting is used. By default the slot size for CPU is the highest reservation of CPU among all VM’s in the cluster (or 256 MHz if no per VM reservations exist). The slot size for Memory is the highest reservation of Memory among all VM’s in the cluster (or 0 MB + Memory Overhead if no per VM reservations exist). If you want to really dive further into HA and slot size calculations, I would highly recommend reading Duncan Epping’s HA Deepdive at Yellow-Bricks.com.

Each Cisco Nexus 1000v Virtual Supervisor Module (VSM) has a reservation of 1500 MHz on the CPU and 2048 MB on the Memory so that these resources can be guaranteed for the VSM’s. …
(more…)

vCenter is one of the most important components of your vSphere 4.x virtual infrastructure. Many advanced capabilities of vSphere 4 (vMotion, DRS, etc.) are not available without vCenter. Prior to vSphere 4.x, it was sufficient to backup the vCenter database and restore vCenter by building a new vCenter server, restoring the database, and reinstalling vCenter to attach to the restored database.

With the introduction of vSphere 4.x, vCenter 4.x started using Active Directory Application Mode (ADAM) on Windows Server 2003 and Active Directory Lightweight Directory Services (AD LDS) on Windows Server 2008 to accommodate Linked Mode for vCenter. The roles and permissions are stored in the ADAM or AD LDS database. In order to restore the roles and permissions, the ADAM or AD LDS database must be backed up.

VMware KB1023985 tells you that you need to back up the SSL certificates, vCenter Database, and the ADAM/AD LDS database. There are many well-known ways to back up a SQL database. However, backing up an AD LDS instance is a lesser known procedure. The following PowerShell script will back up the the AD LDS VMware Instance on Server 2008 and the SSL folder. As always, test it thoroughly before using it. (more…)

As a part of TBL professional services datacenter practice, I perform many health and security checks on virtual infrastructures for clients. One of the common issues that I run into is the use of the default “root” account for administering ESX servers. This is an issue for two reasons:

• The “root” account has a tremendous amount of power and the password for it is typically the same shared password on each ESX host.
• If all administration is done with the “root” account there is no audit trail for accountability. It could have been Joe, Bob, or Sue that logged into the ESX host. You just don’t know.

Of course, most administration should be done through vCenter, but you still occasionally need to log into an ESX host directly. The solution to this that I have recommended in the past has been to create local user accounts coinciding with the Active Directory user name on each ESX host. Then do not use root unless absolutely necessary when performing administrative tasks directly on the host. However, this meant that the IT Administrators would need to manage user accounts in Active Directory and the local accounts on the ESX / ESXi hosts.

There has been a “less than ideal” solution to Active Directory authentication for quite a while (see Scott Lowe’s article). However, this solution was very laborious, involves the command line, and only worked on ESX Classic. Not ESXi.

With the release of vSphere 4.1, native Active Directory authentication is one of the many new features. Here’s how easy it is to implement once you have ESX installed.

1. Connect to your ESX/ESXi server with the vSphere Client.
2. Click on “Inventory” and highlight your ESX/ESXi server.
3. Click on the “Configuration” tab.
4. Navigate to “Software –> Authentication Services”
5. Click on “Properties” on the right hand side.
6. Change the “Directory Service Type” from “Local Authentication” to “Active Directory”
7. Once you do that and enter in your Domain, click “Join Domain” and you will be prompted for appropriate credentials to join the domain.
8. Click “OK” when you are done.

That’s it! Now you can have accountability controlled through Active Directory Authentication. Joe, Bob, and Sue can all use their respective Active Directory accounts for authentication. Accountability!

Permissions can now be added for Active Directory users and groups as well.

You can even use it with the vSphere CLI and the Direct Console User Interface (DCUI) on ESXi.

Should you still need the local “root” account for emergencies, it will still be available to you. Otherwise, do your company a favor and maintain an audit trail for administrative actions on your infrastructure.

This VCDX exam note will be short and sweet. The networking section has a lot of repeated lessons (which is good ). The main takeaways for me in this section are:

Configure Advanced Service Console Networking: Redundant HA Heartbeat

• There are two heartbeats that occur. Inter-node and network connectivity (Service Console default gateway by default).
• Inter-node heartbeat occurs every 5 seconds by default
• Network connectivity (read isolation response) heartbeat occurs every 15 seconds by default.
• Redundant networking for HA can occur with NIC teams, a second service console, or both.
• Advanced HA options for redundancy include:
  • das.isolationaddress (Used to set another isolation heartbeat address. You can have up to 10 using das.isolationaddressX- where X is a number 1-10)
  • das.usedefaultisolationaddress (Value is true or false. Determines if the default isolation address of the Service Console’s default gateway should be used.)
  • das.failuredetectiontime (The timeout for the host (15000 milliseconds by default) to declare another host dead when receiving no heartbeats).
  • das.failuredetectioninterval (Changes the inter-node heartbeat interval)

Configure Hostname Resolution

• Three files
  • /etc/hosts
  • /etc/nsswitch.conf (Look for the “hosts:     files dns” entry. This determines the order of name resolution. Files then DNS).
  • /etc/resolv.conf (Nameservers and search suffix are listed in this file).

This VCDX exam note covers Objective 2.3 . This is a relatively short objective and much of the material has been covered in previous posts. So let’s dive in.

Define Configuration Options for VMKernel Ports: Peer DNS

While I have never used this option, it is my understanding that peer dns is used when you are using DHCP (which should not happen in production) to give the IP information to a vmkernel NIC. It is used to gather the DNS from the DHCP server. What I do not know is what parameters the -P or --peerdns options takes. Does anyone out there know this?

• esxcfg-vmknic -P
• esxcfg-vmknic --peerdns

MTU and TSO have already been covered in a previous post.

Understand VMKernel Routing

Well, the VMKernel doesn’t actually do any routing. I believe the objective is to understand how to set up the routing table (default gateway, etc.) in your VMKernel. The command that you need to use is esxcfg-route.

• esxcfg-route -l (list the current routing table)
• esxcfg-route -a 192.168.1.0 255.255.255.0 192.168.0.1 (add a specific route to the 192.168.1.0 network via 192.168.0.1)
• esxcfg-route 192.168.0.1 (sets 192.168.0.1 as the default gateway)

Troubleshoot VMKernel Configuration Issues

• /var/log/vmkernel (vmkernel log file)
• /var/log/vmkwarning (a subset of the vmkernel log file that just lists warnings)

This VCDX exam note will wrap up objective 2.1 with the skills and abilities section. So, let’s jump right in.

Configure Service Console Network Using CLI

To configure a service console network you need to create a vswitch and port group first (let’s say vSwitch0 and “Service Console”)

• esxcfg-vswitch -a vSwitch0
• esxcfg-vswitch -A "Service Console" vSwitch0

Then you need to link one or more physical nics to the vswitch (let’s say vmnic0 and vmnic2)

• esxcfg-vswitch -L vmnic0 vSwitch0
• esxcfg-vswitch -L vmnic2 vSwitch0

Finally, you need to create the vswif interface on the vSwitch and attach it to the “Service Console” portgroup. (let’s say the IP is 192.168.0.1/24)

• esxcfg-vswif -a -i 192.168.0.1 -n 255.255.255.0 -p "Service Console"

Configure VLANs

To add a VLAN to a portgroup you must create the portgroup, then you must assign a VLAN ID to the portgroup (let’s say “VMNet1″ for the portgroup, vSwitch1 for the virtual switch and 100 for the VLAN ID.

• esxcfg-vswitch -A "VMNet1" vSwitch1
• esxcfg-vswitch -v 100 -p "VMNet1"

Configure TSO and Jumbo Frames has already been covered.

Enable Cisco Discovery Protocol

Use the -B or --set-cdp option (let’s say the virtual switch is vSwitch0)

• esxcfg-vswitch -B down|listen|advertise|both vSwitch0

You will choose one of the four: down, listen, advertise, or both.

This VCDX note will cover Objective 2.1.K.3 “Enable Advanced Networking Capabilities”. Let’s dive right in.

TCP Segmentation Offload (TSO)

TSO is enabled in the VMKernel by default. If you want to disable TSO on a vmkernel interface when you create it you must use the -t or --tso option. This disables TSO on the vmkernel interface that is being created.

• e.g. esxcfg-vmknic -a -i 192.168.0.1 -n 255.255.255.0 -t vmkernel1

TSO must be enabled on Virtual Machines. The supported VM’s are
-Windows Server 2003 Enterprise and Datacenter with SP2 (32 and 64 bit)
-Red Hat Enterprise Linux 4 (64-bit)
-Red Hat Enterprise Linux 5 (32 and 64 bit)
-SUSE Linux Enterprise Server 10 (32 and 64 bit)

The virtual machine must use the Enhanced VMXNet driver for the virtual NIC. It is possible to use TSO in a Windows Server 2003 Standard Edition VM, for example, using this workaround posted by Scott Lowe. The bottom line is that the driver must be Enhanced VMXNet for TSO to work inside the guest VM.

Jumbo Frames

To enable Jumbo Frames for an entire vSwitch, the MTU must be adjusted to the max of 9000. You can also set the MTU when you create the switch. Adjusting the MTU is done with the -m or –mtu=MTU option.

• e.g. esxcfg-vswitch vSwitch3 -m 9000 or esxcfg-vswitch -a vSwitch 3 -m 9000

To enable Jumbo Frames on a single vmkernel interface, it must be done at creation time.

• e.g. esxcfg-vmknic -i 192.168.0.1 -n 255.255.255.0 -m 9000

NetQueue

NetQueue allows each virtual NIC to have a network queue instead of one common queue. This can increase the throughput capability of 10Gbe adapters. As of ESX 3.5 U2 the following network adapters support NetQueue:

• Intel 82598 10 Gigabit Ethernet Controller
• Neterion

NetQueue is disabled by default. To enable NetQueue at the command line, add the following line to the /etc/vmware/esx.conf file:

• /vmkernel/netNetqueueEnabled = "TRUE"

You can also enable NetQueue using the VI Client by highlighting the host and going to the Configuration tab. From there, navigate to “Advanced Settings -> VMKernel” Select the checkbox for VMkernel.Boot.netNetqueueEnabled . Then you must enable NetQueue on the adapter module using the following commands:

• esxcfg-module -s "intr_type=2 rx_ring_num=8" s2io (for the Neterion)
• esxcfg-module -s "InterruptType=2,2 VMDQ=16,16" ixgbe (for the Intel)

You must reboot the ESX server for the changes to take effect. To disable NetQueue, remove the line you added to the esx.conf or deselect the VMkernel.Boot.netNetqueueEnabled checkbox. You must also disable NetQueue on the adapter modules by using the following commands:

• esxcfg-module -s "" s2io (for the Neterion)
• esxcfg-module -s "" ixgbe (for the Intel)

I was recently setting up a Capacity Assessment for a client when I ran into an issue with perfmon. Perfmon is used on the physical windows servers to collect performance information. This information is used to help determine the best route (design, hardware specs., etc.) to virtualization for the physical servers. However, upon trying to collect the information on some systems I received an error that the performance counter “object was not found.” The VMware Capacity Planner data manager will tell you that this error is usually due to traffic not getting through a firewall or anti-virus product on the server.

I did some digging and found that the firewall was not the problem. I opened up perfmon directly on the problem servers and found numbers for counters instead of the names. This told me that I had corrupt counters. The fix in this case was really simple. I copied the counters (*.dat files) from a server that the performance metrics were working on (same OS) to the problem servers.

The two files in question are %systemroot%\system32\perfc009.dat and %systemroot%\system32\perfh009.dat.

Once these files were replaced with the ones from the good server, I restarted perfmon and the counters were there. The data collection could continue. There is a chance that you may need to dig a little deeper to restore the counters in your specific scenario. The following Microsoft KB article should help: KB300956

I’m up to section 2 on my VCDX Enterprise Administration exam review. This VCDX exam note deals specifically with Objective 2.1.K.2 (Create and modify virtual switches and virtual switch policies). You can easily create vSwitches and Portgroups using the VI Client. However, you may be in a situation (like on the Enterprise Administration Exam ) that you need to use the command line interface to create vSwitches and Portgroups. That is what this note will cover.

Notes for Objective 2.1.K.2: Create and modify virtual switches and virtual switch policies.

• List information about the pNICs on your ESX host (useful in helping to identify certain pNICs):
  • esxcfg-nics -l
• Create a new vswitch:
  • esxcfg-vswitch -a "vswitch_name" (where vswitch_name is the name of your new vswitch. Usually in the form of vSwitch#).
• Add a pNIC as an uplink to a vSwitch:
  • esxcfg-vswitch -L vmnic# vswitch# (where vmnic# is the chosen vmnic (pNIC) and vswitch# is the chosen vswitch).
• Remove a pNIC from a vSwitch:
  • esxcfg-vswitch -U vmnic# vswitch# (where vmnic# is the chosen vmnic (pNIC) and vswitch# is the chosen vswitch).
• Create a portgroup on a vswitch:
  • esxcfg-vswitch -A "portgroup_name" vswitch# (where portgroup_name is the name of your new portgroup and vswitch# is the chosen vswitch).
• Add a vmkernel NIC to a portgroup:
  • esxcfg-vmknic -a -i x.x.x.x -n x.x.x.x "portgroup_name" (where the first x.x.x.x is the IP Address and the second x.x.x.x is the subnet mask. Also where portgroup_name is the name of your new portgroup.)
• Enable a vmkernel nic:
  • esxcfg-vmknic -e "portgroup_name" (where portgroup_name is the name of your chosen vmkernel portgroup).
• Set the CDP properties of a vswitch:
  • esxcfg-vswitch -B listen|advertise|both (Choose “listen” , “advertise”, or “both” for CDP configuration).

The load-balancing options and other virtual switch policies can be configured through the VI Client. You can also configure them through the command-line (like enabling VMotion on a vmkernel port) using the unsupported Virtual Infrastructure metashell (vimsh), which is largely undocumented by VMware. A good place to start  learning about vimsh is at VI-Toolkit.com.