Virtual John

Update Manager

2009-10-08T11:45:00.000-04:00

Here is an extremely useful tip if you are attempting to update ESX hosts in a remote datacenter.
I found it nearly impossible to update the hosts in a well connected datacenter. I even tried one update at a time but still was getting failures.

This occurs around the default timeout for tasks of fifteen minutes in Virtual Center.

Increasing this timeout gives the patches time to apply and copy over the WAN. It does have to be changed in the vpxd.cfg of Virtual Center and vpxa.cfg of the ESX hosts.

<task>
<timeout>10800</timeout>
</task>
<vmomi>
<soapstubadapter>
<blockingtimeoutseconds>10800</blockingtimeoutseconds>
</soapstubadapter>
</vmomi>

Fix:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009670

I found some references to editing the vci-integrity.xml recvTimeout and UpdateDownloadReries. However this is incorrect, these settings control the download of the updates from the internet, not the download of the updates to the ESX host.

VMWorld 2009

2009-09-01T18:03:00.002-04:00

Out in San Francisco for VMWorld 2009. Great conference so far. Random observations. VDI Classes are easier to get into. It seems like people are using VDI now but why the low interest? BYOI: The WiFi is slow..... Bring your own aircard. Time to get deep into vSphere. Not enough time in the day or night for the sessions, dinners, and parties. Sessions that are 100% statistics are boring. Way too much advertising on #vmware on Twitter. Everyone loves a flying monkey.

Links for 2009-04-13 [del.icio.us]

2009-04-14T00:00:00-07:00

UK SharePoint Team : Virtualizing SharePoint Series – Introduction

Links for 2009-01-30 [del.icio.us]

2009-01-31T00:00:00-08:00

http://www.npr.org/blogs/allsongs/2009/01/songsmith_brilliant_and_awful_1.html

Links for 2008-10-29 [del.icio.us]

2008-10-30T00:00:00-07:00

Links for 2008-10-28 [del.icio.us]

2008-10-29T00:00:00-07:00

Setting Video Hardware Acceleration Level | PeetersOnline.nl

Links for 2008-09-03 [del.icio.us]

2008-09-04T00:00:00-07:00

VI:OPS: Monitoring ESX Server on Microsoft System Center Operations Manager 2007

Spanning Tree and BPDU Guard

2008-08-08T11:46:00.004-04:00

One of our engineers accidentally "bridged" the interfaces in windows. This is done by selecting multiple interfaces and picking "Bridge". This triggered the PortFast Bridge Protocol Data Unit (BPDU) guard feature when the switch saw BPDU traffic on the port. Which disabled all of the ports (backup and production) on the host to isolate and protect the greater network infrastructure. http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a008009482f.shtml This is a good safety mechanism and isolation response but something that may not be obvious at first if you encounter this issue. We were able to correct this by shutting down and disconnecting the network ports of the offending VM through the service console and bringing the ports back up on the Cisco swtich. We still did have link connectivity issues and ended up having to reboot the ESX host. At this point we got all interfaces back and were able to bring up the offending VM, disconnected from any network, delete the bridge, and reconnect to the network with no further issues. Hope this helps if anyone else runs into it.

DRS Performance and Best Practices

2008-07-31T09:18:00.004-04:00

VMWare seems to have seriously ramped up their documentation machine in a wonderful way. I'm seeing more and more valuable papers posted. http://www.vmware.com/files/pdf/drs_performance_best_practices_wp.pdf There are some interesting conclusions in the document.

When deciding which hosts to group into a DRS cluster, try to choose hosts that are as homogeneous as possible in CPU and memory. This seems to be a bit of a no brainer in any kind of clustered environment, but even more important in DRS where all nodes of the cluster are active partners, even though DRS will account for CPU/Memory size differences.
When more ESX hosts in a DRS cluster are VMotion compatible, DRS has more choices to better balance workloads across the cluster. We recommend clusters of up to 32 hosts. I'm not sure I buy 32 hosts yet. There is overhead involved in the DRS calculations even though they only take place every 5 minutes by default. The limit in a cluster was just recently raised from 16 to 32. There is a definite advantage to larger cluster size though as can be seen in the following chart. One may also wonder why they only tested up to 16 rather than 32 nodes of the cluster. Of course that could just be a lab limitation though as the cap was just lifted in u1 or u2 of 3.5.
The default migration threshold (moderate) works for most configurations. You can set the migration threshold to more aggressive levels when all of the following conditions are satisfied:
- The hosts in the cluster are relatively homogeneous.
- The virtual machines’ resource utilization remains fairly constant.
- The cluster has relatively few constraints on where a virtual machine can be placed
You should set the migration threshold to more conservative levels when the converse is true.
The default DRS frequency is once every five minutes, but you can set it to any period between one and 60 minutes. You should avoid changing the default value. This should never be less than 5 minutes and make sure you adjust aggressiveness of automation before even considering this setting.
In general, do not specify affinity rules unless you have a specific need to do so. In some cases, however, specifying affinity rules can improve performance.
- Keeping virtual machines together can improve performance if the virtual machines need to communicate with each other, because network communication between virtual machines on the same host enjoys lower latencies.
- Separating virtual machines maintains maximal availability of the virtual machines.
- Virtual machines that might need to be separated is virtual machines with I/O‐intensive workloads. If they share a single host, they might saturate the host’s I/O capacity, leading to performance degradation. DRS does not make virtual machine placement decisions based on their usage of I/O resources.
Very good point on the I/O placement. When will I/O be taken into consideration by DRS? Storage I/O is being addressed by VMWare well around proving throughput. But balancing that throughput is still non existent.
Assign resource allocations to virtual machines and resource pools carefully. Be mindful of the impact of limits, reservations and virtual machine memory overhead. Make sure you understand "slices" if you are working with DRS clusters and how they impact admission control. This has been changed made more conservative between 3.0.x & 3.5.x. Always considerer expandable reservations if you need to use reservations.
Virtual machines with smaller memory sizes or fewer virtual CPUs provide more opportunities for DRS to migrate them in order to improve balance across the cluster. Virtual machines with larger memory size or more virtual CPUs add more constraints in migrating the virtual machines. Hence you should configure only as many virtual CPUs and as much memory for a virtual machine as needed. One of the golden rules in VMWare no matter if using DRS or not.
You can specify DRS modes of automatic, manual, or partially automated at the cluster level as well as the virtual machine level. We recommend that you keep the cluster in automatic mode. DRS is now a mature and stable process. I've yet to see anyone that starts in "manual" or "partially automated" stay in that mode.

Kudos to VMWare for another quality white paper.

PlateSpin PowerRecon: Long Term Impressions

2008-07-15T11:37:00.002-04:00

I've been using PowerRecon for about six months now and thought I'd share my long term impressions. Technically, this is a wonderful product and does really do everything it's advertised too. I would hands down recommend this product over the VRA offered by VMWare and their VARs. However, the support is VERY lacking from PlateSpin. Their support isn't advertised as callback support but that's really what it is. Try calling the 800 number and talking to someone live. 9 out of 10 times you will just end up in a voice mail box somewhere. I also have used their support several times thorough email and really didn't get any headway until asking to be contacted over the phone. Once on the phone their support engineers are very capable and engaged. One would have hoped the Novell acquisition would have brought more money to build out a support organization, but at this point it is severely lacking. Bottom line: Wonderful product, woeful support.

Links for 2008-07-14 [del.icio.us]

2008-07-15T00:00:00-07:00

nLite - Deployment Tool for the bootable Unattended Windows installation - About

Links for 2008-06-11 [del.icio.us]

2008-06-12T00:00:00-07:00

ESX Storage Architecture

2008-04-29T09:08:00.005-04:00

So something else interesting from yesterday is it sounds like big changes are coming to ESX storage architecture in the back end. It's still a ways off but it could finally mean Power Path coming to ESX as well as a lot of other storage tools being able to be written and work with Virtual Infrastructure. Power Path has been a major pain point for us. Our storage team requires it on all SAN connected hosts. We worked around this from some rather expensive script writing by EMC. This really seems like a script that EMC should provide customers work around the Power Path not working issue. Issue: When a Storage Processor event occurs such as a flare code upgrade or SP failure/replacement happens, all the load is pushed to the remaining SP. In a Power Path connected world, Power Path manages the trespass and moves the LUN back to the preferred SP. So at this point all ESX hosts are forced to one SP. Solution: We engaged EMC to write a perl script that works of the navi management server. This enumerates all the VMWare LUNS and checks the "default SP" If the current owner is not the default SP, the script issues a trespass and moves the LUN to the correct SP. Seems simple enough. But by the time we included all the features we wanted the functional spec was two pages long. Some examples are exit codes, configuration files, interactive/silent mode, etc...

SAN Pathing on an Active/Passive Array Part II

2008-04-28T17:39:00.003-04:00

In a previous post I spoke of the issues around SAN Pathing on an Active/Passive Array. Today I met with VMWare EMC and our internal storage team to discuss this. I was slightly incorrect on how ESX decides where to assign a path. ESX issues a SCSI inq command and the first path to respond is used. So, because the first card to come up issues the command first, it usually takes the path. But depending on load of the array and fabric, the other card could be used. In talking through the issue though with everyone we came to a couple conclusions.

This can be done.

Build a table of the wwn from both SP.
Determine what SP a LUN is on from the wwn of the SP.
Determine what path on the second HBA is on the same SP.
Check if active path is already on preferred HBA (every other LUN alternates HBA)
If not on preferred HBA disable all but active path and path preferred to move to, the path being the ON but not ON ACTIVE path is not enough.
Disable ON ACTIVE path to move to other HBA.
Re-enable all paths.
Continue though all LUNS

In most environments the effort is not worth the benefit though.

So, we are going to keep an eye on HBA utilization and see if a bottleneck appears. At that point we may pursue the script above.

Optimum LUN Size for ESX Clusters

2008-04-16T15:11:00.003-04:00

I'm curious what other organizations are using as LUN sizes? We have standardized on 250G LUNs. This seems to be what a lot of "best practices" kind of guides suggest and keeps I/O to an individual LUN manageable and tends to result in 6-8 VM's per LUN.

HA = High Availability?

2008-04-16T08:29:00.003-04:00

So, something that has been bothering me with VMWare marketing for some time is HA. What is "High Availability"? It's a pretty subjective term. Wiki defines it as: High availability is a system design protocol and associated implementation that ensures a certain absolute degree of operational continuity during a given measurement period. However look at the Google definitions and you'll find some more and some less restrictive definitions. What is VMWare HA? Let's start with what HA is not, Continuous Availability. I think a lot of times CA is the perception non engineers have of what VMWare HA provides. What HA provides is a method in which all hosts in a cluster continuously monitor for the best way to restart all of the VMs on a given host, if that host fails or the VMs become isolated. In plain English this means, if one of your hosts in a cluster of VMWare Servers goes away the VMs will reboot elsewhere. Reboot = downtime, so is this high availability? Or just higher availability than no fault tolerance? How do you define High Availability?

Virtual Center Upgrade - Part 2

2008-04-15T17:47:00.003-04:00

I had to simplify this when executing on multiple machines, so here is what I came up with. ps -auxwww | grep hostd |awk '{print "kill -9 " $2}' | sh service mgmt-vmware start service vmware-vpxa restart

Virtual Center Upgrade

2008-04-15T16:12:00.003-04:00

I recently upgraded from Virtual Center 2.0.1 to 2.0.2. Now a week later have found the first real issue while attempting to import a VM from VMWare Converter. The import would hang at 1%, if you viewed the job in Virtual Center it would show creating the VM. If you connected directly to the ESX host with the VI client, you would see the VM had completed creation. Basically the communication looked like it would get to the ESX host but not come back to Virtual Center. I was able to run convert directly to the ESX host and start the VM. But none of this translated into Virtual Center. To solve this we had to restart the management services. Due to some bugs in the hostd service you have to kill the process rather than restarting it. There is a known issue in some versions of ESX that when you restart the hostd process (service vmware-hostd restart) any VMs set to shutdown on a reboot the host, may shut down. So first we have to find the pid's of the hostd services. [root@xxxxx24 root]# ps -auxwww | grep hostd root 1933 0.0 0.0 4260 4 ? S Jan30 0:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -q 5 -c /usr/sbin/hostd-support /usr/sbin/vmware-hostd -u root 1939 0.4 11.5 58636 31060 ? S Jan30 462:02 /usr/lib/vmware/hostd/vmware-hostd /etc/vmware/hostd/config.xml -u root 21729 0.0 0.2 3696 664 pts/1 S 16:04 0:00 grep hostd [root@xxxxx24 root]# Then kill the two hostd services. [root@xxxxx24 root]# kill -9 1933 1939 Now we can start mgmt-vmware and restart the vpxa service [root@xxxxxx24 root]# service mgmt-vmware start Starting VMware ESX Server Management services: VMware ESX Server Host Agent (background) [ OK ] Availability report startup (background) [ OK ] [root@ivpesx24 root]# service vmware-vpxa restart Stopping vmware-vpxa: [ OK ] Starting vmware-vpxa: [ OK ] [root@xxxxxx24 root]# This causes the host to disconnect from Virtual Center for a bit and when it reconnects everything synced up correctly. One caveat to this. In ESX 3.5, there are some logfiles of the pids stored in /var/run that may need to be cleaned out before starting the mgmt-vmware service.

Cisco Networking

2008-04-15T13:58:00.002-04:00

I found this through the RTFM website. Always a quality spot for reference. At the end of the session, the participants should be able to:

Objective 1: Understand key concepts of server virtualization architectures as they relate to the network.
Objective 2: Explain the impact of server virtualization on DC network design (Ethernet & Fiber Channel)
Objective 3: Design Cisco DC networks to support server virtualization environments

http://www.cisco.com/web/BE/learn_events/pdfs/Server_Virtualization.pdf

Healthy Console Reminder

2008-04-15T13:35:00.001-04:00

It’s easy to get spoiled these days. Most everything can be installed over RDP without a problem. But I read this today and was of reminded the importance of the console (even using /console for RDP). Thought it would be good information to share. Every couple of months or so, we'll get a call from an Administrator reporting that his system hung when he tried to reboot it after installing patches. More often than not, the patches were installed by an Administrator logging on to the server via Remote Desktop (without using the /console or /admin switch) and using either the Windows Update web site or the "Automatic Updates" tray icon. Both methods do the same thing and use the same processes. After the updates are installed, the Administrator clicks on the "Restart Now" button to complete the installation. The Remote Desktop Session goes away, and the Administrator thinks that the server is in the process of rebooting. However, the problem is that the server may not really be rebooting. When the Administrator tries to connect back into the server via RDP after several minutes, he discovers that he cannot. When he logs on at the console of the machine to investigate, he discovers that the RDP Listener is listening on port 3389 but no-one can connect via RDP. To resolve the issue, he has to reboot the server from the console. So what happened? http://blogs.technet.com/askperf/archive/2008/03/18/hotfix-installs-remote-desktop-and-the-reboot-that-wasn-t.aspx

Tools I Use - PowerRecon (3.1) - Planning Edition

2008-04-14T18:02:00.006-04:00

I have been through a "VMware Virtualization Assessment" from VMWare and frankly wasn't that impressed. We had some very specific questions around capacity planning and Disk I/O that were not captured. I can see the value in this offering for a shop trying to get into virtualization, but in a mature IT shop with very specific questions not, general "fit" it didn't work well. So, when it came time to virtualize one of our DataCenters from an acquisition, another tool was in order. We had around 50-60 workloads to consolidate and a Clariion CX3/40 already on site. So, our big questions were: How many hosts and how fast of disk? Something I've found with many of the third party management tools around virutalization is a certain lack of maturity. PowerRecon was no different. In planning I had created a database on a shared SQL2005 server and an account with dbo rights to only the database I wanted to use. But, this didn't work. PowerRecon wants full rights on the SQL server to create it's own databases, you can scale back those rights later though. The inventory went fairly well with only one email/call to PlateSpin support for a group of problem servers. This was fixed by changing the credentials that the service started as to a Domain Admin account. Once again, a bit lacking in maturity of a product to only grant appropriate security to part that need it. You can use alternate credentials to connect/inventory/monitor, but this didn't seem to work perfectly. Once you get through these steps the rest is easy. Well it's easy if your the patient type which I'm not. Even though I really needed 20-30 days of monitoring to see some trends, I still wanted to peek at the results daily. It's pretty interesting to see the data fill in. Reporting was pretty straight forward and I customized some of the reports with ease to export and plan disk sizing. The two questions were answered very well in number of host, memory was the constraint. In disk, there were 2 very clear levels. Our SQL & File Servers were heavy on I/O and everything else was not. So FC for the SQL & File Servers and SATA for everything else. One thing that could be improved for the capacity planning tool would be the ability to plan around IOPS not just MB/Sec. I was able to get the information I wanted in IOPS from the reports, but could only run scenarios based on MB/Sec. Would I use this product again? Certainly so. I purchased enough monitoring days to have a pad to use on other virtualization projects around the enterprise. Would I buy this product in the PowerRecon Standard Edition with Planning packaging? I don't know. The price point on it seems comparable with the other products on the market that are generating a lot of buzz. I'm also currently evaluating vCharter Pro from Vizioncore and Capacity Bottleneck Analyzer Virtual Appliance from VKernel.

Tools I Use - Veeam

2008-04-14T15:44:00.003-04:00

Veeam seems to have snuck into everyones shop with FastSCP a free SCP client that is FAST for ESX. They have branched out into backup and monitoring as well. ESX Stencils - Everyone loves a pretty picture in Visio RootAccess - Not to allow RootAccess but to add local user accounts FastSCP - Great SCP utility for copying files to ESX and can elevate to root. One caveat, if you have ESX hosts behind a firewall, FastSCP uses high ports and can not be fixed to the SCP port (22). I use FileZilla for these. ESXDiag - Good troubleshooting tool for random errors, or "double checking" a server before deploying.

Unity...

2008-04-14T15:28:00.006-04:00

At my company our desktop standard is Windows XP. For some time I've been running the "corporate desktop" within a VM. I wanted to leverage the a 64bit OS for the host OS. I selected Fedora 64bit (FC5, 6, now 7) as the host as we are also a big RHEL shop. My background is more Windows so the experience would be good for ESX and RHEL. I recently upgraded to the beta version of 6.5 and though the unity interface is still a bit buggy, boy is it cool. It really does blur the presentation layer. The biggest caviat I've found with the beta version of 6.5 is speed. VMWare enables full debug logging on all beta releases. So it's significantly slower than 6.0 is. I have no doubt this will change in the full release though.

SAN Pathing on an Active/Passive Array

2008-04-14T13:59:00.003-04:00

So the best practice pathing on an Active/Passive array is MRU (Most Recently Used). This keeps the host from trying to use more than one SP and only failing The command to observe the startup behavior of paths in ESX 3.0.x is "esxcfg-mpath -l" /dev/sda is the local VMFS and the other path is the CDROM. Next you will see the 4 LUNS. These are on two separate arrays. One CX600 and one CX3/80. [root@xxxxx35 root]# esxcfg-mpath -l Disk vmhba0:0:0 /dev/sda (139392MB) has 1 paths and policy of Fixed Local 2:14.0 vmhba0:0:0 On active preferred Enclosure vmhba0:264:0 (0MB) has 1 paths and policy of Fixed Local 2:14.0 vmhba0:264:0 On active preferred Disk vmhba1:0:0 /dev/sdb (256000MB) has 4 paths and policy of Most Recently Used FC 15:0.0 2100001b3209abad<->500601601060225b vmhba1:0:0 On active preferred FC 15:0.0 2100001b3209abad<->5006016a1060225b vmhba1:1:0 Standby FC 17:0.0 2100001b32091db5<->500601681060225b vmhba2:0:0 Standby FC 17:0.0 2100001b32091db5<->500601621060225b vmhba2:1:0 On Disk vmhba1:0:1 /dev/sdc (256000MB) has 4 paths and policy of Most Recently Used FC 15:0.0 2100001b3209abad<->500601601060225b vmhba1:0:1 On active preferred FC 15:0.0 2100001b3209abad<->5006016a1060225b vmhba1:1:1 Standby FC 17:0.0 2100001b32091db5<->500601681060225b vmhba2:0:1 Standby FC 17:0.0 2100001b32091db5<->500601621060225b vmhba2:1:1 On Disk vmhba1:2:0 /dev/sdd (256000MB) has 4 paths and policy of Most Recently Used FC 15:0.0 2100001b3209abad<->5006016a39a02964 vmhba1:2:0 On active preferred FC 15:0.0 2100001b3209abad<->5006016339a02964 vmhba1:3:0 Standby FC 17:0.0 2100001b32091db5<->5006016239a02964 vmhba2:2:0 Standby FC 17:0.0 2100001b32091db5<->5006016b39a02964 vmhba2:3:0 On Disk vmhba1:2:1 /dev/sde (256000MB) has 4 paths and policy of Most Recently Used FC 15:0.0 2100001b3209abad<->5006016a39a02964 vmhba1:2:1 On active preferred FC 15:0.0 2100001b3209abad<->5006016339a02964 vmhba1:3:1 Standby FC 17:0.0 2100001b32091db5<->5006016239a02964 vmhba2:2:1 Standby FC 17:0.0 2100001b32091db5<->5006016b39a02964 vmhba2:3:1 On [root@xxxxxx35 root]# You can see how ESX loads all the paths to the first hba, in this case vmhba1 (On Active). The array has a preferred SP so the array picks which Storage Processor, and the ESX host picks which hba. This will eventually create a bottleneck on vmhba1(or first HBA). To solve this, I'm working with VMWare on the best way to balance the paths. There will be no intelligence to the balancing based on load, but I would like to put half of the LUNs onto the second HBA. To do this I'm looking at a startup script that would disable the standby path on the first HBA and then disable the On Active path on the first HBA. This would force the path to fail to the second HBA, and allow the path to stay on the preferred Storage Processor. If the On Active path is failed first, this could cause a LUN trespass and move the LUN (and all other ESX hosts connected to the LUN) to the second Storage Processor. After a pause both paths would be re-enabled. I have proposed this to our VMWare/EMC TAM's and am waiting on a responses.

Day 1

2008-04-14T11:20:00.000-04:00

So, I've been working with VMWare in some form since about 98 or 99, and quite extensively over the last 3 years or so. I decided it was time to create a place to paste my notes and thoughts and lessons learned. Maybe someone else will find it useful, maybe I'll just refer back to it for reference. A quick who am I? I am a Senior Engineer in a company with 200+ ESX hosts and on the cross department design team for the architecture. I am personally responsible for 50+ ESX hosts with from stand alone to SAN connected clusters.