I guess we had this discussion in the past as well when 32GB became a commodity. The question I always have is how many eggs do you want to have in one basket. Basically do you want to scale up(larger hosts) or scale out(more hosts).

I guess it’s a common discussion and a lot of people don’t see the impact sizing your hosts correctly. Think about this environment, 250 VMs in total with the need of roughly 480GB of memory:

• 10 Hosts, each having 48GB and 8 Cores, 25 VMs each.
• 5 Hosts, each having 96GB and 16 Cores, 50 VMs each.

If  you look at it from an uptime perspective; Would a failure occur in scenario 1 you will lose 10% of your environment. If you look at scenario 2 this is 20%. Clearly the associated cost with the down time for 20% of your estate is higher than for 10% of your estate.

Now it’s not only the associated cost with the impact of a host failure it is also for instance the ability of DRS to load balance the environment. The less hosts you will have the smaller the chances are DRS will be able to balance the load. Keep in mind DRS uses a deviation to calculate the imbalance and simulates a move to see if it results in a balanced cluster.

Another thing to keep in mind is HA. When you design for N+1 redundancy and need to buy an extra host the costs associated for redundancy is high for a scale up scenario. Not only the costs associated are high, the load when the fail-over needs to occur will also increase immense. If you only have 4 hosts and 1 host fails the added load on the 3 hosts will have a higher impact than it would have on for instance 9 hosts in a scale out scenario.

Licensing is another often used argument for buying larger hosts but for VMware it usually will not make a difference. I’m not the “capacity management” or “capacity planning” guru to be honest but I can recommend VMware Capacity Planner as it can help you to easily create several scenarios. (Or Platespin Recon for that matter.) If you have never tried it and are a VMware partner check it out and run the scenarios based on scale up and scale out principles and do the math.

Now, don’t get me wrong I am not saying you should not buy hosts with 96GB but think before you make this decision. Decide what an acceptable risk is and discuss the impact of the risk with your customer(s). As you can imagine for any company there’s a cost associated with down time. Down time for 20% of your estate will have a different financial impact than down time for 10% of your estate and this needs to be weighted against all the pros and cons of scale out vs scale up.

I read the course material this week and I can honestly say that it rocks! Of course I did not expect anything less becauseVMware’s performance guru Scott Drummonds was involved in developing this course. (Scott wrote an article about the training a month ago, you can find it here.) Below you will find a short description.

This hands-on training course explores the management of performance in a VMware vSphere™ environment. It provides the knowledge and skills necessary to make fundamental design decisions that enhance performance and to meet performance goals in an already-deployed vSphere installation. The course is based on VMware® ESX™ 4.0, ESXi 4.0, and vCenter™ Server 4.0.

source pdf

Like I said, heavily recommended for everyone!

I wasn’t aware that the balloon driver was involved with the Mem.IdleTax. I haven’t seen any documentation stating this…and assumed that the VMkernel just stopped mapping idle memory for the VM without letting it know. If the VM needed the memory again, the VMkernel would just re-map it.

I can be totally wrong about this, but I have not seen any documentation to debunk this theory. It is my belief that the Mem.IdleTax is a totally separate memory saving/shaving technique from the balloon driver or the .vswp file.

If VMware engineering has or would publish an official article on this…I think it would clear up alot of things.

To summarize; How does ESX reclaim idle memory or free memory from a virtual machine? The answer is simple. ESX has two idle memory reclamation mechanisms:

1. Balloon driver
2. vSwap

I would like to refer to page 29 of the Resource Management Guide where the above is stated. I do not think it is a coincidence that the paragraph above “memory reclamation” is “Memory Tax for Idle Virtual Machines”. (There is a third memory “reclamation” mechanism by the way, it is called “TPS”, but this is not used to specifically reclaim Idle Memory but rather to free up memory by sharing pages where possible.)

By default the balloon driver is used to reclaim idle memory. The balloon driver is in fact used as some operating systems only update there internal free memory map. Basically what I am saying is that the hypervisor is unaware of the fact that specific pages are unused as they might still contain data and the GOS(Guest Operating System) will not report to the hypervisor that the pages are not being used anymore. The balloon driver is used to notify the GOS that there is a lack of memory.

When the balloon inflates the GOS will first assign all “unused / free” pages to the balloon driver. If this is enough it will stop. If this isn’t enough the OS will decide which pages it will page out  until it reaches its threshold. The pages will need to be written to GOS swap as they might be needed later, they can’t just be reused without storing them somewhere.

I guess this section of the excellent white-paper “Memory Resource Management in VMware ESX Server” by Carl Waldspruger describes what I explained above.

The guest OS decides which particular pages to reclaim and, if necessary, pages them out to its own virtual disk. The balloon driver communicates the physical page number for each allocated page to ESX Server, which may then reclaim the corresponding machine page.

To be absolutely certain I reached out Carl Waldspruger to verify my statements/claims are correct. (Yes they were…)

By the way this concept is also described in the “VMware vSphere: Manage for Performance” course manual on page 151. Excellent course which I can recommend to everyone as it will not only explain this concept but also how to identify it and how to resolve it.

As you can imagine the C:\ partition is not the ideal place for storing log files. I would personally recommend to use a separate drive for logfiles so avoid it from flooding any OS or Program related drives. You could pick a small size based on the expected log size and if needed increase the amount of logs that are stored and the size of the log file.

Changing this is pretty simple. Open “vpxd.cfg” and add the following line in between <log> and </log>

<directory>D:\VMware\Logs</directory>

Changing the amount of log files stored and the size is also pretty basic, in this example vCenter will store 10 logfiles which are max 10MB each:

<maxFileSize>10485760</maxFileSize>
<maxFileNum>10</maxFileNum>

Keep in mind that you will need to restart the vCenter Service after these changes before they take effect!

Did you know that there are two similar VM event triggers in Alarms in vCenter?

1. VM powered on
2. DRS – VM powered on

The first only works for VMs outside of DRS enabled clusters. The second only works for VMs inside DRS enabled clusters. Now that’s definitely something you should be aware off when enabling Alarms / Event triggers. Imagine you want to know when a VM has been powered on and you enable the first even trigger but didn’t notice it will only sent an alarm when the VMs are not part of DRS cluster… You could be waiting for a very long time before you receive a single event alarm.

Just when I wanted to click “Publish” I received an email from one of my colleagues. Horst Mundt wrote an excellent article about Alarms and created a very handy spreadsheet which contains all alarms / events.

vSphere alarm triggers
In terms of alarms, vCenter 4 has much more to offer than vCenter 2.5. There is a whole range of default alarms available when you install vCenter 4, and they will give you a very good first shot for monitoring your vSphere  environment. If you’ve never wondered what exactly the default alarms mean, or how to tune them – that’s fine. If you’re interested in a bit more detail – read the attached PDF.

• vSphere Alarms v2.xlsx (69.3 K)
• Fun with vSphere Alarms.pdf (656.6 K)

Make sure to visit the VMTN source page and leave a comment or rate the article.

Default install with no redundancy:

VM Network inherits from vSwitch:

Management Network does not inherit from vSwitch:

For the default “VM Network” portgroup everything works as expected. But for the “Management Network” it doesn’t. So what’s the problem? Well it might not be a huge issue but it is something you will need to keep in mind. I wanted to add two NICs to my vSwitch0 and expected that both would be marked as “active” on the vSwitch. And this is what happens on the vSwitch, BUT the “Management Network” does not inherit the vSwitch settings so what do you think will happen? Again see the screenshot below for the details:

For some weird reason one of the vmnics is set to “unused” instead of active… Keep this in mind when installing / configuring ESXi as you might end up with less redundancy then expected. I just did a quick search if it was a known/documented change and it appears that I am not the only one who ran into this, but is does not seem to be a commonly known “issue”/change.

Single initiator zoning is something we have always recommend in the field (VMware PSO Consultants/Architects) and something that is clearly mentioned in our documentation… at least that’s what I thought.

On page 31 of the SAN Design and Deploy guide we clearly state the following:

When a SAN is configured using zoning, the devices outside a zone are not visible to the devices inside the zone. When there is one HBA or initiator to a single storage processor port or target zone, it is commonly referred to as single zone. This type of single zoning protects devices within a zone from fabric notifications, such as Registered State Change Notification (RSCN) changes from other zones. In addition, SAN traffic within each zone is isolated from the other zones. Thus, using single zone is a common industry practice.

That’s crystal clear isn’t it? Unfortunately there’s another document floating around which is called “Fibre Channel SAN Configuration Guide” and this document states the following on page 36:

• ESX Server hosts that use shared storage for virtual machine failover or load balancing must be in one zone.
• If you have a very large deployment, you might need to create separate zones for different areas of functionality. For example, you can separate accounting from human resources.

So which one is correct and which one isn’t? I don’t want any confusion around this. The first document, the SAN Design and Deploy guide is correct. VMware recommends single initiator zoning. Of course if you want to do “single initiator / single target” that would even be better, but single initiator is the bare minimum. Now let’s hope the VMware Tech Writers can get that document fixed…

One would expect that both a CPU and Memory reservation would have the same behavior when it comes to claiming and releasing resources but unfortunately this is not the case. Or should we say fortunately?

The following is taken from the resource management guide:

CPU Reservation:
Consider a virtual machine with reservation=2GHz that is
totally idle. It has 2GHz reserved, but it is not using any of
its reservation. Other virtual machines cannot reserve these 2GHz. Other virtual machines can use these 2GHz, that is, idle
CPU reservations are not wasted.

Memory Reservation:
If a virtual machine has a memory reservation but has not yet accessed its full reservation, the unused memory can be reallocated to other virtual machines. After a virtual machine has accessed its full reservation, ESX Server allows the virtual machine to retain this much memory, and will not reclaim it, even if the virtual machine becomes idle and stops accessing memory.

The above paragraph is a bit misleading , as it seems to imply that a VM has to access its full reservation. What it should really say is “Memory which is protected by a reservation will not be reclaimed by ballooning or Host-level swapping even if it becomes idle,” and “Physical machine memory will not be allocated to the VM until the VM accesses virtual RAM needing physical RAM backing.” Then that pRAM is protected by the reservation and won’t be reclaimed by ballooning or .vswp-file swapping. If there is any .vswp memory at all as no .vswp is created when the reservation is equal to the provisioned memory.

Note, however, that even if pRAM is not allocated to the VM to back vRAM because the VM hasn’t accessed corresponding vRAM yet, the whole reservation is reserved, but the pRAM could still be used  This gets really confusing. But I think of it thus:

1. Reservations can be defined at the VM level or the Resource Pool level.
2. Reservations at the RP level are activated or reserved immediately.
3. Reservations at the VM level are activated or reserved when the VM is powered on.
4. An activated reservation is removed from the total physical Resource “Unreserved” accounting.
5. Reserving and using a resource are distinct: memory or CPU can be reserved but not used or used but not reserved.
6. CPU reservations are friendly.
7. Memory reservations are greedy and hoard memory.
8. Memory reservations are activated at startup, yet pRAM is only allocated as needed. Unallocated pRAM may be used by others.
9. Once pRAM is protected by a memory reservation, it will never be reclaimed by ballooning of .vswp-swapping even if the corresponding vRAM is idle.

Example: A VM has 4 GB of vRAM installed and a 3 GB memory reservation defined. When the VM starts, 3 GB of pRAM are reserved. If the host had 32 GB of RAM installed and no reservations active, it now has 29 GB “unreserved”.

However, if the VM accesses only 500 MB of vRAM, only 500 MB of pRAM are allocated (or granted) to it. Other VMs could use 2500 MB of RAM that you would think is part of the reservation. They cannot reserve that 2500 MB however. As soon as the VM accesses 3 GB of vRAM and so has 3 GB of pRAM backing it, no other VMs can use that 3 GB of pRAM even if the VM never touches it again, because that pRAM is now protected by the 3 GB Reservation.  If the VM uses 4 GB, it gets the 3 GB guaranteed never ballooned or swapped, but the remaining 1 GB is subject to ballooning or swapping.

Simple huh

VMware’s long-term prioritization for managing the most aggressively over-committed memory looks like this:

1. Do not swap if possible.  We will continue to leverage transparent page sharing and ballooning to make swapping a last resort.
2. Use ODMC to a predefined cache to decrease memory utilization.*
3. Swap to persistent memory (SSD) installed locally in the server.**
4. Swap to the array, which may benefit from installed SSDs.

(*) Demonstrated in the lab and coming in a future product.
(**) Part of our vision and not yet demonstrated.

I just love it when we give insights in upcoming features but I am not sure I agree with the prioritization. I think there are several things that one needs to keep in mind. In other words there’s a cost associated to these decisions / features and your design needs to adjusted to these associated effects.

1. TPS -> Although TPS is an amazing way of reducing the memory footprint you will need to figure out what the ratio of deduplication is. Especially when you are using Nehalem processors there’s a serious decrease. The reasons for the decrease of TPS effectiveness are the following:
   • NUMA – By default there is no inter node transparent page sharing (read Frank’s article for more info on this topic)
   • Large Pages – By default TPS does not share large(2MB) pages. TPS only shares small(4KB) pages. It will break large pages down in small pages when memory is scarce but it is definitely something you need to be aware off. (for more info read my article on this topic.
2. Use ODMC -> I haven’t tested with ODMC yet and I don’t know what the associated cost is at the moment.
3. Swap on local SSD -> Swap on local SSD will most definitely improve the speed when swapping occurs. However as Frank already described in his article there is an associated cost:
   • Disk space – You will need to make sure you will have enough disk space available to power on VMs or migrate VMs as these swap files will be created at power on or at migration.
   • Defaults – By default .vswp files are stored in the same folder as the .vmx. Changing this needs to be documented and taken into account during upgrades and design changes.
4. Swap to array (SSD) -> This is the option that most customers use for the simple reason that it doesn’t require a local SSD disk. There are no changes needed to enable it and it’s easier to increase a SAN volume than it is to increase a local disk when needed. The associated costs however are:
   • Costs – Shared storage is relatively expensive compared to local disks
   • Defaults – If .vswp files need to be SSD based you will need to separate the .vswp from the rest of the VMs and created dedicated shared SSD volumes.

I fully agree with Scott that it’s an exciting feature and I can’t wait for it to be available. Keep in mind though that there is a trade off for every decision you make and that the result of a decision might not always end up as you expected it would. Even though Scott’s list makes totally sense there is more than  meets the eye.

When Tech Support is disabled it removes the option to login to the console with “unsupported“. Please keep in mind that the console is the only way to get direct command line access to ESXi as SSH is disabled by default. This also means that in order to get access to the console you will need access to the physical host, or the IP KVM switch / DRAC / ILO for that matter. Hosts are usually located in a secured environment which removes the need for limiting console access in my opinion.

I can still imagine that people have a different opinion, but if you look at it from a support perspective you might change your mind. Troubleshooting an issue can get really complicated when there is no Tech Support access. I guess in a high secure environment you could treat ESXi as a stateless appliance and just install a new version when it fails. Personally I would prefer to find the root cause and try to prevent the same problem from occurring again.

Of course you can enable Tech Support again when needed but a reboot is required. This might cause the symptoms of the problem you were facing to disappear. It’s my recommendation to Keep Tech Support enabled.

[edit] Of course Alan “the king of powershell” Renouf jumped on this topic immediately and created a couple of lines of script which show you the current setting, disable it for all hosts or enable it for all hosts. Thanks Alan! [/edit]