VMware Front Experience

ESXi 6.5 Release Notes for free license and white box users

2016-11-24T07:34:00.000+01:00

VMware has made vSphere 6.5 generally available on November 15th, and this includes the core hypervisor platform ESXi 6.5. In the What's New and Release Notes documents you can read all about the exciting new features, but most of them require paid licenses and a vCenter server to centrally manage your ESXi hosts.

This blog post is my first try to explain what free license users and owners of unsupported ("white box") hardware need to know when they are going for an upgrade or installation of ESXi 6.5 ...

Read more »

[Update] ESXi-Customizer-PS 2.5 - The ESXi image customization script

2016-11-23T14:02:00.002+01:00

This is just a quick heads up for the users of my ESXi-Customizer-PS script for ESXi image builds and customizations. I have just published the new version 2.5 of the script, and that adds

full support of vSphere 6.5 (use the new -v65 switch to limit Image profile selection)
support of PowerCLI 6.5
enhanced logging capabilities

If you have run into issues with older versions of the script then please update now and let me know how it goes.

In the meantime VMware has also published the ESXi 6.5 GA bits to its Online Depot, and that means it is very easy to create an ESXi 6.5 installation ISO or Offline bundle with my script. Just run it without any arguments for the ISO file (or only the -ozip switch for the Offline Bundle), and it will do its magic.

Have fun!

This post first appeared on the VMware Front Experience Blog and was written by Andreas Peetz. Follow him on Twitter to keep up to date with what he posts.

How to deploy Windows Nano Server (TP5) on vSphere

2016-07-15T15:48:00.004+02:00

With the upcoming release of Windows Server 2016 Microsoft will introduce a new edition of its flagship Operating System: Windows Nano Server is a super lightweight version of Windows Server that was "built for the Cloud". When I first heard of it I immediately became curious, because it resembles VMware's ESXi in many ways: It is an embedded-like and headless system with a very small footprint and can be managed only remotely. It has limited use cases, but two of them are acting as a fully-featured Hyper-V virtualization host and as a container host.

When looking for a good way to deploy Nano Server on vSphere I found some guides, but they were all referring to older Tech Preview (TP) versions of Server 2016 (things have changed in the current TP5) and/or did not result in optimal state-of-the art configurations. So I decided to write up my own guide, and here it is ...

Read more »

Active Directory issues with ESXi 6.0 Update 2 and an automated fix

2016-06-07T13:28:00.000+02:00

Lately VMware published two new Knowledge Base (KB) articles that should alarm all people using Active Directory (AD) authentication with their ESXi 6.0 hosts:

Let's have a closer look at these articles and make sure that you draw the right conclusions.

Using haproxy as a PSC load balancer

2016-04-26T15:27:00.003+02:00

When designing a vSphere 6.0 environment with multiple vCenter servers you will - in most cases - end up with the need to deploy external Platform Services Controllers (PSCs). If you are unsure what topology to choose then you should take a look at the PSC Topology Decision Tree that was recently published by VMware. It will guide you to the topology that suits your requirements best.

Since the PSC hosts the critical Single-Sign-On (SSO) component a specific requirement is to make an external PSC highly available so that you are still able to log on to vCenter even if one PSC fails. Currently the only supported way to implement a seamless automatic failover from a failing PSC to another one is to put multiple PSCs (of the same SSO domain and site) behind a load balancer. The process of properly configuring the load balancer and the vCenter servers behind it is quite complex, so most people refrain from it and just deploy a secondary PSC to that they manually re-point the vCenter servers if the primary one should fail (as per KB2113917). But this is a manual process (although it can of course be automated as William Lam explained in this post) and it takes a restart of all vCenter services during which vCenter will be unavailable.

This is why I wanted to try out in the lab how complicated it really is to implement load balanced PSCs and how well they work. However, I did not have a supported load balancer available in the lab - currently only Citrix Netscaler, the F5 BIG-IP and VMware's own NSX-v are officially supported for vSphere 6.0. All quite expensive options and no quick and easy deployments. So I decided to try my luck with the standard Open Source load balancer: haproxy. It turned out that this works very well and can be implemented quite quickly. Here is how:

Read more »

Re-pointing vCenter Server 5.5: A Survival Guide to KB2033620

2016-03-30T16:06:00.000+02:00

vSphere 6.0 has been around for about a year now, but VMware's largest customers are usually one or even two versions behind. With the recent release of Update 2 it looks like the 6.0 version has gained the stability and maturity that enterprise customers are waiting for. This is the reason why I just did extensive testing of vSphere 5.5 to 6.0 upgrades in the lab.

The main challenge of such an upgrade is to transform your vCenter Single Sign-On (SSO) setup into a topology that is fully supported and not deprecated. With vSphere 6.0 the SSO component is now part of the new Platform Services Controller (PSC) role that can be separated from the remaining vCenter services. In fact VMware recommends doing this whenever you want to have two or more vCenter servers in the same SSO domain which is a prerequisite for the new Enhanced Linked mode. The separation of SSO was already possible with vSphere 5.5 (although only with the Windows version of vCenter, but not the VCSA 5.5), but I think most people wanted to keep it simple and installed all vCenter services on the same server. So if they have multiple vCenter servers installed in this way then they need to switch to one or more external PSCs now.

There are many ways and orders in that you can - on the one hand - upgrade all components to 6.0 and - on the other hand - switch to an external PSC/SSO model. But only few of them are documented and supported by VMware. Their general recommendation is to transform into a supported topology first, and then do the upgrade of the PSCs and vCenter servers. KB2130433 e.g. describes how to upgrade/migrate two vCenter 5.5 servers with embedded SSO into the same SSO domain. This and other migration scenarios involve re-pointing your vCenter 5.5 server to a newly installed external SSO 5.5 instance.

So when preparing the upgrade of a complex vSphere 5.5 environment to 6.0 you will sooner or later stumble over KB2033620 which describes how to do this re-pointing. Unfortunately this KB article and the tools that it refers you to are very poorly written and full of issues. Some of them are mentioned in the KB article itself with workarounds to follow, but a lot are not ... Here is a list of the most annoying issues with KB2033620 and how to fix them.

Read more »

An important heads-up for users of the Embedded Host Client!

2016-03-16T09:35:00.001+01:00

My ESXi Patch Tracker bot never sleeps, so when I woke up this morning it already greeted me with the message that VMware has released ESXi 6.0 Update 2 in the middle of the (European) night. As usual with Update releases vCenter was also updated.

For the record here are the most important URLs for you:

With this release there are some great news for the users of the Embedded Host Client, but also a caveat that you should be aware of.

The VIBMatrix has joined the ESXi Patch Tracker!

2016-02-20T17:40:00.000+01:00

My ESXi Patch Tracker service is becoming more and more popular. Many of its users probably don't know that I had a very similar service long before: On the VIBMatrix pages I maintained tables of ESXi Patch releases with lists of all included VIB packages. The original reason why I created these tables more than three years ago was to prove that ESXi 5.x/6.x Patch bundles are cumulative.

Until today the VIBMatrix was a manually maintained Excel sheet that I imported to Google Docs whenever I added a new patch to it. These times are over: I'm happy to announce that the VIBMatrix is now also fully automated and integrated into the ESXi Patch Tracker service!

Read more »

[Unsupported] Defeating the VCSA shell timeout

2016-02-09T20:56:00.001+01:00

In my earlier post vCSA 6.0 tricks: shell access, password expiration and certificate warnings I showed how can set the VCSA shell timeout to an effectively indefinite value using the command

shell.set --enabled true --timeout 2147483647

at the appliance shell prompt. I was notified that in recent versions of the VCSA (probably since Update 1) this does no longer work. The maximum timeout that the command will accept is now 86400 seconds (= 1 day). The VAMI interface of the VCSA 6 (that was added in Update 1) also allows enabling the shell and setting the shell timeout, but there the GUI also limits the input to max. 1 day (resp. 1440 minutes).

When I checked my own vCSA (that was originally installed with 6.0 GA, then upgraded to Update 1 and Update 1b) I found that my old large timeout setting was still in place and functioning. That means even in the latest build of the VCSA 6 it is still possible to set an arbitrarily large shell timeout, just not through the appliance shell or the VAMI. So how do you do this?

How to use the ESXi Patch Tracker to update ESXi

2016-01-31T20:30:00.000+01:00

About a year ago I started my VMware ESXi Patch Tracker service. This is a set of automatically generated web pages that show informations about all available ESXi patches. Its primary purpose is to give you an easy way to track the patch history and get alerted about new patches once they are released. You can subscribe to it by RSS feed, E-mail and Twitter.

Recently I have added new functionality to the Patch Tracker that enables you to update your ESXi hosts in a very convenient way. Here is how:

Read more »

Why upgrading from ESXi 5.5 Update 3b to 6.0 fails

2015-12-30T13:51:00.002+01:00

As of now users who are running ESXi 5.5 with the latest patch level Update 3b will find it difficult to upgrade to ESXi 6.0. Trying that with VMware Update Manager (VUM) or the esxcli software profile update method fails with error messages. esxcli throws this error:

[DependencyError]
VIB VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.78.3223702 requires com.vmware.usb-9.2.2.0, but the requirement cannot be satisfied within the ImageProfile.

and the VUM remediation process fails with the error event

The upgrade contains conflicting VIBs. Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

The issue is mentioned in an updated version of the VMware ESXi 5.5 Update 3b Release Notes, with a refreshingly short list of possible workarounds: None.

What are our options in this situation, are there really no workarounds?

How to build a customized ESXi installation image for your whitebox server

2015-12-20T19:15:00.000+01:00

My ESXi-Customizer tool has come a long way... Originally built in 2011 it was a great tool for building customized VMware ESXi installation images, it was downloaded over 200k times since then and helped a lot of people who had one quest: getting ESXi installed and running on their whitebox (unsupported) hardware. This year it was even mentioned in a VMware KB article ... However, please do NOT use it anymore! Because ...

Since quite a while there is a much better way to build customized ESXi images, and that is my ESXi-Customizer-PS PowerCLI script. I'm writing this post not only to explain how easy it is to create a customized ESXi installation image for your whitebox server with this script, but also to convince the last nonbelievers that it works much better than my legacy ESXi-Customizer tool.

Read more »

How to embed the VMware Tools version 10.0 into ESXi

2015-11-10T13:59:00.001+01:00

Recently I ranted about the current situation with VMware Tools, and my blog post The great VMware Tools dilemma caught some attention and got a lot of agreement. The good news is that VMware is listening to its customers and trying to improve things.

In a new blog post VMware's Brian Graf elaborates on the recommended solution to use a shared productLocker location for VMware Tools with your ESXi hosts. I complained in my post that it is not easily possible to populate such a shared location with the new "standalone" Tools version 10.0. And this has been improved now:

The VMware Tools version 10.0 download now includes all the files that you need to upload to the shared productLocker location, and
Brian has shared some scripts on his own blog to automate the process of creating a shared productLocker, uploading the Tools to it and configuring the hosts to use it.

Great news!

But what if you only have one or few ESXi hosts and want to avoid the overhead of creating a shared productLocker? Wouldn't it be good if we could just update the Tools that are embedded into ESXi to the new version 10.0?

To say it short: Yes, we can! The embedded Tools is just a VIB package (called tools-light) that can be updated. Unfortunately, VMware refuses (so far) to make VMware Tools 10.0 available as a VIB file, but we can build this ourselves! Here is how ...

Read more »

[Update] ESXi Community Packaging Tools updated to version 2.3

2015-11-09T13:35:00.000+01:00

After almost two years I decided to touch my ESXi Community Packaging Tools again and give them an important update. These tools consists of two scripts that enable Community Developers to create software packages for ESXi 5.x/6.x in the VMware proprietary formats VIB (VMware Installation Bundle) and ZIP (VMware Offline Bundle).

Read on to learn what's new and to not miss an exciting sneak peak of what's to come next.

Read more »

VMware introduces support for Intel i219 (Jacksonville) NICs in ESXi

2015-10-08T21:21:00.000+02:00

In this year's summer Intel introduced their new Skylake platform. These motherboards are based on the Z170 chipset and often include onboard Intel i219(-V or -LM) Gigabit Ethernet adapters. VMware ESXi did not properly support the Skylake chipset from the beginning, but it looks like this changed with the release of ESXi 6.0 Update 1 and 5.5 Update 3. I do not own such a system myself, but I have heard reports of users who were able to successfully install ESXi 6.0 U1 on such a system.

However, ESXi 6.0 Update 1 does not (yet) include a driver that supports the i219 NICs. But ESXi 5.5 Update 3 (that was published only few days later) comes with an updated e1000e driver (version 3.2.2.1) that supports these NICs. Here is how you can make best use of it, even if you are already using ESXi 6.0.

Read more »

The great VMware Tools dilemma

2015-10-05T14:09:00.000+02:00

Recently VMware made VMware Tools version 10.0.0 available as a standalone download. This version is now an official downloadable component of vSphere 6.0 with its own Release Notes document and a download page in MyVMware.

In the announcement blog post VMware's Brian Graf writes:

Good news. We have decided that there isn’t any specific reason that VMware Tools builds should be tied to vSphere releases/ESXi builds. Rather, when our engineering teams are ready with key features/updates, we should have the ability to get those benefits out to our customers as quickly as possible.

This release was announced and perceived as a great achievement, but - except for this good news that Brian shared with us - I cannot follow any of the other excited statements that were made about it. It's time to debunk some myths, and it's time to admit that the whole VMware Tools story is still a great mess!

Read more »

Shut up, Windows 10! Here is a free must-have tool to protect your privacy.

2015-08-24T15:18:00.002+02:00

If you install Windows 10 with the default "Express Settings" then it will collect a plethora of data from you and send it to Microsoft and "trusted partners": Not only whatever you ask Cortana (Microsoft's version of Apple's Siri or Google Now), but also what you type or write by hand, where you are, what networks you connect to, what you search for and more...

Sure, they are not alone with this: Google does it with Android, Apple with iOS devices, and - if you ask them - they all collect the data only to optimize the personalized services that they offer you for free. Really for free? No, you pay with your data ... However, more and more people are concerned about data privacy and want to have a choice about what data they disclose. Recently I stumbled over a great tool that helps those people when using Windows 10.

Read more »

A fix for Intel i211 and i350 adapters not being detected by ESXi

2015-08-21T14:51:00.000+02:00

Recently two readers of my blog asked me for help with a strange issue that they encountered when trying to install ESXi on their whitebox hardware. The one was a Shuttle DS57 barebone, and the other one was a Compulab Fitlet-X (an interesting tiny fanless industrial PC with 4 onboard NICs). Both have Intel i211 Gigabit Ethernet adapters that were not detected by ESXi, although they are officially supported by the ESXi 6.0 built-in igb driver (see VMware HCL entry).

Looking for a solution I found that this seems to be a common issue. Multiple reports found in the VMware Communities and other forums convinced me that resolving the issue would help a lot of people. Time for some late night troubleshooting ...

Read more »

Troubleshooting VM network performance using vsish

2015-08-04T21:53:00.000+02:00

Long time, no post ... I'm currently busy with several support requests that I opened with VMware. These keep me busy but they are also good opportunities to learn something new.

One of the cases involves troubleshooting the network performance of Lync 2013 servers running on top of vSphere. It turned out that the VMkernel system information shell (better known as vsish) is a great tool for this.

Read more »

Heads up: The upcoming Leap Second adjustment may affect your IT infrastructure!

2015-06-11T14:23:00.000+02:00

Okay, probably everyone is aware of the upcoming leap second time adjustment on June 30th, 0:00 UTC. I won't explain the story behind it although it's a very interesting topic, but others have done this already. A good starting point is the Wikipedia Leap Second article.

You might wonder: Okay, this is an anomaly that is confusing, but it's only one second, and modern IT systems should not really stumble over this, right? I was of the same opinion until recently, but the nearer the time comes the more IT vendors alert their customers of leap second related issues in their products, and you might be surprised to hear that very critical parts of your infrastructure can fail on June 30th if you do not patch them before!

Read more »

Filtering logs in ESXi 6.0 - A practical example

2015-05-27T16:43:00.000+02:00

Recently I rented a new server with an LSI MegaRAID SAS 9260-4i controller and - of course - installed VMware ESXi 6.0 on it. I will soon migrate my "production workloads" (the V-Front Online Depot, the ESXi Patch Tracker site and my Zimbra E-Mail server) onto this box to improve their availability. A disk failure is one of the most likely disaster scenarios, so a hardware RAID controller with two mirrored disks will certainly help.

After I had the host installed and configured I did a routine check of the log files and found a lot of SCSI errors like these in /var/log/vmkernel.log:

cpu1:34460)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x439d80668b00, 0) to dev "naa.600605b0058b25f01c995a4f0b03da18" on path "vmhba1:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
cpu1:32779)ScsiDeviceIO: 2646: Cmd(0x439d80716d00) 0x4d, CmdSN 0x1 from world 34425 to dev "naa.600605b0058b25f01c995a4f0b03da18" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

These errors were referring to the RAID 1 disk, and first I was worried about them - according to KB289902 the D:0x2 translates to "Device: Check condition" and the sense data codes 0x5 0x24 0x0 and 0x5 0x20 0x0 mean "Illegal request", "Invalid Field in CDB", resp. "Invalid command operation code". I double checked that the RAID controller is on the VMware HCL, I searched the Internet for these errors, found few peoples' posts describing this issue, but no solution. I even updated the firmware of the controller, all to no avail ... and then, finally, I stumbled over KB1031221 which - basically - describes that these errors are "normal" and "can be safely ignored".

Well, that calmed me down... But I'm a log purist: I wanted to get rid of these false error messages in vmkernel.log!

What's new in the V-Front Online Depot - or - how to install ESXi on an HP Pavilion 17 notebook

2015-05-20T21:18:00.001+02:00

Recently I added some new NIC drivers to the V-Front Online Depot. Well, they are not really new, but were created back in 2011/12 by a Japanese guy who goes by the nickname ichi. He was actively participating in Dave Mishchenko's vm-help.com forums and posted the drivers there.

Although I was aware of them since then I did not add them to the Depot before - I thought that no one would really need them these days, because the NIC models that they support are pretty old and rarely used today. I was wrong, at least in one case, because I needed one of these drivers myself to install ESXi on a brand new HP Pavilion 17 notebook that I just purchased ...

Read more »

CPU Microcode Updates revisited: What's new in ESXi 6.0?

2015-05-11T14:44:00.000+02:00

Almost two years ago I wrote an FAQ post on CPU Microcode updates with ESXi 5.x. This still gets a lot of hits, but - although the general information about Microcodes still applies - the ESXi related information needs an update, because things have changed with ESXi 6.0.

So what's new and different now?

Read more »

[Release] ESXi-Customizer-PS 2.4 - The ESXi image customization script

2015-04-25T18:26:00.001+02:00

I have just released version 2.4 of my ESXi-Customizer-PS script - a PowerCLI script to create customized ESXi installation images and the successor of my legacy ESXi-Customizer tool.

This is a minor, but important update, because it fixes an annoyance that users of the latest PowerCLI version 6.0 ran into.

Read more »

ESXi 6.0 hits the VMware Online Depot - What does that mean?

2015-04-17T21:54:00.002+02:00

On April 9th VMware published the first ESXi 6.0 patch (see the ESXi Patch Tracker and KB2111976), and at the same time the Imageprofiles of both this patch and the ESXi 6.0 GA release appeared in the VMware Online Depot. Unfortunately they did not also upload all the ESXi 6.0 VIB packages at that time, but only the one that was updated with the first patch. Today, they finally made up for this, and I cannot only publish this article again, but now the instructions and examples in it will really work :-)

What does it mean to have ESXi 6.0 available in the VMware Online Depot? It makes it a lot easier to upgrade an ESXi host and to build customized ESXi 6.0 installation images, especially for free license users who do not have easy access to the GA Offline bundle. I wrote about this already in my Ultimate guide to upgrade your white box, and this is the promised update to it.

So what are the new possibilities now?

Read more »