Patch Release ESXi550-201601501-BG breaks connectivity to vCenter

A few days back I upgraded an ESXi host from ESXi 5.1 to 5.5.  I had downloaded the latest image my vCenter and hardware would support.  The upgrade went well.  Once the host rebooted and was connected back in vCenter I performed a patch scan to see missing patches.

I had about six missing patches.  One being ESXi550-201601501-BG, released only a few days beforehand on 4 Jan.  It was marked as a Critical Bugfix.  As I have always done with Critical Bugfixes I Remediated.  After all, it was Critical and a Bugfix.  Once the host remediated and rebooted I found that I could no longer connect to the host via vCenter.  Receiving the below error when attempting to connect.

Cannot contact the specific host (hostname). The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.

I opened a C# Client and was able to successfully connect to the ESXi host directly.  So all looked good there.  I decided to look up the KB article of this latest patch (KB 2141207) to see what it fixed.  Nothing came up.  No page could be found!

With a lack of information on this patch the best thing I could do was just to revert the patches.  The fix was simple.  I rebooted the host.  During the Loading VMware Hypervisor I pressed Shift R for Recovery mode.

The VMware Hypervisor Recovery screen then appears.  It shows the current build and the previous build that it will Roll Back to.  Pressing Y starts the process.  It was quick and simple and I was back up and going on the previous 5.5 build I had just upgraded to without the addition patches I installed.

(The above screenshot was not the host that I Rolled Back but rather is a base 5.1 image)

The KB article page now works.  It is quite clear you shouldn’t update to version 5.5 Update 3b before you upgrade your vCenter to the same version of 5.5 Update 3b.  The interesting note here is that I didn’t realise that I was updating the host to ESXi 5.5 Update 3b.  My upgrade image was ESXi 5.5 Update 3a and I thought I was just installing a critical bugfix patch.  Where I can undone was this was an esx-base patch that brought the effective build number of the ESXi host up to the same level of a full ESXi 5.5 Update 3b build.

So what was actually going on with this patch…

Reading the KB article isn’t entirely obvious what’s going on.  Yes, we now know not to update unless vCenter is on Update 3b but why?  The article states that the patch updates the esx-base vib to resolve an issue when upgrading from ESXi 5.5 to 6.x, where some VMs and objects may become inaccessible or report non-compliant.  That’s clearly, though, not what we’re doing here.

The KB article now has a new link to a second KB article (KB 2140304) which describes the issue you will face if you don’t upgrade vCenter first.  What it describes is that SSLv3 is now disabled by default in this esx-base vib.  So I can only presume that vCenter wants to connect to the ESXi host using SSLv3.

Fortunately the resolution wasn’t that complex.  I was able to resolve the issue and get the host up and going relatively quickly and all within my outage window.  This is the first time I have experienced an ESXi patch break a host.  I’m surprised that a patch is actually changing the default behaviour of a host.  I can understand, and even support, VMware making the change in the latest ISO image build of ESXi 5.5 but not in a patch.

The Lesson…

So the lesson here (and there’s probably a few but I’m going with), stay away from any esx-base patches that have been released post 4 Jan 2016 until vCenter is upgraded to 5.5 Update 3b.

As a side note I would assume SSLv3 could be re-enabled on the host but I haven’t looked into this yet.

References

After upgrading an ESXi host to 5.5 Update 3b and later, the host is no longer manageable by vCenter Server (2140304)

VMware ESXi 5.5, Patch ESXi550-201601501-BG: Updates esx-base (2141207)