Mitigation and Threat Hunting Guidance for Unsigned vSphere VIBs

Refer to latest updates at VMware KB89619:

Mitigation and Threat Hunting Guidance for Unsigned vSphere Installation Bundles (VIBs) in ESXi (89619) (vmware.com)

Details

On Thursday September 29th, Mandiant published information on malware they discovered in the wild that leverages unsigned VIBs to install backdoors on a compromised ESXi host. It should be noted that a malicious actor must first obtain administrative privileges (root) on an ESXi host prior to installing a malicious VIB. Also, Mandiant found no evidence that a vulnerability in a VMware product was exploited to gain access to ESXi during their investigations.

For information on operational security best practices, Mandiant’s findings, and general information about this disclosure please review our article entitled Protecting vSphere From Specialized Malware.

This KB Article will focus on mitigation and threat hunting instructions for unsigned VIBs.

Resolution and Mitigation

In addition to implementing various operational security best practices mentioned in Protecting vSphere From Specialized Malware to prevent a potential compromise in the first place, VMware recommends enablement of the Secureboot feature in ESXi to mitigate the risk of malicious actors persisting on a compromised ESXi host via malicious VIB installation. Secure boot was designed to disallow installation of unsigned VIBs on an ESXi host. In addition, secure boot disallows the –force flag which would normally allow an administrator to bypass acceptance level settings on the ESXi host.

To enable Secureboot perform the following steps:

Please contact your hardware vendor for steps on how to enable UEFI / Secureboot for your system.

Enabling Secureboot on ESXi: UEFI Secure Boot for ESXi Hosts (vmware.com)

Run the Secure boot validation script: /usr/lib/vmware/secureboot/bin/secureBoot.py -c

– If 7.0 u2 or later and the host has a TPM, please see the following document: Enable or Disable the Secure Boot Enforcement for a Secure ESXi Configuration (vmware.com)

Threat Hunting

Concerned customers can perform the following instructions in order to audit their ESXi host(s) for unsigned VIBs.
Download the following PowerCLI script Verify_ESXi_VIB_Signature.ps1 (attached to this KB) and run against your vCenter using the SSO admin credentials..

-Requirements:

PowerCLI installed (Install PowerCLI (vmware.com)

443 access to vCenter where the script is running from

Set the PowerShell Execution Policy to unsigned:Set the PowerShell Execution Policy to RemoteSigned (vmware.com)

What to look for in the results:

Example:

KB89619_Verify_Unsigned_VIBs_on_ESXi_(ver_1.2).ps1 - Script Results

Overall Status = Good: This host has no unsigned VIBs.

Overall Status = Not Good: Unsigned VIBs were detected on the host.

Note: 6.5 has a known issue which will show an unsigned VIB on the ESXi base. Please see the following KB:Unable to enable Secure Boot in ESXi 6.x (79790) (vmware.com)

Note: CommunitySupported VIBs are not signed. CommuitySupported VIB’s require an ESXi host to be set to CommunitySupported acceptance level, which is not recommended.

What should I do if I find unsigned VIBs in my environment?

VMware does not recommend using unsigned VIBs but their presence does not definitively prove that an ESXi host has been compromised. VMware recommends that organizations attempt to determine the origin of any unsigned VIB(s) that are found on their ESXi hosts as it is possible that a trusted administrator may have intentionally installed the unsigned VIB(s) for a legitimate purpose. However, organizations who suspect a compromise may have occurred should follow their established incident response processes. For organizations who do not have an in-house Incident Response team, VMware provides a list of trusted partners who offer incident response services, please see: https://www.vmware.com/partners/work-with-partners/incident-response-and-managed-security-service-providers.html

Please follow KB Unable to enable Secure Boot in ESXi 6.x if esx-base VIB verification is failing with error “Failed to verify checksum for payload btldr: Not found”

Secure boot feature will verify the VIBs during the boot and will trigger a PSOD with following error if any Unsigned

VIB is installed on the ESXi host.
UEFI Secure Boot failed:
Failed to verify signatures of the following vibs (XX)