Researchers have identified a new family of malware designed to backdoor and create persistence on VMware ESXi servers by exploiting legitimate features supported by hypervisor software. According to Mandiant researchers who found and analyzed the backdoors, they were packaged and deployed to infected servers as vSphere Installation Bundles (VIBs). VIBs are software packages used to distribute components that extend VMware ESXi functionality. The malicious VIBs provided hackers with the ability to execute and persist commands remotely on servers and the ability to execute commands on guest virtual machines running on the servers.
Hackers used hard-to-detect unsigned VIBs
By default, VMware ESXi is configured to only accept installation of VIBs that are VMWare, VmwareAccepted, or PartnerSupported certified. At these acceptance levels, bundles must be digitally signed by VMware or a partner whose signature is approved by VMware.
However, there is a fourth level of acceptance called CommunitySupported and VIBs in this category do not need to be digitally signed. The downside is that these bundles must be deployed by an administrator intentionally using the –force flag on the install command via the esxcli command line tool.
The malicious VIBs found by Mandiant had their manifest file modified to indicate “partner” as the acceptance level, but in fact did not have a digital signature and were deployed using the –force command. This means that the attackers already had administrative-level access to the servers before deploying them. so they were a late-stage payload.
One of the effects of listing “partner” as a source in the rogue VIB manifest was that they appeared as PartnerSupported when the “esxcli software vib list” command was used when in fact they were not. This oversight in the command that simply displays what the manifest says helped attackers better hide their backdoors from administrators. To discover them, administrators would have had to use the command “esxcli software vib signature verify” which would have verified the digital signature of all VIBs deployed on their servers.
Attackers deployed both hypervisors and virtual machine backdoors
In addition to a manifest file and a signature file, VIBs include a collection of files and directories that will be copied to the system. One of these files was a passive backdoor that used VMware service names to hide and listen for traffic on a hardcoded port number on the ESXi server. The backdoor, named VIRTUALPITA, can run arbitrary commands, upload and download files, and start and stop vmsyslogd, the ESXi service responsible for logging system kernel messages and other components.
“When executing arbitrary commands, the malware also sets the HISTFILE environment variable to 0 to further obscure activity that has occurred on the machine,” the Mandiant researchers said. “Variants of this malware have been found to listen on a Virtual Machine Communication Interface (VMCI) and log this activity in the sysclog file.” VMware VMCI is the high-speed communication interface through which virtual machines communicate with the host kernel.
In the /usr/libexec/setconf/ and /usr/bin directories.
The researchers also found a secondary backdoor in the malicious VIBs which they dubbed VIRTULPIA. This backdoor program was written in Python and listened for IPv6 traffic on port 546. Attackers could use this backdoor to execute arbitrary commands, transfer files, and open a reverse shell. Communications through the port were via a custom protocol that used RC4 encryption.
Finally, some attacks involved a third backdoor called VIRTUALGATE, written for Windows and deployed to guest virtual machines running on compromised ESXi servers. This backdoor allows attackers to execute commands on the guest VM from the hypervisor or between different guest VMs running on the same host through the VMCI.
Researchers observed attackers using VIRTUALPITA to run a shell script that launched a Python script that then executed commands on guest virtual machines. On the virtual machines, the commands were executed by the legitimate VMware Tools service (vmtoolsd.exe). In one case the commands involved listing files from certain directories and then bundling them as CAB archives and in another case the attackers used the MiniDump utility to dump the memory of a process and search through it clear identification information.
Mandiant has not linked these attacks to any known group, so it is tracking them under a new group ID called UNC3886. “Given the highly targeted and evasive nature of this intrusion, we suspect that the motivation for UNC3886 is related to cyber espionage,” the researchers said. “Furthermore, we assess with low confidence that UNC3886 has a connection to China.”
Although there are not yet many incidents where the VIB malware has been used to compromise ESXi servers, Mandiant expects other threat groups to copy this technique in the future.
VMware recommends enabling UEFI Secure Boot
VMware has released an advisory in response to Mandiant’s findings along with a PowerShell script that can be used to scan an environment for malicious VIBs. However, the main recommendation is to enable UEFI Secure Boot on the system, which provides cryptographic attestation of components early in the boot process.
“When Secure Boot is enabled, use of the ‘CommunitySupported’ acceptance level will be blocked, preventing attackers from installing unsigned and missigned VIBs (even with the –force parameter as noted in the report)”, VMware says in its advice. “vSphere 8 takes another step and prevents the execution of unsigned binaries or binaries installed by means other than a properly signed VIB. Efforts by attackers to disable this feature generate unavoidable ESXi alarms as clues that something is happening in an environment.
Copyright © 2022 IDG Communications, Inc.