HOWTO: Tanzu Kubernetes Grid Integrated Edition workaround steps to address CVE-2021-44228

CloudNerve.com -

ISSUE

CVE-2021-44228 has been determined to impact Tanzu Kubernetes Grid Integrated Edition (TKGI) via the Apache Log4j open source component it ships.

This vulnerability and its impact on VMware products are documented in the following VMware Security Advisory (VMSA), please review this document before continuing:

Notice: On December 14, 2021 the Apache Software Foundation notified the community that their initial guidance for CVE-2021-44228 workarounds was not sufficient. We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors. 

We expect to fully address both CVE-2021-44228 and CVE-2021-45046 by updating log4j to version 2.16 in forthcoming releases of Tanzu Kubernetes Grid Integrated Edition, as outlined by our software support policies. VMSA-2021-0028 will be updated when these releases are available. In the interim, we will be updating this Knowledge Base article with revised guidance to remove all JndiLookup classes per Apache Software Foundation guidance. Please subscribe to this article to be informed when updates are published.

RESOLUTION

​​​​​​Impact / Risks

UAA authentication will be unavailable while it’s restarting to apply the mitigation. Change log:

Updated on December 23rd, 2021:

  • Updated resolution section with the following release notices
    • TKGI 1.10.8 and TKGI 1.13.1 released with the fixes for CVE both CVE-2021-44228 and CVE-2021-45046

Updated on December 20th, 2021:

  • Updated resolution section with the following release notices
    • TKGI 1.11.7 and TKGI 1.12.3 released with the fixes for CVE both CVE-2021-44228 and CVE-2021-45046
    • Harbor 2.3.5 and 2.4.1 released with the fixes for CVE both CVE-2021-44228 and CVE-2021-45046.

Updated on December 17th, 2021:

  • Updated instructions to disable the Wavefront monitoring feature as full mitigation for both CVE-2021-44228 and CVE-2021-45046 for TKGI
  • Updated instructions to disable the Wavefront monitoring feature as full mitigation for both CVE-2021-44228 and CVE-2021-45046 for Harbor
  • Removed manual workaround to patch wavefront proxy pod for TKGI
  • Removed manual workaround to patch wavefront for Harbor tile

Updated on December 16th, 2021:

  • Updated the UAA release to 74.5.29, this UAA version uses log4j version 2.16.0 to fully address both CVE-2021-44228 and CVE-2021-45046
  • Removed previous manual workaround

Updated on December 15th, 2021:

  • Add the workaround to patch with new UAA releases in persistent way. This is the recommended workaround.
  • The previous workaround is improved by using “bosh ignore”, this is for customers who are still using older TKGI versions and could not use new UAA version.
  • Disable BOSH resurrection workaround is not recommended.

Resolution

TKGI have been released with the fixes for CVE both CVE-2021-44228 and CVE-2021-45046.

Harbor 2.3.5 and 2.4.1 have been released with the fixes for CVE both CVE-2021-44228 and CVE-2021-45046.

Note: VMware recommends to upgrade to the latest patch versions ASAP. The workaround section below is only applicable if you can not upgrade to the TKGI or Harbor Tile versions listed above.

Workaround  ( manually patch UAA Bosh release)

To patch the UAA in a persistent way for CVE-2021-44228 on TKGI:

  1. SSH into the Ops Manager VM. For more information, refer to Logging Into Ops Manager VMs with SSH.
  2. Download the patched UAA BOSH releases to the Ops Manager VM.
    For TKGI 1.9.x ~ 1.13.x, please use below UAA version (TKGI 1.7.x and 1.8.x can use same version but not validated as end of Support):

    sudo -u tempest-web wget -P /var/tempest/releases/ https://uaa-release-tarballs.s3.us-west-1.amazonaws.com/releases/uaa-release-74.5.29-rc.5.tgz
  3. Find the file paths of the YAML files that define all the versions of the TKGI tile. You want the .yml file from the following command It should look something like:

    sudo grep -l "^name: pivotal-container-service" /var/tempest/workspaces/default/metadata/*

    /var/tempest/workspaces/default/metadata/266bfb993b3c.yml

  4. Confirm the version of TKGI you’re using with the following command on each full file path;
    if there’s more than one file returned by the above, run it on each to identify the version that you have currently deployed, which you’ll need to edit in next steps

    sudo head FULL-FILE-PATH

  5. Make a backup of this YAML file, into your home directory. You can restore this backup over the file you’re about to edit in order to revert the workaround if needed later.

    sudo cp FULL-FILE-PATH ~ubuntu/

  6. Edit the YAML file (using “sudo editor-of-choice”, such as “emacs”, “vi”, or “nano”) to replace the relevant release sections for UAA release.EXAMPLE OF OLD UAA SECTION
    - name: uaa
      version: 74.5.25
      file: uaa-74.5.25-ubuntu-xenial-621.141.tgz
      exported_from:             <-- make sure remove this line
      - os: ubuntu-xenial        <-- make sure remove this line
        version: '621.141'       <-- make sure remove this line

    EXAMPLE OF AFTER UPDATE
    - name: uaa
      version: 74.5.29-rc.5
      file: uaa-release-74.5.29-rc.5.tgz

  7. Apply Changes to the TKGI Tile!
    When you review pending changes for the TKGI tile, it should look something like this – added(+) version/file should reflect the changes made at step 6:

    Note: If you see an error like “Packages must be exported from stemcell ‘ubuntu-xenial/621.160’, but some packages are not compiled for this stemcell”, then you likely forgot to remove the “exported_from” block from a previous step. You can fix that and try again.Your UAA are now patched. They will remain patched, even if VM resurrection takes place, you upgrade the stemcell, or you reconfigure the tile. If you upgrade the TKGI tile to a new version this mitigation will be lost and may need to be reapplied.

If you enabled the Wavefront Integration feature in TKGI
VMware recommends to disable the Wavefront monitoring feature as full mitigation for both CVE-2021-44228 and CVE-2021-45046, until you upgrade to the patch releases.

  • In TKGI Tile, go to “In-Cluster Monitoring”, Select “No” to disable the “Wavefront Integration”, apply the change, then do the clusters upgrade to make it effective.

If you enabled the “VM monitoring with Wavefront” in Harbor tile
VMware recommends to disable the Wavefront monitoring feature as full mitigation for both CVE-2021-44228 and CVE-2021-45046, until you upgrade to the patch releases.

  • In Harbor Tile, go to “VM Monitoring Settings”, Select “Do not enable VM monitor” to disable the feature, apply the change.