Skip to content

Add e2e tier 18: SBOM-driven CVE sweep and zero-downtime bundle remediation - #141

Merged
Bruno Borges (brunoborges) merged 2 commits into
mainfrom
brunoborges-log4j-cve-sbom-hot-redeploy-test
Sep 30, 2026
Merged

Bruno Borges (brunoborges) merged 2 commits into
mainfrom
brunoborges-log4j-cve-sbom-hot-redeploy-test

Conversation

@brunoborges

@brunoborges Bruno Borges (brunoborges) commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR adds a test for the "Log4Shell day" scenario. An app runs on a managed dependency bundle, and a library in that bundle gets a critical CVE. The test answers three questions: how do we find which prod apps are affected, how do we release a fixed bundle, and how do we redeploy without downtime.

Brewlet does not record which bundle a JavaApplication or pod uses. So the new sweep, integration-tests/e2e/cve_sweep.py, starts from what is actually running. For each pod it follows: digest-pinned image -> brewlet inspect managed-dependency evidence -> the bundle's CycloneDX SBOM -> components and purls. It then checks each component against the advisory's version range. If it cannot trace an image to an SBOM, it fails closed instead of reporting that image as clean.

New tier 19 (tier19-cve-remediation.sh):

  • Publishes a bundle with a stub log4j-core 2.14.1. The fixture in integration-tests/fixtures/log4j-cve-app uses a neutral package, not Apache code.
  • Composes three thin-JAR apps: orders and billing on that bundle, and inventory as a control on an unrelated bundle. Deploys them to prod and staging namespaces.
  • Checks that the prod sweep flags exactly prod/orders and prod/billing. Checks that a fleet-wide sweep also sees staging/orders.
  • Publishes a patched bundle (2.17.1) and recomposes the unchanged thin JARs onto it. Asserts that the app layer and applicationJarDigest stay the same and only the classpath layer changes.
  • Rolls the Deployments to the new digests (maxUnavailable: 0) while a client polls the Service continuously. Asserts zero failed requests and that the JVMs now load 2.17.1.
  • Re-sweeps: prod comes back clean, and staging is still flagged.

"Hot redeploy" here means recompose plus a rolling update. Brewlet does not swap classpath layers inside a running JVM (SPECIFICATION section 11).

Other changes:

  • Wires tier 19 into run.sh, the reset.sh namespace list, and the CI matrix in e2e.yml. The CI entry also runs the offline sweep tests. Tier 18 was already taken by the patched-JDK rollout from Add patched-JDK rollout e2e (tier 18) and fix two provisioner bugs #140.
  • Adds offline unit tests for the sweep (cve_sweep_test.py, using a fake CLI).
  • Updates integration-tests/AGENTS.md.
  • Adds a "Responding to a dependency CVE" section to docs/managed-dependency-bundles.md.

Validation

  • integration-tests/e2e/run.sh --tier 19 on Docker Desktop kind (3 nodes, arm64 worker, go1.27.1, JDK 25): 21 passed, 0 failed, 0 skipped. It passed both before and after merging main (it was tier 18 before the merge). In the first run the client made 114 requests during the rollout with 0 failures.
  • PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s integration-tests/e2e -p 'cve_sweep_test.py' -v: 5 tests OK.
  • bash -n run.sh reset.sh tier19-cve-remediation.sh: OK. shellcheck is not installed locally, so it was not run.

Compatibility and operations

None. This PR adds a test tier, test tooling, a fixture, documentation, and a CI matrix entry. It does not change the product. Tier 19 skips when there is no cluster, Docker, Go, Python 3, JDK, or schedulable local containerd node.

Checklist

  • Tests cover changed behavior.
  • Documentation is updated when needed.
  • No credentials, proprietary data, or unrelated generated files are included.

…iation

Rehearses a Log4Shell-style CVE in a managed dependency bundle: identify
affected prod workloads from running pods' SBOMs, publish a patched bundle,
recompose the unchanged thin JARs, and roll out by digest with zero dropped
requests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
main added tier 18 (patched JDK rollout). Move the dependency CVE tier to
19, add it to the CI matrix and reset namespaces, and run its offline
sweep tests in CI.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@brunoborges
Bruno Borges (brunoborges) merged commit 5487893 into main Sep 30, 2026
13 checks passed
@brunoborges
Bruno Borges (brunoborges) deleted the brunoborges-log4j-cve-sbom-hot-redeploy-test branch September 30, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant