Skip to content

Keep :latest patched between releases, not just at them - #211

Merged
thalida merged 6 commits into
mainfrom
chore/issue-209-refresh-latest-image
Sep 3, 2026
Merged

Keep :latest patched between releases, not just at them#211
thalida merged 6 commits into
mainfrom
chore/issue-209-refresh-latest-image

Conversation

@thalida

@thalida thalida commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Closes #209

What was actually wrong

Not the build. The image was just older than the fix.

:latest = v1.11.0, published 2026-08-22
DSA-6465-1 fixes CVE-2026-14456 in trixie (3.5.7-1~deb13u2) 2026-08-25
Weekly scan ran 2026-08-31

The three findings are one CVE across openssl's three binary packages.

Verified rather than assumed:

  • The published image carries openssl / libssl3t64 / openssl-provider-legacy at 3.5.6-1~deb13u2.
  • Replaying the Dockerfile's exact apt sequence on python:3.13-slim today yields 3.5.7-1~deb13u2 for all three. A rebuild clears it, with no Dockerfile change.

So the CVE isn't the bug. The bug is that nothing rebuilds :latest.

The gap

release.yml triggers on push: tags: v*. Between releases nothing refreshes the published image, so every CVE that lands mid-cycle sits in :latest and gets an issue filed every Monday until someone happens to ship a feature.

refresh.yml

Runs Mondays 04:00 UTC (two hours before the scan) and on dispatch:

  1. Resolves the newest release tag and checks out that tag, not main:latest has to keep meaning released code. Only the OS packages underneath it move.
  2. Rebuilds multi-arch with pull: true + no-cache-filters: runtime, so the base and the apt-get upgrade are genuinely fresh.
  3. Pushes it as refresh-<version>-<date> only.
  4. Trivy-scans that image. Still dirty means the fix needs a real dependency change, so it fails here and :latest never moves.
  5. Smoke-tests /api/health, cosign-signs the digest.
  6. Only then imagetools create retags it :latest — a retag of the manifest already in the registry, so :latest resolves to the exact digest that was scanned, tested and signed.
  7. A deploy job on needs: [refresh] dispatches the same Forgejo workflow release.yml does, so production actually moves onto the patched image.

cache-from without cache-to: a weekly multi-arch mode=max export would evict the entries PR builds actually read out of the 10GB cache.

The smoke test is inlined rather than shared with release.yml. It has to be — this job checks out an old release tag, so any repo script added after that release wouldn't exist in the workspace.

Two things found on the way

release.yml was missing the cache guards ci.yml has. It uses cache-from: type=gha with no no-cache-filters: runtime and no pull: true, while sharing the cache ci.yml writes with mode=max. A release cut today could restore a stale runtime layer and republish the same CVEs. The Dockerfile comment already claimed this couldn't happen ("ci.yml excludes this stage from the build cache") — now it's true of every workflow that builds the image, and the comment says so.

AGENTS.md documented a cat:security label that has never existed. The scan workflow creates and applies security + dependencies; idea was undocumented too.

Contract change worth knowing

:latest no longer always resolves to the newest version tag's digest — it tracks that tag's code on today's packages. Version tags themselves never move. README's Release section documents this.

Duplication that's deliberate

The smoke test and the Forgejo dispatch are inlined rather than shared with release.yml. They have to be: this job checks out an old release tag, so a composite action or bin/ script added after that release simply wouldn't exist in the workspace. Sharing them would mean checking the tag out to a subpath and building with context: src — more subtlety than 30 duplicated lines are worth.

Verification

  • actionlint clean on refresh.yml and scan.yml
  • prettier --check clean repo-wide
  • scan.yml's issue-body script run under node against this issue's real Trivy rows: renders "1 HIGH/CRITICAL CVE across 3 package(s)"

After merge

workflow_dispatch only appears once the workflow is on the default branch. So merge, then run Refresh published image manually to clear :latest today rather than waiting for Monday.

🤖 Generated with Claude Code

https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ

thalida and others added 2 commits September 3, 2026 11:44
`:latest` only got rebuilt when a version tag was pushed, so an OS CVE landing
mid-cycle sat in the published image until the next feature release. That is
what happened here: v1.11.0 shipped 2026-08-22, DSA-6465-1 fixed openssl three
days later, and the 08-31 scan found it.

refresh.yml rebuilds the newest release tag's source against a fresh base every
Monday, scans and smoke-tests the result, signs it, and only then retags it
`:latest`. It builds the tag rather than main so `:latest` keeps meaning
released code, and pushes a dated tag first so a bad rebuild can't move
`:latest` at all.

release.yml was also missing the cache guards ci.yml has, while sharing the
`type=gha` cache ci.yml writes with mode=max. A release cut today could have
restored a stale runtime layer and republished the same packages, which the
Dockerfile comment already claimed was impossible.

scan.yml's "time to rebuild against a fresh base" is now false by construction
— a refresh already tried that — so it says what a surviving finding means
instead, and counts distinct CVEs rather than package rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ
AGENTS.md listed `cat:security`, which has never existed. The weekly scan
creates and applies `security` + `dependencies` instead, and `idea` went
undocumented too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ
@thalida thalida linked an issue Sep 3, 2026 that may be closed by this pull request
thalida and others added 2 commits September 3, 2026 11:51
A patched image nobody pulls patches nothing: refresh.yml republished `:latest`
and left production on whatever it last pulled, which made `just deploy` a
weekly chore. It now dispatches the same Forgejo deploy release.yml does, gated
on `needs: [refresh]` so it can't fire before the rebuild has passed its scan,
smoke test and signature.

The README section was four ideas long for two ideas' worth of content; trimmed
to the bullets and the one caveat you'd act on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ
refresh.yml deploys production now, and workflow_dispatch would have let anyone
with write access trigger that — a wider door than release.yml's actor gate.
Scheduled runs stay ungated on purpose: `github.actor` is the last editor of the
workflow on a schedule event, so gating cron would let an unrelated edit quietly
stop the weekly patch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ
Comment thread .github/workflows/refresh.yml Fixed
thalida and others added 2 commits September 3, 2026 11:54
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ
CodeQL flagged refresh.yml's deploy job for inheriting the default
GITHUB_TOKEN scopes, and release.yml's has the same gap (open alert #1).
Neither touches the GitHub API — they curl Forgejo with its own token — so
the minimal grant is none at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Qzxns3a3Q12RfayHZRrGZ
@thalida
thalida merged commit bb586d3 into main Sep 3, 2026
5 checks passed
@thalida
thalida deleted the chore/issue-209-refresh-latest-image branch September 3, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trivy scan: 3 HIGH/CRITICAL CVEs in codecity:latest (2026-08-31)

2 participants