Skip to content

ci: De-flake Build and Publish Images with retried fetches - #2489

Closed
ericcurtin wants to merge 3 commits into
bootc-dev:mainfrom
ericcurtin:ci-deflake-publish-images
Closed

ericcurtin wants to merge 3 commits into
bootc-dev:mainfrom
ericcurtin:ci-deflake-publish-images

Conversation

@ericcurtin

Copy link
Copy Markdown
Contributor

"Build and Publish Images" on main keeps failing intermittently (most recently on ae375f9). I went through all 12 failures on main since late August. Every one was a transient network error:

Count Failure Where
8 unexpected EOF pulling quay.io/{fedora,centos-bootc}/*-bootc from the quay CDN just package (implicit base pull)
3 rpm-md download errors (mirror 500 / checksum mismatch) rpm-ostree compose in target-base
1 quay.io 502 Bad Gateway _pull-lbi-images

ci.yml already guards against this with just build-fetch, which front-loads network access with a retry loop. build-and-publish.yml never used it. build-fetch also missed some network steps:

  • It depended on package, which pulls the base image and builds the buildroot stage (dnf + cargo fetch) without retries.
  • The tools stage, which is always built because sdboot-signed depends on it, runs dnf outside --target=fetch.
  • _pull-lbi-images always contacts the registry, even when the images are already present. It relied only on podman's short internal retry.

Changes

  1. Justfile: Factor out retry helper (no functional change). The retry shell function moves into a shared _retry_fn variable.
  2. Justfile: Retry all network steps reachable from build-fetch.
    • New _package-fetch step: pulls the base image and builds --target=buildroot with retries, so package hits the layer cache.
    • Prefetch --target=tools instead of --target=fetch. tools builds on top of fetch, so this covers both.
    • Wrap _pull-lbi-images in the retry helper.
  3. ci: Fetch dependencies with retry in build-and-publish. Run just build-fetch before just build, with the same BOOTC_CI_RETRIES=10 / BOOTC_CI_DELAY=60 as ci.yml.

Notes / tradeoffs

  • The buildroot cache sharing relies on buildah including only stage-declared ARGs in the RUN cache key. buildroot declares only initramfs and CARGO_INCREMENTAL, and pkgversion / SOURCE_DATE_EPOCH are declared only in build. If the cache ever missed, package would just redo the work as it does today, so this can't make things worse. To confirm, look for Using cache on the buildroot steps of the package build in this PR's CI logs.
  • In the publish job, build-fetch also prefetches the upgrade-source image and buildroot_base, which publishing doesn't use. That adds a few minutes. I kept it simple rather than adding a knob.
  • A permanent failure in a retried step (e.g. a depsolve error) now takes longer to report on main. ci.yml already accepts this tradeoff. Compile errors still fail fast because the build stage isn't retried.

Tested locally with just -n build-fetch and by running _package-fetch / _pull-lbi-images against a fake podman that fails on the first call, to check the retry path and the BOOTC_SKIP_PACKAGE path. I haven't done an end-to-end podman build locally. This PR's CI exercises build-fetch in the integration and upgrade jobs.

Assisted-by: AI

Move the shell `retry` function out of `build-fetch` into a shared
`_retry_fn` variable so other recipes can reuse it. No functional
change.

Assisted-by: AI
`build-fetch` is meant to front-load every network access with retries
so the subsequent `just build` runs offline, but several steps were
missed and show up as flakes in CI:

- `package` implicitly pulled the base image and built the `buildroot`
  stage (install-buildroot, cargo fetch) without retries. Add a
  `_package-fetch` step that pulls the base and builds `--target=buildroot`
  with retries first, so `package` then hits the layer cache.
- The `tools` stage (always built via `sdboot-signed`) runs dnf outside
  of `fetch`. Prefetch `--target=tools` instead, which includes `fetch`.
- `_pull-lbi-images` always contacts the registry, even if the images
  were already pulled by `build-fetch`, relying only on podman's short
  internal retry. Wrap it in the retry helper as well.

Assisted-by: AI
@bootc-bot
bootc-bot Bot requested a review from cgwalters September 23, 2026 09:57
The publish-images jobs ran `just build` directly and so had none of
the network retries that ci.yml gets from `just build-fetch`. Nearly
every failure of this workflow on main in the last month was a
transient quay.io or Fedora mirror error. Run `build-fetch` first with
the same retry parameters as ci.yml.

Assisted-by: AI
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
@ericcurtin
ericcurtin force-pushed the ci-deflake-publish-images branch from 892bc80 to 200fd16 Compare September 23, 2026 10:02
@ericcurtin ericcurtin closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant