ci: De-flake Build and Publish Images with retried fetches - #2489
Closed
ericcurtin wants to merge 3 commits into
Closed
ericcurtin wants to merge 3 commits into
ericcurtin wants to merge 3 commits into
Conversation
Move the shell `retry` function out of `build-fetch` into a shared `_retry_fn` variable so other recipes can reuse it. No functional change. Assisted-by: AI
`build-fetch` is meant to front-load every network access with retries so the subsequent `just build` runs offline, but several steps were missed and show up as flakes in CI: - `package` implicitly pulled the base image and built the `buildroot` stage (install-buildroot, cargo fetch) without retries. Add a `_package-fetch` step that pulls the base and builds `--target=buildroot` with retries first, so `package` then hits the layer cache. - The `tools` stage (always built via `sdboot-signed`) runs dnf outside of `fetch`. Prefetch `--target=tools` instead, which includes `fetch`. - `_pull-lbi-images` always contacts the registry, even if the images were already pulled by `build-fetch`, relying only on podman's short internal retry. Wrap it in the retry helper as well. Assisted-by: AI
The publish-images jobs ran `just build` directly and so had none of the network retries that ci.yml gets from `just build-fetch`. Nearly every failure of this workflow on main in the last month was a transient quay.io or Fedora mirror error. Run `build-fetch` first with the same retry parameters as ci.yml. Assisted-by: AI Signed-off-by: Eric Curtin <eric.curtin@docker.com>
ericcurtin
force-pushed
the
ci-deflake-publish-images
branch
from
September 23, 2026 10:02
892bc80 to
200fd16
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
"Build and Publish Images" on main keeps failing intermittently (most recently on ae375f9). I went through all 12 failures on main since late August. Every one was a transient network error:
unexpected EOFpullingquay.io/{fedora,centos-bootc}/*-bootcfrom the quay CDNjust package(implicit base pull)target-base502 Bad Gateway_pull-lbi-imagesci.ymlalready guards against this withjust build-fetch, which front-loads network access with a retry loop.build-and-publish.ymlnever used it.build-fetchalso missed some network steps:package, which pulls the base image and builds thebuildrootstage (dnf +cargo fetch) without retries.toolsstage, which is always built becausesdboot-signeddepends on it, runs dnf outside--target=fetch._pull-lbi-imagesalways contacts the registry, even when the images are already present. It relied only on podman's short internal retry.Changes
retryshell function moves into a shared_retry_fnvariable._package-fetchstep: pulls the base image and builds--target=buildrootwith retries, sopackagehits the layer cache.--target=toolsinstead of--target=fetch.toolsbuilds on top offetch, so this covers both._pull-lbi-imagesin the retry helper.just build-fetchbeforejust build, with the sameBOOTC_CI_RETRIES=10/BOOTC_CI_DELAY=60asci.yml.Notes / tradeoffs
buildrootdeclares onlyinitramfsandCARGO_INCREMENTAL, andpkgversion/SOURCE_DATE_EPOCHare declared only inbuild. If the cache ever missed,packagewould just redo the work as it does today, so this can't make things worse. To confirm, look forUsing cacheon the buildroot steps of thepackagebuild in this PR's CI logs.build-fetchalso prefetches the upgrade-source image andbuildroot_base, which publishing doesn't use. That adds a few minutes. I kept it simple rather than adding a knob.ci.ymlalready accepts this tradeoff. Compile errors still fail fast because thebuildstage isn't retried.Tested locally with
just -n build-fetchand by running_package-fetch/_pull-lbi-imagesagainst a fakepodmanthat fails on the first call, to check the retry path and theBOOTC_SKIP_PACKAGEpath. I haven't done an end-to-end podman build locally. This PR's CI exercisesbuild-fetchin the integration and upgrade jobs.Assisted-by: AI