Skip to content

ci: retry apt-get update+install together, so a mirror 404 stops turning main red (#1814) #1821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bong-water-water-bong merged 2 commits into from
Nov 10, 2026
Merged

ci: retry apt-get update+install together, so a mirror 404 stops turning main red (#1814) #1821

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3b1a690
ci: retry apt-get update+install together, so a mirror 404 stops turn…
Nov 10, 2026
842a520
merge main into ci/1814-apt-retry
Nov 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,26 @@ jobs:
steps:
- name: Tools for checkout and the build
run: |
pkgs="ca-certificates git python3 python3-venv curl jq file procps make gcc-16 g++-16 cmake ninja-build \
libcurl4-openssl-dev libzstd-dev libssl-dev libdrm-dev libpcre2-dev"
# The archive mirrors behind security.ubuntu.com have served an index whose .debs then 404
# (main 11-10 04:33Z and 05:45Z: perl 5.40.1-7ubuntu0.4), and a re-run passed. Retry the
# update and the install together, so a retry re-reads the index and can reach another mirror.
for try in 1 2 3; do
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-o Acquire::Retries=3 $pkgs && break
[ "$try" = 3 ] && exit 1
echo "apt attempt $try failed; retrying in $((try * 30)) s"; sleep $((try * 30))
# engine#1814: `apt-get update` and the install must be retried TOGETHER, with a back-off. Twice on 2026-11-10 the
# install 404'd on .debs the fresh index had just listed (perl-base/perl-modules-5.40/libperl5.40, mirror IPs
# 91.189.92.22 and 91.189.91.82) and turned main red for every PR; `gh run rerun --failed` passed with no code
# change, so it is the mirror, not the tree. Retrying only the install would reuse the same stale index, so each
# attempt re-reads the index -- which is also what lets it reach a different mirror.
ok=0
for attempt in 1 2 3 4; do
if apt-get update &&
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ca-certificates git \
python3 python3-venv curl jq file procps make gcc-16 g++-16 cmake ninja-build libcurl4-openssl-dev \
libzstd-dev libssl-dev libdrm-dev libpcre2-dev; then
ok=1; break
fi
echo "apt attempt $attempt failed (mirror 404 is a known transient, engine#1814); retrying in $((attempt * 15))s"
sleep $((attempt * 15))
done
if [ "$ok" != 1 ]; then
echo "::error::apt-get update+install failed 4 times -- if the 404s name .debs the index lists, it is the mirror (engine#1814), not this tree; re-run the job"
exit 1
fi
git config --global --add safe.directory "$GITHUB_WORKSPACE"
python3 -m venv /opt/ci-venv && echo /opt/ci-venv/bin >> "$GITHUB_PATH"
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
Expand Down
Loading