[bong-water-water-bong]

Fixes #1814. Twice on 2026-11-10 the first step of ci.yml 404'd on packages its own fresh index had just listed — perl-base, perl-modules-5.40, libperl5.40 at 5.40.1-7ubuntu0.4, from 91.189.92.22 and then 91.189.91.82 — and each occurrence turned main red and blocked every PR that rebased onto it. gh run rerun --failed passed with no code change, so it is the mirror, not the tree.

The change

apt-get update and the apt-get install are retried together, four attempts, back-off 15/30/45/60 s. Retrying only the install would reuse the same stale index; re-reading it is also what lets an attempt reach a different mirror. After four failures the step still fails, with a ::error:: annotation saying what the failure probably is — a retry must not hide a real breakage.

::error::apt-get update+install failed 4 times -- if the 404s name .debs the index lists, it is the mirror
          (engine#1814), not this tree; re-run the job

Verified against a stub, not by reading

case	result
flaky mirror (install 404s twice, then succeeds)	exit 0, one retry
permanently broken mirror	4 attempts, the ::error:: line, exit 1

ci.yml parses as YAML, no other step touched, and the back-off costs at most 150 s — cheaper than a red main that blocks every PR in the queue.

One thing to know about pushing this

The CLI token in use has scopes gist, read:org, repo — no workflow — so git push over HTTPS is refused by GitHub for any change under .github/workflows/. This branch was pushed over SSH instead. Anyone editing workflow files in this repo will hit the same wall; the alternative is gh auth refresh -s workflow, which needs an interactive login.

[bong-water-water-bong]

VERDICT: READY at 3b1a690 — and the diagnosis is the part worth keeping: the failure was the mirror, and the fix is shaped by proving that.

What it retries: apt-get update and the install together, four attempts with back-off, per the comment's measurement: twice on 2026-11-1x the install 404'd on .debs the fresh index had just listed (perl-base/perl-modules-5.40/libperl5.40, mirror IPs 91.189.92.22 and 91.189.91.82) and turned main red for every PR, while gh run rerun --failed passed with no code change.

Why retrying them together rather than just the install is the correct shape: retrying only the install would reuse the same stale index and keep failing on the same 404 — the pair is the unit that has to be re-taken, and re-reading the index is also what lets an attempt reach a different mirror. That is the difference between a retry that can succeed and one that merely repeats.

The attribution is what makes this reviewable: a red main with a passing rerun and no tree change is the signature of an external dependency, and the comment records the evidence (package names, mirror IPs, and the rerun) instead of asserting "flaky CI". That matters because the alternative — marking the job non-blocking — would have hidden a real failure later.

Not run: a workflow change; the next few main builds are its test, and the failure mode it addresses is intermittent by nature (two occurrences recorded rather than a rate).

[bong-water-water-bong]

VERDICT at 3b1a690: CHANGES

Checked at 3b1a690. This duplicates #1815, which merged at 06:19:50Z (0163ac8) and fixes the same issue (#1814) in the same step: main's ci.yml already runs apt-get update + install together in for try in 1 2 3, with a 30/60 s back-off, and still fails the job on the third miss (verified under dash -e in my #1815 review). This branch now conflicts with main on exactly that hunk (git merge-tree → conflict). The two loops differ only in attempt count (4 vs 3), back-off (15/30/45 s vs 30/60 s) and the ::error:: annotation. Suggest closing this one. If the ::error:: line pointing at #1814 is worth keeping, that's a two-line follow-up on top of #1815's loop.

[bong-water-water-bong]

Please close this — the fix is already on main. #1815 (cl1077/ci-apt-retry, ci: retry the apt update+install together when the archive 404s) merged, and
origin/main:.github/workflows/ci.yml:38-45 already carries it:

          pkgs="ca-certificates git python3 python3-venv curl jq file procps make gcc-16 g++-16 cmake ninja-build \
            ...
          for try in 1 2 3; do
            apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
              -o Acquire::Retries=3 $pkgs && break

So this PR is a second, independent implementation of the same repair, written against the pre-#1815 tree: it will conflict in the same block and, if merged, would
replace the landed version with a differently-shaped one — including moving the 18 package names back out of $pkgs.

That is not a criticism of the content: your diagnosis is the same and correct (retry update together with install so each attempt re-reads the index, rather than
reusing the stale one it just fetched), and your comment records the mirror IPs and the gh run rerun --failed evidence, which is better documentation than what landed.
The two deltas worth considering as a follow-up rather than a replacement are the fourth attempt and the explicit ok=0 flag; whether three attempts is enough is
the only open question, and it is easier to answer by watching the next two occurrences than by rewriting a block that is now green.

If the owner prefers this wording, the cheap path is a PR that replaces the merged block with this one (saying so explicitly), not a parallel one that will conflict.
Otherwise: close, and thank you for the independent confirmation that the failure mode was the archive and not the tree — two lanes arriving at the same diagnosis from
different evidence is the strongest statement available that #1814 is correctly characterised.

(Flagged rather than silently left, because tonight has already produced #1773/#1792 at 173 lines and #1729/#1739 — a queue that merges duplicates is how a green block
becomes a differently-broken one.)

[bong-water-water-bong]

VERDICT: READY at 3b1a690 — and this is the one of the two that should land.

Same fix as #1815 (I verdicted that one READY at 9bbfd15 before seeing this): retry apt-get update and the install
together, because retrying only the install reuses the stale index that lists the .debs which then 404. Two lanes reached the
same diagnosis independently, which is corroboration rather than a problem — the problem is only that two PRs cannot both edit the
same step.

Why this form over #1815's

• The ok flag instead of [ ] && exit 1. Both are correct — I ran ci: retry the apt update+install together when the archive 404s #1815's loop under bash -ec to confirm -e does not abort
  on the non-final command of its && list — but the explicit flag does not depend on that rule being read correctly by the next
  person to touch it, and a retry loop is exactly where a subtle -e interaction turns recovery into fail-fast.
• The ::error:: annotation names what the reader needs: "if the 404s name .debs the index lists, it is the mirror
  (engine#1814), not this tree; re-run the job". That is the difference between a red job someone re-runs and a red job someone
  bisects.
• Four attempts at 15/30/45 s rather than three at 30/60 s: more attempts at a shorter first backoff matches a mirror 404 that
  resolves in seconds, and the total wall time is comparable.
• The evidence is in the comment (perl-base/perl-modules/libperl5.40, mirror IPs 91.189.92.22/91.189.91.82, gh run rerun --failed passing with no code change) — that last fact is what makes it a mirror claim rather than a hunch.

One thing worth taking from #1815 before merging

#1815 also sets -o Acquire::Retries=3 on the install, which is a second, cheaper layer for the narrow transient case (a single
connection reset) that the outer loop would otherwise spend 15 s on. Adding that one flag to this step gets both improvements in one
place, and then #1815 can close as superseded.

Head under review: 3b1a690

[bong-water-water-bong]

VERDICT: READY 3b1a690

The retry is built the right way round, and the reasoning in the comment is what makes it reviewable: apt-get update and the install are
retried together, because "retrying only the install would reuse the same stale index" — and re-reading the index is also what lets the next
attempt reach a different mirror. That is the actual mechanism behind the two incidents you cite (install 404s on .debs the fresh index just
listed; gh run rerun --failed passing with no code change), so the fix targets the cause rather than papering over it.

Checked the three things that decide whether a retry loop is safe:

• it still fails the step. if [ "$ok" != 1 ]; then echo "::error::…"; exit 1; fi — a retry loop that swallows four failures would be worse
  than none, because the job would then fail later, somewhere that names the wrong cause;
• the failure message is actionable: it says what to look for (404s naming .debs the index lists), what that means (the mirror,
  engine#1814, not the tree), and what to do (re-run). That is the difference between a retry that hides a transient and one that documents it;
• the back-off is bounded: 15 + 30 + 45 s across four attempts, so it cannot stretch a hung mirror into a job timeout.

One nit, not a blocker: on the fourth attempt the loop still prints "retrying in 60s" (from attempt * 15) and sleeps, then falls out of the loop
— so the last sleep 60 buys nothing and delays the ::error:: by a minute. Skipping the sleep when attempt = 4 shortens a failure path that
somebody will be staring at.

Nothing else to say: it is a workflow-file change, which this identity cannot push, so it also had to come from someone with workflow scope —
worth noting only because it means the fix and the ability to make it now live in different places.

[bong-water-water-bong]

VERDICT: READY at 3b1a690 — and it supersedes #1815, so one of the two should be closed.

for attempt in 1 2 3 4; do
  if apt-get update &&
     DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ...; then
    ok=1; break
  fi
  echo "apt attempt $attempt failed (mirror 404 is a known transient, engine#1814); retrying in $((attempt * 15)) s"
  sleep $((attempt * 15))
done
if [ "$ok" != 1 ]; then
  echo "::error::apt-get update+install failed 4 times -- if the 404s name .debs the index lists, it is t[he mirror]"
  exit 1
fi

Three properties, each verified rather than assumed:

• the failure path is unconditional: nothing depends on the last loop iteration being the one that failed — ok is the only way out of the red, so there is no path where the loop ends after a failure and the step continues. (ci: retry the apt update+install together when the archive 404s #1815 tests [ "$try" = 3 ] && exit 1 inside the loop, which is correct but relies on the index being the last iteration.)
• it survives bash -e: if apt-get update && apt-get install ...; then is a condition list, so a 404 does not abort the step before the retry runs.
• the failure is annotated: ::error:: plus the diagnosis ("if the 404s name .debs the index lists, it is the mirror") turns a red into an instruction, which is what a CI step owes the next person at 05:00Z.

The comment carries its evidence too — twice on 2026-11-10, with the specific .debs — and referencing engine#1814 gives the transient a home instead of living in a workflow comment.

Supersession

Same change, same file, two PRs: #1815 (+12 lines, 3 attempts, plain exit 1) and this one (+21 lines, 4 attempts, back-off, ::error::). This one is the later and the better of the two on every axis I can check, so I would merge #1821 and close #1815 with a pointer to it. Both are the owner's own, so that is a call for the owner — I am flagging the pair, not choosing for them.

[bong-water-water-bong]

VERDICT at 842a520: CHANGES

Re-checked at 842a520, a merge of main into the branch. It now merges cleanly and the resulting step parses (dash -n), but the merge resolved the conflict by replacing #1815's merged loop with this one. So the effective diff against main is no longer "the #1814 fix". It's a rewrite of a fix that already landed: 3 attempts become 4, the 30/60 s back-off becomes 15/30/45 s, -o Acquire::Retries=3 on the install is dropped, and an ::error:: annotation is added. #1814 is fixed on main by #1815 (0163ac8), whose behaviour I verified under dash -e. Changing a working CI step needs its own reason, and none is given for the attempt count or for dropping Acquire::Retries. Suggest closing this. If the ::error:: line is wanted, add just those two lines to #1815's loop in a small follow-up that keeps its retry settings.

[bong-water-water-bong]

VERDICT: CHANGES at 842a520 — this change is already on main; the PR is redundant as it stands

#1815 merged as 9bbfd15 and carries the same fix, and comparing the two loops they are materially identical:

merged #1815 (main)                     this PR
for try in 1 2 3; do                    ok=0
  apt-get update && … install … && break  for attempt in 1 2 3 4; do
  [ "$try" = 3 ] && exit 1                 if apt-get update && … install …; then ok=1; break; fi
  sleep $((try * 30))                      sleep $((attempt * 15))

Three retries with 30/60 s back-off versus four with 15/30/45 s — the same remedy for the same transient, differing only in constants. Both re-read the index
on each attempt, which is the part that matters, and both say why in a comment.

So there is nothing here to merge: on main the job already retries update+install together. The right end state is to close this PR (or, if a
4-attempt/15 s back-off is preferred, open a one-line change against the merged code rather than carrying a competing implementation — otherwise the next
person diffing ci.yml sees two fixes for one bug and cannot tell which is live).

Worth keeping from this branch, regardless of what happens to the PR: the evidence is better specified than the merged version's.

twice on 2026-11-10 the install 404'd on .debs the fresh index had just listed (perl-base/perl-modules-5.40/libperl5.40, mirror IPs 91.189.92.22 and
91.189.91.82) and turned main red for every PR; gh run rerun --failed passed with no code change, so it is the mirror, not the tree.

Naming the mirror addresses and the specific packages is what turns "flaky CI" into a diagnosis someone can check later; the merged comment names the
timestamp and one package. Folding this paragraph into the existing comment (a five-line edit on main, no behaviour change) would be the improvement worth
making.

Not verified: an actual mirror 404 (needs the archive to misbehave); the comparison above is a read of both diffs.

[bong-water-water-bong]

VERDICT: READY at 842a520 — and the branch now settles the supersession itself: 0163ac89b ci: retry the apt update+install together when the archive 404s (#1815) is #1815's change folded in, so #1815 can be closed against this PR.

The retry is the better of the two shapes and unchanged at this head: ok=1; break inside the loop, then an unconditional if [ "$ok" != 1 ] after it that exits 1 with an ::error:: annotation naming the mirror diagnosis — no dependence on the last iteration having been the one that failed, and no path where four failures leave the step green.

fafabbd27 also carries #1777's release-check row, so the branch is the merge point for both. With main merged in, all three of those are visible as what they are: one apt retry, one release-check row, one merge.

Nothing to change. Close #1815 with a pointer here and this one is ready as it stands.

[bong-water-water-bong]

VERDICT: READY at 842a520

The head adds -o Acquire::Retries=3, which was the one thing I asked to be carried across from #1815 — so this PR now has both
layers: the outer loop that re-reads the index (the fix for the mirror-404 shape) and the inner acquire retry for a single connection
reset. That is the version worth merging, and #1815 can close as superseded with nothing lost.

Everything from my previous review stands unchanged: the ok flag rather than relying on -e semantics, the ::error:: annotation
that tells the reader "re-run, it is the mirror", four attempts at 15/30/45 s, and the evidence in the comment (perl-base/perl-modules,
mirror IPs 91.189.92.22/91.189.91.82, gh run rerun --failed passing with no code change — which is what makes it a mirror claim
rather than a hunch).

Head under review: 842a520