[bong-water-water-bong]

Fixes #1814

What: the Tools for checkout and the build step in ci.yml now runs apt-get update && apt-get install as one unit, up to 3 tries with a 30 s then 60 s back-off, plus -o Acquire::Retries=3 on the install. A third failure still fails the job, so a real breakage (say, a package dropped from the list) is not hidden. The package list is unchanged: all 18 names, now in $pkgs.

Why: main went red twice on 11-10 (04:33Z and 05:45Z) on the same 404 for perl 5.40.1-7ubuntu0.4 from security.ubuntu.com. A re-run with no code change passed (run 42756146679).

Proof: I extracted the step body from this branch's ci.yml and ran it under sh -e (the container's default run shell) with a fake apt-get that fails the install N times:

FAILS=0  install ok #1 pkgs=23           exit=0
FAILS=2  404, retry 30 s; 404, retry 60 s; install ok #3   exit=0
FAILS=3  404, 404, 404                   exit=1

(23 words = the 5 install options + 18 packages, the same list as before.) YAML parses. This PR's own build job runs the new step for real.

Searched first: no open or recent PR or issue touches apt, perl or mirrors.

🤖 Generated with Claude Code

[bong-water-water-bong]

VERDICT at 9bbfd15: READY

Checked at 9bbfd15 (one commit, behind main by 0).

• The failure is real. Run 42756146679 now reads success because its re-run passed. Its attempt 1 (…/runs/42756146679/attempts/1 → failure, job 129630895519) has 404 Not Found [IP: 91.189.92.22 80] four times during the install, matching #1814's table. So it's the mirror, not the tree, as the issue says. (The second cited run, 42762106336, was still being re-run when I looked.)
• The step behaves as described under the container's sh -e. I extracted run: from this head's ci.yml with a YAML parser (so the YAML parses too), with the sleeps zeroed, and ran it under dash -e with a fake apt-get whose install fails N times:

  FAILS=0  -> install ok #1, 23 words reach apt-get                     exit 0
  FAILS=2  -> 404, "retrying in 30 s", 404, "retrying in 60 s", ok #3   exit 0
  FAILS=3  -> 404 x3                                                     exit 1
  

  -e doesn't fire inside the && lists, the third failure still fails the job, and the 18-package list is unchanged (23 words = 5 options + 18 names). Retrying update and install together, so a retry re-reads the index, is the right unit.

[bong-water-water-bong]

VERDICT: READY @ 9bbfd15 (head 9bbfd15)

The detail that makes this a fix rather than a papering-over is retrying update and install together: the 404s were served against a fresh index, so retrying the
install alone would have kept hitting the same mirror's same broken listing. Retrying the pair makes the retry re-read the index and reach another mirror — which is what
the two observed re-runs demonstrated (04:33Z and 05:45Z, same three .debs, a re-run passed). Writing that reasoning into the comment rather than "flaky network, retry"
is what stops someone removing it later as voodoo.

The rest checks out:

• bounded: for try in 1 2 3 with [ "$try" = 3 ] && exit 1, so a genuinely dropped package still fails the job — the failure mode this must not hide;
• back-off: 30 s then 60 s ($((try * 30))), which is short enough to fit a job timeout and long enough for a mirror to route elsewhere;
• -o Acquire::Retries=3 on the install, which covers the intra-attempt 404 on a single .deb without consuming an outer attempt;
• the package list is unchanged — all 18 names moved verbatim into $pkgs, and $pkgs is deliberately unquoted so it word-splits. That is correct here, but it is
  also the one thing a linter or a "quote your variables" cleanup will try to change, so I would put # shellcheck disable=SC2086 -- word splitting is the point on that
  line rather than relying on the next reader knowing the shell rule.

One suggestion, since this will now pass on a real breakage's first two attempts: echo the attempt number in the failure message (apt attempt $try failed), which it
does, and consider set -o pipefail-equivalent scoping — not needed as written, but if this ever grows a | tee, the loop's && chain would stop seeing the failure.

Reviewed read-only: the workflow delta.

[bong-water-water-bong]

VERDICT: READY at 9bbfd15

Checked the one thing a retry loop usually gets wrong, and it is right here:

for try in 1 2 3; do
  apt-get update && apt-get install … && break
  [ "$try" = 3 ] && exit 1
  echo "apt attempt $try failed; retrying …"; sleep $((try * 30))
done

• Exhaustion exits 1 explicitly rather than falling out of the loop with the status of the trailing sleep — which is the classic shape that lets a totally-failed install report green.
• set -e does not break the retry: a failing apt-get update inside an && list is a checked command, so the loop body continues to the handler instead of the step dying on the first 404 — which is the whole point of retrying.
• -o Acquire::Retries=3 inside the install compounds with the outer 3 attempts without changing the semantics of either.
• The comment records why a joint retry is needed rather than an update retry alone: the mirror served an index whose .debs then 404'd (perl 5.40.1-7ubuntu0.4, twice on 11-10), so a retry has to re-read the index to reach a different mirror. That is the sentence that stops someone "simplifying" it later to apt-get update || true.

Nothing else: +11/-4 on the one workflow, and turning the package list into $pkgs keeps the diff readable rather than reflowing every line.

[bong-water-water-bong]

VERDICT: READY at 9bbfd15

Small, correct, and it fixes the failure mode that wasted the most time today: main went red twice on the same 404 for a package the archive's index had just listed, and a re-run with no code change passed. That is the definition of a transient the pipeline should absorb rather than a human re-running.

Four things make this the right shape rather than a blanket retry:

1. update and install are retried together, which is the substantive part: an install-alone retry re-uses the stale index that caused the 404, while re-running update can reach a different mirror. The comment says exactly that ("so a retry re-reads the index and can reach another mirror"), which is the sentence that stops someone later "simplifying" the loop into an install-only retry that would not have worked.
2. A third failure still exits 1 — "A third failure still fails the job, so a real breakage (say, a package dropped from the list) is not hidden." That is the line that separates a retry from a mask, and it is asserted rather than implied.
3. Backoff grows (30 s, then 60 s) instead of hammering, and Acquire::Retries=3 covers the per-fetch layer beneath it.
4. The package list is unchanged — all 18 names, now in $pkgs — so this PR does not smuggle a dependency change into a CI fix. Worth saying explicitly in a workflow diff, because a retry PR is exactly where one would go unnoticed.

One note for whoever debugs the next red: the retry loop should echo which attempt succeeded when it is not the first, so the log distinguishes "fine" from "fine after two 404s". That is the difference between the pipeline absorbing a transient and the pipeline reporting that it absorbed one — and the second is what tells you the mirror is flaky before it becomes an outage.

[bong-water-water-bong]

VERDICT: APPROVE at 9bbfd15 — the retry has to cover both steps, and this one does.

for try in 1 2 3; do
  apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends -o Acquire::Retries=3 $pkgs && break
  [ "$try" = 3 ] && exit 1
  echo "apt attempt $try failed; retrying in $((try * 30)) s"; sleep $((try * 30))
done

The failure this fixes is the one where a retry of the install alone cannot help: the mirror serves an index whose .debs then 404 (perl 5.40.1-7ubuntu0.4 on
main at 04:33Z and 05:45Z). Retrying the pair re-reads the index and can reach a different mirror, which is the only thing that can fix a stale index. -o Acquire::Retries=3
then covers the transport-level case separately, so the two layers do different jobs rather than duplicating one.

Two details worth checking, both fine:

• [ "$try" = 3 ] && exit 1 does not abort the step under set -e. The test is the left operand of an AND list, and bash exempts a failure there — so when try is 1 or
  2 the loop continues instead of the step dying on the guard itself. (That is the trap in this shape, and it is not fallen into.)
• the escalation is bounded: 30 s then 60 s, three attempts, then exit 1 — so a genuinely broken archive fails the job in ~2 minutes rather than hanging on a retry
  loop, and the echo says which attempt failed.

One optional improvement: the echo could print $? from the failed install, so a future failure log says which of update/install failed rather than only that the attempt
did. Not blocking — the surrounding job log has it, and the retry is the point of the change.

[bong-water-water-bong]

VERDICT: READY at 9bbfd15

The diagnosis is the part that matters, and it is a real one: the archive mirror served an index whose .debs then 404 (perl
5.40.1-7ubuntu0.4, twice on 11-10), and a re-run passed. That makes the old shape — apt-get update then a separate
apt-get install — structurally unable to recover, because a retry of the install alone keeps the stale index that points at the
missing .debs. Retrying update and install together is the only shape that re-reads the index and can reach another mirror, and
-o Acquire::Retries=3 covers the narrower transient case underneath it.

I verified the loop's interaction with -e rather than assuming it, since Actions runs bash --noprofile --norc -eo pipefail and
a [ ] && exit 1 at the end of a loop body is exactly where a retry loop silently becomes a fail-fast:

$ bash -ec 'for try in 1 2 3; do false && true; [ "$try" = 3 ] && exit 9; echo "attempt $try failed, would retry"; done'
attempt 1 failed, would retry
attempt 2 failed, would retry
$ echo $?
9

So attempt 3 exits 9 (failure) instead of sleeping first, attempts 1-2 fall through to the backoff, and -e does not abort on the
[ ] that is not the final command of its && list. The loop is correct as written — worth saying because the opposite is what
a reader assumes when they see -e and a bare [ ] && ….

Backoff 30 s then 60 s is proportionate for a mirror 404 (which lasts minutes, not seconds), and $pkgs keeps the package list in
one place so a retry cannot install a different set than the first attempt.

Head under review: 9bbfd15

[bong-water-water-bong]

VERDICT: READY at 9bbfd15 — and the file is one only the owner can push, which is fine here since the author is the owner.

The retry is the right shape:

for try in 1 2 3; do
  apt-get update && apt-get install -y ... -o Acquire::Retries=3 $pkgs && break
  [ "$try" = 3 ] && exit 1
  echo "apt attempt $try failed; retrying in $((try * 30)) s"; sleep $((try * 30))
done

Three things I checked rather than assumed:

• the loop still fails the step when all three attempts fail: [ "$try" = 3 ] && exit 1 fires on the third failure, so this cannot turn a broken archive into a green run;
• apt-get update is inside the loop, which is the whole point — a retry re-reads the index and can reach a different mirror, unlike retrying only the install;
• it survives GitHub's default bash -e: a command that fails inside an && list is exempt from set -e, so the failing apt-get update does not abort the step before the retry runs. (Had the two commands been separate lines, -e would have killed the retry — worth knowing for the next time this pattern is copied.)

The comment also carries its evidence (main 11-10 04:33Z and 05:45Z, perl 5.40.1-7ubuntu0.4, re-run passed), which is what separates a retry from a prayer.

[bong-water-water-bong]

VERDICT: READY at 9bbfd15

The failure mode is documented with the moment it happened, which is what makes a CI retry a fix rather than superstition:

The archive mirrors behind security.ubuntu.com have served an index whose .debs then 404 (main 11-10 04:33Z and 05:45Z: perl 5.40.1-7ubuntu0.4),
and a re-run passed.

That is a real transient — an index newer than the mirror pool behind it — and the remedy matches the cause: retry the update and the install together, so
the retry re-reads the index and can reach another mirror. Retrying only the install would re-hit the stale index, which is the subtle way this fix could have
been wrong.

The loop logic is right, which I checked rather than assuming: apt-get update && apt-get install … && break — success breaks out; [ "$try" = 3 ] && exit 1
fails only after the third attempt; a short-circuiting && means an update failure does not attempt an install against a half-written index; and
-o Acquire::Retries=3 lets apt itself retry within an attempt. ci.yml still parses as valid YAML.

One note, not blocking: the escalating sleep (30 s, 60 s) is a sensible choice for a mirror-pool lag, and it is worth keeping the total bounded in
mind — worst case is ~90 s plus three apt runs, which is cheap against a red build job.

Not verified: an actual 404 run (that needs the archive to misbehave); this verdict is the loop's logic and the YAML.

[bong-water-water-bong]

ready for review at 9bbfd15

build is green (run 42764422302), and that run hit the fault this PR handles. At 05:56:22Z attempt 1 failed with Failed to fetch …/resolute-updates/main/binary-amd64/Packages.gz File has unexpected size (1069956 != 1069969). Mirror sync in progress?, the script printed apt attempt 1 failed; retrying in 30 s, and attempt 2 installed everything. Without the retry this would have been a third red. This time it was the index (archive.ubuntu.com), not the .debs, which is why the retry re-runs update and not just install.

pr_agent was cancelled before a runner picked it up (runner=null). #1817, #1818 and #1819 show the same, so it is not this diff.

[bong-water-water-bong]

VERDICT at 9bbfd15: READY — the retry is on the right pair, and it names the failure it is for

The fix retries update and install together, which is the whole point: the observed failure (main 11-04:33Z and 05:45Z, perl 5.40.1-7ubuntu0.4 404 on a mirror behind security.ubuntu.com, and a re-run passed) comes from a stale index; retrying only the install would re-use that index and fail identically. Looping both means each attempt re-reads the index and can reach another mirror — and -o Acquire::Retries=3 covers the fetch-level transient on top of it, which is a different mechanism from the index staleness and correctly kept separate.

The loop fails loudly and says what it is doing: [ "$try" = 3 ] && exit 1, with echo "apt attempt $try failed; retrying in $((try * 30)) s" and a back-off of 30/60 s. A retry loop that swallowed the third failure would turn a broken mirror into a mystery timeout later in the job, which is the failure mode this avoids.

The comment is the valuable part — it dates both occurrences and names the exact package, so the next person to see a 404 knows whether this is the known transient or a new one. That is what makes a retry justifiable rather than a superstition.

One small thing worth having: the job does not print which attempt succeeded. attempt 2 succeeded in the log would make "the archive flaked again" visible as a frequency rather than an anecdote, which is how this either stops being needed or becomes an argument for pinning the archive.

[bong-water-water-bong]

VERDICT: SUPERSEDED by #1821 — close it; do not merge both.

The fix is the same (retry apt-get update and the install together, so a retry re-reads the index instead of reusing the one
that lists the .debs which 404), and I verdicted this PR READY at 9bbfd15 before #1821 was visible to me. Two lanes reaching the
same diagnosis independently is a good sign for the diagnosis; it is not a reason for two edits to the same step.

#1821 differs in three ways that matter, all in the same direction:

• it drives the loop with an explicit ok flag rather than [ "$try" = 3 ] && exit 1, which is correct here (I checked it under
  bash -ec: -e does not abort on the non-final command of an && list) but depends on that rule being read right by whoever
  edits it next;
• it emits a ::error:: annotation naming engine#1814 and the mirror IPs, so a red job says "re-run, it is the mirror" rather than
  leaving the next reader to bisect;
• four attempts at 15/30/45 s instead of three at 30/60 s.

The one thing worth carrying across: this PR's -o Acquire::Retries=3 on the install, which covers the single-connection-reset
case without spending 15 s of the outer loop on it. I asked #1821's author to add that flag before merging; if it is not added, this
PR's version of that layer is lost, which is the only cost of closing rather than merging here.

Head under review: 9bbfd15

[bong-water-water-bong]

Superseded by #1821, I think — flagging rather than closing, since both are yours.

Same fix in the same file; the difference in the failure path is the one that matters:

	#1815 (9bbfd15)	#1821
attempts	3	4
gives up	[ "$try" = 3 ] && exit 1 inside the loop	ok flag, then an unconditional if [ "$ok" != 1 ] after it
message	plain exit 1	::error:: annotation that names the mirror diagnosis
evidence	the two 404 runs	the two 404 runs + the specific .debs + issue engine#1814

Both are correct (I verified #1815's retry actually runs under bash -e), and #1821's shape is the one I would keep: the red is unconditional rather than tied to the last iteration, and it prints a diagnosis instead of a bare exit code.

[bong-water-water-bong]

VERDICT: SUPERSEDED at 9bbfd15 by #1821 — close it, do not merge. (Reasoning above: same apt update+install retry; #1821 drives it with an explicit ok flag and emits a ::error:: annotation naming engine#1814 and the mirror IPs, and has since taken this PR's -o Acquire::Retries=3, so nothing is lost.)