Skip to content

fix(ci): shrink per-attempt install timeout, add more retries - #1268

Merged
Wikid82 merged 2 commits into
mainfrom
fix/apt-shrink-per-attempt-timeout
Aug 19, 2026
Merged

fix(ci): shrink per-attempt install timeout, add more retries#1268
Wikid82 merged 2 commits into
mainfrom
fix/apt-shrink-per-attempt-timeout

Conversation

@Wikid82

@Wikid82 Wikid82 commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Summary

PR #1267's "chore: scope E2E pull_request paths" run surfaced a real gap in the apt reliability hardening from #1263/#1265: the apt.conf.d Acquire::http::Timeout/Acquire::Retries tuning does not reliably abort a connection that stalls completely mid-transfer after already receiving some bytes. Observed directly in the failing job (E2E Firefox Security Enforcement):

19:06:45  Get:2 fonts-ipafont-gothic [3513 kB]   ← completes
19:10:16  Get:3 fonts-freefont-ttf [5641 kB]     ← request issued
19:15:49  (5.5 min of total silence, then our 10m outer timeout fires)

On this runner's apt implementation, Acquire::http::Timeout appears to mostly govern connection setup rather than acting as an ongoing no-progress watchdog once a transfer is already in flight — a connection that goes silent mid-download isn't aborted by that config, regardless of the configured value.

Fix

The one mechanism that has proven reliable every single time this class of issue has come up this session is the outer bash timeout wrapper — it fires and kills cleanly on schedule without exception. Shrink it from 10m to 3m per attempt and increase attempts from 3 to 5, so a stalled connection gets abandoned much faster and there are more chances at a working connection, while keeping worst-case total time per install (5 × 3m + 4 × 15s retry sleeps ≈ 16 min) well below the previous worst case (~30+ min for 3 × 10m).

The apt.conf.d hardening from #1263 is left in place — still correct/harmless, just insufficient on its own for this specific stall shape.

Test plan

  • actionlint clean
  • lefthook run pre-commit clean
  • Confirmed all 10 install-retry sites updated consistently (timeout 3m, 5 attempts, updated messages), no stale 10m/1 2 3 references remain

PR #1267 surfaced a real gap in the apt reliability hardening: the
apt.conf.d Acquire::http::Timeout/Acquire::Retries tuning does not
reliably abort a connection that stalls completely mid-transfer after
already receiving some bytes -- observed directly in a failing run:

  19:06:45  Get:2 fonts-ipafont-gothic [3513 kB]   -- completes
  19:10:16  Get:3 fonts-freefont-ttf [5641 kB]     -- request issued
  19:15:49  (5.5 min of total silence, then our 10m outer timeout fires)

On this runner's apt implementation, Acquire::http::Timeout appears to
mostly govern connection setup rather than acting as an ongoing
no-progress watchdog once a transfer is already in flight, so a
connection that goes silent mid-download isn't aborted by that config
regardless of the configured value.

The one mechanism that has proven reliable every time this class of
issue has come up this session is the outer bash `timeout` wrapper --
it fires and kills cleanly on schedule without exception. Shrink it
from 10m to 3m per attempt and increase attempts from 3 to 5, so a
stalled connection gets abandoned much faster and there are more
chances at a working connection, while keeping worst-case total time
per install (5 * 3m + 4 * 15s retry sleeps = ~16 min) well below the
previous worst case (~30+ min for 3 * 10m).

The apt.conf.d hardening from the prior commit is left in place --
it's still correct/harmless, just insufficient on its own for this
specific stall shape.
@codecov

codecov Bot commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-advanced-security

Copy link
Copy Markdown
Contributor

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

⚠️ Supply Chain Verification Results

⚠️ WARNING

📦 SBOM Summary

  • Components: 1753

🔍 Vulnerability Scan

Severity Count
🔴 Critical 0
🟠 High 2
🟡 Medium 5
🟢 Low 2
Total 13

📎 Artifacts

  • SBOM (CycloneDX JSON) and Grype results available in workflow artifacts

Generated by Supply Chain Verification workflow • View Details

WebKit's --with-deps pulls in the full GStreamer/media-codec stack
(~170 packages, ~90MB) that Chromium/Firefox don't need. The 3m
per-attempt timeout from the previous commit was killing healthy,
non-stalled WebKit installs mid-download on a slow mirror, since a
full download legitimately takes close to 10 minutes at reduced
throughput. WebKit's two install steps go back to a 10m/3-attempt
budget; Chromium/Firefox stay on the fail-fast 3m/5-attempt scheme.

Also mask the per-run ephemeral CHARON_ENCRYPTION_KEY before writing
it to GITHUB_ENV — it's generated via openssl rand rather than a
registered secrets.* value, so GitHub's automatic log redaction never
picked it up and it was appearing in plaintext in every job's env dump.
@Wikid82
Wikid82 merged commit 67afe53 into main Aug 19, 2026
57 of 66 checks passed
@Wikid82
Wikid82 deleted the fix/apt-shrink-per-attempt-timeout branch August 19, 2026 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants