fix(ci): shrink per-attempt install timeout, add more retries - #1268
Merged
Conversation
PR #1267 surfaced a real gap in the apt reliability hardening: the apt.conf.d Acquire::http::Timeout/Acquire::Retries tuning does not reliably abort a connection that stalls completely mid-transfer after already receiving some bytes -- observed directly in a failing run: 19:06:45 Get:2 fonts-ipafont-gothic [3513 kB] -- completes 19:10:16 Get:3 fonts-freefont-ttf [5641 kB] -- request issued 19:15:49 (5.5 min of total silence, then our 10m outer timeout fires) On this runner's apt implementation, Acquire::http::Timeout appears to mostly govern connection setup rather than acting as an ongoing no-progress watchdog once a transfer is already in flight, so a connection that goes silent mid-download isn't aborted by that config regardless of the configured value. The one mechanism that has proven reliable every time this class of issue has come up this session is the outer bash `timeout` wrapper -- it fires and kills cleanly on schedule without exception. Shrink it from 10m to 3m per attempt and increase attempts from 3 to 5, so a stalled connection gets abandoned much faster and there are more chances at a working connection, while keeping worst-case total time per install (5 * 3m + 4 * 15s retry sleeps = ~16 min) well below the previous worst case (~30+ min for 3 * 10m). The apt.conf.d hardening from the prior commit is left in place -- it's still correct/harmless, just insufficient on its own for this specific stall shape.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
Contributor
|
| Severity | Count |
|---|---|
| 🔴 Critical | 0 |
| 🟠 High | 2 |
| 🟡 Medium | 5 |
| 🟢 Low | 2 |
| Total | 13 |
📎 Artifacts
- SBOM (CycloneDX JSON) and Grype results available in workflow artifacts
Generated by Supply Chain Verification workflow • View Details
WebKit's --with-deps pulls in the full GStreamer/media-codec stack (~170 packages, ~90MB) that Chromium/Firefox don't need. The 3m per-attempt timeout from the previous commit was killing healthy, non-stalled WebKit installs mid-download on a slow mirror, since a full download legitimately takes close to 10 minutes at reduced throughput. WebKit's two install steps go back to a 10m/3-attempt budget; Chromium/Firefox stay on the fail-fast 3m/5-attempt scheme. Also mask the per-run ephemeral CHARON_ENCRYPTION_KEY before writing it to GITHUB_ENV — it's generated via openssl rand rather than a registered secrets.* value, so GitHub's automatic log redaction never picked it up and it was appearing in plaintext in every job's env dump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #1267's "chore: scope E2E pull_request paths" run surfaced a real gap in the apt reliability hardening from #1263/#1265: the
apt.conf.dAcquire::http::Timeout/Acquire::Retriestuning does not reliably abort a connection that stalls completely mid-transfer after already receiving some bytes. Observed directly in the failing job (E2E Firefox Security Enforcement):On this runner's apt implementation,
Acquire::http::Timeoutappears to mostly govern connection setup rather than acting as an ongoing no-progress watchdog once a transfer is already in flight — a connection that goes silent mid-download isn't aborted by that config, regardless of the configured value.Fix
The one mechanism that has proven reliable every single time this class of issue has come up this session is the outer bash
timeoutwrapper — it fires and kills cleanly on schedule without exception. Shrink it from10mto3mper attempt and increase attempts from 3 to 5, so a stalled connection gets abandoned much faster and there are more chances at a working connection, while keeping worst-case total time per install (5 × 3m + 4 × 15s retry sleeps ≈ 16 min) well below the previous worst case (~30+ min for 3 × 10m).The
apt.conf.dhardening from #1263 is left in place — still correct/harmless, just insufficient on its own for this specific stall shape.Test plan
actionlintcleanlefthook run pre-commitcleantimeout 3m, 5 attempts, updated messages), no stale10m/1 2 3references remain