Skip to content

Release only green commits, and let a Runner installed from a release update itself - #86

Merged
codefusion-automerge[bot] merged 6 commits into
mainfrom
feat/release-self-update
Nov 12, 2026
Merged

codefusion-automerge[bot] merged 6 commits into
mainfrom
feat/release-self-update

Conversation

@XeonFX

@XeonFX XeonFX commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

lon1, pdx1, str1 and waw1 run the Runner from a release, and the console flagged both "runs Runner commits that aren't green" and "its Runner can't update itself". Two causes, two fixes.

Only green commits are released

release.yml published on every push to main, before CI ended, so runner-0.1.0-4768026b6a4c went out with red CI. Now:

  • Runner build (runner-build.yml) builds the two Linux programs on every push to main and keeps them as artifacts for a week. It publishes nothing.
  • Runner release (release.yml) runs when CI, Image or Runner build ends for a push to main. scripts/release-gate.sh publishes only when the commit is on main and the push's newest CI run, Image run (if the push made one) and Runner build run are green. That is the console's rollout rule plus the build. Otherwise it leaves a notice saying "wait" or "skip". The gate checks out main's scripts and holds no write permission. The publish job runs no repository code and takes the programs from the build run the gate named. workflow_dispatch with a commit does the same check by hand.

A Runner from a release updates itself

A Runner with no build_folder now uses ReleaseUpdate (crates/runner/src/release_update.rs). It lists update-to only, and refuses plain update with a reason. On update-to <commit> it:

  • finds that commit's release through GitHub's API with no token. The release must not be a draft, must be by github-actions[bot], at the whole commit, on main and newer than the commit that runs.
  • waits up to 45 minutes for a release that is not out yet.
  • fetches SHA256SUMS and this host's codefusion-runner-linux-{amd64,arm64}, following GitHub's redirect to its storage with no token, and checks the SHA-256.
  • runs the program's version, which must run on this host and name the commit.
  • drains, installs and guards the start exactly as a build update does. That is the Installer now shared with CheckoutUpdate.

It refuses a commit with no release, an older one, a bad checksum, a missing file, the wrong architecture or a full disk, and the running program stays. The new cf_ghapp::Releases streams and hashes downloads, starts again from byte 0 after a break, 429 or 5xx, and stops at once on a full disk or when the Runner ends. Two changes reach every GitHub call: a used-up rate limit (x-ratelimit-remaining: 0) is now waited out until its reset, and RetryPolicy::wait_after is the one retry rule.

host-setup.sh install --version X is now a minimum. An enrolled host whose Runner runs a newer commit of main, by GitHub's compare (retrying 429 and 5xx), keeps it. If GitHub can't say (rate limit, 5xx, an unknown commit), the run fails, changes nothing and says why, so a requested release is never silently skipped.

Tests

Mocks cover network and time only. Each new check was broken on purpose and its test went red:

  • crates/runner/src/release_update.rs unit tests: checksum mismatch, missing asset or SHA256SUMS, malformed list, no release (404 and empty list), a release that comes out during the wait, 429/500/502/503, a download interrupted once or always, wrong architecture, a binary of another commit, older, diverged and off-main commits, author, draft and commit checks, the wait ending with the Runner, and a health-check rollback after MAX_STARTS.
  • crates/ghapp/tests/release.rs: redirect to another host with no auth header, interrupted download written once, full disk, size limit, non-HTTPS, stall and retry-wait stopped, used-up rate limit.
  • crates/daemon/tests/service.rs: a release Runner lists only update-to, waits for its running machine before installing, refuses a second update-to, and shows "fetching its release".
  • xtask/tests/host_setup.rs: runs_newer (keep, install, unknown, 429, 5xx, 404, malformed) and release-gate.sh (green, red, waiting, newest run counts, other workflows, commits, events and branches ignored, GitHub failures).

Not provable before merge

  • The workflow_run gate and the artifact handover only run once these workflows are on main. On this PR, the gate's dry run for the base commit says "wait", or fails if GitHub can't list a runner-build.yml that main doesn't have yet.
  • GitHub's real redirect to its storage host, real ENOSPC on Linux, the exec failure for a wrong-architecture program, and Handle::block_on in the live daemon are covered by fakes only.
  • Today's hosts run a Runner that cannot update itself. Each needs one host-setup.sh install --version of the first release this PR produces. After that, the console's update-to keeps them current.
  • Releases are not signed. Anyone with write access to the repository can publish or replace one. Signing SHA256SUMS with a key only main's environment holds, as images are signed, is the next step.

… itself

A Runner installed from a release (no build_folder) could not update at
all, and the release workflow put out commits whose CI was red.

- "Runner build" builds the Linux programs on every push to main.
  "Runner release" runs on CI, Image and Runner build ending and publishes
  a commit only once all of its push's runs are green on main
  (scripts/release-gate.sh), the rule the console rolls out by.
- ReleaseUpdate: a Runner with no build_folder lists only update-to, finds
  that commit's release (by github-actions[bot], at the whole commit, on
  main and newer than the running one), fetches SHA256SUMS and its own
  architecture's program, checks the checksum and that the program runs
  here and names the commit, then drains, installs and guards the start as
  a build update does. It waits up to 16 minutes for a release not out yet.
- cf-ghapp Releases: tokenless reads, compare, and a streamed, hashed
  download that restarts after a break and stops on a full disk.
- host-setup.sh --version is a minimum: an enrolled Runner on a newer
  commit of main stays, and stays when GitHub cannot say.
…ses more sparingly

The review of the release update found:
- a 403 with x-ratelimit-remaining 0 and no Retry-After was final; it is
  now waited out until x-ratelimit-reset, for every GitHub call;
- a stalled download or a retry wait outlived the Runner's end;
- the 16-minute wait could run out on a slow Runner build: 45 minutes now,
  asked every 3, the newest releases read only every fifth look and at
  the last, 30 a page;
- SHA256SUMS had a second, looser parser: cf_image::parse_sums now;
- the console showed 'building' for a release: 'fetching its release';
- the fetch lock repeated the daemon's one-update guard.
- Http::send and a release download take their waits from one
  RetryPolicy::wait_after; the file client sits beside the API client.
- One sleep_unless for every wait the Runner's end cuts short.
- Downloaded carries its SHA-256 in hex; one check of plain names.
- SelfUpdate says where its updates come from (UpdateFrom), which decides
  the plain update and what the console shows, in place of two methods.
- host-setup.sh's compare_status and main_head serve runs_newer and the
  release gate, which reads the push's runs in one call.
- The release takes both build artifacts in one download.
Kept with status 0, a Runner older than the release asked for stayed while the run ended
'host-setup: unchanged'. A job on the host can use up the address's rate limit of reads
without a token, and a commit GitHub does not have answered the same way every run.
@XeonFX
XeonFX marked this pull request as ready for review November 12, 2026 20:42
@XeonFX XeonFX added the automerge Merges itself once every check is green label Nov 12, 2026
@codefusion-automerge
codefusion-automerge Bot merged commit 985cbce into main Nov 12, 2026
14 checks passed
@codefusion-automerge
codefusion-automerge Bot deleted the feat/release-self-update branch November 12, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge Merges itself once every check is green

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant