Repository navigation
Release only green commits, and let a Runner installed from a release update itself - #86
Merged
Merged
Conversation
… itself A Runner installed from a release (no build_folder) could not update at all, and the release workflow put out commits whose CI was red. - "Runner build" builds the Linux programs on every push to main. "Runner release" runs on CI, Image and Runner build ending and publishes a commit only once all of its push's runs are green on main (scripts/release-gate.sh), the rule the console rolls out by. - ReleaseUpdate: a Runner with no build_folder lists only update-to, finds that commit's release (by github-actions[bot], at the whole commit, on main and newer than the running one), fetches SHA256SUMS and its own architecture's program, checks the checksum and that the program runs here and names the commit, then drains, installs and guards the start as a build update does. It waits up to 16 minutes for a release not out yet. - cf-ghapp Releases: tokenless reads, compare, and a streamed, hashed download that restarts after a break and stops on a full disk. - host-setup.sh --version is a minimum: an enrolled Runner on a newer commit of main stays, and stays when GitHub cannot say.
…ses more sparingly The review of the release update found: - a 403 with x-ratelimit-remaining 0 and no Retry-After was final; it is now waited out until x-ratelimit-reset, for every GitHub call; - a stalled download or a retry wait outlived the Runner's end; - the 16-minute wait could run out on a slow Runner build: 45 minutes now, asked every 3, the newest releases read only every fifth look and at the last, 30 a page; - SHA256SUMS had a second, looser parser: cf_image::parse_sums now; - the console showed 'building' for a release: 'fetching its release'; - the fetch lock repeated the daemon's one-update guard.
- Http::send and a release download take their waits from one RetryPolicy::wait_after; the file client sits beside the API client. - One sleep_unless for every wait the Runner's end cuts short. - Downloaded carries its SHA-256 in hex; one check of plain names. - SelfUpdate says where its updates come from (UpdateFrom), which decides the plain update and what the console shows, in place of two methods. - host-setup.sh's compare_status and main_head serve runs_newer and the release gate, which reads the push's runs in one call. - The release takes both build artifacts in one download.
Kept with status 0, a Runner older than the release asked for stayed while the run ended 'host-setup: unchanged'. A job on the host can use up the address's rate limit of reads without a token, and a commit GitHub does not have answered the same way every run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lon1, pdx1, str1 and waw1 run the Runner from a release, and the console flagged both "runs Runner commits that aren't green" and "its Runner can't update itself". Two causes, two fixes.
Only green commits are released
release.ymlpublished on every push to main, before CI ended, sorunner-0.1.0-4768026b6a4cwent out with red CI. Now:runner-build.yml) builds the two Linux programs on every push to main and keeps them as artifacts for a week. It publishes nothing.release.yml) runs when CI, Image or Runner build ends for a push to main.scripts/release-gate.shpublishes only when the commit is on main and the push's newest CI run, Image run (if the push made one) and Runner build run are green. That is the console's rollout rule plus the build. Otherwise it leaves a notice saying "wait" or "skip". The gate checks out main's scripts and holds no write permission. The publish job runs no repository code and takes the programs from the build run the gate named.workflow_dispatchwith a commit does the same check by hand.A Runner from a release updates itself
A Runner with no
build_foldernow usesReleaseUpdate(crates/runner/src/release_update.rs). It listsupdate-toonly, and refuses plainupdatewith a reason. Onupdate-to <commit>it:github-actions[bot], at the whole commit, on main and newer than the commit that runs.SHA256SUMSand this host'scodefusion-runner-linux-{amd64,arm64}, following GitHub's redirect to its storage with no token, and checks the SHA-256.version, which must run on this host and name the commit.Installernow shared withCheckoutUpdate.It refuses a commit with no release, an older one, a bad checksum, a missing file, the wrong architecture or a full disk, and the running program stays. The new
cf_ghapp::Releasesstreams and hashes downloads, starts again from byte 0 after a break, 429 or 5xx, and stops at once on a full disk or when the Runner ends. Two changes reach every GitHub call: a used-up rate limit (x-ratelimit-remaining: 0) is now waited out until its reset, andRetryPolicy::wait_afteris the one retry rule.host-setup.sh install --version Xis now a minimum. An enrolled host whose Runner runs a newer commit of main, by GitHub's compare (retrying 429 and 5xx), keeps it. If GitHub can't say (rate limit, 5xx, an unknown commit), the run fails, changes nothing and says why, so a requested release is never silently skipped.Tests
Mocks cover network and time only. Each new check was broken on purpose and its test went red:
crates/runner/src/release_update.rsunit tests: checksum mismatch, missing asset orSHA256SUMS, malformed list, no release (404 and empty list), a release that comes out during the wait, 429/500/502/503, a download interrupted once or always, wrong architecture, a binary of another commit, older, diverged and off-main commits, author, draft and commit checks, the wait ending with the Runner, and a health-check rollback afterMAX_STARTS.crates/ghapp/tests/release.rs: redirect to another host with no auth header, interrupted download written once, full disk, size limit, non-HTTPS, stall and retry-wait stopped, used-up rate limit.crates/daemon/tests/service.rs: a release Runner lists onlyupdate-to, waits for its running machine before installing, refuses a secondupdate-to, and shows "fetching its release".xtask/tests/host_setup.rs:runs_newer(keep, install, unknown, 429, 5xx, 404, malformed) andrelease-gate.sh(green, red, waiting, newest run counts, other workflows, commits, events and branches ignored, GitHub failures).Not provable before merge
workflow_rungate and the artifact handover only run once these workflows are on main. On this PR, the gate's dry run for the base commit says "wait", or fails if GitHub can't list arunner-build.ymlthat main doesn't have yet.Handle::block_onin the live daemon are covered by fakes only.host-setup.sh install --versionof the first release this PR produces. After that, the console'supdate-tokeeps them current.SHA256SUMSwith a key only main's environment holds, as images are signed, is the next step.