Retry stapling notarized macOS pkg on transient failure#14718
Merged
Conversation
xcrun stapler staple can fail right after notarytool submit --wait reports success, because Apple's notarization ticket can take a moment to propagate before it is fetchable. Retry a few times with a delay instead of failing the whole release build immediately.
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
src/core/retry.ts already provides the attempt/backoff pattern used elsewhere (e.g. src/core/download.ts); reuse it instead of a hand-rolled loop plus the sync sleep polyfill.
Member
Author
|
Running the build installer workflow to verify it works |
Member
Author
|
And it is still failing... we have a Mac signing problem again 😢 |
xcrun stapler (and potentially other tools invoked here) can write its actual failure reason to stdout rather than stderr. Without this, a failing command only surfaces an empty error message in CI logs.
The retry loop wasn't giving the notarization ticket enough time to become queryable in Apple's CloudKit-backed notary database: both a production run and a dry-run dispatch exhausted the previous ~80s retry window with the identical failure every attempt: CloudKit query for ... failed due to "Record not found". Could not find base64 encoded ticket in response for ... That message was only visible once runCmd started including stdout in its thrown error. Widen the retry window (6 attempts, 20-30s spacing, ~2.5min worst case) and only retry on that specific transient signature so a real signing/notarization failure still surfaces immediately instead of being retried for minutes.
withRetry's attempts option counts retries after the first call, so attempts: 6 was actually running 7 total invocations / ~3min worst case instead of the intended 6 / ~2.5min. Also require both the CloudKit and "Record not found" signature fragments (not either alone) so an unrelated CloudKit error still fails fast instead of being retried for minutes.
Member
Author
|
All failure are from Julia problem on Windows it seems... |
A bounded staple retry alone cannot fix this: every post-binary-bump CI run exhausted the retry window with the same CloudKit "Record not found" error and never recovered, and the evidence could not rule out the ticket simply never being issued. notarizeAndWait only trusted notarytool's exit code and scraped an id from stdout -- it never checked the terminal Accepted/Invalid status, which is the signal that actually distinguishes a propagation delay from a rejected submission, and that status never appeared in our CI logs at all. Parse and require status: Accepted before stapling (dumping the developer log on rejection), and always echo notarytool's output so the verdict is visible. Stapling only buys offline Gatekeeper validation -- a notarized-but-unstapled package still validates online on first launch -- so once the package is verified notarized, a persistent CloudKit propagation delay is downgraded to a warning and the release proceeds instead of failing. Any non-propagation staple error still throws.
Dart Sass 1.101.0 (bundled via #14664) switched its macOS AOT snapshot, dart-sass/src/sass.snapshot, from an ELF blob to a native Mach-O. Apple's notary ignores non-native files but requires every Mach-O in the payload to be signed, so the never-signed snapshot flipped notarization to Invalid ("Archive contains critical validation errors") and no ticket was issued. The subsequent "CloudKit ... Record not found" from stapler was a symptom of the missing ticket, not a propagation race; the earlier exit-code-only notarization check masked the rejection entirely. Sign the snapshot alongside the sass wrapper (no entitlements: it is loaded by the already-entitled dart VM, so it needs only a valid signature and secure timestamp). The staple retry is kept as narrow insurance against a genuine, rare CloudKit timing gap but no longer swallows failures: with notarization now verified Accepted before stapling, a persistent staple failure is a real anomaly and fails the build. Document the format-flip incident and add a pre-merge step to the dependency upgrade checklist: bundled-binary bumps must be validated by dispatching create-release.yml on the branch, since normal CI never builds, signs, or notarizes the installers.
Member
Author
|
I am merging despited windows failure, all Julia related. Something else to look at |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The macOS release build failed at the pkg stapling step (https://github.com/quarto-dev/quarto-cli/actions/runs/29915034977/job/88906897397), right after notarization had just reported success.
Root Cause
stapleNotary()inpackage/src/macos/installer.ts:422callsxcrun stapler stapleonce, immediately afternotarytool submit --waitreports success. Apple's notarization ticket can take a few seconds to propagate before it's fetchable, so the immediate staple attempt fails with exit code 65 even though notarization succeeded.Fix
Retry stapling up to 5 times with a 15s delay between attempts before giving up.
Verified with
deno check(no local way to exercisexcrun/stapleroutside macOS CI).