Retry stapling notarized macOS pkg on transient failure

.claude/rules/copyright.md

@@ -0,0 +1,12 @@
+---
+paths:
+  - "**/*.ts"
+  - "**/*.js"
+  - "**/*.lua"
+---
+
+# Copyright headers
+
+New source files get a single current-year header — `Copyright (C) <year> Posit Software, PBC` (e.g. `2026`), not a back-dated `2020-<year>` range. Leave existing files' headers alone unless you're already substantially editing them.
+
+Don't guess the year — read it from the system (`date +%Y`, or the current date stated at session start). Copying a header from an existing file carries a stale year.

dev-docs/upgrade-dependencies.md

@@ -2,6 +2,17 @@ Change version numbers in `./configuration` to correspond to new versions.
 
 Update hardcoded version strings in `src/command/check/check.ts` (`versionConstraints` array, ~line 249) so that they match the new versions in `configuration`. The `configuration` file warns about this in a comment.
 
+## Verify installer signing & notarization before merging (bundled binaries)
+
+Regular CI does **not** build, sign, or notarize the installers — that only happens in `create-release.yml`. A bundled binary bump (Deno, Pandoc, Dart Sass, Typst, esbuild, …) can change what the platform code-signing/notarization steps must cover, so a bump that passes normal CI can still break the macOS/Windows release build. **Before merging any PR that bumps a bundled binary**, dispatch the release workflow on the PR branch without publishing and confirm the installer jobs pass:
+
+```bash
+gh workflow run create-release.yml --repo quarto-dev/quarto-cli --ref <branch> -f publish-release=false
+# then watch make-installer-mac and make-installer-win in the resulting run
+```
+
+Why this bites (real incident, #14664): Dart Sass 1.101.0 changed its macOS AOT snapshot (`dart-sass/src/sass.snapshot`) container from ELF to Mach-O. Apple's notary ignores non-native files but requires every Mach-O signed, so the never-signed snapshot flipped notarization from Accepted to Invalid — invisible to normal CI, only caught at release time. When a bump does require a new signing entry, add it in `package/src/macos/installer.ts` / the Windows `sign-files` paths; see `llm-docs/code-signing-installers.md`.
+
 ## Upgrade deno
 
 ### Upgrade standard library

llm-docs/code-signing-installers.md

@@ -82,7 +82,7 @@ Sequence:
 1. **Decide whether to sign** — reads `QUARTO_APPLE_APP_DEV_ID`. Empty → skip everything below with a warning.
 2. **Sign individual binaries** with `codesign` (via `signCode()` at `package/src/macos/installer.ts:293`). Two lists:
    - **With entitlements** (`entitlements.plist`): per-arch `deno`, `dart-sass/src/dart`, `esbuild`, `pandoc`, `typst`, optional `typst-gather`, optional `deno_dom/libplugin.dylib`. Entitlements grant JIT (`com.apple.security.cs.allow-jit`) and unsigned-executable-memory (`com.apple.security.cs.allow-unsigned-executable-memory`) — required by Deno's V8 and by the other JIT-using tools.
-   - **Without entitlements**: shell wrappers and JS bundle — `quarto`, `quarto.js`, `dart-sass/sass`.
+   - **Without entitlements**: shell wrappers, JS bundle, and the Dart Sass AOT snapshot — `quarto`, `quarto.js`, `dart-sass/sass`, `dart-sass/src/sass.snapshot`. The snapshot is a Mach-O loaded by the (entitled) `dart` VM, not its own process, so it needs a valid signature + secure timestamp but no entitlements. See the notarization-Invalid troubleshooting below for why it must be signed.
    - `codesign` invocation: `-s <id> --timestamp --options=runtime --force --deep --verbose=4` (`+ --entitlements <plist>` for the first list). `--options=runtime` opts into Hardened Runtime, a notarization prerequisite.
 3. **Build the tarball** (`quarto-<v>-macos.tar.gz`) from the signed `pkg-working` tree. The tarball ships unwrapped binaries with the signatures embedded — no `.pkg`, no notarization, but each Mach-O is signed individually.
 4. **Build the core `.pkg`** with `pkgbuild` (identifier `org.rstudio.quarto`), then sign it via `productsign --sign <installer-id> --timestamp` (function `signPackage()` at `package/src/macos/installer.ts:282`). Renames signed output back over the original. Note: the `pkgbuild` invocation passes `--install-location` twice (`/Library/Quarto` then `/Applications/quarto`); CLI argv semantics mean the later flag wins, so the effective install location is `/Applications/quarto`. The earlier `/Library/Quarto` value is dead and should be cleaned up next time the file is touched.
@@ -121,6 +121,21 @@ If the Account Holder is unreachable (departed staff, etc.):
 
 Determining which type the current signing team uses: inspect any past released `.pkg` from a non-CI environment (CI logs mask the Team ID portion of cert Common Names). On a Mac, `pkgutil --check-signature quarto-X.Y.Z-macos.pkg` shows the full chain. On Linux/WSL, the `.pkg` is a xar archive — its TOC contains base64-encoded X.509 certs that can be extracted with Python's stdlib and parsed with `openssl x509`. The cert subject's `O = ...` field is a company name for org enrollment, a person name for individual enrollment.
 
+### Troubleshooting: notarization Invalid "Archive contains critical validation errors"
+
+Symptom: `notarizeAndWait()` throws `Notarization was not accepted (status: Invalid)`, and the dumped `xcrun notarytool log <id>` lists per-file `issues` such as:
+
+```text
+"The binary is not signed with a valid Developer ID certificate."
+"The signature does not include a secure timestamp."
+```
+
+Root cause: the named file is a Mach-O binary inside the bundle that no `codesign` call covered. Every native Mach-O in the payload must be individually signed (with `--timestamp`); Apple's notary rejects the whole submission if any is unsigned. Fix: add the file to `signWithEntitlements` (if it JITs / is a directly-executed hardened-runtime process) or `signWithoutEntitlements` (data-like code loaded by another signed process) in `package/src/macos/installer.ts`, then re-run the release build.
+
+This is easy to miss because a bundled binary can *change format across a version bump*. Concrete case (#14664): Dart Sass `src/sass.snapshot` shipped as an **ELF** blob through 1.87.0 — Apple's notary ignored it (not native code), so it was never signed and notarization passed for years. Dart Sass 1.101.0 switched the macOS AOT snapshot to a native **Mach-O** (`cf fa ed fe`), which the notary then required signed; the never-signed snapshot flipped the result to Invalid. Diagnosing this depended on the terminal-status check in `notarizeAndWait()` — `notarytool submit --wait` had exited 0, so an exit-code-only check would (and previously did) mask the rejection and proceed to `stapler staple`, whose downstream "CloudKit … Record not found" error is a misleading symptom of the missing ticket, not a propagation race.
+
+Prevention: dispatch `create-release.yml` (publish-release=false) on the branch before merging any bundled-binary bump — see `dev-docs/upgrade-dependencies.md`.
+
 ### Historical: `waitForNotaryStatus` / `altool` (candidate for removal)
 
 `waitForNotaryStatus()` at `package/src/macos/installer.ts:360` is **dead code** — defined and never called. Kept here so we have enough context to decide whether to delete it next time the file is touched.

package/src/macos/installer.ts

@@ -18,6 +18,8 @@ import { Configuration } from "../common/config.ts";
 import { runCmd } from "../util/cmd.ts";
 import { getEnv } from "../util/utils.ts";
 import { makeTarball } from "../util/tar.ts";
+import { withRetry } from "../../../src/core/retry.ts";
+import { parseNotarizationResult } from "./notarize-status.ts";
 
 // Packaging specific configuration
 // (Some things are global others may be platform specific)
@@ -102,6 +104,11 @@ export async function makeInstallerMac(config: Configuration) {
         "dart",
       ));
       signWithoutEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "dart-sass", "sass"));
+      // Dart Sass 1.101.0 ships src/sass.snapshot as a Mach-O binary that
+      // Apple's notary service requires signed; the dart VM that loads it
+      // carries the runtime entitlements, so the snapshot itself needs only a
+      // valid Developer ID signature + secure timestamp.
+      signWithoutEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "dart-sass", "src", "sass.snapshot"));
 
       signWithEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "esbuild"));
       signWithEntitlements.push(join(config.directoryInfo.pkgWorking.bin, "tools", arch, "pandoc"));
@@ -253,14 +260,13 @@ export async function makeInstallerMac(config: Configuration) {
     const password = getEnv("QUARTO_APPLE_CONNECT_PW", "");
     const teamId = getEnv("QUARTO_APPLE_CONNECT_TEAMID", "");
     if (username.length > 0 && password.length > 0) {
-      const requestId = await notarizeAndWait(
+      await notarizeAndWait(
         packagePath,
         username,
         password,
-        teamId
+        teamId,
       );
 
-
       // Staple the notary to the package
       await stapleNotary(packagePath);
 
@@ -344,16 +350,66 @@ async function notarizeAndWait(
     ],
   );
 
-  if (result.status.success) {
-    const match = result.stdout.match(/id: (.*)/);
-    if (match) {
-      const id = match[1];
-      return id;
-    } else {
-      throw new Error("Notarization Failed to return an Id:\n" + result.stdout);
+  // Always surface notarytool's own output (submission id + terminal status).
+  // runCmd only logs stdout at debug level, so without this the Accepted /
+  // Invalid verdict never appears in CI logs.
+  info(result.stdout);
+
+  if (!result.status.success) {
+    throw new Error("Notarization Failed\n" + result.stdout + result.stderr);
+  }
+
+  const { id, status } = parseNotarizationResult(result.stdout);
+  if (!id) {
+    throw new Error("Notarization Failed to return an Id:\n" + result.stdout);
+  }
+
+  // `notarytool submit --wait` can exit 0 even when the service rejects the
+  // submission; the trustworthy signal is the terminal status. Only a genuinely
+  // notarized package should reach the (offline-only) stapling step below.
+  if (status !== "Accepted") {
+    await dumpNotarizationLog(id, username, password, teamId);
+    throw new Error(
+      `Notarization was not accepted (status: ${status ?? "unknown"}) for ${input}`,
+    );
+  }
+
+  return id;
+}
+
+// Best-effort dump of the full developer log for a submission, so a rejected
+// notarization shows its reasons in CI. Never masks the caller's own error.
+async function dumpNotarizationLog(
+  id: string,
+  username: string,
+  password: string,
+  teamId: string,
+) {
+  try {
+    const log = await runCmd(
+      "xcrun",
+      [
+        "notarytool",
+        "log",
+        id,
+        "--apple-id",
+        username,
+        "--password",
+        password,
+        "--team-id",
+        teamId,
+      ],
+    );
+    info(log.stdout);
+    if (log.stderr) {
+      error(log.stderr);
     }
-  } else {
-    throw new Error("Notarization Failed\n" + result.stderr);
+  } catch (err) {
+    warning(
+      `Could not fetch notarization log for ${id}: ${
+        err instanceof Error ? err.message : String(err)
+      }`,
+    );
   }
 }
 
@@ -419,9 +475,32 @@ async function waitForNotaryStatus(
   return notaryResult;
 }
 
+// Once notarizeAndWait confirms the submission is Accepted, the ticket exists
+// and stapling normally succeeds immediately. Rarely, the ticket can take a
+// moment to become queryable in Apple's CloudKit-backed notary database, so an
+// immediate staple fails with:
+//   CloudKit query for ... failed due to "Record not found".
+//   Could not find base64 encoded ticket in response for ...
+// A short bounded retry absorbs that rare timing gap. Anything else - or a
+// persistent "Record not found" that outlasts the retry - is a real problem and
+// must fail the build (a rejected submission never reaches here; its status is
+// verified in notarizeAndWait).
+const isCloudKitPropagation = (err: Error) =>
+  err.message.includes("CloudKit") &&
+  err.message.includes("Record not found");
+
 async function stapleNotary(input: string) {
-  await runCmd(
-    "xcrun",
-    ["stapler", "staple", input],
-  );
+  await withRetry(async () => {
+    await runCmd(
+      "xcrun",
+      ["stapler", "staple", input],
+    );
+  }, {
+    // withRetry's `attempts` counts retries after the first call (see
+    // src/core/retry.ts), so 3 here yields 4 total staple invocations.
+    attempts: 3,
+    minWait: 20000,
+    maxWait: 30000,
+    retry: isCloudKitPropagation,
+  });
 }

package/src/macos/notarize-status.ts

@@ -0,0 +1,32 @@
+/*
+ * notarize-status.ts
+ *
+ * Copyright (C) 2026 Posit Software, PBC
+ *
+ */
+
+export interface NotarizationResult {
+  id?: string;
+  status?: string;
+}
+
+// Parses the stdout of `xcrun notarytool submit --wait`.
+//
+// The output carries several `id:` lines (Submission ID, uploaded-file id,
+// processing-complete id) and a terminal `status:` line inside the final
+// "Processing complete" block. We take the first `id:` (the submission id,
+// which is what `xcrun notarytool log <id>` expects) and the terminal
+// `status:` (Accepted / Invalid / Rejected).
+export function parseNotarizationResult(stdout: string): NotarizationResult {
+  // First `id:` line is the submission id (also what `notarytool log` wants).
+  const idMatch = stdout.match(/^\s*id:\s*(\S+)/m);
+  // Terminal `status:` line lives in the final "Processing complete" block.
+  // (`Current status:` lines are streamed progress, not the verdict, and don't
+  // match this anchored `status:` prefix.) Take the last one to be safe.
+  const statusMatches = [...stdout.matchAll(/^\s*status:\s*(\S+)/gm)];
+  const lastStatus = statusMatches.at(-1);
+  return {
+    id: idMatch?.[1],
+    status: lastStatus?.[1],
+  };
+}

package/src/util/cmd.ts

@@ -37,8 +37,14 @@ export async function runCmd(
   const code = output.code;
   info(`Status ${code}`);
   if (code !== 0) {
+    // Some tools (e.g. xcrun stapler) write their actual failure reason to
+    // stdout rather than stderr; without this, that reason is only visible
+    // at debug level and failures show an empty error message.
+    error(stdout);
     error(stderr);
-    throw Error(`Command ${[runCmd, ...args]} failed.`);
+    throw Error(
+      `Command ${[runCmd, ...args]} failed.\n${stdout}\n${stderr}`,
+    );
   }
 
   return {

tests/unit/macos-notarize-status.test.ts

@@ -0,0 +1,63 @@
+/*
+ * macos-notarize-status.test.ts
+ *
+ * Tests the parser that extracts the submission id and terminal status from
+ * `xcrun notarytool submit --wait` output. macOS release builds gate stapling
+ * (and, since PR #14718, whether the release proceeds at all) on the notary
+ * service returning `status: Accepted` -- a check the build previously skipped,
+ * trusting only the process exit code.
+ *
+ * Inputs are hand-written samples of notarytool's own output (the contract
+ * Apple hands us), so a change to Quarto's own build logic is what these guard,
+ * not Apple's service.
+ *
+ * Copyright (C) 2026 Posit Software, PBC
+ */
+
+import { unitTest } from "../test.ts";
+import { assertEquals } from "testing/asserts";
+import { parseNotarizationResult } from "../../package/src/macos/notarize-status.ts";
+
+const acceptedOutput = `Conducting pre-submission checks for quarto-1.10.17-macos.pkg and initiating connection to the Apple notary service...
+Submission ID received
+  id: 2efe2717-52ef-43a5-96dc-0797e4ca1041
+Successfully uploaded file
+  id: 2efe2717-52ef-43a5-96dc-0797e4ca1041
+  path: /Users/runner/work/quarto-cli/quarto-1.10.17-macos.pkg
+Waiting for processing to complete.
+Current status: Accepted..........
+Processing complete
+  id: 2efe2717-52ef-43a5-96dc-0797e4ca1041
+  status: Accepted
+`;
+
+const invalidOutput = `Conducting pre-submission checks for quarto-1.10.17-macos.pkg and initiating connection to the Apple notary service...
+Submission ID received
+  id: 9a9b9c9d-1111-2222-3333-444455556666
+Successfully uploaded file
+  id: 9a9b9c9d-1111-2222-3333-444455556666
+  path: /Users/runner/work/quarto-cli/quarto-1.10.17-macos.pkg
+Waiting for processing to complete.
+Current status: Invalid..........
+Processing complete
+  id: 9a9b9c9d-1111-2222-3333-444455556666
+  status: Invalid
+`;
+
+unitTest("parseNotarizationResult - accepted submission", async () => {
+  const result = parseNotarizationResult(acceptedOutput);
+  assertEquals(result.id, "2efe2717-52ef-43a5-96dc-0797e4ca1041");
+  assertEquals(result.status, "Accepted");
+});
+
+unitTest("parseNotarizationResult - invalid submission", async () => {
+  const result = parseNotarizationResult(invalidOutput);
+  assertEquals(result.id, "9a9b9c9d-1111-2222-3333-444455556666");
+  assertEquals(result.status, "Invalid");
+});
+
+unitTest("parseNotarizationResult - no id or status when absent", async () => {
+  const result = parseNotarizationResult("some unrelated output\n");
+  assertEquals(result.id, undefined);
+  assertEquals(result.status, undefined);
+});