Distinguish compilation provenance digests

[raghubetina]

Summary

• keep the artifact Head digest pinned to the retained Compilation Head
• accept a distinct canonical Foundation Plan digest inside artifact bytes authenticated by retained artifact SHA-256
• exercise that distinction through focused artifact tests, the retained-download journey, and the freshly packed executable smoke
• commit the cross-platform runtime-digest recipe used by external evidence

Why

The controlled Movie Catalog journey reached a successful Publication and Compilation, then exposed that exact submitted Head bytes and normalized Compiler-input bytes legitimately have different digests. The CLI incorrectly required them to be equal and rejected the valid retained artifact.

No service API expansion is needed: retained artifact.sha256 authenticates the full envelope, while head_source_sha256 continues to pin exact submitted Head provenance.

Integration

CLI #17, #16, and #18 have landed. This branch now targets current main while preserving the exact reviewed behavioral revision f55edffc… in its ancestry. The service response and CLI package remain an intentional coordinated release pair; no mixed-version compatibility layer is included.

Verification

• focused provenance contracts: 4 tests
• npm run check: 147 tests plus typecheck, lint, format, package allowlist, and packed executable smoke
• npm audit: 0 vulnerabilities
• node scripts/runtime-digest.js at current head e5d375164e4bc9536b9dff5cf34d11aa19e06612: 3ef1252eaa30c3d436339cb238b099821b1bde9b97b518fdac182f13781c9e10
• controlled journey evidence remains pinned to behavioral revision f55edffc9e88924f9a4c95f41c4d0bc9b72422f8 and its runtime digest 9e5a4bd0f16f49ab2e17c04f7defc59366f8fa073f772b310d8f684177890eab; the later source-runtime change is the reviewed Unicode explanation propagated from Unify local application identity tools #16

[raghubetina]

The Node 24 quality failure is the same transitive audit advisory isolated in #17; this stacked diff does not change the lockfile. The Node 22 jobs are green or still running. After #17 lands and #18 is updated, this branch should be rebased and CI rerun.

[raghubetina]

Review

Correct fix, and the reasoning in the description holds up under checking. The relaxation does not weaken authentication, because the chain that authenticates the value it stops comparing is independent and intact.

Verification

npm run check: 147 tests / 147 pass, plus typecheck, lint, format, package allowlist, and the packed executable smoke.

I could not reproduce the packed runtime-tree digest 9e5a4bd0.... Neither the README nor scripts/ documents how it is computed, so there is no recipe to follow. Worth committing the command that produces it, since a digest nobody else can recompute cannot serve as the provenance pin it is presumably meant to be.

The bug is real and the diagnosis is right

The change:

-    value.foundation_plan.sha256 !== expected.headSourceSha256 ||
+    typeof value.foundation_plan.sha256 !== "string" ||
+    !SHA256_PATTERN.test(value.foundation_plan.sha256) ||

The old line asserted that the digest of the canonical Foundation Plan inside the artifact equals the digest of the exact bytes the user submitted. Those are different bytes, so that equality was wrong whenever normalization changed anything, which is essentially always.

I reintroduced the old check and reran the suite: 3 failures, reporting The compilation artifact provenance is invalid, which is the same message the field journey hit. So the fix is covered rather than merely applied.

Coverage reaches the packed executable too. scripts/smoke-package.js:329 deliberately constructs the two-digest case:

const headSha256 = sha256(plan);
const foundationPlanSha256 = sha256(
  Buffer.from("canonical Foundation Plan snapshot"),
);

Two visibly different values rather than the same digest reused, which is exactly the scenario that used to be rejected. Good choice of fixture; a smoke that happened to use one digest for both would have passed before and after.

What still authenticates the value

The relaxation replaces a value comparison with a shape check, so the question is what proves foundation_plan.sha256 is the right digest rather than an arbitrary well-formed one.

Everything else in parseProvenance is retained. head_source_sha256 still has to equal the expected Head digest, and so do compilation_id, project_id, graph_version, analysis.id, along with hasExactKeys on both the provenance block and the foundation_plan object and the exact format match. So submitted-Head provenance is unchanged; only the canonical digest moved from pinned to validated.

The outer chain is the part that matters, and I checked it rather than taking the description's word. compilation.js:437:

response.headers.get("cache-control") !== "no-store, no-transform" ||
response.headers.get("content-encoding") !== null ||
contentLength === null ||
contentLength !== String(metadata.byte_size) ||
source.byteLength !== metadata.byte_size ||
response.headers.get("etag") !== `"sha256:${metadata.sha256}"` ||
sha256(source) !== metadata.sha256

The last line is the one I most wanted to see. The CLI hashes the bytes it actually received and compares, rather than trusting the ETag header, so a server sending a matching ETag over different bytes is caught. Refusing any content-encoding matters too, since a compressed transfer would make the byte-length and digest checks compare the wrong thing.

metadata.sha256 comes from the compilation response, which is itself validated. So the whole envelope, including the canonical digest inside it, is authenticated by a digest the service committed to before the download. The comment in the diff says exactly this and is accurate.

The tradeoff worth naming

There is a real reduction here, and it is the right call, but it should be explicit somewhere durable.

Before, the CLI could independently verify that the Plan inside the artifact was the Plan it submitted, by recomputing one digest. Now it cannot. It verifies that the artifact is the artifact the service said it was, and it trusts the service's canonicalization to have been faithful to the submitted bytes.

That trust is unavoidable without the CLI implementing the same canonicalization the Compiler does, which would mean duplicating normalization logic in a second language and keeping it in step forever. Not worth it.

But the property being given up is one a reader might assume still holds, particularly given how much of this codebase is built on content addressing. Worth a line in the README or the design note saying that foundation_plan.sha256 is authenticated by the enclosing artifact digest and is not independently checkable against the submitted Head. Otherwise somebody later reads parseProvenance, sees a shape check where a value check would fit, and tightens it back.

Stacking

The description's ordering is clear: land #17 and #16, update #18, then rebase this. Worth adding that #18 carries the flag-day coupling with firstdraft#301 that I detailed on both of those, since this branch inherits it. The head_source_sha256 key list here is the same exact-set check, so the same constraint applies: this CLI cannot ship before the service sends the field, and the service cannot send it before this CLI ships, without one release tolerating both shapes.

[raghubetina]

The same document, two different fingerprints

This PR fixes a bug that only appeared when somebody actually ran the thing. The fix is two lines. The misunderstanding behind it is one you will make yourself, because it comes from a habit that is normally correct.

The habit

Hash something, compare the hash, know you have the same thing. It is the foundation of content addressing, and this codebase leans on it everywhere: Git commit SHAs, ETags, artifact digests.

So when the CLI downloads a compiled artifact and wants to check the Plan inside is the Plan it submitted, the obvious move:

value.foundation_plan.sha256 !== expected.headSourceSha256 ||

Same Plan, same digest. Reject if they differ.

Reasonable, and wrong.

Why it is wrong

The submitted Plan and the Plan inside the artifact are the same document and not the same bytes.

Here is what happened in between. The user's file went to the server. The server parsed it and stored it in a normalized form: keys in a canonical order, consistent indentation, arrays in a defined sequence, no incidental whitespace. Then the Compiler read that normalized form and recorded its digest in the artifact.

Two JSON files, semantically identical, byte-wise different:

{"name":"Movie","fields":[{"key":"title"}]}

{
  "fields": [{ "key": "title" }],
  "name": "Movie"
}

Same meaning. Same object if you parse both. Completely different SHA-256.

So the check demanded that two legitimately different byte sequences produce the same digest, which is precisely what a hash function guarantees will not happen.

The general trap

A digest identifies bytes, not meaning. Whenever you hash something to establish identity, the real question is which serialization you are hashing, and whether everyone in the chain agrees.

This bites in ordinary work too:

• Hashing a JSON payload for a webhook signature, where the sender hashed the raw body and you hashed to_json of the parsed hash. Different key order, different digest, signature fails.
• Caching on a digest of a serialized object whose serializer changes format in a library upgrade, invalidating your entire cache overnight.
• Comparing file digests across platforms where one side has CRLF line endings.
• Hashing a Unicode string that arrived normalized differently, which is the same trap another PR in this batch handles by normalizing before digesting.

The pattern: two parties each hash "the document" and disagree, because they were never hashing the same bytes.

How the fix works

The digest is no longer compared to a value. It is checked for shape:

typeof value.foundation_plan.sha256 !== "string" ||
!SHA256_PATTERN.test(value.foundation_plan.sha256) ||

Must be a string, must look like a SHA-256. That is all.

Which raises the obvious objection: if you are not checking its value, is it not meaningless? Could the server not put any 64 hex characters there?

No, and the reason is worth following because it is a nice piece of layered reasoning.

The download path checks this, in compilation.js:

response.headers.get("content-encoding") !== null ||
contentLength !== String(metadata.byte_size) ||
source.byteLength !== metadata.byte_size ||
response.headers.get("etag") !== `"sha256:${metadata.sha256}"` ||
sha256(source) !== metadata.sha256

That last line is the load-bearing one. The CLI hashes the bytes it actually received and compares to the digest the service declared earlier, in the compilation response. Not the ETag header, which a server could set to anything; the real bytes.

So the chain is:

1. The service reports artifact.sha256 in the compilation response.
2. The CLI downloads and proves the bytes hash to that value.
3. Therefore everything inside those bytes is exactly what the service committed to.
4. foundation_plan.sha256 is inside those bytes.

The canonical digest does not need its own comparison because it is covered by the digest of the envelope containing it. One check authenticates the whole payload.

Note the content-encoding check as part of that. If the server gzipped the response, the bytes you hashed would be the compressed ones and the comparison would be against something else entirely. Refusing any content-encoding keeps "the bytes I hashed" and "the bytes the digest describes" the same thing.

What was genuinely given up

Being honest about this matters, because it is easy to present a fix as pure improvement.

Before, the CLI could prove on its own that the Plan in the artifact was the Plan it submitted, by recomputing a digest it already had. Now it cannot. It proves the artifact is what the service said it was, and it trusts that the service's normalization faithfully represents the submitted bytes.

That is a real reduction in what the client verifies independently. It is also the right call: the alternative is implementing the server's exact canonicalization rules in JavaScript and keeping the two implementations in step forever. Every divergence would surface as a false rejection of a valid artifact, which is the bug this PR is fixing, permanently.

The lesson is that layered verification lets you stop checking something only if an outer layer covers it. Before relaxing an inner check, trace what the outer one actually proves. Here it proves plenty. Sometimes it proves less than you assumed.

How this was found

Worth noticing:

The controlled Movie Catalog journey reached a successful Publication and Compilation, then exposed that exact submitted Head bytes and normalized Compiler-input bytes legitimately have different digests.

Not a code review, not a unit test. Somebody ran the whole thing end to end and it rejected a valid artifact.

The bug was invisible to tests because the tests constructed both digests from the same source. When your fixture builds the input and the expectation the same way, an assertion that they are equal always passes and tells you nothing.

You can see the fix handle that too. The packed smoke now uses deliberately different values:

const headSha256 = sha256(plan);
const foundationPlanSha256 = sha256(
  Buffer.from("canonical Foundation Plan snapshot"),
);

If those were both sha256(plan), the regression would be free to come back.

The habit worth taking: when a test asserts two things are equal, check whether your fixture made them equal by construction. If it did, you have tested your fixture.

[raghubetina]

Addressed the reproducibility follow-up in 6a9c8e2: node scripts/runtime-digest.js now commits the exact length-delimited recipe. Its normalized repository-relative sort key keeps the result platform-independent.

At current head ccdc2df, it prints 3ef1252eaa30c3d436339cb238b099821b1bde9b97b518fdac182f13781c9e10; the historical controlled journey remains explicitly pinned to behavioral revision f55edffc and digest 9e5a4bd0…. The full 147-test check, package allowlist, and packed smoke pass.