Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions pilots/action-boundary/AUTHORITY_CHECKPOINT.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ node pilots/action-boundary/inspect-file-authority.mjs \

The optional second argument is the file anchor directory. The command opens it with `create: false`; it never creates a missing anchor or authority directory. It exits zero for `observed_consistent` and 2 for a lock, mismatch, unreadable state, or other `needs_reconciliation` result. A present authority or anchor lock might belong to a live worker or a crashed one. `checkpoints.comparison` distinguishes a local/external mismatch; `committed_state.comparison` distinguishes a protocol-2 or protocol-3 state mismatch while the checkpoints still agree. `committed_state.includes_authority_mode` is true only for protocol 3. Protocol 1 reports `not_committed`, not a successful state verification. Omit the anchor only to inspect an unanchored store; an anchored local checkpoint without its anchor reports `authority_anchor_not_supplied`.

`reconciliation` lists claimed grants with no local outcome or an `outcome_unknown` record, even if the lock is gone and the checkpoints agree. Each candidate has only a grant ID and opaque operation/idempotency hashes for looking up the host's external claim and provider history; it does not include the prepared request or recipient. It also flags missing or mismatched claim, grant, operation, and outcome links. `returned` counts callbacks that returned locally; it is not a delivery count. A missing outcome does not prove the callback ran: the external claim could have rejected or the worker could have stopped before handoff. These candidates require human review, never automatic retry.

The command performs no unlock, mode initialization, checkpoint advance, grant issue, clock update, provider call, or recovery. A missing mode reports `authority_mode_missing`, including in an empty directory: a never initialized store cannot be distinguished from one whose mode was deleted. Its scan is not an atomic snapshot of live workers, and `observed_consistent` cannot establish provider delivery, an independently protected anchor or external claim backend, or permission to resume an uncertain operation. Preserve the records and reconcile the provider's operation and idempotency history before any operator-controlled recovery. `inspectFileRevocationHandoff({ directory, anchor })` exposes the same observation to a host without invoking the CLI.

Run the project-authored child-process and failure tests:
Expand Down
2 changes: 1 addition & 1 deletion pilots/action-boundary/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ The first operation on a fresh, empty authority store writes a durable `authorit

**Detect a restored local snapshot:** [The opt-in checkpoint](AUTHORITY_CHECKPOINT.md) records issue, revoke, and claim transitions in a separate host-controlled anchor before their local records become usable. New protocol-3 stores bind their grant, epoch, operation, spent-claim, and mode records to that anchor. A child process with an older restored authority directory or selectively missing epoch or spent marker blocks before its simulated provider callback. Existing protocol-1 stores retain only whole-directory rollback detection; protocol-2 stores bind decision files but exclude mode. The bundled `createFileAuthorityAnchor` is a separate local file example, not an independently operated service; restoring both directories together still defeats this check. Anchored and unanchored workers cannot safely share one authority.

**Inspect a stranded authority:** The [read-only inspection command](AUTHORITY_CHECKPOINT.md#inspect-before-reconciliation) shows whether an authority lock exists, whether local and external checkpoints agree, and whether protocol-2 or protocol-3 committed state matches. It neither clears the lock nor changes any authority record. Its observations can race live workers and require provider-side reconciliation before an uncertain operation can be resolved.
**Inspect a stranded authority:** The [read-only inspection command](AUTHORITY_CHECKPOINT.md#inspect-before-reconciliation) shows whether an authority lock exists, whether local and external checkpoints agree, and whether protocol-2 or protocol-3 committed state matches. It also lists spent claims missing an outcome or recording an unknown outcome, with opaque IDs and hashes for provider-side reconciliation. It neither clears the lock nor changes any authority record. Its observations can race live workers, and a local callback return does not establish provider delivery.

The checkpoint and `epochAnchor` are alternative revocation protections and cannot be configured together. Protocol 3 detects selective changes to committed local decision files and mode before issue or dispatch if its separately retained state stays current. The epoch-aware claim checks the independently held issuer epoch at dispatch. The checkpoint can be combined with spent-only `claimOnce`.

Expand Down
90 changes: 90 additions & 0 deletions pilots/action-boundary/file-revocation-handoff.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,89 @@ function checkpointRecord(value) {
"head_sha256|protocol|sequence|state_sha256|store_id");
}

function claimReconciliation(directory) {
const claimed = path.join(directory, "claimed");
const outcomes = path.join(directory, "outcomes");
const names = fs.readdirSync(claimed).sort();
const outcomeNames = fs.readdirSync(outcomes).sort();
const pattern = /^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\.json$/;
const candidates = [], issues = [];
const counts = { claims: names.length, outcomes: outcomeNames.length,
returned: 0, missing_outcome: 0, unknown_outcome: 0 };
const claimNames = new Set(names);
const outcomeNamesSet = new Set(outcomeNames);
for (const name of names) {
if (!pattern.test(name)) {
issues.push({ collection: "claimed", problem: "unexpected_filename" });
continue;
}
const grantId = name.slice(0, -5);
let claim, grant, outcome, grantUnreadable = false;
try { claim = readRecord(path.join(claimed, name)); }
catch { issues.push({ grant_id: grantId, problem: "claim_unreadable" }); continue; }
if (claim?.protocol !== "aml-file-authority-claim/1" ||
claim.grant_id !== grantId || !DIGEST.test(claim.binding_sha256) ||
!DIGEST.test(claim.idempotency_key_sha256)) {
issues.push({ grant_id: grantId, problem: "claim_invalid" });
continue;
}
try { grant = readRecord(path.join(directory, "grants", name)); }
catch { issues.push({ grant_id: grantId, problem: "grant_unreadable" });
grantUnreadable = true; }
if (!grantUnreadable && (!grant || grant.protocol !== "aml-file-authority-grant/1" ||
grant.grant_id !== grantId || grant.binding_sha256 !== claim.binding_sha256 ||
(grant.operation_sha256 !== undefined && !DIGEST.test(grant.operation_sha256)))) {
issues.push({ grant_id: grantId, problem: "grant_missing_or_mismatch" });
}
if (grant?.operation_sha256 && DIGEST.test(grant.operation_sha256)) {
let operation, operationUnreadable = false;
try { operation = readRecord(path.join(directory, "operations",
`${grant.operation_sha256}.json`)); }
catch { issues.push({ grant_id: grantId, problem: "operation_unreadable" });
operationUnreadable = true; }
if (!operationUnreadable && (!operation ||
operation.protocol !== "aml-file-authority-operation/1" ||
operation.operation_sha256 !== grant.operation_sha256 ||
operation.grant_id !== grantId ||
operation.binding_sha256 !== claim.binding_sha256)) {
issues.push({ grant_id: grantId, problem: "operation_missing_or_mismatch" });
}
}
if (outcomeNamesSet.has(name)) {
try { outcome = readRecord(path.join(outcomes, name)); }
catch { issues.push({ grant_id: grantId, problem: "outcome_unreadable" }); continue; }
if (!outcome || outcome.protocol !== "aml-file-authority-outcome/1" ||
outcome.grant_id !== grantId ||
outcome.binding_sha256 !== claim.binding_sha256 ||
!["returned", "outcome_unknown"].includes(outcome.status) ||
typeof outcome.reason !== "string") {
issues.push({ grant_id: grantId, problem: "outcome_invalid_or_mismatch" });
continue;
}
}
if (!outcome) counts.missing_outcome++;
else if (outcome.status === "outcome_unknown") counts.unknown_outcome++;
else counts.returned++;
if (!outcome || outcome.status === "outcome_unknown") {
candidates.push({ grant_id: grantId,
operation_sha256: DIGEST.test(grant?.operation_sha256) ?
grant.operation_sha256 : null,
idempotency_key_sha256: claim.idempotency_key_sha256,
local_outcome: outcome?.status ?? "missing" });
}
}
for (const name of outcomeNames) {
if (!pattern.test(name)) {
issues.push({ collection: "outcomes", problem: "unexpected_filename" });
} else if (!claimNames.has(name)) {
issues.push({ grant_id: name.slice(0, -5), problem: "outcome_without_claim" });
}
}
return { status: issues.length ? "invalid_records" : candidates.length ?
"provider_review_needed" : "no_unresolved_local_claims",
counts, candidates, issues };
}

// One-machine example of the host's independent monotonic checkpoint store.
// Keep this directory under a separate backup/retention domain. Restoring
// both directories together can still replay an older authority snapshot.
Expand Down Expand Up @@ -312,6 +395,12 @@ export function inspectFileRevocationHandoff({ directory, anchor = null } = {})
if (committedState === "mismatch") reasons.push("committed_state_mismatch");
}
} catch { reasons.push("committed_state_unreadable"); }
let reconciliation = null;
try {
reconciliation = claimReconciliation(directory);
if (reconciliation.issues.length) reasons.push("reconciliation_records_invalid");
if (reconciliation.candidates.length) reasons.push("claims_need_provider_review");
} catch { reasons.push("reconciliation_unavailable"); }
// A genuinely fresh directory and an older empty store with a deleted mode
// are indistinguishable from a read-only scan. Neither is verified state.
if (modeStatus === "missing") reasons.push("authority_mode_missing");
Expand Down Expand Up @@ -358,6 +447,7 @@ export function inspectFileRevocationHandoff({ directory, anchor = null } = {})
(localRead || !localReadable || anchor ? "unknown" : "unanchored"),
authority_mode: { status: modeStatus, record: modeRecord },
checkpoints: { local, external, comparison },
reconciliation,
committed_state: { comparison: committedState, observed_sha256: observedDigest,
includes_authority_mode: local?.protocol === "aml-file-authority-checkpoint/3", counts }
};
Expand Down
77 changes: 77 additions & 0 deletions test/action-boundary-file-authority-inspection.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,87 @@ test("inspection and its CLI report consistent protocol-3 state without writes",
} });
assert.deepEqual(report.committed_state.counts,
{ epochs: 1, grants: 1, operations: 1, claimed: 1 });
assert.deepEqual(report.reconciliation, {
status: "no_unresolved_local_claims",
counts: { claims: 1, outcomes: 1, returned: 1,
missing_outcome: 0, unknown_outcome: 0 }, candidates: [], issues: []
});
assert.deepEqual(command, report);
assert.deepEqual(snapshot(parent), before);
}));

test("a spent claim without an outcome is listed even when the lock is gone", () => {
const parent = fs.mkdtempSync(path.join(os.tmpdir(), "aml-inspection-claim-"));
return (async () => {
try {
const directory = path.join(parent, "authority");
const host = createFileRevocationHandoff({ directory, claimOnce: () => false });
const grant = host.issue(input(), { operationId: "host-intent-review-001" });
const result = await host.dispatch({ ...input(), grant_id: grant.grant_id,
execute: () => { throw Error("unexpected callback"); } });
assert.deepEqual([result.status, result.reason], ["blocked", "external_claim_spent"]);
const before = snapshot(parent);
const report = inspectFileRevocationHandoff({ directory });
assert.equal(report.lock, "absent");
assert.equal(report.committed_state.comparison, "not_committed");
assert.equal(report.status, "needs_reconciliation");
assert.ok(report.reasons.includes("claims_need_provider_review"));
assert.equal(report.reconciliation.status, "provider_review_needed");
assert.deepEqual(report.reconciliation.counts,
{ claims: 1, outcomes: 0, returned: 0, missing_outcome: 1,
unknown_outcome: 0 });
assert.deepEqual(report.reconciliation.candidates, [{ grant_id: grant.grant_id,
operation_sha256: JSON.parse(fs.readFileSync(path.join(directory, "grants",
`${grant.grant_id}.json`))).operation_sha256,
idempotency_key_sha256: JSON.parse(fs.readFileSync(path.join(directory, "claimed",
`${grant.grant_id}.json`))).idempotency_key_sha256,
local_outcome: "missing" }]);
assert.equal(JSON.stringify(report).includes("team@example.test"), false);
assert.deepEqual(snapshot(parent), before);
} finally { fs.rmSync(parent, { recursive: true, force: true }); }
})();
});

test("unknown callback outcome and mismatched outcome are not called resolved", () =>
fixture(async ({ parent, directory, anchor, host }) => {
const grant = host.issue(input());
const result = await host.dispatch({ ...input(), grant_id: grant.grant_id,
execute: async () => { throw Error("provider timeout"); } });
assert.equal(result.status, "outcome_unknown");
const before = snapshot(parent);
const report = inspectFileRevocationHandoff({ directory, anchor });
assert.equal(report.checkpoints.comparison, "match");
assert.equal(report.committed_state.comparison, "match");
assert.equal(report.reconciliation.candidates[0].local_outcome, "outcome_unknown");
assert.ok(report.reasons.includes("claims_need_provider_review"));
assert.deepEqual(snapshot(parent), before);
fs.writeFileSync(path.join(directory, "outcomes", `${grant.grant_id}.json`),
JSON.stringify({ protocol: "aml-file-authority-outcome/1",
grant_id: grant.grant_id, binding_sha256: "a".repeat(64),
status: "returned", reason: "callback_returned" }) + "\n");
const mismatched = inspectFileRevocationHandoff({ directory, anchor });
assert.equal(mismatched.committed_state.comparison, "match");
assert.equal(mismatched.reconciliation.status, "invalid_records");
assert.ok(mismatched.reconciliation.issues.some(item =>
item.problem === "outcome_invalid_or_mismatch"));
}));

test("an outcome with no spent claim is reported as an invalid local link", () =>
fixture(async ({ directory, anchor, host }) => {
const grant = host.issue(input());
fs.writeFileSync(path.join(directory, "outcomes", `${grant.grant_id}.json`),
JSON.stringify({ protocol: "aml-file-authority-outcome/1",
grant_id: grant.grant_id, binding_sha256: grant.binding_sha256,
status: "returned", reason: "callback_returned" }) + "\n");
const report = inspectFileRevocationHandoff({ directory, anchor });
assert.equal(report.checkpoints.comparison, "match");
assert.equal(report.committed_state.comparison, "match");
assert.equal(report.reconciliation.status, "invalid_records");
assert.deepEqual(report.reconciliation.issues,
[{ grant_id: grant.grant_id, problem: "outcome_without_claim" }]);
assert.ok(report.reasons.includes("reconciliation_records_invalid"));
}));

test("a file anchor lock is visible even when both checkpoint records agree", () =>
fixture(async ({ anchorDirectory, directory, anchor, host }) => {
host.issue(input());
Expand Down
8 changes: 7 additions & 1 deletion test/action-boundary-file-revocation-handoff.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ import { promisify } from "node:util";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { createFileRevocationHandoff } from "../pilots/action-boundary/file-revocation-handoff.mjs";
import { createFileRevocationHandoff,
inspectFileRevocationHandoff } from "../pilots/action-boundary/file-revocation-handoff.mjs";
import { createTestingClaimOnce } from "../pilots/action-boundary/testing-claim-anchor.mjs";
import { createTestingEpochAnchor } from "../pilots/action-boundary/testing-epoch-anchor.mjs";

Expand Down Expand Up @@ -175,6 +176,11 @@ test("crash after claim strands the lock and cannot silently retry", () =>
await assert.rejects(() => exec(process.execPath, [worker, directory, grant.grant_id, "crash"]),
error => error.code === 48);
assert.equal(fs.readdirSync(path.join(directory, "claimed")).length, 1);
const report = inspectFileRevocationHandoff({ directory });
assert.equal(report.lock, "present");
assert.equal(report.reconciliation.candidates[0].grant_id, grant.grant_id);
assert.equal(report.reconciliation.candidates[0].local_outcome, "missing");
assert.ok(report.reasons.includes("claims_need_provider_review"));
let calls = 0;
const result = await host.dispatch({ ...context(), grant_id: grant.grant_id,
execute: () => { calls++; } });
Expand Down
Loading