diff --git a/docs/packages/overlays/overlay-discovery-services.md b/docs/packages/overlays/overlay-discovery-services.md index 0aa1bbd52..20c3fda4d 100644 --- a/docs/packages/overlays/overlay-discovery-services.md +++ b/docs/packages/overlays/overlay-discovery-services.md @@ -4,7 +4,7 @@ title: '@bsv/overlay-discovery-services' kind: package domain: overlays npm: '@bsv/overlay-discovery-services' -version: '2.2.3' +version: '2.2.4' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/overlays/overlay-express.md b/docs/packages/overlays/overlay-express.md index 24c1ff7ea..aa7388b91 100644 --- a/docs/packages/overlays/overlay-express.md +++ b/docs/packages/overlays/overlay-express.md @@ -4,7 +4,7 @@ title: '@bsv/overlay-express' kind: package domain: overlays npm: '@bsv/overlay-express' -version: '2.7.0' +version: '2.7.1' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/overlays/overlay-topics.md b/docs/packages/overlays/overlay-topics.md index 63205b89f..f6a066e02 100644 --- a/docs/packages/overlays/overlay-topics.md +++ b/docs/packages/overlays/overlay-topics.md @@ -4,7 +4,7 @@ title: '@bsv/overlay-topics' kind: package domain: overlays npm: '@bsv/overlay-topics' -version: '1.8.2' +version: '1.8.3' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/packages/overlays/overlay.md b/docs/packages/overlays/overlay.md index 312d65026..942d47265 100644 --- a/docs/packages/overlays/overlay.md +++ b/docs/packages/overlays/overlay.md @@ -4,7 +4,7 @@ title: '@bsv/overlay' kind: package domain: overlays npm: '@bsv/overlay' -version: '2.5.0' +version: '2.6.0' last_updated: '2026-09-18' last_verified: '2026-09-18' review_cadence_days: 30 diff --git a/docs/reference/package-api-migrations.md b/docs/reference/package-api-migrations.md index 0c370b6bb..4f12c4cf0 100644 --- a/docs/reference/package-api-migrations.md +++ b/docs/reference/package-api-migrations.md @@ -42,10 +42,10 @@ and clean-consumer tests remain the executable type authority. | `@bsv/gasp` | `1.3.1` | `1.3.6` | patch | [API and usage](../packages/overlays/gasp.md) | No consumer migration is required; existing constructor calls, imports, synchronization behavior, and wire semantics are unchanged. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | | `@bsv/lch` | `0.0.0` | `0.1.0` | minor | [API and usage](../packages/content/lch.md) | No consumer migration is required; this is the first release of a new additive package. Applications must give quote() the verified Asset encryption descriptor and selected Offer key-delivery mechanism, keep createPayment behind explicit wallet authorization, persist the funded transaction and every partial settlement proof through recovery, retry with that same transaction, distinguish finalized from accepted evidence, fail closed on unknown settlement or evidence profiles, and supply an address-pinning connector for DNS endpoints. Payees should select receipt-complete unless they explicitly accept authorized-output provider, privacy, and pre-internalization key-release tradeoffs. Distributors must retain THIRD_PARTY_NOTICES.md with the package; the new profile adds no dependency. Published BRC-170 remains authoritative if the implementation and standard differ. | | `@bsv/message-box-client` | `2.4.0` | `2.5.1` | minor | [API and usage](../packages/messaging/message-box-client.md) | No API migration is required. socketOptions is optional and nothing is forwarded by default, so socket transport negotiation and every HTTP code path are unchanged for existing callers; wallet and originator stay client-owned and cannot be overridden through it. managerOptions excludes autoConnect and retries: JavaScript callers receive an immediate error for autoConnect: false or nonzero message retries because the live socket starts immediately and AuthSocket does not send Socket.IO acknowledgements. Connection reconnection options remain supported. Upgrade @bsv/sdk and @bsv/message-box-client together; historical number-array wallets, current Uint8Array substrates, and already-pending numeric-key messages interoperate through the same portable transaction form. Distributors who copy the UMD file must keep THIRD_PARTY_NOTICES.md and LICENSES/ with it. No migration is required for the bundled payment optimization. Module consumers can use SDK 2.5.0 or later to enable the same optional extension; existing compatible SDK peer versions remain supported. Recipients must advertise already-validated transaction IDs through x-bsv-payment-known-txids, an optional SDK extension rather than a standardized BRC-105 header; services that omit it retain existing payment behavior. Existing return shapes and payment envelopes remain unchanged. Failed or incomplete notification payments stay queued. Resolve uncertain refund-send outcomes before retrying; this patch adds ordering checks, not an exactly-once refund journal. listMessages/listMessagesLite envelope behavior and basket-insertion semantics remain tracked in issue #503. | -| `@bsv/overlay` | `2.2.1` | `2.5.0` | minor | [API and usage](../packages/overlays/overlay.md) | Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included. | -| `@bsv/overlay-discovery-services` | `2.1.1` | `2.2.3` | minor | [API and usage](../packages/overlays/overlay-discovery-services.md) | Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | -| `@bsv/overlay-express` | `2.5.0` | `2.7.0` | minor | [API and usage](../packages/overlays/overlay-express.md) | Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. Valid legacy JSON shapes, numeric-string heights, public CORS, topic-free raw transactions, configured limits and synchronization defaults remain supported. No storage migration is required. Malformed/coercive inputs are rejected and missing BASM engine/storage capabilities return HTTP 400 with code BASM_UNSUPPORTED. Existing header providers remain valid without a blockTransactionCount field. | -| `@bsv/overlay-topics` | `1.6.10` | `1.8.2` | minor | [API and usage](../packages/overlays/overlay-topics.md) | Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | +| `@bsv/overlay` | `2.2.1` | `2.6.0` | minor | [API and usage](../packages/overlays/overlay.md) | Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Knex/SQL submit remains the default path; Engine uses commitAdmission only when storage advertises a complete overlay-admission-v1 provider. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included. | +| `@bsv/overlay-discovery-services` | `2.1.1` | `2.2.4` | minor | [API and usage](../packages/overlays/overlay-discovery-services.md) | Existing mainnet and testnet advertisers are unchanged. TTN operators pass chain ttn and provision the staging storage and overlay endpoints before advertising. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | +| `@bsv/overlay-express` | `2.5.0` | `2.7.1` | minor | [API and usage](../packages/overlays/overlay-express.md) | Existing mainnet and testnet servers are unchanged. TTN servers call configureNetwork('ttn'), configureArcade with the TTN endpoint, and configureChaintracks or configureChainTracker before engine initialization. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. Valid legacy JSON shapes, numeric-string heights, public CORS, topic-free raw transactions, configured limits and synchronization defaults remain supported. No storage migration is required. Malformed/coercive inputs are rejected and missing BASM engine/storage capabilities return HTTP 400 with code BASM_UNSUPPORTED. Existing header providers remain valid without a blockTransactionCount field. | +| `@bsv/overlay-topics` | `1.6.10` | `1.8.3` | minor | [API and usage](../packages/overlays/overlay-topics.md) | Existing topic and lookup identifiers remain unchanged. Production UMP overlays must give UMPTopicManager and the UMP lookup service Mongo-backed stores that use the same database, then roll out before updated wallet clients; the no-argument manager is bounded but intended only for isolated single-process use. The reservation and bootstrap-marker collections are additive and initialize from currently indexed UMP UTXOs; take a MongoDB backup before rollout. Legacy ambiguous rows remain visible and can be resolved with WAB pinning rather than deleted. Valid uora-anchor-v3 outputs retain their bytes, signatures, and admission result. Readers now reject nonconforming key, tail, and text shapes. Coordinate reader upgrades across nodes serving tm_uora_dpp and audit any previously indexed nonconforming outputs before rebuilding that topic. No wire prefix, export, or persisted schema changes are required. Mandala custom state adapters must implement isAdminOutpoint against admitted per-asset admin history and retain matching owner rows during admission. Registration must omit assetId or use an empty string. Back up and audit historical admin and ownership records before replay; missing records must be restored from verified admission evidence. Valid wire fields and encodings are unchanged, and no collection migration is needed. Wallets that reveal input linkage must reveal the spender's own child key. No migration is required for this dependency candidate update; default storage selection and existing runtime behavior remain unchanged. No migration is required for this dependency candidate update. | | `@bsv/paymail` | `2.4.2` | `2.4.8` | patch | [API and usage](../packages/messaging/paymail.md) | Existing Paymail client APIs and protocol semantics are retained. Consumers provide one Express 4.18 or 5 runtime and matching type graph; browser bundles continue to exclude the server router implementation. Consumers of the former bundled Money Button or Tokenized specification documents must follow the authoritative links in docs/specs/README.md. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | | `@bsv/payment-express-middleware` | `2.1.1` | `2.1.6` | patch | [API and usage](../packages/middleware/payment-express-middleware.md) | No consumer migration is required; legacy x-bsv-payment JSON behavior remains supported, and Express 4 and 5 applications use their own peer-provided Express installation. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. | | `@bsv/sdk` | `2.7.0` | `2.7.1` | patch | [API and usage](../packages/sdk/bsv-sdk.md) | No API or wire migration is required. Upgrade to @bsv/sdk 2.7.1 or later. Valid BRC-103 v0.1 peers keep the same message fields, payload bytes, nonces, and signature derivation; authenticated messages whose transport identity differs from the established session now fail closed. No API or valid BUMP wire migration is required for the offset correction. | @@ -277,8 +277,8 @@ CLI entry points: `{"lch":"./dist/cli.js"}`. - Package documentation: [docs/packages/overlays/overlay.md](../packages/overlays/overlay.md) - Source: [packages/overlays/overlay](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay) -- Release note: Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. Adds bounded BASM peer JSON validation, canonical header/proof/raw binding, optional trusted full-block-count position checks, and classified capability/resource errors. Forward sync pages fit the standard 1,000-anchor HTTP limit. Claimed admitted-list indices are bound to the compound path even when every remote txid is already local, and inclusion uses chain-tracker root/height rather than coinbase maturity. -- Migration: Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included. +- Release note: Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. Adds bounded BASM peer JSON validation, canonical header/proof/raw binding, optional trusted full-block-count position checks, and classified capability/resource errors. Forward sync pages fit the standard 1,000-anchor HTTP limit. Claimed admitted-list indices are bound to the compound path even when every remote txid is already local, and inclusion uses chain-tracker root/height rather than coinbase maturity. Adds an opt-in Mongo AdmissionStorage adapter and Engine submit path that uses commitAdmission when overlay-admission-v1 is advertised, with enlisted indexes or a durable projection outbox and majority ACK. +- Migration: Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Knex/SQL submit remains the default path; Engine uses commitAdmission only when storage advertises a complete overlay-admission-v1 provider. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included. | Public subpath | Runtime target(s) | Declaration target(s) | | ---------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | diff --git a/docs/reference/stack-facts.md b/docs/reference/stack-facts.md index 7f8878d17..970147e1a 100644 --- a/docs/reference/stack-facts.md +++ b/docs/reference/stack-facts.md @@ -58,10 +58,10 @@ authorized release action. | network | `@bsv/chirp` | `0.1.1` | browser-library | browser-bundler, browser-esm, cli, node-esm | browser, node | `>=22` | [packages/network/chirp](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/network/chirp) | | network | `@bsv/teranode-listener` | `1.1.5` | node-library | node-esm | node | `>=22` | [packages/network/ts-p2p](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/network/ts-p2p) | | overlays | `@bsv/gasp` | `1.3.6` | browser-library | browser-bundler, browser-esm, node-cjs, node-esm | browser, node | `>=22` | [packages/overlays/gasp-core](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/gasp-core) | -| overlays | `@bsv/overlay` | `2.5.0` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay) | -| overlays | `@bsv/overlay-discovery-services` | `2.2.3` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-discovery-services](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-discovery-services) | -| overlays | `@bsv/overlay-express` | `2.7.0` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-express](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-express) | -| overlays | `@bsv/overlay-topics` | `1.8.2` | node-library | node-esm | node | `>=22` | [packages/overlays/topics](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/topics) | +| overlays | `@bsv/overlay` | `2.6.0` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay) | +| overlays | `@bsv/overlay-discovery-services` | `2.2.4` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-discovery-services](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-discovery-services) | +| overlays | `@bsv/overlay-express` | `2.7.1` | node-library | node-cjs, node-esm | node | `>=22` | [packages/overlays/overlay-express](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay-express) | +| overlays | `@bsv/overlay-topics` | `1.8.3` | node-library | node-esm | node | `>=22` | [packages/overlays/topics](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/topics) | | sdk | `@bsv/sdk` | `2.7.1` | browser-library | browser-bundler, browser-esm, node-cjs, node-esm, umd-global | browser, node, umd | `>=22` | [packages/sdk](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/sdk) | | sdk | `@bsv/verifast` | `0.3.5` | wasm-library | browser-bundler, browser-esm, node-cjs, node-esm, umd-global, wasm-worker | browser, node, umd, wasm, worker | `>=22` | [packages/verifast](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/verifast) | | wallet | `@bsv/btms` | `1.2.2` | node-library | node-cjs, node-esm | node | `>=22` | [packages/wallet/btms](https://github.com/bsv-blockchain/ts-stack/tree/main/packages/wallet/btms) | diff --git a/governance/package-release-notes.json b/governance/package-release-notes.json index e9990915c..0b54842ee 100644 --- a/governance/package-release-notes.json +++ b/governance/package-release-notes.json @@ -126,8 +126,8 @@ "name": "@bsv/overlay", "publishedVersion": "2.2.1", "releaseType": "minor", - "summary": "Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. Adds bounded BASM peer JSON validation, canonical header/proof/raw binding, optional trusted full-block-count position checks, and classified capability/resource errors. Forward sync pages fit the standard 1,000-anchor HTTP limit. Claimed admitted-list indices are bound to the compound path even when every remote txid is already local, and inclusion uses chain-tracker root/height rather than coinbase maturity.", - "migration": "Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Current Engine/Knex behavior is unchanged; no production adapter provides this atomic receipt yet. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No Engine activation, default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included." + "summary": "Adds an engine lookup-result cardinality ceiling before transaction/proof hydration and an optional TopicManager abort hook for provisional admission state when strict transaction broadcast fails. Retains the hash-pinned pre-uniformization Open BSV License version 4 grant as a scoped continuity notice. Adds optional v1 admission/storage capability types, deterministic semantic identity and exact integer helpers, recovery fences and cursor-publication predicates, with shared reference conformance fixtures. Adds an opt-in MongoDB foundation for validated scoped schema bootstrap, content-addressed payload publication, transactionally guarded references, and recoverable payload collection. Adds bounded BASM peer JSON validation, canonical header/proof/raw binding, optional trusted full-block-count position checks, and classified capability/resource errors. Forward sync pages fit the standard 1,000-anchor HTTP limit. Claimed admitted-list indices are bound to the compound path even when every remote txid is already local, and inclusion uses chain-tracker root/height rather than coinbase maturity. Adds an opt-in Mongo AdmissionStorage adapter and Engine submit path that uses commitAdmission when overlay-admission-v1 is advertised, with enlisted indexes or a durable projection outbox and majority ACK.", + "migration": "Existing Engine and TopicManager implementations remain valid. Lookup results default to 1,000 formulas; pass -1 only when an equivalent deployment bound exists. Topic managers whose validation creates provisional external state should implement abortAdmissibleOutputs, while read-only managers require no change. Distributors must retain THIRD_PARTY_NOTICES.md and LICENSES/ with the package. No migration is required for the additive storage capability. Knex/SQL submit remains the default path; Engine uses commitAdmission only when storage advertises a complete overlay-admission-v1 provider. MongoDB is an optional peer: applications that import a Mongo storage deep entry point must install mongodb ^7.5.0 and explicitly operate the replica-set foundation. No default-adapter change, SQL-to-Mongo migration, or mixed-writer database topology is included. Existing Engine, TopicManager and header providers remain valid. BASM reconciliation requires a canonical header resolver; blockTransactionCount is optional and must come from the trusted provider for the same block hash. Header-only proof position assurance is reported explicitly. A remote admitted list is bound to its compound Merkle path before it is treated as divergence or repair evidence. Proof inclusion checks the canonical root at the claimed height and does not apply coinbase 100-block spendability; historical submit still runs standard SPV. BASM admission therefore submits in the historical-tx-no-spv mode, so topic managers that branch on the submission mode observe historical-tx-no-spv rather than historical-tx for BASM-admitted transactions. Untrusted bootstrap prefixes and over-limit proof/raw requests fail closed pending durable recovery work; automatic sync remains off and no schema migration is included." }, { "name": "@bsv/overlay-discovery-services", diff --git a/governance/repository-health/baselines.json b/governance/repository-health/baselines.json index b439a1bed..56191b7df 100644 --- a/governance/repository-health/baselines.json +++ b/governance/repository-health/baselines.json @@ -318,10 +318,10 @@ "@bsv/payment-express-middleware": "2.1.6", "@bsv/teranode-listener": "1.1.5", "@bsv/gasp": "1.3.6", - "@bsv/overlay": "2.5.0", - "@bsv/overlay-discovery-services": "2.2.3", - "@bsv/overlay-express": "2.7.0", - "@bsv/overlay-topics": "1.8.2", + "@bsv/overlay": "2.6.0", + "@bsv/overlay-discovery-services": "2.2.4", + "@bsv/overlay-express": "2.7.1", + "@bsv/overlay-topics": "1.8.3", "@bsv/sdk": "2.7.1", "@bsv/verifast": "0.3.5", "@bsv/btms": "1.2.2", diff --git a/packages/overlays/overlay-discovery-services/package.json b/packages/overlays/overlay-discovery-services/package.json index fa870977a..9276562c6 100644 --- a/packages/overlays/overlay-discovery-services/package.json +++ b/packages/overlays/overlay-discovery-services/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay-discovery-services", - "version": "2.2.3", + "version": "2.2.4", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/overlays/overlay-express/package.json b/packages/overlays/overlay-express/package.json index 27c767b98..63394cc6d 100644 --- a/packages/overlays/overlay-express/package.json +++ b/packages/overlays/overlay-express/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay-express", - "version": "2.7.0", + "version": "2.7.1", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/overlays/overlay/CHANGELOG.md b/packages/overlays/overlay/CHANGELOG.md index 0f74ad78b..e52e09986 100644 --- a/packages/overlays/overlay/CHANGELOG.md +++ b/packages/overlays/overlay/CHANGELOG.md @@ -24,8 +24,11 @@ All notable changes to this project will be documented in this file. The format - Adds `storageHasAdmission` for the optional `Storage.admission` field. - Adds an opt-in MongoDB schema, payload-publication, reference-guard, and payload-GC foundation behind an optional `mongodb` peer. It does not activate - MongoDB in Engine, select a new default adapter, or provide an admission - receipt path. + MongoDB as the overlay default or select a new SQL adapter. +- Adds an opt-in Mongo `AdmissionStorage` adapter and Engine submit path that + uses `commitAdmission` when `overlay-admission-v1` is advertised, with + enlisted same-session indexes or a durable projection outbox, majority ACK, + and SQL/Knex compatibility retained. - Added bounded BASM JSON peer validation, classified capability/resource errors, canonical header and optional full-block-count position checks, and explicit position assurance in sync reports. Fixed default forward pages to fit the @@ -55,6 +58,30 @@ All notable changes to this project will be documented in this file. The format ### Fixed - (Document bugs that were fixed since the last release.) +- Exclude already-applied (dupe) topics from the Mongo admission plan's + identity, decisions and commit, instead of resubmitting them for admission. + A dupe topic no longer reaches `commitAdmission`; when every remaining + topic is a dupe or a failure, `Engine.submit` returns the in-memory STEAK + without building or committing a plan at all, and on a mixed submission + only the genuinely new topics are committed while the STEAK still reports + the dupe as accepted-with-nothing-new. Previously a resubmitted dupe was + still treated as accepted and given a decision, which + `MongoAdmissionStorage.assertAppliedAvailable` rejects whenever that topic + was already applied under a different admission operation — breaking + historical-then-live resubmits and multi-topic retries with + `Overlay admission rejected: invalid-plan` or `digest-mismatch`. +- `MongoOverlayStorage` no longer maps an evicted output document to a live + `Output`. `toOutput` now returns `null` for a `state: 'evicted'` document + (evictions persist their row for audit/history, they do not delete it), + and every read path (`findOutput`, `findOutputsForTransaction`, + `findUTXOsForTopic`) drops a `null` mapping the same way it already + excludes `evicted` at the query level, so a future read path that forgets + that query-level filter cannot resurface an evicted UTXO as unspent. + `findOutput`/`findOutputsForTransaction` also hydrate `outputsConsumed` + and `consumedBy` from the `consumptionEdges` persisted by the admission + commit (or by `updateConsumedBy` on the classic path) instead of always + returning them empty, so Engine history/delete paths can see consumption + edges written during admission. ### Security - (Notify of any improvements related to security vulnerabilities or potential risks.) diff --git a/packages/overlays/overlay/README.md b/packages/overlays/overlay/README.md index 3f9bbef58..4c3a822b3 100644 --- a/packages/overlays/overlay/README.md +++ b/packages/overlays/overlay/README.md @@ -83,11 +83,13 @@ applications should prefer the root entry point wherever possible. `AdmissionStorage` defines an additive v1 atomic admission contract for future adapters. `storageHasAdmission(storage)` reports whether the optional -`admission` field is present. `getAdmissionStorage(storage)` additionally -requires the v1 protocol and both commit and reconciliation methods. Existing -Knex and injected legacy adapters remain supported; their individual methods do -not imply atomic submission. Current `Engine.submit` does not call this -capability, and its early STEAK callback is not a durable commit receipt. +`admission` field is present. `getAdmissionStorage(storage)` detects an explicit provider with both +commit and reconciliation methods. Existing Knex and injected legacy adapters +remain supported; their individual methods do not imply atomic submission. +When `getAdmissionStorage(storage)` observes a complete `overlay-admission-v1` +provider, `Engine.submit` builds an admission plan and returns the saved STEAK +only after majority commit. The SQL/Knex path and its early STEAK callback are +unchanged; that callback is not a durable commit receipt. The contract separates local commit, index visibility and propagation. It binds operation identity to verified transaction, topic/policy and off-chain context; @@ -101,9 +103,9 @@ in this release candidate. ## Optional Mongo foundation The package also contains an opt-in MongoDB foundation for schema bootstrap, -content-addressed payload publication, reference guards, and payload collection. -It is not an Engine integration, an `AdmissionStorage` implementation, or a -default storage selection; importing `@bsv/overlay` alone does not load MongoDB. +content-addressed payload publication, reference guards, payload collection, +and an explicit `AdmissionStorage` adapter. Mongo is not the default Engine +storage selection; importing `@bsv/overlay` alone does not load MongoDB. Applications using a Mongo deep entry point install the optional peer first: @@ -112,15 +114,16 @@ npm install @bsv/overlay mongodb@^7.5.0 ``` The initial entry points are -`@bsv/overlay/storage/mongo/MongoSchema` and -`@bsv/overlay/storage/mongo/MongoPayloadStore`. They require an explicitly +`@bsv/overlay/storage/mongo/MongoSchema`, +`@bsv/overlay/storage/mongo/MongoPayloadStore`, +`@bsv/overlay/storage/mongo/MongoAdmissionStorage`, and +`@bsv/overlay/storage/mongo/MongoOverlayStorage`. They require an explicitly operated unsharded replica set; the supported deployment profile is three members. Payload publication makes GridFS bytes physically `published` before the guarded payload row becomes `ready`; caller-session reference and GC operations share that row guard. See the [Mongo v1 foundation](https://github.com/bsv-blockchain/ts-stack/blob/main/specs/overlay/mongo-v1.md) -for operational bounds, recovery rules, and the still-pending admission -transaction integration. +for operational bounds, recovery rules, and the opt-in admission path. ## Runtime and package formats diff --git a/packages/overlays/overlay/package.json b/packages/overlays/overlay/package.json index de23ed672..c352f34e7 100644 --- a/packages/overlays/overlay/package.json +++ b/packages/overlays/overlay/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay", - "version": "2.5.0", + "version": "2.6.0", "sideEffects": false, "engines": { "node": ">=22" diff --git a/packages/overlays/overlay/src/Engine.ts b/packages/overlays/overlay/src/Engine.ts index 2f0481ee0..906f3aa30 100644 --- a/packages/overlays/overlay/src/Engine.ts +++ b/packages/overlays/overlay/src/Engine.ts @@ -45,6 +45,13 @@ import { import { BASMRemote } from './BASMRemote.js' import { basmHash, basmInteger, requireBASM } from './BASMValidation.js' import { serializeErrorForLog, serializeLogValue } from './SafeLog.js' +import { + buildOverlayAdmissionPlan, + getOverlayAdmissionHost, + overlayAdmissionMode, + selectNewAdmissionTopics, + waitForAdmissionReceipt +} from './EngineAdmission.js' const DEFAULT_GASP_SYNC_LIMIT = 10000 const DEFAULT_BASM_RANGE_LIMIT = 1024 @@ -109,6 +116,19 @@ type StorageMutationContext = { offChainValues: number[] | undefined } +type OverlayAdmissionSubmitContext = { + taggedBEEF: TaggedBEEF + steak: STEAK + tx: Transaction + txid: string + mode: SubmissionMode + offChainValues: number[] | undefined + validations: TopicValidation[] + failedTopics: Set + anyTopicAccepted: boolean + onSteakReady?: (steak: STEAK) => void +} + function findSpendingInputIndex(tx: Transaction, output: Output): number { return tx.inputs.findIndex(input => { const realSource = input.sourceTXID || input.sourceTransaction?.id('hex') @@ -1048,6 +1068,82 @@ export class Engine { this.endTime(`transactionPropagation_${txid.substring(0, 10)}`) } + private assertSupportedTopics(topics: string[]): void { + for (const topic of topics) { + if (this.managers[topic] === undefined || this.managers[topic] === null) { + throw new Error(`This server does not support this topic: ${topic}`) + } + } + } + + private shouldSkipPropagation(mode: SubmissionMode): boolean { + return ( + this.advertiser === undefined || + mode === 'historical-tx' || + mode === 'historical-tx-no-spv' + ) + } + + private async acknowledgeOverlayAdmission( + context: OverlayAdmissionSubmitContext + ): Promise { + const admissionHost = getOverlayAdmissionHost(this.storage) + if (admissionHost === undefined) return undefined + const { + taggedBEEF, + steak, + tx, + txid, + mode, + offChainValues, + validations, + failedTopics, + anyTopicAccepted, + onSteakReady + } = context + if (!anyTopicAccepted) { + if (onSteakReady !== undefined) onSteakReady(steak) + return steak + } + // A topic counts toward `anyTopicAccepted` when it is a dupe (so the + // caller still gets broadcast/propagation for a retried submission), but + // a dupe must never be resubmitted for admission: it was already + // committed under a (possibly different) operation, and re-including it + // here would make commitAdmission reject the whole plan. When nothing + // left over is a genuinely new admission, skip the plan/commit entirely + // and hand back the STEAK already computed from validation — it reports + // each dupe topic as accepted-with-nothing-new, same as the classic + // (non-admission) storage path does. + if (selectNewAdmissionTopics(validations, failedTopics).length === 0) { + if (onSteakReady !== undefined) onSteakReady(steak) + return steak + } + const applied = await this.buildAppliedTransactionRecord(tx) + const buildPlan = async () => + await buildOverlayAdmissionPlan({ + host: admissionHost, + tx, + txid, + beef: taggedBEEF.beef, + topics: taggedBEEF.topics, + mode: overlayAdmissionMode(mode), + offChainValues, + validations, + failedTopics, + lookupServices: this.lookupServices, + includePropagation: !this.shouldSkipPropagation(mode), + applied + }) + const committed = await waitForAdmissionReceipt( + admissionHost.admission, + await buildPlan(), + buildPlan + ) + const acknowledged = JSON.parse(committed.receipt.steak) as STEAK + if (onSteakReady !== undefined) onSteakReady(acknowledged) + return acknowledged + } + /** * Submits a transaction for processing by Overlay Services. * @param {TaggedBEEF} taggedBEEF - The transaction to process @@ -1060,11 +1156,7 @@ export class Engine { * @returns {Promise} The submitted transaction execution acknowledgement */ async submit(taggedBEEF: TaggedBEEF, onSteakReady?: (steak: STEAK) => void, mode: 'historical-tx' | 'current-tx' | 'historical-tx-no-spv' = 'current-tx', offChainValues?: number[]): Promise { - for (const t of taggedBEEF.topics) { - if (this.managers[t] === undefined || this.managers[t] === null) { - throw new Error(`This server does not support this topic: ${t}`) - } - } + this.assertSupportedTopics(taggedBEEF.topics) // Validate the transaction SPV information const tx = Transaction.fromBEEF(taggedBEEF.beef) @@ -1131,6 +1223,20 @@ export class Engine { throw error } + const admissionSteak = await this.acknowledgeOverlayAdmission({ + taggedBEEF, + steak, + tx, + txid, + mode, + offChainValues, + validations, + failedTopics, + anyTopicAccepted, + onSteakReady + }) + if (admissionSteak !== undefined) return admissionSteak + // Call the callback function with STEAK if it is provided (before storage mutations) if (onSteakReady !== undefined) { onSteakReady(steak) @@ -1159,7 +1265,7 @@ export class Engine { }) // If we don't have an advertiser or we are dealing with historical transactions, just return the steak - if (this.advertiser === undefined || mode === 'historical-tx' || mode === 'historical-tx-no-spv') { + if (this.shouldSkipPropagation(mode)) { return steak } diff --git a/packages/overlays/overlay/src/EngineAdmission.ts b/packages/overlays/overlay/src/EngineAdmission.ts new file mode 100644 index 000000000..630c64197 --- /dev/null +++ b/packages/overlays/overlay/src/EngineAdmission.ts @@ -0,0 +1,404 @@ +import { createHash } from 'node:crypto' +import type { STEAK, Transaction } from '@bsv/sdk' +import { extractMerkleProofMetadata } from './BASM.js' +import { + admissionSemanticDigest, + asStorageUint64, + getAdmissionStorage, + type AdmissionCommit, + type AdmissionCommitResult, + type AdmissionOutboxIntent, + type AdmissionPayloadRef, + type AdmissionReconcileResult, + type AdmissionStorage, + type AdmissionTopicDecision, + type HistoryFence, + type StorageScope +} from './storage/AdmissionStorage.js' +import type { Output } from './Output.js' +import type { LookupService } from './LookupService.js' + +export const OVERLAY_ENGINE_POLICY_ID = 'overlay-engine-submit-v1' + +export type OverlayAdmissionMode = 'live' | 'historical' + +type TopicValidationLike = { + topic: string + isDupe: boolean + previousCoins: number[] + previousOutputs: Array + admissibleOutputs: { + outputsToAdmit: number[] + coinsToRetain: number[] + coinsRemoved?: number[] + } +} + +export interface OverlayAdmissionHost { + admission: AdmissionStorage + admissionScope: StorageScope + publishAdmissionPayload?: (input: { + kind: AdmissionPayloadRef['kind'] + bytes: Uint8Array + txid?: string + }) => Promise + enlistedIndexTargets?: () => readonly string[] + getHistoryFence?: (topic: string) => Promise +} + +export function getOverlayAdmissionHost(storage: unknown): OverlayAdmissionHost | undefined { + const admission = getAdmissionStorage(storage) + if (admission === undefined || typeof storage !== 'object' || storage === null) return undefined + if (!('admissionScope' in storage)) return undefined + const scope = (storage as { admissionScope?: StorageScope }).admissionScope + if ( + scope === undefined || + typeof scope.network !== 'string' || + typeof scope.genesisHash !== 'string' || + typeof scope.nodeId !== 'string' || + !/^[0-9a-f]{64}$/.test(scope.genesisHash) + ) { + return undefined + } + const host = storage as OverlayAdmissionHost + return { + admission, + admissionScope: { ...scope }, + // Adapters are class instances whose methods read instance state, so each + // optional method stays bound to the storage it came from. + publishAdmissionPayload: host.publishAdmissionPayload?.bind(host), + enlistedIndexTargets: host.enlistedIndexTargets?.bind(host), + getHistoryFence: host.getHistoryFence?.bind(host) + } +} + +export function overlayAdmissionContextDigest(offChainValues?: number[]): string { + const hash = createHash('sha256') + if (offChainValues !== undefined) hash.update(Buffer.from(offChainValues)) + return hash.digest('hex') +} + +export function overlayAdmissionOperationId( + _mode: OverlayAdmissionMode, + txid: string, + topics: string[] +): string { + const topicKey = [...topics].sort((a, b) => a.localeCompare(b, 'en')).join('\n') + // Historical GASP replay and a later live retry represent the same + // transaction admission. Keep their id stable so the durable admission + // record is replayed instead of being submitted a second time. + const raw = `submit:${txid}:${topicKey}` + if (Buffer.byteLength(raw, 'utf8') <= 512 && raw.isWellFormed()) return raw + return `submit:${createHash('sha256').update(raw, 'utf8').digest('hex')}` +} + +export function overlayAdmissionMode( + mode: 'historical-tx' | 'current-tx' | 'historical-tx-no-spv' +): OverlayAdmissionMode { + return mode === 'current-tx' ? 'live' : 'historical' +} + +async function localPayload( + host: OverlayAdmissionHost, + kind: AdmissionPayloadRef['kind'], + bytes: Uint8Array, + txid?: string +): Promise { + if (host.publishAdmissionPayload !== undefined) { + return await host.publishAdmissionPayload({ kind, bytes, txid }) + } + return { + kind, + digest: createHash('sha256').update(bytes).digest('hex'), + byteLength: asStorageUint64(String(bytes.byteLength)) + } +} + +/** + * Selects the topics that require a new admission decision: topics that did + * not fail validation, are not already-applied dupes, and actually admit or + * retain something (or consume a previously-admitted coin). Already-applied + * (dupe) topics are excluded so that a retry — including a historical GASP + * replay after a live admit, or a multi-topic resubmit where only some + * topics were previously applied — never re-submits an admission decision + * for a topic that MongoAdmissionStorage.assertAppliedAvailable would reject + * as belonging to a different operation. + */ +export function selectNewAdmissionTopics( + validations: readonly T[], + failedTopics: Set +): T[] { + return validations.filter( + validation => + !failedTopics.has(validation.topic) && + !validation.isDupe && + (validation.admissibleOutputs.outputsToAdmit.length > 0 || + validation.admissibleOutputs.coinsToRetain.length > 0 || + validation.previousCoins.length > 0) + ) +} + +export async function buildOverlayAdmissionPlan(input: { + host: OverlayAdmissionHost + tx: Transaction + txid: string + beef: number[] + topics: string[] + mode: OverlayAdmissionMode + offChainValues?: number[] + validations: TopicValidationLike[] + failedTopics: Set + lookupServices: { [key: string]: LookupService } + includePropagation: boolean + applied?: { + firstSeenHeight?: number + blockHeight?: number + blockHash?: string + blockIndex?: number + merkleRoot?: string + } +}): Promise { + // Only topics with a genuinely new admission get an identity/decision entry + // and take part in the commit. Dupe topics are still reflected in `steak` + // below (with their empty admissibleOutputs), matching what the classic + // storage path already returns for a dupe, but they must never be + // resubmitted to commitAdmission: MongoAdmissionStorage.assertAppliedAvailable + // rejects an applied row whose admissionId belongs to a different + // operation, which a dupe re-included here would trigger on retry. + const accepted = selectNewAdmissionTopics(input.validations, input.failedTopics) + const identityTopics = accepted.map(validation => ({ + topic: validation.topic, + policyId: OVERLAY_ENGINE_POLICY_ID + })) + if (identityTopics.length === 0) throw new Error('Overlay admission plan has no topics') + const identity = { + scope: input.host.admissionScope, + txid: input.txid, + mode: input.mode, + contextDigest: overlayAdmissionContextDigest(input.offChainValues), + topics: identityTopics + } + const raw = await localPayload( + input.host, + 'raw-transaction', + Buffer.from(input.tx.toBinary()), + input.txid + ) + const payloads: AdmissionPayloadRef[] = [raw] + let proof: AdmissionPayloadRef | undefined + if (input.tx.merklePath !== undefined) { + proof = await localPayload( + input.host, + 'merkle-path', + Buffer.from(input.tx.merklePath.toBinary()) + ) + payloads.push(proof) + } + const merkle = extractMerkleProofMetadata(input.txid, input.tx.merklePath) + const decisions: AdmissionTopicDecision[] = [] + const steak: STEAK = {} + for (const validation of input.validations) { + steak[validation.topic] = { + outputsToAdmit: validation.admissibleOutputs.outputsToAdmit, + coinsToRetain: validation.admissibleOutputs.coinsToRetain, + coinsRemoved: validation.admissibleOutputs.coinsRemoved ?? [] + } + } + for (const validation of accepted) { + const { outputsConsumed, outputsToMarkStale } = classifyCoins(input.tx, validation) + const fence = + input.host.getHistoryFence === undefined + ? { chainEpoch: asStorageUint64('0'), topicHistoryGeneration: asStorageUint64('0') } + : await input.host.getHistoryFence(validation.topic) + const outputs: AdmissionTopicDecision['outputs'] = [] + for (const outputIndex of validation.admissibleOutputs.outputsToAdmit) { + const txOut = input.tx.outputs[outputIndex] + if (typeof txOut?.satoshis !== 'number' || !Number.isSafeInteger(txOut.satoshis)) continue + const scriptBytes = Buffer.from(txOut.lockingScript.toBinary()) + const script = await localPayload(input.host, 'locking-script', scriptBytes) + payloads.push(script) + outputs.push({ + txid: input.txid, + outputIndex: asStorageUint64(String(outputIndex)), + satoshis: asStorageUint64(String(txOut.satoshis)), + score: asStorageUint64(String(Date.now())), + script: { payload: script, offset: asStorageUint64('0'), byteLength: script.byteLength } + }) + } + const spends = validation.previousOutputs.flatMap(output => + output === null + ? [] + : [ + { + outpoint: { txid: output.txid, outputIndex: asStorageUint64(String(output.outputIndex)) }, + expectedVersion: '1', + spender: input.txid + } + ] + ) + decisions.push({ + topic: validation.topic, + expectedHistory: fence, + reads: [], + spends, + evictions: outputsToMarkStale.map(item => ({ + txid: item.txid, + outputIndex: asStorageUint64(String(item.previousOutputIndex)) + })), + outputs, + edges: outputsConsumed.flatMap(source => + outputs.map(output => ({ + source: { txid: source.txid, outputIndex: asStorageUint64(String(source.outputIndex)) }, + consumer: { txid: output.txid, outputIndex: output.outputIndex } + })) + ), + applied: appliedRecord(input, proof, merkle) + }) + steak[validation.topic] = { + outputsToAdmit: validation.admissibleOutputs.outputsToAdmit, + coinsToRetain: validation.admissibleOutputs.coinsToRetain, + coinsRemoved: outputsToMarkStale.map(item => item.inputIndex) + } + } + const enlisted = new Set(input.host.enlistedIndexTargets?.() ?? []) + const outbox: AdmissionOutboxIntent[] = Object.keys(input.lookupServices) + .filter(target => !enlisted.has(target)) + .map(target => ({ + eventId: `${overlayAdmissionOperationId(input.mode, input.txid, input.topics)}:lookup:${target}`, + kind: 'lookup' as const, + target, + payloads: [raw] + })) + if (input.includePropagation && input.mode === 'live') { + outbox.push({ + eventId: `${overlayAdmissionOperationId(input.mode, input.txid, input.topics)}:propagate`, + kind: 'propagation', + target: 'ship', + payloads: [raw] + }) + } + return { + key: { + scope: input.host.admissionScope, + operationId: overlayAdmissionOperationId(input.mode, input.txid, input.topics), + semanticDigest: admissionSemanticDigest(identity) + }, + identity, + payloads: uniquePayloads(payloads), + decisions, + outbox, + steak: JSON.stringify(steak) + } +} + +export async function waitForAdmissionReceipt( + admission: AdmissionStorage, + plan: AdmissionCommit, + rebuild: () => Promise +): Promise { + let current = plan + for (let attempt = 0; attempt < 8; attempt += 1) { + const result = await admission.commitAdmission(current) + if (result.state === 'committed') return result + if (result.state === 'rejected') { + if (result.code === 'read-conflict' && attempt < 7) { + current = await rebuild() + continue + } + throw new Error(`Overlay admission rejected: ${result.code}`) + } + const reconciled: AdmissionReconcileResult = await admission.reconcileAdmission( + current.key, + result.attemptId + ) + if (reconciled.state === 'committed') return reconciled + if (reconciled.state === 'aborted') { + current = await rebuild() + continue + } + if (reconciled.state === 'rejected') { + throw new Error(`Overlay admission rejected: ${reconciled.code}`) + } + } + throw new Error('Overlay admission commit is pending') +} + +function appliedRecord( + input: { + txid: string + applied?: { + firstSeenHeight?: number + blockHeight?: number + blockHash?: string + blockIndex?: number + merkleRoot?: string + } + }, + proof: AdmissionPayloadRef | undefined, + merkle: ReturnType +): AdmissionTopicDecision['applied'] { + const applied: AdmissionTopicDecision['applied'] = { txid: input.txid } + const firstSeen = input.applied?.firstSeenHeight ?? merkle?.blockHeight + if (firstSeen !== undefined) applied.firstSeenHeight = asStorageUint64(String(firstSeen)) + if (proof !== undefined) applied.proof = proof + const blockHash = input.applied?.blockHash + const height = input.applied?.blockHeight ?? merkle?.blockHeight + const index = input.applied?.blockIndex ?? merkle?.blockIndex + const merkleRoot = input.applied?.merkleRoot ?? merkle?.merkleRoot + if ( + blockHash !== undefined && + height !== undefined && + index !== undefined && + merkleRoot !== undefined + ) { + applied.block = { + height: asStorageUint64(String(height)), + hash: blockHash, + index: asStorageUint64(String(index)), + merkleRoot + } + } + return applied +} + +function uniquePayloads(payloads: AdmissionPayloadRef[]): AdmissionPayloadRef[] { + const seen = new Set() + const result: AdmissionPayloadRef[] = [] + for (const payload of payloads) { + const key = `${payload.kind}:${payload.digest}:${payload.byteLength}` + if (seen.has(key)) continue + seen.add(key) + result.push(payload) + } + return result +} + +function classifyCoins( + tx: Transaction, + validation: TopicValidationLike +): { + outputsConsumed: Array<{ txid: string; outputIndex: number }> + outputsToMarkStale: Array<{ txid: string; previousOutputIndex: number; inputIndex: number }> +} { + const outputsConsumed: Array<{ txid: string; outputIndex: number }> = [] + const outputsToMarkStale: Array<{ + txid: string + previousOutputIndex: number + inputIndex: number + }> = [] + for (const inputIndex of validation.previousCoins) { + const input = tx.inputs[inputIndex] + const previousTXID = input.sourceTXID ?? input.sourceTransaction?.id('hex') + if (typeof previousTXID !== 'string') continue + if (validation.admissibleOutputs.coinsToRetain.includes(inputIndex)) { + outputsConsumed.push({ txid: previousTXID, outputIndex: input.sourceOutputIndex }) + } else { + outputsToMarkStale.push({ + txid: previousTXID, + previousOutputIndex: input.sourceOutputIndex, + inputIndex + }) + } + } + return { outputsConsumed, outputsToMarkStale } +} diff --git a/packages/overlays/overlay/src/__tests/Engine.admission.test.ts b/packages/overlays/overlay/src/__tests/Engine.admission.test.ts new file mode 100644 index 000000000..355e97001 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/Engine.admission.test.ts @@ -0,0 +1,420 @@ +import { Engine } from '../Engine.js' +import type { LookupService } from '../LookupService.js' +import type { TopicManager } from '../TopicManager.js' +import type { Storage } from '../storage/Storage.js' +import type { OverlayAdmissionHost } from '../EngineAdmission.js' +import type { + AdmissionCommit, + AdmissionReceipt, + AdmissionStorage, + StorageScope +} from '../storage/AdmissionStorage.js' +import { getAdmissionStorage } from '../storage/AdmissionStorage.js' +import { KnexStorage } from '../storage/knex/KnexStorage.js' +import type { Knex } from 'knex' +import { Transaction, type AdmittanceInstructions } from '@bsv/sdk' + +const mockChainTracker = { + isValidRootForHeight: jest.fn(async () => true), + currentHeight: jest.fn(async () => 800000) +} + +const BRC62Hex = + '0100beef01fe636d0c0007021400fe507c0c7aa754cef1f7889d5fd395cf1f785dd7de98eed895dbedfe4e5bc70d1502ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e010b00bc4ff395efd11719b277694cface5aa50d085a0bb81f613f70313acd28cf4557010400574b2d9142b8d28b61d88e3b2c3f44d858411356b49a28a4643b6d1a6a092a5201030051a05fc84d531b5d250c23f4f886f6812f9fe3f402d61607f977b4ecd2701c19010000fd781529d58fc2523cf396a7f25440b409857e7e221766c57214b1d38c7b481f01010062f542f45ea3660f86c013ced80534cb5fd4c19d66c56e7e8c5d4bf2d40acc5e010100b121e91836fd7cd5102b654e9f72f3cf6fdbfd0b161c53a9c54b12c841126331020100000001cd4e4cac3c7b56920d1e7655e7e260d31f29d9a388d04910f1bbd72304a79029010000006b483045022100e75279a205a547c445719420aa3138bf14743e3f42618e5f86a19bde14bb95f7022064777d34776b05d816daf1699493fcdf2ef5a5ab1ad710d9c97bfb5b8f7cef3641210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013e660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000001000100000001ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e000000006a47304402203a61a2e931612b4bda08d541cfb980885173b8dcf64a3471238ae7abcd368d6402204cbf24f04b9aa2256d8901f0ed97866603d2be8324c2bfb7a37bf8fc90edd5b441210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013c660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000000' + +const exampleTX = Transaction.fromHexBEEF(BRC62Hex) +const exampleBeef = exampleTX.toBEEF() + +const admissionScope: StorageScope = { + network: 'testnet', + genesisHash: '11'.repeat(32), + nodeId: 'engine-node' +} + +function receiptFor(plan: AdmissionCommit): AdmissionReceipt { + return { + operationId: plan.key.operationId, + semanticDigest: plan.key.semanticDigest, + durability: 'atomic-local', + steak: plan.steak, + indexes: plan.outbox + .filter(intent => intent.kind === 'lookup') + .map(intent => ({ target: intent.target, state: 'pending' as const })), + propagation: plan.outbox.some(intent => intent.kind === 'propagation') + ? 'pending' + : 'not-requested' + } +} + +describe('Engine admission submit', () => { + let mockTopicManager: TopicManager + let mockLookupService: LookupService + let mockStorage: Storage & + OverlayAdmissionHost & { admission: AdmissionStorage; admissionScope: StorageScope } + let commitAdmission: jest.Mock + let saved: AdmissionReceipt | undefined + + beforeEach(() => { + jest.clearAllMocks() + saved = undefined + mockTopicManager = { + identifyAdmissibleOutputs: jest.fn(async (): Promise => ({ + outputsToAdmit: [0], + coinsToRetain: [] + })), + getDocumentation: async () => 'docs', + getMetaData: async () => ({ name: 'Mock', shortDescription: 'Mock' }) + } + mockLookupService = { + outputAdmittedByTopic: jest.fn(), + outputSpent: jest.fn(), + lookup: jest.fn(), + outputNoLongerRetainedInHistory: jest.fn(), + outputEvicted: jest.fn(), + admissionMode: 'locking-script', + spendNotificationMode: 'none', + getDocumentation: async () => 'docs', + getMetaData: async () => ({ name: 'Mock', shortDescription: 'Mock' }) + } + commitAdmission = jest.fn(async (plan: AdmissionCommit) => { + if (saved !== undefined) return { state: 'committed' as const, receipt: saved } + saved = receiptFor(plan) + return { state: 'committed' as const, receipt: saved } + }) + mockStorage = { + admission: { + protocol: 'overlay-admission-v1', + commitAdmission, + reconcileAdmission: jest.fn() + }, + admissionScope, + doesAppliedTransactionExist: jest.fn(async () => false), + insertAppliedTransaction: jest.fn(), + insertOutput: jest.fn(), + findOutput: jest.fn(async () => null), + findOutputsForTransaction: jest.fn(async () => []), + markUTXOAsSpent: jest.fn(), + updateConsumedBy: jest.fn(), + updateTransactionBEEF: jest.fn(), + deleteOutput: jest.fn(), + findUTXOsForTopic: jest.fn(), + updateLastInteraction: jest.fn(), + getLastInteraction: jest.fn(async () => 0) + } + }) + + test('SQL Knex storage still has no admission capability', () => { + const knex = jest.fn() as unknown as Knex + expect(getAdmissionStorage(new KnexStorage(knex))).toBeUndefined() + }) + + test('uses commitAdmission and does not mutate via CRUD or lookup callbacks', async () => { + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + const steak = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }) + expect(commitAdmission).toHaveBeenCalledTimes(1) + expect(mockStorage.insertOutput).not.toHaveBeenCalled() + expect(mockStorage.markUTXOAsSpent).not.toHaveBeenCalled() + expect(mockLookupService.outputAdmittedByTopic).not.toHaveBeenCalled() + expect(steak.Hello.outputsToAdmit).toEqual([0]) + }) + + test('returns the saved STEAK after a lost admission ACK', async () => { + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + const first = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }) + const second = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }) + expect(second).toEqual(first) + expect(commitAdmission).toHaveBeenCalledTimes(2) + expect(mockLookupService.outputAdmittedByTopic).not.toHaveBeenCalled() + }) + + test('historical submit omits propagation outbox', async () => { + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker, + undefined, + undefined, + undefined, + undefined, + { + createAdvertisements: jest.fn(), + revokeAdvertisements: jest.fn(), + findAllAdvertisements: jest.fn(), + parseAdvertisement: jest.fn() + } + ) + await engine.submit({ beef: exampleBeef, topics: ['Hello'] }, undefined, 'historical-tx') + const plan = commitAdmission.mock.calls[0][0] as AdmissionCommit + expect(plan.identity.mode).toBe('historical') + expect(plan.outbox.some(intent => intent.kind === 'propagation')).toBe(false) + }) + + test('live submit with an advertiser records a pending propagation outbox', async () => { + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker, + undefined, + undefined, + undefined, + undefined, + { + createAdvertisements: jest.fn(), + revokeAdvertisements: jest.fn(), + findAllAdvertisements: jest.fn(), + parseAdvertisement: jest.fn() + } + ) + const onReady = jest.fn() + const steak = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }, onReady) + const plan = commitAdmission.mock.calls[0][0] as AdmissionCommit + expect(plan.identity.mode).toBe('live') + expect(plan.outbox.some(intent => intent.kind === 'propagation')).toBe(true) + expect(onReady).toHaveBeenCalledWith(steak) + }) + + test('calls onSTEAKReady only after the admission receipt', async () => { + const order: string[] = [] + commitAdmission = jest.fn(async (plan: AdmissionCommit) => { + order.push('commit') + saved = receiptFor(plan) + return { state: 'committed' as const, receipt: saved } + }) + mockStorage.admission.commitAdmission = commitAdmission + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + await engine.submit({ beef: exampleBeef, topics: ['Hello'] }, () => { + order.push('callback') + }) + expect(order).toEqual(['commit', 'callback']) + }) + + test('keeps failed-topic STEAK keys and attaches merkle applied metadata', async () => { + const rejectManager: TopicManager = { + identifyAdmissibleOutputs: jest.fn(async () => { + throw new Error('topic rejected') + }), + getDocumentation: async () => 'docs', + getMetaData: async () => ({ name: 'Reject', shortDescription: 'Reject' }) + } + const proven = exampleTX.inputs[0].sourceTransaction + if (proven === undefined || proven.merklePath === undefined) { + throw new Error('expected the BEEF ancestor to carry a merkle path') + } + const engine = new Engine( + { Hello: mockTopicManager, Reject: rejectManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + engine.logger = { ...console, error: jest.fn() } + const steak = await engine.submit({ + beef: proven.toBEEF(), + topics: ['Hello', 'Reject'] + }) + expect(Object.keys(steak).sort()).toEqual(['Hello', 'Reject']) + expect(steak.Hello.outputsToAdmit).toEqual([0]) + expect(steak.Reject.outputsToAdmit).toEqual([]) + const plan = commitAdmission.mock.calls[0][0] as AdmissionCommit + expect(JSON.parse(plan.steak)).toEqual(steak) + expect(plan.decisions).toHaveLength(1) + expect(plan.decisions[0].applied.proof?.kind).toBe('merkle-path') + expect(plan.decisions[0].applied.firstSeenHeight).toBe('800000') + expect(plan.decisions[0].applied.block).toBeUndefined() + }) + + test('returns STEAK without commit when no topic is accepted', async () => { + mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => ({ + outputsToAdmit: [], + coinsToRetain: [] + })) + const onReady = jest.fn() + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + const steak = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }, onReady) + expect(steak).toEqual({ Hello: { outputsToAdmit: [], coinsToRetain: [] } }) + expect(commitAdmission).not.toHaveBeenCalled() + expect(onReady).toHaveBeenCalledWith(steak) + }) + + test('does not throw when no topic is accepted and no onSteakReady callback is supplied', async () => { + mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => ({ + outputsToAdmit: [], + coinsToRetain: [] + })) + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + const steak = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }) + expect(steak).toEqual({ Hello: { outputsToAdmit: [], coinsToRetain: [] } }) + expect(commitAdmission).not.toHaveBeenCalled() + }) + + test('does not call commitAdmission and returns the in-memory STEAK when every topic is a dupe', async () => { + mockStorage.doesAppliedTransactionExist = jest.fn(async () => true) + const onReady = jest.fn() + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + const steak = await engine.submit({ beef: exampleBeef, topics: ['Hello'] }, onReady) + expect(steak).toEqual({ Hello: { outputsToAdmit: [], coinsToRetain: [] } }) + expect(commitAdmission).not.toHaveBeenCalled() + expect(onReady).toHaveBeenCalledWith(steak) + expect(mockTopicManager.identifyAdmissibleOutputs).not.toHaveBeenCalled() + }) + + test('commits only the new topic on a mixed dupe/new-admission submission, while STEAK still reports the dupe', async () => { + mockStorage.doesAppliedTransactionExist = jest.fn( + async (params: { topic: string }) => params.topic === 'Dupe' + ) + const engine = new Engine( + { Hello: mockTopicManager, Dupe: mockTopicManager }, + { Hello: mockLookupService, Dupe: mockLookupService }, + mockStorage, + mockChainTracker + ) + const steak = await engine.submit({ beef: exampleBeef, topics: ['Hello', 'Dupe'] }) + expect(commitAdmission).toHaveBeenCalledTimes(1) + const plan = commitAdmission.mock.calls[0][0] as AdmissionCommit + expect(plan.decisions.map(decision => decision.topic)).toEqual(['Hello']) + expect(plan.identity.topics.map(entry => entry.topic)).toEqual(['Hello']) + const planSteak = JSON.parse(plan.steak) as Record + expect(planSteak.Dupe).toEqual({ outputsToAdmit: [], coinsToRetain: [], coinsRemoved: [] }) + expect(steak.Hello.outputsToAdmit).toEqual([0]) + expect(steak.Dupe).toEqual({ outputsToAdmit: [], coinsToRetain: [], coinsRemoved: [] }) + }) + + test('resubmit after historical admission (same tx, same topic) does not re-commit and does not throw', async () => { + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + const first = await engine.submit( + { beef: exampleBeef, topics: ['Hello'] }, + undefined, + 'historical-tx-no-spv' + ) + expect(first.Hello.outputsToAdmit).toEqual([0]) + expect(commitAdmission).toHaveBeenCalledTimes(1) + + mockStorage.doesAppliedTransactionExist = jest.fn(async () => true) + const second = await engine.submit( + { beef: exampleBeef, topics: ['Hello'] }, + undefined, + 'current-tx' + ) + expect(commitAdmission).toHaveBeenCalledTimes(1) + expect(second).toEqual({ Hello: { outputsToAdmit: [], coinsToRetain: [] } }) + }) + + test('uses host payload, fence, and enlisted index hooks', async () => { + const publishAdmissionPayload = jest.fn( + async (input: { kind: string; bytes: Uint8Array; txid?: string }) => ({ + kind: input.kind, + digest: 'ab'.repeat(32), + byteLength: String(input.bytes.byteLength) + }) + ) + const getHistoryFence = jest.fn(async () => ({ + chainEpoch: '1', + topicHistoryGeneration: '2' + })) + mockStorage.publishAdmissionPayload = publishAdmissionPayload + mockStorage.enlistedIndexTargets = () => ['Hello'] + mockStorage.getHistoryFence = getHistoryFence + mockStorage.findOutput = jest.fn(async () => ({ + txid: + exampleTX.inputs[0].sourceTXID ?? + '3ecead27a44d013ad1aae40038acbb1883ac9242406808bb4667c15b4f164eac', + outputIndex: exampleTX.inputs[0].sourceOutputIndex, + outputScript: [], + satoshis: 1, + topic: 'Hello', + spent: false, + outputsConsumed: [], + consumedBy: [] + })) + mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => ({ + outputsToAdmit: [0], + coinsToRetain: [0] + })) + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + await engine.submit({ beef: exampleBeef, topics: ['Hello'] }, undefined, 'current-tx', [7, 8]) + const submitted = commitAdmission.mock.calls[0][0] as AdmissionCommit + expect(publishAdmissionPayload).toHaveBeenCalled() + expect(getHistoryFence).toHaveBeenCalledWith('Hello') + expect(submitted.decisions[0].expectedHistory).toEqual({ + chainEpoch: '1', + topicHistoryGeneration: '2' + }) + expect(submitted.decisions[0].spends).toHaveLength(1) + expect(submitted.outbox.some(intent => intent.target === 'Hello')).toBe(false) + expect(submitted.identity.contextDigest).toHaveLength(64) + }) + + test('surfaces admission rejection and recovers a pending receipt', async () => { + const engine = new Engine( + { Hello: mockTopicManager }, + { Hello: mockLookupService }, + mockStorage, + mockChainTracker + ) + commitAdmission.mockResolvedValueOnce({ state: 'rejected', code: 'spend-conflict' }) + await expect(engine.submit({ beef: exampleBeef, topics: ['Hello'] })).rejects.toThrow( + 'Overlay admission rejected: spend-conflict' + ) + + commitAdmission + .mockResolvedValueOnce({ state: 'rejected', code: 'read-conflict' }) + .mockImplementation(async (plan: AdmissionCommit) => { + saved = receiptFor(plan) + return { state: 'committed' as const, receipt: saved } + }) + await expect(engine.submit({ beef: exampleBeef, topics: ['Hello'] })).resolves.toEqual( + expect.objectContaining({ Hello: expect.objectContaining({ outputsToAdmit: [0] }) }) + ) + + commitAdmission.mockResolvedValue({ state: 'pending', attemptId: 'lost-ack' }) + const lastPlan = commitAdmission.mock.calls[commitAdmission.mock.calls.length - 1]?.[0] as + AdmissionCommit | undefined + mockStorage.admission.reconcileAdmission = jest.fn(async () => ({ + state: 'committed' as const, + receipt: + saved ?? receiptFor(lastPlan ?? (commitAdmission.mock.calls[0][0] as AdmissionCommit)) + })) + await expect(engine.submit({ beef: exampleBeef, topics: ['Hello'] })).resolves.toBeDefined() + }) +}) diff --git a/packages/overlays/overlay/src/__tests/EngineAdmission.test.ts b/packages/overlays/overlay/src/__tests/EngineAdmission.test.ts new file mode 100644 index 000000000..1b788390a --- /dev/null +++ b/packages/overlays/overlay/src/__tests/EngineAdmission.test.ts @@ -0,0 +1,527 @@ +import { Transaction } from '@bsv/sdk' +import type { LookupService } from '../LookupService.js' +import type { Output } from '../Output.js' +import { + asStorageUint64, + type AdmissionCommit, + type AdmissionStorage, + type StorageScope +} from '../storage/AdmissionStorage.js' +import { + buildOverlayAdmissionPlan, + getOverlayAdmissionHost, + overlayAdmissionContextDigest, + overlayAdmissionMode, + overlayAdmissionOperationId, + waitForAdmissionReceipt, + type OverlayAdmissionHost +} from '../EngineAdmission.js' + +const BRC62Hex = + '0100beef01fe636d0c0007021400fe507c0c7aa754cef1f7889d5fd395cf1f785dd7de98eed895dbedfe4e5bc70d1502ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e010b00bc4ff395efd11719b277694cface5aa50d085a0bb81f613f70313acd28cf4557010400574b2d9142b8d28b61d88e3b2c3f44d858411356b49a28a4643b6d1a6a092a5201030051a05fc84d531b5d250c23f4f886f6812f9fe3f402d61607f977b4ecd2701c19010000fd781529d58fc2523cf396a7f25440b409857e7e221766c57214b1d38c7b481f01010062f542f45ea3660f86c013ced80534cb5fd4c19d66c56e7e8c5d4bf2d40acc5e010100b121e91836fd7cd5102b654e9f72f3cf6fdbfd0b161c53a9c54b12c841126331020100000001cd4e4cac3c7b56920d1e7655e7e260d31f29d9a388d04910f1bbd72304a79029010000006b483045022100e75279a205a547c445719420aa3138bf14743e3f42618e5f86a19bde14bb95f7022064777d34776b05d816daf1699493fcdf2ef5a5ab1ad710d9c97bfb5b8f7cef3641210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013e660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000001000100000001ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e000000006a47304402203a61a2e931612b4bda08d541cfb980885173b8dcf64a3471238ae7abcd368d6402204cbf24f04b9aa2256d8901f0ed97866603d2be8324c2bfb7a37bf8fc90edd5b441210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013c660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000000' + +const exampleTX = Transaction.fromHexBEEF(BRC62Hex) +const exampleBeef = exampleTX.toBEEF() +const exampleTxid = exampleTX.id('hex') +const previousTxid = + exampleTX.inputs[0].sourceTXID ?? exampleTX.inputs[0].sourceTransaction?.id('hex') +if (typeof previousTxid !== 'string') + throw new Error('expected the example transaction to name its source') + +const admissionScope: StorageScope = { + network: 'testnet', + genesisHash: '11'.repeat(32), + nodeId: 'engine-node' +} + +const lookupServices = { Hello: {} as LookupService } + +const previousOutput: Output = { + txid: previousTxid, + outputIndex: exampleTX.inputs[0].sourceOutputIndex, + outputScript: [], + satoshis: 1, + topic: 'Hello', + spent: false, + outputsConsumed: [], + consumedBy: [] +} + +function host(overrides: Partial = {}): OverlayAdmissionHost { + return { + admission: { + protocol: 'overlay-admission-v1', + commitAdmission: jest.fn(), + reconcileAdmission: jest.fn() + }, + admissionScope, + ...overrides + } +} + +function validation( + overrides: Partial<{ + topic: string + isDupe: boolean + previousCoins: number[] + previousOutputs: Array + outputsToAdmit: number[] + coinsToRetain: number[] + coinsRemoved: number[] + }> = {} +) { + return { + topic: overrides.topic ?? 'Hello', + isDupe: overrides.isDupe ?? false, + previousCoins: overrides.previousCoins ?? [], + previousOutputs: overrides.previousOutputs ?? [], + admissibleOutputs: { + outputsToAdmit: overrides.outputsToAdmit ?? [0], + coinsToRetain: overrides.coinsToRetain ?? [], + coinsRemoved: overrides.coinsRemoved + } + } +} + +describe('Engine overlay admission helpers', () => { + test('getOverlayAdmissionHost requires v1 admission and a hex genesis', () => { + expect(getOverlayAdmissionHost(undefined)).toBeUndefined() + expect(getOverlayAdmissionHost(null)).toBeUndefined() + expect(getOverlayAdmissionHost({})).toBeUndefined() + const admission = { + protocol: 'overlay-admission-v1' as const, + commitAdmission: jest.fn(), + reconcileAdmission: jest.fn() + } + expect(getOverlayAdmissionHost({ admission })).toBeUndefined() + expect( + getOverlayAdmissionHost({ + admission, + admissionScope: { ...admissionScope, genesisHash: 'zz'.repeat(32) } + }) + ).toBeUndefined() + expect( + getOverlayAdmissionHost({ + admission, + admissionScope: { network: 1, genesisHash: admissionScope.genesisHash, nodeId: 'n' } + }) + ).toBeUndefined() + const publishAdmissionPayload = jest.fn() + const enlistedIndexTargets = jest.fn(() => ['Hello']) + const getHistoryFence = jest.fn() + const storage = { + admission, + admissionScope, + publishAdmissionPayload, + enlistedIndexTargets, + getHistoryFence + } + const found = getOverlayAdmissionHost(storage) + expect(found?.admission).toBe(admission) + expect(found?.admissionScope).toEqual(admissionScope) + expect(found?.admissionScope).not.toBe(admissionScope) + // Host methods are forwarded to the storage and keep it as their receiver. + const payload = { kind: 'raw-tx' as const, bytes: Uint8Array.of(1) } + void found?.publishAdmissionPayload?.(payload) + expect(publishAdmissionPayload).toHaveBeenCalledWith(payload) + expect(publishAdmissionPayload.mock.contexts[0]).toBe(storage) + expect(found?.enlistedIndexTargets?.()).toEqual(['Hello']) + expect(enlistedIndexTargets.mock.contexts[0]).toBe(storage) + void found?.getHistoryFence?.('Hello') + expect(getHistoryFence).toHaveBeenCalledWith('Hello') + expect(getHistoryFence.mock.contexts[0]).toBe(storage) + }) + + test('operation ids stay raw when short and hash when oversized or ill-formed', () => { + const short = overlayAdmissionOperationId('live', exampleTxid, ['Hello']) + expect(short).toBe(`submit:${exampleTxid}:Hello`) + const hashed = overlayAdmissionOperationId('historical', exampleTxid, ['t'.repeat(600)]) + expect(hashed.startsWith('submit:')).toBe(true) + expect(hashed).not.toContain('t'.repeat(600)) + expect(overlayAdmissionOperationId('live', exampleTxid, ['\ud800']).startsWith('submit:')).toBe( + true + ) + expect(overlayAdmissionMode('current-tx')).toBe('live') + expect(overlayAdmissionMode('historical-tx')).toBe('historical') + expect(overlayAdmissionMode('historical-tx-no-spv')).toBe('historical') + expect(overlayAdmissionContextDigest()).not.toBe(overlayAdmissionContextDigest([1, 2])) + }) + + test('buildOverlayAdmissionPlan throws when no topic is accepted', async () => { + await expect( + buildOverlayAdmissionPlan({ + host: host(), + tx: exampleTX, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello'], + mode: 'live', + validations: [validation({ outputsToAdmit: [], coinsToRetain: [] })], + failedTopics: new Set(), + lookupServices, + includePropagation: true + }) + ).rejects.toThrow('Overlay admission plan has no topics') + }) + + test('builds spends, evictions, enlisted outbox, and published payloads', async () => { + const publishAdmissionPayload = jest.fn(async input => ({ + kind: input.kind, + digest: overlayAdmissionContextDigest(Array.from(input.bytes)), + byteLength: asStorageUint64(String(input.bytes.byteLength)) + })) + const getHistoryFence = jest.fn(async () => ({ + chainEpoch: asStorageUint64('4'), + topicHistoryGeneration: asStorageUint64('5') + })) + const plan = await buildOverlayAdmissionPlan({ + host: host({ + publishAdmissionPayload, + getHistoryFence, + enlistedIndexTargets: () => ['Hello'] + }), + tx: exampleTX, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello'], + mode: 'live', + offChainValues: [9], + validations: [ + validation({ + previousCoins: [0], + previousOutputs: [previousOutput, null], + coinsToRetain: [0], + outputsToAdmit: [0, 0, 99] + }), + validation({ topic: 'Failed', outputsToAdmit: [0] }) + ], + failedTopics: new Set(['Failed']), + lookupServices: { Hello: {} as LookupService, Other: {} as LookupService }, + includePropagation: true, + applied: { + firstSeenHeight: 10, + blockHeight: 11, + blockHash: 'aa'.repeat(32), + blockIndex: 2, + merkleRoot: 'bb'.repeat(32) + } + }) + expect(plan.identity.mode).toBe('live') + expect(plan.identity.contextDigest).toBe(overlayAdmissionContextDigest([9])) + expect(plan.decisions).toHaveLength(1) + expect(plan.decisions[0].expectedHistory).toEqual({ + chainEpoch: '4', + topicHistoryGeneration: '5' + }) + expect(plan.decisions[0].spends).toEqual([ + { + outpoint: { txid: previousTxid, outputIndex: asStorageUint64('0') }, + expectedVersion: '1', + spender: exampleTxid + } + ]) + expect(plan.decisions[0].evictions).toEqual([]) + expect(plan.decisions[0].edges).toHaveLength(2) + expect(plan.decisions[0].outputs).toHaveLength(2) + expect(plan.decisions[0].applied.block).toEqual({ + height: '11', + hash: 'aa'.repeat(32), + index: '2', + merkleRoot: 'bb'.repeat(32) + }) + expect(plan.outbox.some(intent => intent.target === 'Hello')).toBe(false) + expect(plan.outbox.some(intent => intent.kind === 'propagation')).toBe(true) + expect(publishAdmissionPayload).toHaveBeenCalled() + expect(getHistoryFence).toHaveBeenCalledWith('Hello') + }) + + test('historical plans omit propagation and attach merkle proof metadata', async () => { + const proven = exampleTX.inputs[0].sourceTransaction + if (proven === undefined) throw new Error('expected a proven ancestor') + const plan = await buildOverlayAdmissionPlan({ + host: host(), + tx: proven, + txid: proven.id('hex'), + beef: proven.toBEEF(), + topics: ['Hello'], + mode: 'historical', + validations: [validation()], + failedTopics: new Set(), + lookupServices, + includePropagation: true + }) + expect(plan.identity.mode).toBe('historical') + expect(plan.outbox.some(intent => intent.kind === 'propagation')).toBe(false) + expect(plan.decisions[0].applied.proof?.kind).toBe('merkle-path') + expect(plan.decisions[0].applied.firstSeenHeight).toBeDefined() + }) + + test('excludes a dupe topic from identity and decisions but keeps it, empty, in steak', async () => { + const plan = await buildOverlayAdmissionPlan({ + host: host(), + tx: exampleTX, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello', 'World'], + mode: 'live', + validations: [ + validation({ isDupe: true, outputsToAdmit: [] }), + validation({ topic: 'World' }) + ], + failedTopics: new Set(), + lookupServices, + includePropagation: false + }) + expect(plan.identity.topics.map(entry => entry.topic)).toEqual(['World']) + expect(plan.decisions.map(decision => decision.topic)).toEqual(['World']) + expect(JSON.parse(plan.steak).Hello).toEqual({ + outputsToAdmit: [], + coinsToRetain: [], + coinsRemoved: [] + }) + expect(JSON.parse(plan.steak).World.outputsToAdmit).toEqual([0]) + }) + + test('throws when every topic is a dupe or failed, since callers must not commit an empty plan', async () => { + await expect( + buildOverlayAdmissionPlan({ + host: host(), + tx: exampleTX, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello'], + mode: 'live', + validations: [validation({ isDupe: true, outputsToAdmit: [] })], + failedTopics: new Set(), + lookupServices, + includePropagation: false + }) + ).rejects.toThrow('Overlay admission plan has no topics') + }) + + test('marks unmatched previous coins stale and retains consumed coins', async () => { + const stale = await buildOverlayAdmissionPlan({ + host: host(), + tx: exampleTX, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello'], + mode: 'live', + validations: [ + validation({ + previousCoins: [0], + previousOutputs: [previousOutput], + coinsToRetain: [] + }) + ], + failedTopics: new Set(), + lookupServices, + includePropagation: false + }) + expect(stale.decisions[0].evictions).toEqual([ + { + txid: previousTxid, + outputIndex: asStorageUint64(String(exampleTX.inputs[0].sourceOutputIndex)) + } + ]) + expect(JSON.parse(stale.steak).Hello.coinsRemoved).toEqual([0]) + }) + + test('carries an explicit coinsRemoved through to a topic STEAK entry that is not re-derived', async () => { + // The top-level STEAK pass (covering every validation, including dupes + // and topics with no new admission) must reflect a caller-supplied + // coinsRemoved rather than always defaulting it to [] -- only the + // *accepted* loop re-derives coinsRemoved from outputsToMarkStale. + const plan = await buildOverlayAdmissionPlan({ + host: host(), + tx: exampleTX, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello', 'World'], + mode: 'live', + validations: [ + validation({ isDupe: true, outputsToAdmit: [], coinsRemoved: [3] }), + validation({ topic: 'World' }) + ], + failedTopics: new Set(), + lookupServices, + includePropagation: false + }) + expect(JSON.parse(plan.steak).Hello).toEqual({ + outputsToAdmit: [], + coinsToRetain: [], + coinsRemoved: [3] + }) + }) + + test('recovers a previous txid from sourceTransaction when sourceTXID is absent', async () => { + const provenSource = exampleTX.inputs[0].sourceTransaction + if (provenSource === undefined) throw new Error('expected a proven ancestor') + const tx = { + toBinary: () => exampleTX.toBinary(), + merklePath: undefined, + outputs: exampleTX.outputs, + inputs: [ + { + sourceTXID: undefined, + sourceTransaction: provenSource, + sourceOutputIndex: exampleTX.inputs[0].sourceOutputIndex + } + ] + } as unknown as Transaction + const plan = await buildOverlayAdmissionPlan({ + host: host(), + tx, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello'], + mode: 'live', + validations: [validation({ previousCoins: [0], previousOutputs: [previousOutput] })], + failedTopics: new Set(), + lookupServices, + includePropagation: false + }) + expect(plan.decisions[0].evictions).toEqual([ + { + txid: provenSource.id('hex'), + outputIndex: asStorageUint64(String(exampleTX.inputs[0].sourceOutputIndex)) + } + ]) + }) + + test('skips previous coins whose source txid cannot be recovered', async () => { + const tx = { + toBinary: () => exampleTX.toBinary(), + merklePath: undefined, + outputs: exampleTX.outputs, + inputs: [{ sourceOutputIndex: 0 }] + } as unknown as Transaction + const plan = await buildOverlayAdmissionPlan({ + host: host(), + tx, + txid: exampleTxid, + beef: exampleBeef, + topics: ['Hello'], + mode: 'live', + validations: [validation({ previousCoins: [0], previousOutputs: [previousOutput] })], + failedTopics: new Set(), + lookupServices, + includePropagation: false + }) + expect(plan.decisions[0].evictions).toEqual([]) + expect(plan.decisions[0].edges).toEqual([]) + expect(plan.outbox.some(intent => intent.kind === 'propagation')).toBe(false) + }) +}) + +describe('waitForAdmissionReceipt', () => { + const plan = { + key: { + scope: admissionScope, + operationId: 'op', + semanticDigest: 'aa'.repeat(32) + } + } as AdmissionCommit + + test('returns a committed result and retries a read-conflict rebuild', async () => { + const receipt = { + operationId: 'op', + semanticDigest: 'aa'.repeat(32), + durability: 'atomic-local' as const, + steak: '{}', + indexes: [], + propagation: 'not-requested' as const + } + const admission: AdmissionStorage = { + protocol: 'overlay-admission-v1', + commitAdmission: jest + .fn() + .mockResolvedValueOnce({ state: 'rejected', code: 'read-conflict' }) + .mockResolvedValueOnce({ state: 'committed', receipt }), + reconcileAdmission: jest.fn() + } + const rebuild = jest.fn(async () => plan) + await expect(waitForAdmissionReceipt(admission, plan, rebuild)).resolves.toEqual({ + state: 'committed', + receipt + }) + expect(rebuild).toHaveBeenCalledTimes(1) + }) + + test('throws a non-retryable rejection and a rejected reconcile', async () => { + await expect( + waitForAdmissionReceipt( + { + protocol: 'overlay-admission-v1', + commitAdmission: jest.fn(async () => ({ state: 'rejected', code: 'spend-conflict' })), + reconcileAdmission: jest.fn() + }, + plan, + async () => plan + ) + ).rejects.toThrow('Overlay admission rejected: spend-conflict') + + await expect( + waitForAdmissionReceipt( + { + protocol: 'overlay-admission-v1', + commitAdmission: jest.fn(async () => ({ state: 'pending', attemptId: 'a1' })), + reconcileAdmission: jest.fn(async () => ({ state: 'rejected', code: 'invalid-plan' })) + }, + plan, + async () => plan + ) + ).rejects.toThrow('Overlay admission rejected: invalid-plan') + }) + + test('reconciles a lost ack, rebuilds after abort, and times out while pending', async () => { + const receipt = { + operationId: 'op', + semanticDigest: 'aa'.repeat(32), + durability: 'atomic-local' as const, + steak: '{}', + indexes: [], + propagation: 'pending' as const + } + await expect( + waitForAdmissionReceipt( + { + protocol: 'overlay-admission-v1', + commitAdmission: jest.fn(async () => ({ state: 'pending', attemptId: 'lost' })), + reconcileAdmission: jest.fn(async () => ({ state: 'committed', receipt })) + }, + plan, + async () => plan + ) + ).resolves.toEqual({ state: 'committed', receipt }) + + const rebuild = jest.fn(async () => plan) + await expect( + waitForAdmissionReceipt( + { + protocol: 'overlay-admission-v1', + commitAdmission: jest + .fn() + .mockResolvedValueOnce({ state: 'pending', attemptId: 'aborted' }) + .mockResolvedValueOnce({ state: 'committed', receipt }), + reconcileAdmission: jest.fn(async () => ({ state: 'aborted' })) + }, + plan, + rebuild + ) + ).resolves.toEqual({ state: 'committed', receipt }) + expect(rebuild).toHaveBeenCalledTimes(1) + + await expect( + waitForAdmissionReceipt( + { + protocol: 'overlay-admission-v1', + commitAdmission: jest.fn(async () => ({ state: 'pending', attemptId: 'stuck' })), + reconcileAdmission: jest.fn(async () => ({ state: 'pending', attemptId: 'stuck' })) + }, + plan, + async () => plan + ) + ).rejects.toThrow('Overlay admission commit is pending') + }) +}) diff --git a/packages/overlays/overlay/src/__tests/EngineAdmissionHostBinding.test.ts b/packages/overlays/overlay/src/__tests/EngineAdmissionHostBinding.test.ts new file mode 100644 index 000000000..b7a2f9ddc --- /dev/null +++ b/packages/overlays/overlay/src/__tests/EngineAdmissionHostBinding.test.ts @@ -0,0 +1,83 @@ +import type { + AdmissionPayloadRef, + AdmissionStorage, + HistoryFence, + StorageScope +} from '../storage/AdmissionStorage' +import { getOverlayAdmissionHost } from '../EngineAdmission' + +// Storage adapters are classes whose optional host methods read instance state. +// The host object must call them on the adapter, not as detached functions. +class InstanceStateStorage { + public readonly admissionScope: StorageScope = { + network: 'testnet', + genesisHash: '11'.repeat(32), + nodeId: 'binding-node' + } + + public readonly admission: AdmissionStorage + private readonly published: string[] = [] + private readonly targets = ['ls_example'] + private readonly fence: HistoryFence = { chainEpoch: '7', topicHistoryGeneration: '3' } + + constructor(admission: AdmissionStorage) { + this.admission = admission + } + + async publishAdmissionPayload(input: { + kind: AdmissionPayloadRef['kind'] + bytes: Uint8Array + }): Promise { + this.published.push(input.kind) + return { kind: input.kind, digest: 'ab'.repeat(32), byteLength: String(input.bytes.byteLength) } + } + + enlistedIndexTargets(): readonly string[] { + return this.targets + } + + async getHistoryFence(): Promise { + return this.fence + } + + publishedKinds(): readonly string[] { + return this.published + } +} + +const admission = { + protocol: 'overlay-admission-v1', + commitAdmission: jest.fn(), + reconcileAdmission: jest.fn() +} as unknown as AdmissionStorage + +describe('getOverlayAdmissionHost', () => { + it('invokes optional host methods on the storage instance', async () => { + const storage = new InstanceStateStorage(admission) + const host = getOverlayAdmissionHost(storage) + expect(host).toBeDefined() + + const ref = await host?.publishAdmissionPayload?.({ + kind: 'raw-tx', + bytes: Uint8Array.of(1, 2, 3) + }) + expect(ref).toEqual({ kind: 'raw-tx', digest: 'ab'.repeat(32), byteLength: '3' }) + expect(storage.publishedKinds()).toEqual(['raw-tx']) + expect(host?.enlistedIndexTargets?.()).toEqual(['ls_example']) + await expect(host?.getHistoryFence?.('tm_example')).resolves.toEqual({ + chainEpoch: '7', + topicHistoryGeneration: '3' + }) + }) + + it('leaves host methods undefined when the storage does not provide them', () => { + const host = getOverlayAdmissionHost({ + admission, + admissionScope: new InstanceStateStorage(admission).admissionScope + }) + expect(host).toBeDefined() + expect(host?.publishAdmissionPayload).toBeUndefined() + expect(host?.enlistedIndexTargets).toBeUndefined() + expect(host?.getHistoryFence).toBeUndefined() + }) +}) diff --git a/packages/overlays/overlay/src/__tests/StorageContract.test.ts b/packages/overlays/overlay/src/__tests/StorageContract.test.ts index e613d7315..f8f5d38d3 100644 --- a/packages/overlays/overlay/src/__tests/StorageContract.test.ts +++ b/packages/overlays/overlay/src/__tests/StorageContract.test.ts @@ -4,6 +4,7 @@ import type { Knex } from 'knex' import { KnexStorage } from '../storage/knex/KnexStorage.js' import { admissionSemanticDigest, + asStorageUint64, getAdmissionStorage, isReplaySafeProjection, parseStorageOutputIndex, @@ -93,6 +94,13 @@ describe('S01 portable persistence contract', () => { expect(() => parseStorageUint64(1 as unknown as string)).toThrow() }) + test('asStorageUint64 returns the canonical decimal after validation', () => { + expect(asStorageUint64('0')).toBe('0') + expect(asStorageUint64('18446744073709551615')).toBe('18446744073709551615') + expect(() => asStorageUint64('01')).toThrow('Invalid storage uint64') + expect(() => asStorageUint64('-1')).toThrow('Invalid storage uint64') + }) + test.each(fixture.outputIndices)('exact output index: "$value"', ({ value, valid }) => { if (valid) expect(String(parseStorageOutputIndex(value))).toBe(value) else expect(() => parseStorageOutputIndex(value)).toThrow() diff --git a/packages/overlays/overlay/src/__tests/mongo/EngineAdmissionDupeRetry.mongo.test.ts b/packages/overlays/overlay/src/__tests/mongo/EngineAdmissionDupeRetry.mongo.test.ts new file mode 100644 index 000000000..b3b28038b --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/EngineAdmissionDupeRetry.mongo.test.ts @@ -0,0 +1,102 @@ +import { Transaction, type AdmittanceInstructions } from '@bsv/sdk' +import { Engine } from '../../Engine.js' +import type { LookupService } from '../../LookupService.js' +import type { TopicManager } from '../../TopicManager.js' +import { MongoOverlayStorage } from '../../storage/mongo/MongoOverlayStorage.js' +import { bootstrapMongoOverlay } from '../../storage/mongo/MongoSchema.js' +import { createMongoReplicaFixture, type MongoReplicaFixture } from './MongoReplicaFixture.js' + +// Same fixture BEEF used throughout the admission test suite: a single, +// unconfirmed transaction whose one input spends a proven ancestor output. +const BRC62Hex = + '0100beef01fe636d0c0007021400fe507c0c7aa754cef1f7889d5fd395cf1f785dd7de98eed895dbedfe4e5bc70d1502ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e010b00bc4ff395efd11719b277694cface5aa50d085a0bb81f613f70313acd28cf4557010400574b2d9142b8d28b61d88e3b2c3f44d858411356b49a28a4643b6d1a6a092a5201030051a05fc84d531b5d250c23f4f886f6812f9fe3f402d61607f977b4ecd2701c19010000fd781529d58fc2523cf396a7f25440b409857e7e221766c57214b1d38c7b481f01010062f542f45ea3660f86c013ced80534cb5fd4c19d66c56e7e8c5d4bf2d40acc5e010100b121e91836fd7cd5102b654e9f72f3cf6fdbfd0b161c53a9c54b12c841126331020100000001cd4e4cac3c7b56920d1e7655e7e260d31f29d9a388d04910f1bbd72304a79029010000006b483045022100e75279a205a547c445719420aa3138bf14743e3f42618e5f86a19bde14bb95f7022064777d34776b05d816daf1699493fcdf2ef5a5ab1ad710d9c97bfb5b8f7cef3641210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013e660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000001000100000001ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e000000006a47304402203a61a2e931612b4bda08d541cfb980885173b8dcf64a3471238ae7abcd368d6402204cbf24f04b9aa2256d8901f0ed97866603d2be8324c2bfb7a37bf8fc90edd5b441210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013c660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000000' + +const mockChainTracker = { + isValidRootForHeight: async () => true, + currentHeight: async () => 800000 +} + +function admittingManager(): TopicManager { + return { + identifyAdmissibleOutputs: async (): Promise => ({ + outputsToAdmit: [0], + coinsToRetain: [] + }), + getDocumentation: async () => 'docs', + getMetaData: async () => ({ name: 'Mock', shortDescription: 'Mock' }) + } +} + +function noopLookup(): LookupService { + return { + outputAdmittedByTopic: async () => {}, + outputSpent: async () => {}, + lookup: async () => [], + outputNoLongerRetainedInHistory: async () => {}, + outputEvicted: async () => {}, + admissionMode: 'locking-script', + spendNotificationMode: 'none', + getDocumentation: async () => 'docs', + getMetaData: async () => ({ name: 'Mock', shortDescription: 'Mock' }) + } +} + +describe('Engine admission retries against real MongoAdmissionStorage', () => { + let fixture: MongoReplicaFixture + + beforeAll(async () => { + fixture = await createMongoReplicaFixture() + await bootstrapMongoOverlay(fixture.db, fixture.scope) + }, 120000) + + afterAll(async () => { + await fixture.close() + }, 60000) + + test( + 'a multi-topic resubmit that overlaps an already-applied topic commits only the new topic instead of rejecting invalid-plan', + async () => { + const storage = new MongoOverlayStorage(fixture.db, fixture.scope) + // getOverlayAdmissionHost() lifts these off the storage instance into a + // plain object, so they must be pre-bound here or they lose `this` + // when EngineAdmission invokes them later — unrelated to the F1 fix + // under test, this is only how MongoOverlayStorage's admission host is + // wired up for direct use outside of a DI container. + storage.publishAdmissionPayload = storage.publishAdmissionPayload.bind(storage) + storage.enlistedIndexTargets = storage.enlistedIndexTargets.bind(storage) + storage.getHistoryFence = storage.getHistoryFence.bind(storage) + const engine = new Engine( + { TopicA: admittingManager(), TopicC: admittingManager() }, + { TopicA: noopLookup(), TopicC: noopLookup() }, + storage, + mockChainTracker + ) + + const tx = Transaction.fromHexBEEF(BRC62Hex) + const beef = tx.toBEEF() + + // Live admit under a single topic, TopicA. + const first = await engine.submit( + { beef, topics: ['TopicA'] }, + undefined, + 'historical-tx-no-spv' + ) + expect(first.TopicA.outputsToAdmit).toEqual([0]) + + // A later resubmission of the same tx names TopicA again (now a dupe) + // alongside a brand-new topic, TopicC — the "multi-topic retry" shape + // the finding describes. This must not throw + // "Overlay admission rejected: invalid-plan" / "digest-mismatch". + const second = await engine.submit( + { beef, topics: ['TopicA', 'TopicC'] }, + undefined, + 'historical-tx-no-spv' + ) + expect(second.TopicC.outputsToAdmit).toEqual([0]) + expect(second.TopicA).toEqual({ outputsToAdmit: [], coinsToRetain: [], coinsRemoved: [] }) + + await storage.close() + }, + 30000 + ) +}) diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionHarness.ts b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionHarness.ts new file mode 100644 index 000000000..953ae241f --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionHarness.ts @@ -0,0 +1,479 @@ +import { randomUUID } from 'node:crypto' +import { Binary } from 'mongodb' +import { + type AdmissionCommitResult, + type AdmissionOperationKey, + type AdmissionOutpoint, + type AdmissionPayloadRef, + type AdmissionReconcileResult, + type AdmissionStorage, + type StorageScope +} from '../../storage/AdmissionStorage.js' +import type { RecoveryLease } from '../../storage/RecoveryContract.js' +import type { + AdmissionStorageContractHarness, + AdmissionStorageFaults, + AdmissionStorageSeeds, + AdmissionStorageTestSnapshot +} from '../admission/ReferenceAdmissionStorage.js' +import { MongoAdmissionStorage } from '../../storage/mongo/MongoAdmissionStorage.js' +import { MongoReadGuards } from '../../storage/mongo/MongoReadGuards.js' +import { + MongoCollectionNames, + decodeMongoUint64, + encodeMongoOutputIndex, + encodeMongoUint64, + mongoChainKey, + mongoNodeKey, + mongoRecordKey +} from '../../storage/mongo/MongoSchema.js' +import type { MongoReplicaFixture } from './MongoReplicaFixture.js' +import { referenceScope } from '../admission/ReferenceAdmissionStorage.js' + +const clone = (value: T): T => structuredClone(value) +const scopeKey = (scope: StorageScope): string => + JSON.stringify([scope.network, scope.genesisHash, scope.nodeId]) +const operationKey = (key: AdmissionOperationKey): string => + JSON.stringify([scopeKey(key.scope), key.operationId]) +const outputKey = (scope: StorageScope, topic: string, outpoint: AdmissionOutpoint): string => + JSON.stringify([scopeKey(scope), topic, outpoint.txid, outpoint.outputIndex]) +const appliedKey = (scope: StorageScope, topic: string, txid: string): string => + JSON.stringify([scope.network, scope.genesisHash, scope.nodeId, topic, txid]) +const edgeKey = ( + scope: StorageScope, + topic: string, + source: AdmissionOutpoint, + consumer: AdmissionOutpoint +): string => + JSON.stringify([scope.network, scope.genesisHash, scope.nodeId, topic, source, consumer]) + +class FaultAdmission implements AdmissionStorage { + readonly protocol = 'overlay-admission-v1' as const + inner: AdmissionStorage + loseReply = false + abortBefore = false + unresolved = new Map() + + constructor(inner: AdmissionStorage) { + this.inner = inner + } + + async commitAdmission( + plan: Parameters[0] + ): Promise { + const key = operationKey(plan.key) + const active = this.unresolved.get(key) + if (active !== undefined) return { state: 'pending', attemptId: active.attemptId } + if (this.abortBefore) { + this.abortBefore = false + const attemptId = randomUUID() + this.unresolved.set(key, { attemptId, aborted: true }) + return { state: 'pending', attemptId } + } + const result = await this.inner.commitAdmission(plan) + if (this.loseReply && result.state === 'committed') { + this.loseReply = false + const attemptId = randomUUID() + this.unresolved.set(key, { attemptId, aborted: false }) + return { state: 'pending', attemptId } + } + return result + } + + async reconcileAdmission( + key: AdmissionOperationKey, + attemptId?: string + ): Promise { + const record = this.unresolved.get(operationKey(key)) + if (record !== undefined) { + if (attemptId !== undefined && attemptId !== record.attemptId) { + return { state: 'pending', attemptId: record.attemptId } + } + this.unresolved.delete(operationKey(key)) + if (record.aborted) return { state: 'aborted' } + return await this.inner.reconcileAdmission(key, attemptId) + } + return await this.inner.reconcileAdmission(key, attemptId) + } +} + +export class MongoAdmissionHarness implements AdmissionStorageContractHarness { + private inner: MongoAdmissionStorage + private readonly wrapper: FaultAdmission + private readonly leases = new Map() + private readonly seededPayloads = new Set() + readonly seed: AdmissionStorageSeeds + readonly faults: AdmissionStorageFaults + + constructor(private readonly fixture: MongoReplicaFixture) { + this.inner = new MongoAdmissionStorage(fixture.db, referenceScope) + this.wrapper = new FaultAdmission(this.inner) + this.seed = { + readyPayload: async ref => { + this.seededPayloads.add(`${ref.kind}:${ref.digest}`) + await this.insertReadyPayload(ref) + }, + read: async (scope, _topic, key, version) => { + const guards = new MongoReadGuards(this.fixture.db) + await guards.initialize(scope, key) + await this.fixture.db + .collection(MongoCollectionNames.readGuards) + .updateOne( + { _id: mongoRecordKey(mongoNodeKey(scope), 'read-guard', key) }, + { $set: { version, updatedAt: new Date() } } + ) + }, + spendable: async (scope, topic, outpoint, version) => { + await this.insertSpendable(scope, topic, outpoint, version) + }, + history: async (scope, topic, fence) => { + await this.insertHistory(scope, topic, fence) + }, + lease: async lease => { + this.leases.set(`${lease.topic}:${lease.peerId}:${lease.jobId}`, clone(lease)) + await this.insertLease(lease) + }, + now: () => undefined + } + this.faults = { + loseReplyAfterCommitOnce: () => { + this.wrapper.loseReply = true + }, + abortBeforeCommitOnce: () => { + this.wrapper.abortBefore = true + } + } + } + + get storage(): { admission: AdmissionStorage } { + return { admission: this.wrapper } + } + + restart(): AdmissionStorage { + this.inner = new MongoAdmissionStorage(this.fixture.db, referenceScope) + this.wrapper.inner = this.inner + return this.wrapper + } + + async reset(): Promise { + this.wrapper.loseReply = false + this.wrapper.abortBefore = false + this.wrapper.unresolved.clear() + this.leases.clear() + this.seededPayloads.clear() + await this.inner.close().catch(() => undefined) + for (const name of Object.values(MongoCollectionNames)) { + if (name === MongoCollectionNames.schema) continue + await this.fixture.db.collection(name).deleteMany({}) + } + await this.fixture.db.collection('enlisted_index').deleteMany({}) + this.inner = new MongoAdmissionStorage(this.fixture.db, referenceScope) + this.wrapper.inner = this.inner + } + + async snapshot(): Promise { + const payloads = await this.fixture.db + .collection(MongoCollectionNames.payloads) + .find() + .toArray() + const refs = await this.fixture.db + .collection(MongoCollectionNames.payloadReferences) + .find() + .toArray() + const pinned = new Set(refs.map(item => item.payloadId as string)) + const reads = await this.fixture.db.collection(MongoCollectionNames.readGuards).find().toArray() + const fences = await this.fixture.db + .collection(MongoCollectionNames.topicGenerations) + .find() + .toArray() + const jobs = await this.fixture.db + .collection(MongoCollectionNames.basmRecoveryJobs) + .find() + .toArray() + const outputs = await this.fixture.db.collection(MongoCollectionNames.outputs).find().toArray() + const edges = await this.fixture.db + .collection(MongoCollectionNames.consumptionEdges) + .find() + .toArray() + const applied = await this.fixture.db + .collection(MongoCollectionNames.appliedTransactions) + .find() + .toArray() + const lookup = await this.fixture.db + .collection(MongoCollectionNames.lookupOutbox) + .find() + .toArray() + const propagation = await this.fixture.db + .collection(MongoCollectionNames.propagationOutbox) + .find() + .toArray() + const historyRefs = refs.filter(item => String(item.slot).startsWith('history-update')) + const historyUpdates = [] + for (const reference of historyRefs) { + const payload = payloads.find(item => item._id === reference.payloadId) + const bytes = + payload?.inlineData instanceof Binary + ? Buffer.from(payload.inlineData.buffer) + : Buffer.alloc(0) + const parsed = JSON.parse(bytes.toString('utf8')) as { + affectedFromHeight: string + checkpoint?: string + } + historyUpdates.push({ topic: reference.ownerId as string, ...parsed }) + } + return { + payloads: payloads + .filter( + item => + this.seededPayloads.has(`${item.kind}:${item.digest}`) || pinned.has(item._id as string) + ) + .map(item => ({ + ref: { + kind: item.kind, + digest: item.digest, + byteLength: decodeMongoUint64(item.byteLength as string) + } as AdmissionPayloadRef, + pinned: pinned.has(item._id as string) + })), + reads: reads + .filter(item => item.version !== null) + .map(item => ({ + key: JSON.stringify([scopeKey(this.scopeOf(item)), '', item.key]), + version: item.version as string + })), + fences: fences.map(item => ({ + topic: item.topic as string, + fence: { + chainEpoch: decodeMongoUint64(item.chainEpoch as string), + topicHistoryGeneration: decodeMongoUint64(item.topicHistoryGeneration as string) + } + })), + leases: jobs.map(job => { + const seeded = this.leases.get(`${job.topic}:${job.peerId}:${job.jobId}`) + const base = seeded ?? { + scope: this.scopeOf(job), + topic: job.topic as string, + peerId: job.peerId as string, + jobId: job.jobId as string, + leaseToken: decodeMongoUint64(job.leaseToken as string), + expiresAtMs: String((job.leaseUntil as Date).getTime()), + chainEpoch: decodeMongoUint64(job.chainEpoch as string), + topicHistoryGeneration: decodeMongoUint64(job.topicHistoryGeneration as string) + } + return { + ...base, + topicHistoryGeneration: decodeMongoUint64(job.topicHistoryGeneration as string) + } + }), + outputs: outputs.map(item => { + const outpoint = { txid: item.txid as string, outputIndex: item.outputIndex as string } + const topic = item.topic as string + const scope = this.scopeOf(item) + return { + key: outputKey(scope, topic, outpoint), + version: item.version as string, + ...(item.spender === undefined ? {} : { spentBy: item.spender as string }), + topic, + ...(item.state === 'unspent' + ? { + output: { + txid: item.txid as string, + outputIndex: item.outputIndex as string, + satoshis: decodeMongoUint64(item.satoshis as string), + score: decodeMongoUint64(item.score as string), + script: { + payload: { + digest: '00'.repeat(32), + byteLength: '0', + kind: 'locking-script' as const + }, + offset: decodeMongoUint64(item.scriptOffset as string), + byteLength: decodeMongoUint64(item.scriptByteLength as string) + } + } + } + : {}) + } + }), + edges: edges + .map(item => + edgeKey( + this.scopeOf(item), + item.topic as string, + { txid: item.sourceTxid as string, outputIndex: item.sourceOutputIndex as string }, + { txid: item.consumerTxid as string, outputIndex: item.consumerOutputIndex as string } + ) + ) + .sort(), + applied: applied.map(item => ({ + key: appliedKey(this.scopeOf(item), item.topic as string, item.txid as string), + record: { txid: item.txid as string } + })), + outbox: [ + ...lookup.map(item => ({ scope: this.scopeOf(item), eventId: item.eventId as string })), + ...propagation.map(item => ({ scope: this.scopeOf(item), eventId: item.eventId as string })) + ], + handoffs: jobs + .filter(item => typeof item.checkpoint === 'string' && item.checkpoint.length > 0) + .map(item => ({ topic: item.topic as string, checkpoint: item.checkpoint as string })), + historyUpdates, + operations: [] + } + } + + get adapter(): MongoAdmissionStorage { + return this.inner + } + + private scopeOf(document: { + network?: unknown + genesisHash?: unknown + nodeId?: unknown + }): StorageScope { + return { + network: document.network as string, + genesisHash: document.genesisHash as string, + nodeId: document.nodeId as string + } + } + + private async insertReadyPayload(ref: AdmissionPayloadRef): Promise { + const now = new Date() + const id = mongoRecordKey(mongoChainKey(this.fixture.scope), ref.kind, ref.digest) + try { + await this.fixture.db.collection(MongoCollectionNames.payloads).insertOne({ + _id: id, + schemaVersion: 1, + network: this.fixture.scope.network, + genesisHash: this.fixture.scope.genesisHash, + kind: ref.kind, + digest: ref.digest, + byteLength: encodeMongoUint64(ref.byteLength), + state: 'ready', + guard: randomUUID(), + ownerNodeId: this.fixture.scope.nodeId, + ownerId: randomUUID(), + fencingToken: encodeMongoUint64('1'), + leaseUntil: now, + createdAt: now, + updatedAt: now + }) + } catch (error) { + if (typeof error === 'object' && error !== null && 'code' in error && error.code === 11000) + return + throw error + } + } + + private async insertSpendable( + scope: StorageScope, + topic: string, + outpoint: AdmissionOutpoint, + version: string + ): Promise { + const now = new Date() + const id = mongoRecordKey( + mongoNodeKey(scope), + 'output', + topic, + outpoint.txid, + encodeMongoOutputIndex(outpoint.outputIndex) + ) + try { + await this.fixture.db.collection(MongoCollectionNames.outputs).insertOne({ + _id: id, + schemaVersion: 1, + network: scope.network, + genesisHash: scope.genesisHash, + nodeId: scope.nodeId, + topic, + txid: outpoint.txid, + outputIndex: encodeMongoOutputIndex(outpoint.outputIndex), + satoshis: encodeMongoUint64('1'), + score: encodeMongoUint64('1'), + scriptPayloadId: mongoRecordKey(mongoChainKey(scope), 'locking-script', 'bb'.repeat(32)), + scriptOffset: encodeMongoUint64('0'), + scriptByteLength: encodeMongoUint64('1'), + state: 'unspent', + version, + createdAt: now, + updatedAt: now + }) + } catch (error) { + if (typeof error === 'object' && error !== null && 'code' in error && error.code === 11000) { + await this.fixture.db + .collection(MongoCollectionNames.outputs) + .updateOne({ _id: id }, { $set: { version, state: 'unspent', updatedAt: now } }) + return + } + throw error + } + } + + private async insertHistory( + scope: StorageScope, + topic: string, + fence: { chainEpoch: string; topicHistoryGeneration: string } + ): Promise { + const now = new Date() + const id = mongoRecordKey(mongoNodeKey(scope), 'generation', topic) + await this.fixture.db.collection(MongoCollectionNames.topicGenerations).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion: 1, + network: scope.network, + genesisHash: scope.genesisHash, + nodeId: scope.nodeId, + topic, + policyId: 'policy-1', + createdAt: now + }, + $set: { + chainEpoch: encodeMongoUint64(fence.chainEpoch), + topicHistoryGeneration: encodeMongoUint64(fence.topicHistoryGeneration), + updatedAt: now + } + }, + { upsert: true } + ) + } + + private async insertLease(lease: RecoveryLease): Promise { + const now = new Date() + const id = mongoRecordKey( + mongoNodeKey(lease.scope), + 'job', + lease.topic, + lease.peerId, + lease.jobId + ) + await this.fixture.db.collection(MongoCollectionNames.basmRecoveryJobs).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion: 1, + network: lease.scope.network, + genesisHash: lease.scope.genesisHash, + nodeId: lease.scope.nodeId, + topic: lease.topic, + peerId: lease.peerId, + jobId: lease.jobId, + createdAt: now + }, + $set: { + chainEpoch: encodeMongoUint64(lease.chainEpoch), + topicHistoryGeneration: encodeMongoUint64(lease.topicHistoryGeneration), + leaseToken: encodeMongoUint64(lease.leaseToken), + leaseUntil: new Date(Date.now() + 60 * 60 * 1000), + state: 'leased', + checkpoint: lease.jobId, + updatedAt: now + } + }, + { upsert: true } + ) + } +} diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionPlan.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionPlan.test.ts new file mode 100644 index 000000000..03fec866d --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionPlan.test.ts @@ -0,0 +1,322 @@ +import { + admissionSemanticDigest, + type AdmissionCommit, + type AdmissionPayloadRef +} from '../../storage/AdmissionStorage.js' +import { + AdmissionRejectedError, + admissionPlanPayloads, + admissionReceiptFor, + isBoundSteak, + lookupOutboxIntents, + propagationOutboxIntents, + rejectAdmission, + samePayload, + sameScope, + validateAdmissionPlan +} from '../../storage/mongo/MongoAdmissionPlan.js' +import { admissionPlan } from '../admission/AdmissionStorageContract.js' +import { referenceScope } from '../admission/ReferenceAdmissionStorage.js' + +const clone = (value: T): T => structuredClone(value) +const hash = (character: string): string => character.repeat(64) + +function bind(plan: AdmissionCommit): AdmissionCommit { + plan.key.semanticDigest = admissionSemanticDigest(plan.identity) + return plan +} + +describe('Mongo admission plan validation', () => { + test('accepts the contract plan and classifies outbox intents', () => { + const plan = admissionPlan() + expect(validateAdmissionPlan(plan)).toBeUndefined() + expect(lookupOutboxIntents(plan).map(intent => intent.target)).toEqual(['ls_contract']) + expect(propagationOutboxIntents(plan).map(intent => intent.kind)).toEqual(['propagation']) + expect(admissionPlanPayloads(plan).length).toBeGreaterThan(plan.payloads.length) + }) + + test('sameScope and samePayload compare every identity field', () => { + expect(sameScope(referenceScope, { ...referenceScope })).toBe(true) + expect(sameScope(referenceScope, { ...referenceScope, nodeId: 'other' })).toBe(false) + const payload: AdmissionPayloadRef = { + digest: hash('a'), + byteLength: '100', + kind: 'raw-transaction' + } + expect(samePayload(payload, payload)).toBe(true) + expect(samePayload(undefined, payload)).toBe(false) + expect(samePayload({ ...payload, kind: 'locking-script' }, payload)).toBe(false) + expect(samePayload({ ...payload, byteLength: '99' }, payload)).toBe(false) + }) + + test('rejects a digest that does not bind the identity or scope', () => { + const mismatched = clone(admissionPlan()) + mismatched.key.semanticDigest = hash('0') + expect(validateAdmissionPlan(mismatched)).toBe('digest-mismatch') + + const scoped = clone(admissionPlan()) + scoped.key.scope = { ...referenceScope, nodeId: 'other-node' } + expect(validateAdmissionPlan(scoped)).toBe('digest-mismatch') + + const invalid = clone(admissionPlan()) + invalid.identity.txid = 'not-a-hash' + expect(validateAdmissionPlan(invalid)).toBe('digest-mismatch') + }) + + test('rejects historical propagation and inconsistent topic sets', () => { + const historical = bind(clone(admissionPlan('historical-operation'))) + historical.identity.mode = 'historical' + bind(historical) + expect(validateAdmissionPlan(historical)).toBe('invalid-plan') + + const missingDecision = bind(clone(admissionPlan('missing-decision'))) + missingDecision.identity.topics.push({ topic: 'tm_other', policyId: 'policy-2' }) + bind(missingDecision) + expect(validateAdmissionPlan(missingDecision)).toBe('invalid-plan') + + const unknownTopic = clone(admissionPlan('unknown-topic')) + unknownTopic.decisions[0].topic = 'tm_other' + expect(validateAdmissionPlan(unknownTopic)).toBe('invalid-plan') + + const duplicateEvents = clone(admissionPlan('duplicate-events')) + duplicateEvents.outbox[1].eventId = duplicateEvents.outbox[0].eventId + expect(validateAdmissionPlan(duplicateEvents)).toBe('invalid-plan') + }) + + test('rejects payloads that are not ready for reference', () => { + const badDigest = clone(admissionPlan('payload-digest')) + badDigest.payloads[0] = { ...badDigest.payloads[0], digest: 'zz' } + expect(validateAdmissionPlan(badDigest)).toBe('payload-not-ready') + + const badLength = clone(admissionPlan('payload-length')) + badLength.payloads[0] = { ...badLength.payloads[0], byteLength: '01' } + expect(validateAdmissionPlan(badLength)).toBe('payload-not-ready') + }) + + test('rejects STEAK that is not bound to the admitted outputs', () => { + const notJson = clone(admissionPlan('steak-json')) + notJson.steak = '{' + expect(validateAdmissionPlan(notJson)).toBe('invalid-plan') + + const illFormed = clone(admissionPlan('steak-ill')) + illFormed.steak = '\ud800' + expect(validateAdmissionPlan(illFormed)).toBe('invalid-plan') + + const arraySteak = clone(admissionPlan('steak-array')) + arraySteak.steak = '[]' + expect(validateAdmissionPlan(arraySteak)).toBe('invalid-plan') + + const scalar = clone(admissionPlan('steak-scalar')) + scalar.steak = '1' + expect(validateAdmissionPlan(scalar)).toBe('invalid-plan') + + const retainType = clone(admissionPlan('steak-retain')) + retainType.steak = JSON.stringify({ + tm_contract: { outputsToAdmit: [0], coinsToRetain: 'nope' } + }) + expect(validateAdmissionPlan(retainType)).toBe('invalid-plan') + + const removedType = clone(admissionPlan('steak-removed')) + removedType.steak = JSON.stringify({ + tm_contract: { outputsToAdmit: [0], coinsRemoved: 1 } + }) + expect(validateAdmissionPlan(removedType)).toBe('invalid-plan') + + const fractional = clone(admissionPlan('steak-fraction')) + fractional.steak = JSON.stringify({ tm_contract: { outputsToAdmit: [0.5] } }) + expect(validateAdmissionPlan(fractional)).toBe('invalid-plan') + + const negative = clone(admissionPlan('steak-negative')) + negative.steak = JSON.stringify({ tm_contract: { outputsToAdmit: [-1] } }) + expect(validateAdmissionPlan(negative)).toBe('invalid-plan') + + const overflow = clone(admissionPlan('steak-overflow')) + overflow.steak = JSON.stringify({ tm_contract: { outputsToAdmit: [4294967296] } }) + expect(validateAdmissionPlan(overflow)).toBe('invalid-plan') + + const mismatch = clone(admissionPlan('steak-mismatch')) + mismatch.steak = JSON.stringify({ + tm_contract: { outputsToAdmit: [7], coinsToRetain: [], coinsRemoved: [] } + }) + expect(validateAdmissionPlan(mismatch)).toBe('invalid-plan') + + const notString = clone(admissionPlan('steak-type')) + ;(notString as { steak: unknown }).steak = { tm_contract: { outputsToAdmit: [0] } } + expect(validateAdmissionPlan(notString)).toBe('invalid-plan') + }) + + test('rejects malformed outbox intents', () => { + const emptyId = clone(admissionPlan('outbox-empty-id')) + emptyId.outbox[0].eventId = '' + expect(validateAdmissionPlan(emptyId)).toBe('invalid-plan') + + const illId = clone(admissionPlan('outbox-ill-id')) + illId.outbox[0].eventId = '\ud800' + expect(validateAdmissionPlan(illId)).toBe('invalid-plan') + + const emptyTarget = clone(admissionPlan('outbox-empty-target')) + emptyTarget.outbox[0].target = '' + expect(validateAdmissionPlan(emptyTarget)).toBe('invalid-plan') + + const illTarget = clone(admissionPlan('outbox-ill-target')) + illTarget.outbox[0].target = '\ud800' + expect(validateAdmissionPlan(illTarget)).toBe('invalid-plan') + + const kind = clone(admissionPlan('outbox-kind')) + ;(kind.outbox[0] as { kind: string }).kind = 'index' + expect(validateAdmissionPlan(kind)).toBe('invalid-plan') + }) + + test('rejects a decision whose expected history fence is not canonical uint64', () => { + const badEpoch = clone(admissionPlan('history-epoch-format')) + badEpoch.decisions[0].expectedHistory.chainEpoch = 'not-a-number' + expect(validateAdmissionPlan(badEpoch)).toBe('invalid-plan') + + const badGeneration = clone(admissionPlan('history-generation-format')) + badGeneration.decisions[0].expectedHistory.topicHistoryGeneration = '01' + expect(validateAdmissionPlan(badGeneration)).toBe('invalid-plan') + }) + + test('rejects malformed spends, edges, outputs, and applied history', () => { + const outputIndex = clone(admissionPlan('output-index')) + outputIndex.decisions[0].spends[0].outpoint.outputIndex = '4294967296' + expect(validateAdmissionPlan(outputIndex)).toBe('invalid-plan') + + const spendTxid = clone(admissionPlan('spend-txid')) + spendTxid.decisions[0].spends[0].outpoint.txid = 'nope' + expect(validateAdmissionPlan(spendTxid)).toBe('invalid-plan') + + const spender = clone(admissionPlan('spend-spender')) + spender.decisions[0].spends[0].spender = hash('1') + expect(validateAdmissionPlan(spender)).toBe('invalid-plan') + + const eviction = clone(admissionPlan('eviction')) + eviction.decisions[0].evictions = [{ txid: 'nope', outputIndex: '0' }] + expect(validateAdmissionPlan(eviction)).toBe('invalid-plan') + + const edge = clone(admissionPlan('edge')) + edge.decisions[0].edges[0].source.txid = 'nope' + expect(validateAdmissionPlan(edge)).toBe('invalid-plan') + + const outputTxid = clone(admissionPlan('output-txid')) + outputTxid.decisions[0].outputs[0].txid = hash('7') + expect(validateAdmissionPlan(outputTxid)).toBe('invalid-plan') + + const satoshis = clone(admissionPlan('satoshis')) + satoshis.decisions[0].outputs[0].satoshis = '01' + expect(validateAdmissionPlan(satoshis)).toBe('invalid-plan') + + const scriptRange = clone(admissionPlan('script-range')) + scriptRange.decisions[0].outputs[0].script.offset = '20' + scriptRange.decisions[0].outputs[0].script.byteLength = '10' + expect(validateAdmissionPlan(scriptRange)).toBe('invalid-plan') + + const appliedTxid = clone(admissionPlan('applied-txid')) + appliedTxid.decisions[0].applied.txid = hash('7') + expect(validateAdmissionPlan(appliedTxid)).toBe('invalid-plan') + + const firstSeen = clone(admissionPlan('first-seen')) + firstSeen.decisions[0].applied.firstSeenHeight = '01' + expect(validateAdmissionPlan(firstSeen)).toBe('invalid-plan') + + const proof = clone(admissionPlan('applied-proof')) + proof.decisions[0].applied.proof = { digest: 'zz', byteLength: '1', kind: 'merkle-path' } + expect(validateAdmissionPlan(proof)).toBe('payload-not-ready') + + const block = clone(admissionPlan('applied-block')) + block.decisions[0].applied.block = { + height: '1', + hash: 'not-a-hash', + index: '0', + merkleRoot: hash('4') + } + expect(validateAdmissionPlan(block)).toBe('invalid-plan') + + const history = clone(admissionPlan('history')) + history.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '3', + affectedFromHeight: '99' + } + expect(validateAdmissionPlan(history)).toBe('invalid-plan') + + const historyType = clone(admissionPlan('history-type')) + historyType.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '4', + affectedFromHeight: '01' + } + expect(validateAdmissionPlan(historyType)).toBe('invalid-plan') + }) + + test('receipts mark enlisted indexes visible and reject overlapping lookup targets', () => { + const plan = admissionPlan() + expect(admissionReceiptFor(plan, [])).toEqual({ + operationId: plan.key.operationId, + semanticDigest: plan.key.semanticDigest, + durability: 'atomic-local', + steak: plan.steak, + indexes: [{ target: 'ls_contract', state: 'pending' }], + propagation: 'pending' + }) + expect(admissionReceiptFor(plan, ['ls_enlisted']).indexes).toEqual([ + { target: 'ls_enlisted', state: 'visible' }, + { target: 'ls_contract', state: 'pending' } + ]) + expect(() => admissionReceiptFor(plan, ['ls_contract'])).toThrow(AdmissionRejectedError) + try { + admissionReceiptFor(plan, ['ls_contract']) + } catch (error) { + expect(error).toMatchObject({ name: 'AdmissionRejectedError', code: 'invalid-plan' }) + } + + const noPropagation = clone(plan) + noPropagation.outbox = noPropagation.outbox.filter(intent => intent.kind !== 'propagation') + expect(admissionReceiptFor(noPropagation, []).propagation).toBe('not-requested') + }) + + test('rejectAdmission throws a coded admission error', () => { + expect(() => rejectAdmission('digest-mismatch')).toThrow(AdmissionRejectedError) + expect(() => rejectAdmission('spend-conflict')).toThrow('spend-conflict') + }) + + test('isBoundSteak rejects a non-object entry and a topic missing from the STEAK record', () => { + const plan = admissionPlan('steak-entry-shape') + + // A STEAK record whose value for a real topic key is not an + // outputsToAdmit-shaped object at all (here a bare number) must fail + // isSteakEntry's own `typeof`/array guard rather than being coerced. + const scalarEntry = clone(plan) + scalarEntry.steak = JSON.stringify({ tm_contract: 5 }) + expect(isBoundSteak(scalarEntry)).toBe(false) + expect(validateAdmissionPlan(scalarEntry)).toBe('invalid-plan') + + // An object entry that is missing outputsToAdmit entirely (as opposed to + // having the wrong type for coinsToRetain/coinsRemoved) must also fail. + const missingOutputsToAdmit = clone(plan) + missingOutputsToAdmit.steak = JSON.stringify({ tm_contract: {} }) + expect(isBoundSteak(missingOutputsToAdmit)).toBe(false) + expect(validateAdmissionPlan(missingOutputsToAdmit)).toBe('invalid-plan') + + // A STEAK record that never mentions the plan's own decision topic must + // fail the per-decision binding check even though every key it does have + // is individually well-shaped. + const missingTopic = clone(plan) + missingTopic.steak = JSON.stringify({ some_other_topic: { outputsToAdmit: [] } }) + expect(isBoundSteak(missingTopic)).toBe(false) + expect(validateAdmissionPlan(missingTopic)).toBe('invalid-plan') + }) + + test('isBoundSteak rejects an output index that cannot be parsed while binding STEAK', () => { + const overflow = clone(admissionPlan('steak-output-overflow')) + // The STEAK entry itself stays well-shaped and in range so isSteakEntry + // accepts it; the decision's own output carries the out-of-range index, + // so isBoundSteak's decisions.every must reach the parseStorageOutputIndex + // call for the *decision's* outputs and fail closed via its try/catch + // rather than throwing out of validateAdmissionPlan. + overflow.decisions[0].outputs[0].outputIndex = '4294967296' + overflow.steak = JSON.stringify({ + tm_contract: { outputsToAdmit: [0], coinsToRetain: [], coinsRemoved: [] } + }) + expect(isBoundSteak(overflow)).toBe(false) + expect(validateAdmissionPlan(overflow)).toBe('invalid-plan') + }) +}) diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionRetry.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionRetry.test.ts new file mode 100644 index 000000000..b47c50ee9 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionRetry.test.ts @@ -0,0 +1,342 @@ +import type { Db } from 'mongodb' +import { + admissionSemanticDigest, + asStorageUint64, + type AdmissionCommit, + type AdmissionReceipt +} from '../../storage/AdmissionStorage.js' +import { MongoReadGuardConflictError } from '../../storage/mongo/MongoReadGuards.js' +import { + AdmissionRejectedError, + admissionReceiptFor +} from '../../storage/mongo/MongoAdmissionPlan.js' +import { MongoAdmissionStorage } from '../../storage/mongo/MongoAdmissionStorage.js' +import { MongoOverlayStorage } from '../../storage/mongo/MongoOverlayStorage.js' +import type { MongoTransactionRunner } from '../../storage/mongo/MongoTransactionRunner.js' +import { encodeMongoUint64 } from '../../storage/mongo/MongoSchema.js' +import { admissionPlan } from '../admission/AdmissionStorageContract.js' +import { referenceScope } from '../admission/ReferenceAdmissionStorage.js' + +const dummyDb = { collection: () => ({}) } as unknown as Db + +function cursorDb(score?: string): Db { + return { + collection: () => ({ + findOne: async () => (score === undefined ? null : { score }) + }) + } as unknown as Db +} + +const clone = (value: T): T => structuredClone(value) + +function plan(): AdmissionCommit { + const next = clone(admissionPlan('retry')) + next.decisions[0].reads = [] + return next +} + +function receiptFor(value: AdmissionCommit): AdmissionReceipt { + return admissionReceiptFor(value, []) +} + +function runner( + overrides: Partial> = {} +): MongoTransactionRunner { + return { + reconcile: + overrides.reconcile ?? (async () => ({ state: 'pending' as const, attemptId: 'unlocated' })), + run: + overrides.run ?? + (async () => ({ + state: 'committed' as const, + receipt: receiptFor(plan()) + })), + close: async () => {} + } as unknown as MongoTransactionRunner +} + +function storage(overrides: ConstructorParameters[2] = {}) { + return new MongoAdmissionStorage(dummyDb, referenceScope, { + runner: runner(), + readGuards: { initialize: async () => {} } as never, + ...overrides + }) +} + +describe('Mongo admission commit guards and write-conflict retry', () => { + test('constructor rejects invalid and duplicate enlisted indexes', () => { + expect( + () => + new MongoAdmissionStorage(dummyDb, referenceScope, { + enlistedIndexes: [ + { protocol: 'other' as 'overlay-mongo-index-v1', target: 'ls', apply: async () => {} } + ] + }) + ).toThrow('Invalid enlisted Mongo lookup index') + expect( + () => + new MongoAdmissionStorage(dummyDb, referenceScope, { + enlistedIndexes: [ + { protocol: 'overlay-mongo-index-v1', target: '', apply: async () => {} } + ] + }) + ).toThrow('Invalid enlisted Mongo lookup index') + expect( + () => + new MongoAdmissionStorage(dummyDb, referenceScope, { + enlistedIndexes: [ + { protocol: 'overlay-mongo-index-v1', target: 'ls', apply: async () => {} }, + { protocol: 'overlay-mongo-index-v1', target: 'ls', apply: async () => {} } + ] + }) + ).toThrow('Duplicate enlisted Mongo lookup index target') + }) + + test('rejects digest mismatch and unsupported projectors before the body', async () => { + const adapter = storage() + const mismatched = plan() + mismatched.key.semanticDigest = '00'.repeat(32) + expect(await adapter.commitAdmission(mismatched)).toEqual({ + state: 'rejected', + code: 'digest-mismatch' + }) + const invalid = plan() + invalid.identity.txid = 'not-a-hash' + expect(await adapter.commitAdmission(invalid)).toEqual({ + state: 'rejected', + code: 'digest-mismatch' + }) + expect( + await storage({ projector: { protocol: 'not-replay-safe' } }).commitAdmission(plan()) + ).toEqual({ state: 'rejected', code: 'unsupported-projection' }) + const historical = plan() + historical.identity.mode = 'historical' + historical.key.semanticDigest = admissionSemanticDigest(historical.identity) + expect(await adapter.commitAdmission(historical)).toEqual({ + state: 'rejected', + code: 'invalid-plan' + }) + expect( + await storage({ + enlistedIndexes: [ + { + protocol: 'overlay-mongo-index-v1', + target: 'ls_contract', + apply: async () => {} + } + ] + }).commitAdmission(plan()) + ).toEqual({ state: 'rejected', code: 'invalid-plan' }) + }) + + test('returns an already committed or rejected reconcile without running the body', async () => { + const committed: AdmissionReceipt = receiptFor(plan()) + const run = jest.fn() + expect( + await storage({ + runner: runner({ + reconcile: async () => ({ state: 'committed', receipt: committed }), + run + }) + }).commitAdmission(plan()) + ).toEqual({ state: 'committed', receipt: committed }) + expect( + await storage({ + runner: runner({ + reconcile: async () => ({ state: 'rejected', code: 'spend-conflict' }), + run + }) + }).commitAdmission(plan()) + ).toEqual({ state: 'rejected', code: 'spend-conflict' }) + expect(run).not.toHaveBeenCalled() + }) + + test('retries Mongo write conflicts then commits', async () => { + const value = plan() + const committed = receiptFor(value) + const run = jest + .fn() + .mockRejectedValueOnce({ code: 112, message: 'transient' }) + .mockRejectedValueOnce({ code: '112' }) + .mockRejectedValueOnce({ codeName: 'WriteConflict' }) + .mockRejectedValueOnce(new Error('Write conflict')) + .mockRejectedValueOnce({ errmsg: 'Write conflict' }) + .mockResolvedValueOnce({ state: 'committed', receipt: committed }) + expect(await storage({ runner: runner({ run }) }).commitAdmission(value)).toEqual({ + state: 'committed', + receipt: committed + }) + expect(run).toHaveBeenCalledTimes(6) + }) + + test('maps typed body failures and rethrows unknown errors', async () => { + expect( + await storage({ + runner: runner({ + run: async () => { + throw new AdmissionRejectedError('spend-conflict') + } + }) + }).commitAdmission(plan()) + ).toEqual({ state: 'rejected', code: 'spend-conflict' }) + expect( + await storage({ + runner: runner({ + run: async () => { + throw new MongoReadGuardConflictError({ + scope: referenceScope, + key: 'selection', + expectedVersion: '1' + }) + } + }) + }).commitAdmission(plan()) + ).toEqual({ state: 'rejected', code: 'read-conflict' }) + expect( + await storage({ + runner: runner({ + run: async () => { + throw new Error('Mongo payload is not ready for reference') + } + }) + }).commitAdmission(plan()) + ).toEqual({ state: 'rejected', code: 'payload-not-ready' }) + await expect( + storage({ + runner: runner({ + run: async () => { + throw new Error('disk full') + } + }) + }).commitAdmission(plan()) + ).rejects.toThrow('disk full') + }) + + test('waits for a pending attempt, including abort then a fresh body', async () => { + const committed = receiptFor(plan()) + const reconcile = jest + .fn() + .mockResolvedValueOnce({ state: 'pending', attemptId: 'att-1' }) + .mockResolvedValueOnce({ state: 'committed', receipt: committed }) + expect(await storage({ runner: runner({ reconcile }) }).commitAdmission(plan())).toEqual({ + state: 'committed', + receipt: committed + }) + + const run = jest.fn(async () => ({ state: 'committed' as const, receipt: committed })) + const aborted = jest + .fn() + .mockResolvedValueOnce({ state: 'pending', attemptId: 'att-2' }) + .mockResolvedValueOnce({ state: 'aborted' }) + expect( + await storage({ runner: runner({ reconcile: aborted, run }) }).commitAdmission(plan()) + ).toEqual({ state: 'committed', receipt: committed }) + expect(run).toHaveBeenCalledTimes(1) + + const fromRun = jest + .fn() + .mockResolvedValueOnce({ state: 'pending', attemptId: 'unlocated' }) + .mockResolvedValueOnce({ state: 'committed', receipt: committed }) + expect( + await storage({ + runner: runner({ + reconcile: fromRun, + run: async () => ({ state: 'pending', attemptId: 'body-1' }) + }) + }).commitAdmission(plan()) + ).toEqual({ state: 'committed', receipt: committed }) + }) + + test('publishes history-update payloads before the write-conflict retry loop', async () => { + const value = plan() + value.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '4', + affectedFromHeight: '99', + handoff: { + expected: { + scope: referenceScope, + topic: 'tm_contract', + peerId: 'peer-a', + jobId: 'repair-a', + leaseToken: '9', + expiresAtMs: '100', + chainEpoch: '7', + topicHistoryGeneration: '3' + }, + checkpoint: 'repair-checkpoint' + } + } + const publish = jest.fn(async () => ({ + kind: 'outbox-data' as const, + digest: 'aa'.repeat(32), + byteLength: '1' + })) + const committed = receiptFor(value) + expect( + await storage({ + payloads: { publish } as never, + runner: runner({ + run: async () => ({ state: 'committed', receipt: committed }) + }) + }).commitAdmission(value) + ).toEqual({ state: 'committed', receipt: committed }) + expect(publish).toHaveBeenCalledTimes(1) + }) + + test('rejects an out-of-range outbox lease before touching storage', async () => { + const adapter = storage() + await expect(adapter.claimOutbox('lookup', 0)).rejects.toThrow('Invalid Mongo outbox lease') + await expect(adapter.claimOutbox('lookup', 60_001)).rejects.toThrow( + 'Invalid Mongo outbox lease' + ) + await expect(adapter.claimOutbox('propagation', 1.5)).rejects.toThrow( + 'Invalid Mongo outbox lease' + ) + }) +}) + +describe('Mongo overlay storage admission guards', () => { + test('preserves safe GASP cursors and rejects unsafe persisted values', async () => { + const maxSafe = BigInt(Number.MAX_SAFE_INTEGER) + const getCursor = (score?: string): Promise => + new MongoOverlayStorage(cursorDb(score), referenceScope).getLastInteraction( + 'peer.example', + 'tm_contract' + ) + + await expect(getCursor(encodeMongoUint64('42'))).resolves.toBe(42) + await expect(getCursor()).resolves.toBe(0) + await expect(getCursor(encodeMongoUint64(maxSafe.toString()))).resolves.toBe( + Number.MAX_SAFE_INTEGER + ) + await expect(getCursor(encodeMongoUint64((maxSafe + 1n).toString()))).rejects.toThrow( + 'Mongo GASP cursor exceeds a safe JavaScript integer' + ) + await expect(getCursor(encodeMongoUint64((maxSafe + 3n).toString()))).rejects.toThrow( + 'Mongo GASP cursor exceeds a safe JavaScript integer' + ) + // These specific values rounded down/up in the old Number-only decoder. + await expect(getCursor(encodeMongoUint64('9007199254740993'))).rejects.toThrow( + 'Mongo GASP cursor exceeds a safe JavaScript integer' + ) + await expect(getCursor(encodeMongoUint64('9007199254740995'))).rejects.toThrow( + 'Mongo GASP cursor exceeds a safe JavaScript integer' + ) + }) + + test('updateTransactionBEEF is intentionally unimplemented', async () => { + const overlay = new MongoOverlayStorage(dummyDb, referenceScope) + const txid = 'ab'.repeat(32) + await expect(overlay.updateTransactionBEEF(txid, [])).rejects.toThrow( + `Mongo overlay storage does not implement updateTransactionBEEF for ${txid}` + ) + expect(overlay.enlistedIndexTargets()).toEqual([]) + await expect( + overlay.publishAdmissionPayload({ + kind: 'not-a-kind' as 'locking-script', + bytes: new Uint8Array([1]) + }) + ).rejects.toThrow('Invalid Mongo admission payload kind') + expect(asStorageUint64('0')).toBe('0') + }) +}) diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionStorage.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionStorage.test.ts new file mode 100644 index 000000000..e4430e8ca --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionStorage.test.ts @@ -0,0 +1,1056 @@ +import { Transaction } from '@bsv/sdk' +import { admissionPlan, admissionStorageContract } from '../admission/AdmissionStorageContract.js' +import { getAdmissionStorage } from '../../storage/AdmissionStorage.js' +import { MongoAdmissionStorage } from '../../storage/mongo/MongoAdmissionStorage.js' +import { MongoOverlayStorage } from '../../storage/mongo/MongoOverlayStorage.js' +import { + MongoPayloadStore, + type MongoPayloadReference +} from '../../storage/mongo/MongoPayloadStore.js' +import type { ClientSession } from 'mongodb' +import { + bootstrapMongoOverlay, + encodeMongoOutputIndex, + encodeMongoUint64, + MongoCollectionNames, + mongoChainKey, + mongoNodeKey, + mongoRecordKey +} from '../../storage/mongo/MongoSchema.js' +import { createMongoReplicaFixture, type MongoReplicaFixture } from './MongoReplicaFixture.js' +import { MongoAdmissionHarness } from './MongoAdmissionHarness.js' +import type { MongoEnlistedLookupIndex } from '../../storage/mongo/MongoAdmissionStorage.js' +import type { AdmissionCommit, StorageScope } from '../../storage/AdmissionStorage.js' +import { referenceScope } from '../admission/ReferenceAdmissionStorage.js' + +const clone = (value: T): T => structuredClone(value) + +describe('Mongo admission storage', () => { + let fixture: MongoReplicaFixture + let harness: MongoAdmissionHarness + + beforeAll(async () => { + fixture = await createMongoReplicaFixture() + await bootstrapMongoOverlay(fixture.db, fixture.scope) + harness = new MongoAdmissionHarness(fixture) + }, 120000) + + afterAll(async () => { + await fixture.close() + }, 60000) + + /** Seeds everything a plan's decisions and payloads need to be admissible, mirroring the shared contract helper. */ + async function seedPlan( + plan: AdmissionCommit, + options: { skipHistory?: boolean } = {} + ): Promise { + for (const decision of plan.decisions) { + if (!options.skipHistory) { + await harness.seed.history(plan.identity.scope, decision.topic, decision.expectedHistory) + } + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + plan.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + plan.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of plan.payloads) await harness.seed.readyPayload(payload) + } + + admissionStorageContract(() => harness) + + test('advertises overlay-admission-v1 only on the complete adapter', () => { + const storage = new MongoOverlayStorage(fixture.db, fixture.scope) + expect(getAdmissionStorage(storage)?.protocol).toBe('overlay-admission-v1') + expect(getAdmissionStorage(storage.admission)).toBeUndefined() + }) + + test('lost ACK after majority commit returns the saved STEAK without a new body', async () => { + const plan = admissionPlan('lost-ack') + await harness.reset() + for (const decision of plan.decisions) { + await harness.seed.history(plan.identity.scope, decision.topic, decision.expectedHistory) + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + plan.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + plan.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of plan.payloads) await harness.seed.readyPayload(payload) + const first = await harness.adapter.commitAdmission(plan) + expect(first.state).toBe('committed') + const replacement = new MongoAdmissionStorage(fixture.db, fixture.scope) + let bodies = 0 + const original = replacement.commitAdmission.bind(replacement) + replacement.commitAdmission = async (next: AdmissionCommit) => { + bodies += 1 + return await original(next) + } + const retried = await replacement.commitAdmission(plan) + expect(retried).toEqual(first) + expect(bodies).toBe(1) + await replacement.close() + }) + + test('enlisted indexes are visible at commit and external indexes stay pending', async () => { + await harness.reset() + const visible: string[] = [] + const enlisted: MongoEnlistedLookupIndex = { + protocol: 'overlay-mongo-index-v1', + target: 'ls_enlisted', + apply: async (context, plan) => { + visible.push(plan.identity.txid) + await fixture.db + .collection('enlisted_index') + .insertOne({ txid: plan.identity.txid, target: 'ls_enlisted' }, context.options()) + } + } + const adapter = new MongoAdmissionStorage(fixture.db, fixture.scope, { + enlistedIndexes: [enlisted] + }) + const plan = admissionPlan('enlisted-visible', 'e1'.repeat(32)) + plan.outbox = plan.outbox.filter(intent => intent.kind !== 'lookup') + for (const decision of plan.decisions) { + await harness.seed.history(plan.identity.scope, decision.topic, decision.expectedHistory) + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + plan.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + plan.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of plan.payloads) await harness.seed.readyPayload(payload) + const result = await adapter.commitAdmission(plan) + expect(result.state).toBe('committed') + if (result.state !== 'committed') throw new Error('expected commit') + expect(result.receipt.indexes).toEqual([{ target: 'ls_enlisted', state: 'visible' }]) + expect( + await fixture.db.collection('enlisted_index').findOne({ txid: plan.identity.txid }) + ).toEqual(expect.objectContaining({ target: 'ls_enlisted' })) + expect(visible).toEqual([plan.identity.txid]) + await adapter.close() + }) + + test('external lookup index is not visible at commit and duplicate delivery is idempotent', async () => { + const plan = admissionPlan('outbox-delivery') + await harness.reset() + for (const decision of plan.decisions) { + await harness.seed.history(plan.identity.scope, decision.topic, decision.expectedHistory) + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + plan.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + plan.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of plan.payloads) await harness.seed.readyPayload(payload) + const result = await harness.adapter.commitAdmission(plan) + expect(result.state).toBe('committed') + if (result.state !== 'committed') throw new Error('expected commit') + expect(result.receipt.indexes).toEqual([{ target: 'ls_contract', state: 'pending' }]) + expect( + await fixture.db.collection('enlisted_index').findOne({ txid: plan.identity.txid }) + ).toBeNull() + const first = await harness.adapter.claimOutbox('lookup') + expect(first?.eventId).toBe(`${plan.key.operationId}:lookup`) + await harness.adapter.acknowledgeOutbox('lookup', first!.eventId) + await harness.adapter.acknowledgeOutbox('lookup', first!.eventId) + expect(await harness.adapter.claimOutbox('lookup')).toBeNull() + const stored = await fixture.db.collection(MongoCollectionNames.lookupOutbox).findOne({ + eventId: first!.eventId + }) + expect(stored?.state).toBe('delivered') + }) + + test('competing spend of a different spender is rejected', async () => { + await harness.reset() + const first = admissionPlan('race-a', '6'.repeat(64)) + const second = admissionPlan('race-b', '7'.repeat(64)) + for (const plan of [first, second]) { + for (const decision of plan.decisions) { + await harness.seed.history(plan.identity.scope, decision.topic, decision.expectedHistory) + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + plan.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + plan.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of plan.payloads) await harness.seed.readyPayload(payload) + } + const results = await Promise.all([ + harness.adapter.commitAdmission(first), + harness.adapter.commitAdmission(second) + ]) + expect(results.filter(result => result.state === 'committed')).toHaveLength(1) + expect( + results.filter(result => result.state === 'rejected' && result.code === 'spend-conflict') + ).toHaveLength(1) + }) + + test('sequential history updates replace the generation-scoped pin', async () => { + await harness.reset() + const first = admissionPlan('history-first') + first.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '4', + affectedFromHeight: '99' + } + const second = admissionPlan('history-second', '6'.repeat(64)) + second.decisions[0].spends = [] + second.decisions[0].expectedHistory = { chainEpoch: '7', topicHistoryGeneration: '4' } + second.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '5', + affectedFromHeight: '100' + } + await harness.seed.history( + first.identity.scope, + first.decisions[0].topic, + first.decisions[0].expectedHistory + ) + for (const read of first.decisions[0].reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + first.identity.scope, + first.decisions[0].topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of first.decisions[0].spends) { + await harness.seed.spendable( + first.identity.scope, + first.decisions[0].topic, + spend.outpoint, + spend.expectedVersion + ) + } + for (const payload of first.payloads) await harness.seed.readyPayload(payload) + expect((await harness.adapter.commitAdmission(first)).state).toBe('committed') + await harness.seed.history( + second.identity.scope, + second.decisions[0].topic, + second.decisions[0].expectedHistory + ) + for (const read of second.decisions[0].reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + second.identity.scope, + second.decisions[0].topic, + read.key, + read.expectedVersion + ) + } + } + for (const payload of second.payloads) await harness.seed.readyPayload(payload) + expect((await harness.adapter.commitAdmission(second)).state).toBe('committed') + const snapshot = await harness.snapshot() + expect(snapshot.fences.find(item => item.topic === 'tm_contract')?.fence).toEqual({ + chainEpoch: '7', + topicHistoryGeneration: '5' + }) + expect(snapshot.historyUpdates).toEqual([{ topic: 'tm_contract', affectedFromHeight: '100' }]) + }) + + test('replaceHistoryPin tolerates a previous pin whose payload document is already gone', async () => { + await harness.reset() + const first = admissionPlan('history-pin-gc-first') + first.decisions[0].historyUpdate = { nextTopicHistoryGeneration: '4', affectedFromHeight: '99' } + await seedPlan(first) + expect((await harness.adapter.commitAdmission(first)).state).toBe('committed') + + // Simulate the previous history-update payload having since been + // garbage collected while its pin reference row is still in place: + // replaceHistoryPin must skip the release rather than crash on it. + const reference = await fixture.db.collection(MongoCollectionNames.payloadReferences).findOne({ + network: first.identity.scope.network, + genesisHash: first.identity.scope.genesisHash, + nodeId: first.identity.scope.nodeId, + ownerKind: 'basm-job', + ownerId: 'tm_contract', + slot: 'history-update:4' + }) + expect(typeof reference?.payloadId).toBe('string') + await fixture.db + .collection(MongoCollectionNames.payloads) + .deleteOne({ _id: reference?.payloadId as string }) + + const second = admissionPlan('history-pin-gc-second', '6'.repeat(64)) + second.decisions[0].spends = [] + second.decisions[0].expectedHistory = { chainEpoch: '7', topicHistoryGeneration: '4' } + second.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '5', + affectedFromHeight: '100' + } + await seedPlan(second) + expect((await harness.adapter.commitAdmission(second)).state).toBe('committed') + }) + + test('hydrates output scripts and BEEF from payload bytes', async () => { + await harness.reset() + const storage = new MongoOverlayStorage(fixture.db, fixture.scope) + const script = [0x76, 0xa9, 0x14, 0x00, 0x88, 0xac] + const beef = hydrationBeef() + const txid = Transaction.fromBEEF(beef).id('hex') + await storage.insertOutput({ + txid, + outputIndex: 0, + outputScript: script, + satoshis: 1234, + topic: 'tm_contract', + spent: false, + outputsConsumed: [], + consumedBy: [], + score: 10, + beef + }) + const found = await storage.findOutput(txid, 0, 'tm_contract', false, true) + expect(found?.outputScript).toEqual(script) + expect(found?.satoshis).toBe(1234) + expect(found?.score).toBe(10) + expect(found?.beef?.length).toBeGreaterThan(0) + await storage.close() + }) + + test('majority admission effects survive killing the acknowledged primary', async () => { + const plan = admissionPlan('kill-primary') + await harness.reset() + for (const decision of plan.decisions) { + await harness.seed.history(plan.identity.scope, decision.topic, decision.expectedHistory) + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + plan.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + plan.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of plan.payloads) await harness.seed.readyPayload(payload) + const committed = await harness.adapter.commitAdmission(plan) + expect(committed.state).toBe('committed') + await fixture.killPrimary() + const client = await fixture.connect() + fixture.db = client.db(fixture.db.databaseName) + const replacement = new MongoAdmissionStorage(fixture.db, fixture.scope) + expect(await replacement.commitAdmission(plan)).toEqual(committed) + await replacement.close() + }, 60000) + + test('covers overlay storage fences, CRUD filters, and unimplemented BEEF updates', async () => { + await harness.reset() + const storage = new MongoOverlayStorage(fixture.db, fixture.scope) + expect(await storage.getHistoryFence('tm_missing')).toEqual({ + chainEpoch: '0', + topicHistoryGeneration: '0' + }) + const now = new Date() + await storage.admission.generations().insertOne({ + _id: storage.admission.generationId('tm_contract'), + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + nodeId: fixture.scope.nodeId, + topic: 'tm_contract', + chainEpoch: encodeMongoUint64('3'), + topicHistoryGeneration: encodeMongoUint64('4'), + policyId: 'policy-1', + createdAt: now, + updatedAt: now + }) + expect(await storage.getHistoryFence('tm_contract')).toEqual({ + chainEpoch: '3', + topicHistoryGeneration: '4' + }) + + const published = await storage.publishAdmissionPayload({ + kind: 'locking-script', + bytes: new Uint8Array([0x76, 0xa9]) + }) + expect(published.byteLength).toBe('2') + expect(published.digest).toHaveLength(64) + + const beef = hydrationBeef() + const txid = Transaction.fromBEEF(beef).id('hex') + const script = [0x76, 0xa9, 0x14, 0x00, 0x88, 0xac] + await storage.insertOutput({ + txid, + outputIndex: 0, + outputScript: script, + satoshis: 50, + topic: 'tm_contract', + spent: false, + outputsConsumed: [], + consumedBy: [], + score: 8, + beef + }) + expect(await storage.findOutput(txid, 0, undefined, false)).toEqual( + expect.objectContaining({ txid, spent: false, satoshis: 50 }) + ) + expect(await storage.findOutput(txid, 0, 'tm_contract', true)).toBeNull() + expect(await storage.findOutputsForTransaction(txid)).toHaveLength(1) + expect(await storage.findUTXOsForTopic('tm_contract', 0, 0)).toHaveLength(1) + expect(await storage.findUTXOsForTopic('tm_contract', 8, 1, true)).toHaveLength(1) + expect(await storage.findUTXOsForTopic('tm_contract', 9, 1)).toHaveLength(0) + + await storage.updateConsumedBy(txid, 0, 'tm_contract', [ + { txid: 'ab'.repeat(32), outputIndex: 1 } + ]) + await storage.insertAppliedTransaction({ txid, topic: 'tm_contract', proven: false }) + await storage.insertAppliedTransaction({ + txid: 'dd'.repeat(32), + topic: 'tm_contract', + proven: true + }) + expect(await storage.doesAppliedTransactionExist({ txid, topic: 'tm_contract' })).toBe(true) + expect( + await storage.doesAppliedTransactionExist({ txid: 'cd'.repeat(32), topic: 'tm_contract' }) + ).toBe(false) + await storage.updateLastInteraction('peer.example', 'tm_contract', 42) + expect(await storage.getLastInteraction('peer.example', 'tm_contract')).toBe(42) + expect(await storage.getLastInteraction('missing', 'tm_contract')).toBe(0) + + await storage.markUTXOAsSpent(txid, 0, 'tm_contract') + expect(await storage.findOutput(txid, 0, 'tm_contract', true)).toEqual( + expect.objectContaining({ spent: true }) + ) + await storage.deleteOutput(txid, 0, 'tm_contract') + expect(await storage.findOutput(txid, 0, 'tm_contract', false)).toBeNull() + await expect(storage.updateTransactionBEEF(txid, [])).rejects.toThrow( + `Mongo overlay storage does not implement updateTransactionBEEF for ${txid}` + ) + + const outputDoc = { + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + nodeId: fixture.scope.nodeId, + scriptPayloadId: storage.admission.payloadId(published), + scriptOffset: encodeMongoUint64('0'), + scriptByteLength: encodeMongoUint64(published.byteLength), + state: 'unspent' as const, + version: '1', + createdAt: now, + updatedAt: now + } + await fixture.db.collection(MongoCollectionNames.outputs).insertOne({ + ...outputDoc, + _id: mongoRecordKey( + mongoNodeKey(fixture.scope), + 'output', + 'tm_unsafe', + 'ee'.repeat(32), + encodeMongoOutputIndex('0') + ), + topic: 'tm_unsafe', + txid: 'ee'.repeat(32), + outputIndex: encodeMongoOutputIndex('0'), + satoshis: encodeMongoUint64('9007199254740993'), + score: encodeMongoUint64('1') + }) + await expect(storage.findOutput('ee'.repeat(32), 0, 'tm_unsafe')).rejects.toThrow( + 'Mongo satoshis exceeds a safe JavaScript integer' + ) + + await fixture.db.collection(MongoCollectionNames.outputs).insertOne({ + ...outputDoc, + _id: mongoRecordKey( + mongoNodeKey(fixture.scope), + 'output', + 'tm_score', + 'cc'.repeat(32), + encodeMongoOutputIndex('0') + ), + topic: 'tm_score', + txid: 'cc'.repeat(32), + outputIndex: encodeMongoOutputIndex('0'), + satoshis: encodeMongoUint64('1'), + score: encodeMongoUint64('9007199254740993') + }) + await expect(storage.findOutput('cc'.repeat(32), 0, 'tm_score')).rejects.toThrow( + 'Mongo score exceeds a safe JavaScript integer' + ) + + await fixture.db.collection(MongoCollectionNames.outputs).insertOne({ + ...outputDoc, + _id: mongoRecordKey( + mongoNodeKey(fixture.scope), + 'output', + 'tm_script', + 'ff'.repeat(32), + encodeMongoOutputIndex('0') + ), + topic: 'tm_script', + txid: 'ff'.repeat(32), + outputIndex: encodeMongoOutputIndex('0'), + satoshis: encodeMongoUint64('1'), + score: encodeMongoUint64('1'), + scriptPayloadId: 'missing-payload' + }) + await expect(storage.findOutput('ff'.repeat(32), 0, 'tm_script')).rejects.toThrow( + 'Mongo output script payload is not ready' + ) + await expect(harness.adapter.acknowledgeOutbox('lookup', 'missing-event')).rejects.toThrow( + 'Mongo outbox event is not leased' + ) + await storage.close() + }) + + test('rejects a missing eviction and a history update from an empty fence', async () => { + await harness.reset() + const missingEviction = admissionPlan('missing-eviction') + missingEviction.decisions[0].evictions = [{ txid: 'aa'.repeat(32), outputIndex: '0' }] + for (const decision of missingEviction.decisions) { + await harness.seed.history( + missingEviction.identity.scope, + decision.topic, + decision.expectedHistory + ) + for (const read of decision.reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + missingEviction.identity.scope, + decision.topic, + read.key, + read.expectedVersion + ) + } + } + for (const spend of decision.spends) { + await harness.seed.spendable( + missingEviction.identity.scope, + decision.topic, + spend.outpoint, + spend.expectedVersion + ) + } + } + for (const payload of missingEviction.payloads) await harness.seed.readyPayload(payload) + expect(await harness.adapter.commitAdmission(missingEviction)).toEqual({ + state: 'rejected', + code: 'invalid-plan' + }) + + await harness.reset() + const fromZero = admissionPlan('history-zero') + fromZero.decisions[0].expectedHistory = { chainEpoch: '0', topicHistoryGeneration: '0' } + fromZero.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '1', + affectedFromHeight: '0' + } + fromZero.decisions[0].spends = [] + for (const read of fromZero.decisions[0].reads) { + if (read.expectedVersion !== null) { + await harness.seed.read( + fromZero.identity.scope, + fromZero.decisions[0].topic, + read.key, + read.expectedVersion + ) + } + } + for (const payload of fromZero.payloads) await harness.seed.readyPayload(payload) + expect((await harness.adapter.commitAdmission(fromZero)).state).toBe('committed') + }) + + test('a decision whose expected history fence no longer matches the topic is rejected as a read conflict', async () => { + await harness.reset() + const plan = admissionPlan('fence-conflict') + // Leave the topic's generation row unseeded so checkHistory sees the + // default zero fence against the plan's own nonzero expectation. + await seedPlan(plan, { skipHistory: true }) + expect(await harness.adapter.commitAdmission(plan)).toEqual({ + state: 'rejected', + code: 'read-conflict' + }) + }) + + test('re-admitting the same tx and topic under a different operation id is fenced off as invalid', async () => { + await harness.reset() + const txid = 'fe'.repeat(32) + const first = admissionPlan('reapply-a', txid) + await seedPlan(first) + expect((await harness.adapter.commitAdmission(first)).state).toBe('committed') + + const second = admissionPlan('reapply-b', txid) + const result = await harness.adapter.commitAdmission(second) + expect(result).toEqual({ state: 'rejected', code: 'invalid-plan' }) + }) + + test('applyEviction transitions a real, currently-unspent output to evicted', async () => { + await harness.reset() + const plan = admissionPlan('real-eviction') + plan.decisions[0].spends = [] + const eviction = { txid: 'ab'.repeat(32), outputIndex: '0' } + plan.decisions[0].evictions = [eviction] + await seedPlan(plan) + await harness.seed.spendable(plan.identity.scope, plan.decisions[0].topic, eviction, '1') + const result = await harness.adapter.commitAdmission(plan) + expect(result.state).toBe('committed') + const raw = await fixture.db + .collection(MongoCollectionNames.outputs) + .findOne({ _id: harness.adapter.outputId(plan.decisions[0].topic, eviction) }) + expect(raw?.state).toBe('evicted') + }) + + test('an output insert that collides with an existing row is rejected as invalid-plan', async () => { + await harness.reset() + const plan = admissionPlan('duplicate-output') + await seedPlan(plan) + const output = plan.decisions[0].outputs[0] + const now = new Date() + // Pre-existing row at the exact _id the plan's own output insert will + // target. Nothing else in applyPlan checks for output existence ahead + // of the insert itself, so this exercises Mongo's own uniqueness guard. + await fixture.db.collection(MongoCollectionNames.outputs).insertOne({ + _id: harness.adapter.outputId(plan.decisions[0].topic, output), + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + nodeId: fixture.scope.nodeId, + topic: plan.decisions[0].topic, + txid: output.txid, + outputIndex: encodeMongoOutputIndex(output.outputIndex), + satoshis: encodeMongoUint64(output.satoshis), + score: encodeMongoUint64(output.score), + scriptPayloadId: 'unrelated-payload', + scriptOffset: encodeMongoUint64('0'), + scriptByteLength: encodeMongoUint64('1'), + state: 'unspent', + version: 'pre-existing', + createdAt: now, + updatedAt: now + }) + const result = await harness.adapter.commitAdmission(plan) + expect(result).toEqual({ state: 'rejected', code: 'invalid-plan' }) + }) + + test('a non-duplicate-key failure inserting an output is rethrown unchanged rather than swallowed', async () => { + await harness.reset() + const plan = admissionPlan('output-insert-failure') + await seedPlan(plan) + // Scope the injected failure to the outputs namespace specifically, so + // it lands on insertOutput's own insertOne rather than the transaction + // runner's unrelated claim-row insert: duplicateKey(error) must be + // false here, so the original error propagates as-is unclassified. + await fixture.failCommands({ + failCommands: ['insert'], + errorCode: 1, + namespace: `${fixture.db.databaseName}.${MongoCollectionNames.outputs}` + }) + await expect(harness.adapter.commitAdmission(plan)).rejects.toMatchObject({ code: 1 }) + await fixture.disableFailPoint() + }) + + test('an applied-history insert that collides under the same admission id is rejected as invalid-plan', async () => { + await harness.reset() + const plan = admissionPlan('duplicate-applied') + await seedPlan(plan) + const appliedId = mongoRecordKey( + mongoNodeKey(plan.identity.scope), + 'applied', + plan.decisions[0].topic, + plan.decisions[0].applied.txid + ) + const now = new Date() + // The admissionId matches this plan's own operationId, so + // assertAppliedAvailable's mismatch guard does not reject it ahead of + // time; only Mongo's own uniqueness guard on the insert itself can. + await fixture.db.collection(MongoCollectionNames.appliedTransactions).insertOne({ + _id: appliedId, + schemaVersion: 1, + network: plan.identity.scope.network, + genesisHash: plan.identity.scope.genesisHash, + nodeId: plan.identity.scope.nodeId, + topic: plan.decisions[0].topic, + txid: plan.decisions[0].applied.txid, + state: 'unproven', + admissionId: plan.key.operationId, + createdAt: now, + updatedAt: now + }) + const result = await harness.adapter.commitAdmission(plan) + expect(result).toEqual({ state: 'rejected', code: 'invalid-plan' }) + }) + + test('a non-duplicate-key failure inserting an applied-history row is rethrown unchanged rather than swallowed', async () => { + await harness.reset() + const plan = admissionPlan('applied-insert-failure') + await seedPlan(plan) + // Scope the injected failure to the appliedTransactions namespace so it + // lands on insertApplied's own insertOne (which runs after the output + // has already been inserted successfully), not an earlier, unrelated + // insert -- and duplicateKey(error) must be false here, so the original + // error propagates as-is instead of being reclassified. + await fixture.failCommands({ + failCommands: ['insert'], + errorCode: 1, + namespace: `${fixture.db.databaseName}.${MongoCollectionNames.appliedTransactions}` + }) + await expect(harness.adapter.commitAdmission(plan)).rejects.toMatchObject({ code: 1 }) + await fixture.disableFailPoint() + }) + + test('omits firstSeenHeight from the applied-history row when the plan does not supply one', async () => { + await harness.reset() + const plan = clone(admissionPlan('no-first-seen')) + delete (plan.decisions[0].applied as { firstSeenHeight?: string }).firstSeenHeight + await seedPlan(plan) + expect((await harness.adapter.commitAdmission(plan)).state).toBe('committed') + const appliedId = mongoRecordKey( + mongoNodeKey(plan.identity.scope), + 'applied', + plan.decisions[0].topic, + plan.decisions[0].applied.txid + ) + const raw = await fixture.db + .collection(MongoCollectionNames.appliedTransactions) + .findOne({ _id: appliedId }) + expect(raw?.firstSeenHeight).toBeUndefined() + }) + + test('upsertTransaction omits rawPayloadId and records manifestPayloadId when the plan carries no raw payload', async () => { + await harness.reset() + const plan = clone(admissionPlan('no-raw-with-manifest')) + plan.outbox = [] + plan.payloads = plan.payloads.filter(ref => ref.kind !== 'raw-transaction') + plan.payloads.push({ kind: 'beef-manifest', digest: 'ef'.repeat(32), byteLength: '12' }) + await seedPlan(plan) + expect((await harness.adapter.commitAdmission(plan)).state).toBe('committed') + const id = mongoRecordKey(mongoChainKey(plan.identity.scope), 'transaction', plan.identity.txid) + const raw = await fixture.db.collection(MongoCollectionNames.transactions).findOne({ _id: id }) + expect(raw?.rawPayloadId).toBeUndefined() + expect(raw?.manifestPayloadId).toBe( + harness.adapter.payloadId({ kind: 'beef-manifest', digest: 'ef'.repeat(32) }) + ) + }) + + test('an output payload ref with an unrecognized kind is rejected even though it is otherwise well-formed', async () => { + await harness.reset() + const plan = clone(admissionPlan('weird-payload-kind')) + const weirdRef = { kind: 'weird-kind', digest: 'cd'.repeat(32), byteLength: '2' } + // The script's own offset/byteLength must stay consistent with the + // (weird-kind) payload's declared byteLength, or isInvalidOutput's + // range check rejects the plan for a different reason before applyPlan + // -- and pin() -- is ever reached. + plan.decisions[0].outputs[0].script.offset = '0' + plan.decisions[0].outputs[0].script.byteLength = weirdRef.byteLength + ;( + plan.decisions[0].outputs[0].script as unknown as { + payload: { kind: string; digest: string; byteLength: string } + } + ).payload = weirdRef + await seedPlan(plan) + await harness.seed.readyPayload( + weirdRef as unknown as Parameters[0] + ) + const result = await harness.adapter.commitAdmission(plan) + expect(result).toEqual({ state: 'rejected', code: 'invalid-plan' }) + }) + + test('a payload store failure while pinning an output script surfaces as payload-not-ready', async () => { + await harness.reset() + // admissionPlan() binds its identity/key to referenceScope, not the + // fixture's own randomly-named scope, so the adapter under test must be + // constructed with the same scope -- otherwise commitAdmission's own + // cross-scope guard would silently delegate to an unwrapped peer. + const realPayloads = new MongoPayloadStore(fixture.db, referenceScope) + const payloads = { + publish: realPayloads.publish.bind(realPayloads), + addReference: async ( + session: ClientSession, + reference: MongoPayloadReference, + operation?: { timeoutMS?: number; signal?: AbortSignal } + ) => { + if (reference.ownerKind === 'output' && reference.slot === 'script') { + throw new Error('Mongo payload is not ready for reference') + } + await realPayloads.addReference(session, reference, operation) + }, + releaseReference: realPayloads.releaseReference.bind(realPayloads) + } as unknown as MongoPayloadStore + const adapter = new MongoAdmissionStorage(fixture.db, referenceScope, { payloads }) + const plan = admissionPlan('pin-store-failure') + await seedPlan(plan) + const result = await adapter.commitAdmission(plan) + expect(result).toEqual({ state: 'rejected', code: 'payload-not-ready' }) + await adapter.close() + }) + + test('an unrelated payload store failure while pinning propagates unchanged rather than being reclassified', async () => { + await harness.reset() + const realPayloads = new MongoPayloadStore(fixture.db, referenceScope) + const payloads = { + publish: realPayloads.publish.bind(realPayloads), + addReference: async ( + session: ClientSession, + reference: MongoPayloadReference, + operation?: { timeoutMS?: number; signal?: AbortSignal } + ) => { + if (reference.ownerKind === 'output' && reference.slot === 'script') { + throw new Error('Mongo payload store is temporarily unavailable') + } + await realPayloads.addReference(session, reference, operation) + }, + releaseReference: realPayloads.releaseReference.bind(realPayloads) + } as unknown as MongoPayloadStore + const adapter = new MongoAdmissionStorage(fixture.db, referenceScope, { payloads }) + const plan = admissionPlan('pin-unrelated-failure') + await seedPlan(plan) + await expect(adapter.commitAdmission(plan)).rejects.toThrow( + 'Mongo payload store is temporarily unavailable' + ) + await adapter.close() + }) + + test('claimOutbox omits a pinned payload whose underlying document has since been removed', async () => { + await harness.reset() + const plan = admissionPlan('outbox-missing-payload') + await seedPlan(plan) + expect((await harness.adapter.commitAdmission(plan)).state).toBe('committed') + const outboxDataRef = plan.payloads.find(ref => ref.kind === 'outbox-data') + if (outboxDataRef === undefined) throw new Error('expected an outbox-data payload in the plan') + await fixture.db + .collection(MongoCollectionNames.payloads) + .deleteOne({ _id: harness.adapter.payloadId(outboxDataRef) }) + const claimed = await harness.adapter.claimOutbox('lookup') + expect(claimed?.eventId).toBe(`${plan.key.operationId}:lookup`) + expect(claimed?.payloads).toEqual([]) + }) + + test('reconcileAdmission for a different scope delegates to, and reuses, a scoped peer instance', async () => { + await harness.reset() + const otherScope: StorageScope = { ...fixture.scope, nodeId: `${fixture.scope.nodeId}-peer` } + const key = { + scope: otherScope, + operationId: 'never-committed-on-peer-scope', + semanticDigest: 'ab'.repeat(32) + } + // A never-committed operation reconciles to a fresh "pending/unlocated" + // result from the peer's own MongoTransactionRunner; a second call for + // the same foreign scope must reuse the cached peer rather than + // constructing (and reconnecting) a fresh one every time. + const first = await harness.adapter.reconcileAdmission(key) + const second = await harness.adapter.reconcileAdmission(key) + expect(first).toEqual({ state: 'pending', attemptId: 'unlocated' }) + expect(second).toEqual(first) + }) + + test('applyHandoff advances a matching, unexpired lease and records its checkpoint', async () => { + await harness.reset() + const plan = admissionPlan('handoff-happy') + plan.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '4', + affectedFromHeight: '99', + handoff: { + expected: { + scope: plan.identity.scope, + topic: plan.decisions[0].topic, + peerId: 'peer-a', + jobId: 'job-a', + leaseToken: '9', + expiresAtMs: '1', + chainEpoch: '7', + topicHistoryGeneration: '3' + }, + checkpoint: 'checkpoint-a' + } + } + await seedPlan(plan) + await harness.seed.lease(plan.decisions[0].historyUpdate.handoff.expected) + expect((await harness.adapter.commitAdmission(plan)).state).toBe('committed') + const jobId = mongoRecordKey( + mongoNodeKey(plan.identity.scope), + 'job', + plan.decisions[0].topic, + 'peer-a', + 'job-a' + ) + const raw = await fixture.db + .collection(MongoCollectionNames.basmRecoveryJobs) + .findOne({ _id: jobId }) + expect(raw?.checkpoint).toBe('checkpoint-a') + expect(raw?.topicHistoryGeneration).toBe(encodeMongoUint64('4')) + }) + + test('applyHandoff rejects a handoff whose expected fence does not match the decision', async () => { + await harness.reset() + const plan = admissionPlan('handoff-mismatch') + plan.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '4', + affectedFromHeight: '99', + handoff: { + expected: { + scope: plan.identity.scope, + topic: plan.decisions[0].topic, + peerId: 'peer-b', + jobId: 'job-b', + leaseToken: '9', + expiresAtMs: '1', + chainEpoch: '1', + topicHistoryGeneration: '3' + }, + checkpoint: 'checkpoint-b' + } + } + await seedPlan(plan) + await harness.seed.lease(plan.decisions[0].historyUpdate.handoff.expected) + expect(await harness.adapter.commitAdmission(plan)).toEqual({ + state: 'rejected', + code: 'read-conflict' + }) + }) + + test('applyHandoff rejects when no matching, unexpired lease exists for the topic revision handoff', async () => { + await harness.reset() + const plan = admissionPlan('handoff-missing-job') + plan.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '4', + affectedFromHeight: '99', + handoff: { + expected: { + scope: plan.identity.scope, + topic: plan.decisions[0].topic, + peerId: 'peer-c', + jobId: 'job-c', + leaseToken: '9', + expiresAtMs: '1', + chainEpoch: '7', + topicHistoryGeneration: '3' + }, + checkpoint: 'checkpoint-c' + } + } + await seedPlan(plan) + // No lease is seeded for job-c: applyHandoff's own findOneAndUpdate must + // fail closed rather than silently skipping the checkpoint update. + expect(await harness.adapter.commitAdmission(plan)).toEqual({ + state: 'rejected', + code: 'read-conflict' + }) + }) + + test('two admissions racing to bootstrap the same topic generation resolve to exactly one winner', async () => { + await harness.reset() + const bootstrapPlan = (operationId: string, txid: string): AdmissionCommit => { + const value = admissionPlan(operationId, txid) + value.decisions[0].expectedHistory = { chainEpoch: '0', topicHistoryGeneration: '0' } + value.decisions[0].historyUpdate = { + nextTopicHistoryGeneration: '1', + affectedFromHeight: '0' + } + value.decisions[0].spends = [] + return value + } + const a = bootstrapPlan('bootstrap-race-a', 'aa'.repeat(32)) + const b = bootstrapPlan('bootstrap-race-b', 'bb'.repeat(32)) + for (const value of [a, b]) await seedPlan(value) + const results = await Promise.all([ + harness.adapter.commitAdmission(a), + harness.adapter.commitAdmission(b) + ]) + expect(results.filter(result => result.state === 'committed')).toHaveLength(1) + expect( + results.filter(result => result.state === 'rejected' && result.code === 'read-conflict') + ).toHaveLength(1) + }) + + test('two admissions racing on the same outbox event id resolve to exactly one winner', async () => { + await harness.reset() + const sharedEventId = 'shared-outbox-event' + const racePlan = (operationId: string, txid: string): AdmissionCommit => { + const value = admissionPlan(operationId, txid) + value.decisions[0].spends = [] + value.outbox = [{ ...value.outbox[0], eventId: sharedEventId }] + return value + } + const a = racePlan('outbox-race-a', 'cc'.repeat(32)) + const b = racePlan('outbox-race-b', 'dd'.repeat(32)) + for (const value of [a, b]) await seedPlan(value) + const results = await Promise.all([ + harness.adapter.commitAdmission(a), + harness.adapter.commitAdmission(b) + ]) + expect(results.filter(result => result.state === 'committed')).toHaveLength(1) + expect(results.filter(result => result.state === 'rejected')).toHaveLength(1) + }) +}) + +function hydrationBeef(): number[] { + return Transaction.fromHexBEEF( + '0100beef01fe636d0c0007021400fe507c0c7aa754cef1f7889d5fd395cf1f785dd7de98eed895dbedfe4e5bc70d1502ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e010b00bc4ff395efd11719b277694cface5aa50d085a0bb81f613f70313acd28cf4557010400574b2d9142b8d28b61d88e3b2c3f44d858411356b49a28a4643b6d1a6a092a5201030051a05fc84d531b5d250c23f4f886f6812f9fe3f402d61607f977b4ecd2701c19010000fd781529d58fc2523cf396a7f25440b409857e7e221766c57214b1d38c7b481f01010062f542f45ea3660f86c013ced80534cb5fd4c19d66c56e7e8c5d4bf2d40acc5e010100b121e91836fd7cd5102b654e9f72f3cf6fdbfd0b161c53a9c54b12c841126331020100000001cd4e4cac3c7b56920d1e7655e7e260d31f29d9a388d04910f1bbd72304a79029010000006b483045022100e75279a205a547c445719420aa3138bf14743e3f42618e5f86a19bde14bb95f7022064777d34776b05d816daf1699493fcdf2ef5a5ab1ad710d9c97bfb5b8f7cef3641210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013e660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000001000100000001ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e000000006a47304402203a61a2e931612b4bda08d541cfb980885173b8dcf64a3471238ae7abcd368d6402204cbf24f04b9aa2256d8901f0ed97866603d2be8324c2bfb7a37bf8fc90edd5b441210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013c660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000000' + ).toBEEF() +} diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionStorageOrchestration.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionStorageOrchestration.test.ts new file mode 100644 index 000000000..d5c310053 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoAdmissionStorageOrchestration.test.ts @@ -0,0 +1,111 @@ +import type { Db } from 'mongodb' +import type { AdmissionCommit, AdmissionReceipt } from '../../storage/AdmissionStorage.js' +import { admissionReceiptFor } from '../../storage/mongo/MongoAdmissionPlan.js' +import { MongoAdmissionStorage } from '../../storage/mongo/MongoAdmissionStorage.js' +import type { MongoTransactionRunner } from '../../storage/mongo/MongoTransactionRunner.js' +import { admissionPlan } from '../admission/AdmissionStorageContract.js' +import { referenceScope } from '../admission/ReferenceAdmissionStorage.js' + +// These tests exercise MongoAdmissionStorage's own orchestration logic -- +// write-conflict retry exhaustion, the pending-wait poll loop, and its +// classification of thrown errors -- entirely through the injected +// MongoTransactionRunner seam, with no real MongoDB required. Deep +// applyPlan()-internal behavior (the transaction body itself) is covered +// separately against a real replica set in MongoAdmissionStorage.test.ts. + +const dummyDb = { collection: () => ({}) } as unknown as Db + +const clone = (value: T): T => structuredClone(value) + +function plan(): AdmissionCommit { + const next = clone(admissionPlan('orchestration')) + next.decisions[0].reads = [] + return next +} + +function receiptFor(value: AdmissionCommit): AdmissionReceipt { + return admissionReceiptFor(value, []) +} + +function runner( + overrides: Partial> = {} +): MongoTransactionRunner { + return { + reconcile: + overrides.reconcile ?? (async () => ({ state: 'pending' as const, attemptId: 'unlocated' })), + run: + overrides.run ?? + (async () => ({ + state: 'committed' as const, + receipt: receiptFor(plan()) + })), + close: async () => {} + } as unknown as MongoTransactionRunner +} + +function storage(overrides: ConstructorParameters[2] = {}) { + return new MongoAdmissionStorage(dummyDb, referenceScope, { + runner: runner(), + readGuards: { initialize: async () => {} } as never, + ...overrides + }) +} + +describe('MongoAdmissionStorage write-conflict retry exhaustion and pending-wait', () => { + test('exhausting every write-conflict retry attempt surfaces the last raw conflict rather than looping forever', async () => { + const conflict = { code: 112, message: 'transient' } + const run = jest.fn(async () => { + throw conflict + }) + // The retry loop's own bound (attempt < 8, continuing only while + // attempt < 7) guarantees it always returns/throws by the 8th attempt, + // so the last attempt's write conflict -- not classified into any typed + // rejection code by asResult, since it is not an Error instance -- + // propagates as a rejection of commitAdmission itself. + await expect(storage({ runner: runner({ run }) }).commitAdmission(plan())).rejects.toBe( + conflict + ) + expect(run).toHaveBeenCalledTimes(8) + }) + + test('a stuck pending attempt exhausts its 50-poll wait and commitAdmission then submits a fresh attempt', async () => { + const reconcile = jest.fn(async () => ({ + state: 'pending' as const, + attemptId: 'stuck-forever' + })) + const run = jest.fn(async () => ({ state: 'committed' as const, receipt: receiptFor(plan()) })) + // The pre-existing attempt never resolves to committed/rejected/aborted + // across all 50 polls (waitForPending's own timeout, ~1s of 20ms + // delays), so commitAdmission falls through and submits a brand-new + // attempt via the write-conflict retry path instead of hanging forever. + const result = await storage({ runner: runner({ reconcile, run }) }).commitAdmission(plan()) + expect(result).toEqual({ state: 'committed', receipt: receiptFor(plan()) }) + // The initial reconcile at commit entry, plus 50 polls inside waitForPending. + expect(reconcile).toHaveBeenCalledTimes(51) + expect(run).toHaveBeenCalledTimes(1) + }, 10000) + + test('a non-Mongo, non-object rejection from the transaction body is neither retried nor swallowed', async () => { + const run = jest + .fn() + .mockRejectedValueOnce('disk exploded') + .mockResolvedValue({ + state: 'committed' as const, + receipt: receiptFor(plan()) + }) + await expect(storage({ runner: runner({ run }) }).commitAdmission(plan())).rejects.toBe( + 'disk exploded' + ) + // A bare string is not classified as a write conflict, so there is no retry. + expect(run).toHaveBeenCalledTimes(1) + + const runNull = jest.fn(async () => { + // eslint-disable-next-line no-throw-literal + throw null + }) + await expect( + storage({ runner: runner({ run: runNull }) }).commitAdmission(plan()) + ).rejects.toBe(null) + expect(runNull).toHaveBeenCalledTimes(1) + }) +}) diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoOverlayStorageEviction.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoOverlayStorageEviction.test.ts new file mode 100644 index 000000000..0e2179aeb --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoOverlayStorageEviction.test.ts @@ -0,0 +1,332 @@ +import { randomUUID } from 'node:crypto' +import { Transaction } from '@bsv/sdk' +import type { Output } from '../../Output.js' +import { MongoOverlayStorage } from '../../storage/mongo/MongoOverlayStorage.js' +import { MongoPayloadStore } from '../../storage/mongo/MongoPayloadStore.js' +import { + bootstrapMongoOverlay, + encodeMongoOutputIndex, + encodeMongoUint64, + MongoCollectionNames +} from '../../storage/mongo/MongoSchema.js' +import { createMongoReplicaFixture, type MongoReplicaFixture } from './MongoReplicaFixture.js' +import { admissionSemanticDigest, type AdmissionCommit } from '../../storage/AdmissionStorage.js' + +const BRC62Hex = + '0100beef01fe636d0c0007021400fe507c0c7aa754cef1f7889d5fd395cf1f785dd7de98eed895dbedfe4e5bc70d1502ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e010b00bc4ff395efd11719b277694cface5aa50d085a0bb81f613f70313acd28cf4557010400574b2d9142b8d28b61d88e3b2c3f44d858411356b49a28a4643b6d1a6a092a5201030051a05fc84d531b5d250c23f4f886f6812f9fe3f402d61607f977b4ecd2701c19010000fd781529d58fc2523cf396a7f25440b409857e7e221766c57214b1d38c7b481f01010062f542f45ea3660f86c013ced80534cb5fd4c19d66c56e7e8c5d4bf2d40acc5e010100b121e91836fd7cd5102b654e9f72f3cf6fdbfd0b161c53a9c54b12c841126331020100000001cd4e4cac3c7b56920d1e7655e7e260d31f29d9a388d04910f1bbd72304a79029010000006b483045022100e75279a205a547c445719420aa3138bf14743e3f42618e5f86a19bde14bb95f7022064777d34776b05d816daf1699493fcdf2ef5a5ab1ad710d9c97bfb5b8f7cef3641210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013e660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000001000100000001ac4e164f5bc16746bb0868404292ac8318bbac3800e4aad13a014da427adce3e000000006a47304402203a61a2e931612b4bda08d541cfb980885173b8dcf64a3471238ae7abcd368d6402204cbf24f04b9aa2256d8901f0ed97866603d2be8324c2bfb7a37bf8fc90edd5b441210263e2dee22b1ddc5e11f6fab8bcd2378bdd19580d640501ea956ec0e786f93e76ffffffff013c660000000000001976a9146bfd5c7fbe21529d45803dbcf0c87dd3c71efbc288ac0000000000' + +// Direct access to the private mapping function this finding names: it is +// the single place responsible for never surfacing a non-serving (evicted) +// document as a live output. Every current public read path (findOutput, +// findOutputsForTransaction, findUTXOsForTopic) already filters `evicted` +// out of its Mongo query before a document ever reaches this function, so +// exercising the defect end-to-end through those methods cannot fail on the +// pre-fix code — the mapping bug is only observable by feeding `toOutput` +// the raw (evicted) document directly, which is exactly what a future read +// path that forgets the query-level filter would do. +type ToOutputHost = { + toOutput: (document: Record, includeBEEF: boolean) => Promise +} + +function randomHex(bytes: number): string { + return Array.from({ length: bytes }, () => + Math.floor(Math.random() * 256) + .toString(16) + .padStart(2, '0') + ).join('') +} + +describe('MongoOverlayStorage eviction and consumption-edge hydration', () => { + let fixture: MongoReplicaFixture + let storage: MongoOverlayStorage + + beforeAll(async () => { + fixture = await createMongoReplicaFixture() + await bootstrapMongoOverlay(fixture.db, fixture.scope) + storage = new MongoOverlayStorage(fixture.db, fixture.scope) + }, 120000) + + afterAll(async () => { + await fixture.close() + }, 60000) + + test('toOutput never maps an evicted document to a live output', async () => { + const txid = randomHex(32) + const topic = 'Hello' + const output: Output = { + txid, + outputIndex: 0, + outputScript: [1, 2, 3], + satoshis: 1000, + topic, + spent: false, + outputsConsumed: [], + consumedBy: [] + } + await storage.insertOutput(output) + await storage.deleteOutput(txid, 0, topic) + + const raw = await fixture.db.collection(MongoCollectionNames.outputs).findOne({ + txid, + topic, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash + }) + expect(raw?.state).toBe('evicted') + if (raw === null) throw new Error('expected the evicted document to still exist for audit') + + const mapped = await (storage as unknown as ToOutputHost).toOutput(raw, false) + expect(mapped).toBeNull() + }) + + test('findOutput and findOutputsForTransaction hydrate outputsConsumed/consumedBy from persisted consumption edges', async () => { + const topic = `Edges-${randomUUID()}` + const sourceTxid = randomHex(32) + const consumerTxid = randomHex(32) + + const source: Output = { + txid: sourceTxid, + outputIndex: 0, + outputScript: [9], + satoshis: 500, + topic, + spent: false, + outputsConsumed: [], + consumedBy: [] + } + const consumer: Output = { + txid: consumerTxid, + outputIndex: 0, + outputScript: [7], + satoshis: 400, + topic, + spent: false, + outputsConsumed: [], + consumedBy: [] + } + await storage.insertOutput(source) + await storage.insertOutput(consumer) + + // The admission transaction that created `consumer` also spent `source`; + // this is how both MongoAdmissionStorage.insertEdge (during commit) and + // MongoOverlayStorage.updateConsumedBy (classic path) persist that edge. + await storage.updateConsumedBy(sourceTxid, 0, topic, [{ txid: consumerTxid, outputIndex: 0 }]) + + const foundSource = await storage.findOutput(sourceTxid, 0, topic) + expect(foundSource?.outputsConsumed).toEqual([]) + expect(foundSource?.consumedBy).toEqual([{ txid: consumerTxid, outputIndex: 0 }]) + + const foundConsumer = await storage.findOutput(consumerTxid, 0, topic) + expect(foundConsumer?.outputsConsumed).toEqual([{ txid: sourceTxid, outputIndex: 0 }]) + expect(foundConsumer?.consumedBy).toEqual([]) + + const forTransaction = await storage.findOutputsForTransaction(consumerTxid) + expect(forTransaction).toHaveLength(1) + expect(forTransaction[0].outputsConsumed).toEqual([{ txid: sourceTxid, outputIndex: 0 }]) + }) + + test('uses a caller-supplied payload store instead of constructing its own', async () => { + const real = new MongoPayloadStore(fixture.db, fixture.scope) + const calls: string[] = [] + const payloads = { + publish: async (input: Parameters[0]) => { + calls.push(input.kind) + return await real.publish(input) + } + } as unknown as MongoPayloadStore + const overlay = new MongoOverlayStorage(fixture.db, fixture.scope, { payloads }) + await overlay.publishAdmissionPayload({ kind: 'locking-script', bytes: new Uint8Array([7]) }) + // Proves the constructor actually used the injected store (the + // `options.payloads ?? new MongoPayloadStore(...)` branch) rather than + // silently falling back to one it constructs itself. + expect(calls).toEqual(['locking-script']) + await overlay.close() + }) + + test('insertOutput records a spent output as spent from the start', async () => { + const txid = randomHex(32) + const topic = `Spent-${randomUUID()}` + await storage.insertOutput({ + txid, + outputIndex: 0, + outputScript: [4, 5], + satoshis: 10, + topic, + spent: true, + outputsConsumed: [], + consumedBy: [] + }) + expect(await storage.findOutput(txid, 0, topic, true)).toEqual( + expect.objectContaining({ spent: true }) + ) + expect(await storage.findOutput(txid, 0, topic, false)).toBeNull() + }) + + test('findOutput with includeBEEF leaves beef unset when no transaction row exists for the output', async () => { + const txid = randomHex(32) + const topic = `NoBeef-${randomUUID()}` + // insertOutput without a `beef` field never calls persistTransactionBeef, + // so no row exists in the transactions collection for this txid at all. + await storage.insertOutput({ + txid, + outputIndex: 0, + outputScript: [9, 9], + satoshis: 1, + topic, + spent: false, + outputsConsumed: [], + consumedBy: [] + }) + const found = await storage.findOutput(txid, 0, topic, false, true) + expect(found).not.toBeNull() + expect(found?.beef).toBeUndefined() + }) + + test('readScript returns an empty script when the output document carries no scriptPayloadId', async () => { + const txid = randomHex(32) + const topic = `NoScript-${randomUUID()}` + const now = new Date() + // The live schema validator requires scriptPayloadId on every write, so + // a document missing it can only arrive here as a pre-existing (e.g. + // migrated, or written under an older schema) row -- simulated with + // bypassDocumentValidation rather than a normal insertOutput() + $unset, + // which the validator would reject outright. + await fixture.db.collection(MongoCollectionNames.outputs).insertOne( + { + _id: `no-script-${randomUUID()}`, + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + nodeId: fixture.scope.nodeId, + topic, + txid, + outputIndex: encodeMongoOutputIndex('0'), + satoshis: encodeMongoUint64('1'), + score: encodeMongoUint64('1'), + scriptOffset: encodeMongoUint64('0'), + scriptByteLength: encodeMongoUint64('0'), + state: 'unspent', + version: '1', + createdAt: now, + updatedAt: now + }, + { bypassDocumentValidation: true } + ) + const found = await storage.findOutput(txid, 0, topic) + expect(found?.outputScript).toEqual([]) + }) + + test('findOutput with includeBEEF leaves beef unset when the transaction row names a payload that is no longer ready', async () => { + const tx = Transaction.fromHexBEEF(BRC62Hex) + const txid = tx.id('hex') + const beef = tx.toBEEF() + const topic = `EvictedRaw-${randomUUID()}` + await storage.insertOutput({ + txid, + outputIndex: 0, + outputScript: [1, 1], + satoshis: 1, + topic, + spent: false, + outputsConsumed: [], + consumedBy: [], + beef + }) + // Simulate a raw-transaction payload that has since been garbage + // collected (or is otherwise no longer ready) while the transaction row + // that named it is left in place -- readBeef must fail closed to + // "no BEEF" rather than surfacing a null-pointer or stale bytes. + const transaction = await fixture.db + .collection(MongoCollectionNames.transactions) + .findOne({ txid }) + expect(typeof transaction?.rawPayloadId).toBe('string') + await fixture.db + .collection(MongoCollectionNames.payloads) + .updateOne({ _id: transaction?.rawPayloadId as string }, { $set: { state: 'deleted' } }) + const found = await storage.findOutput(txid, 0, topic, false, true) + expect(found?.beef).toBeUndefined() + }) + + test('readBeef hydrates a merkle path pinned by an admission commit into the returned atomic BEEF', async () => { + const exampleTX = Transaction.fromHexBEEF(BRC62Hex) + const proven = exampleTX.inputs[0].sourceTransaction + if (proven === undefined || proven.merklePath === undefined) { + throw new Error('expected the BEEF ancestor to carry a merkle path') + } + const txid = proven.id('hex') + const topic = `Merkle-${randomUUID()}` + const raw = await storage.publishAdmissionPayload({ + kind: 'raw-transaction', + bytes: Buffer.from(proven.toBinary()), + txid + }) + const merkle = await storage.publishAdmissionPayload({ + kind: 'merkle-path', + bytes: Buffer.from(proven.merklePath.toBinary()) + }) + const script = await storage.publishAdmissionPayload({ + kind: 'locking-script', + bytes: Buffer.from(proven.outputs[0].lockingScript.toBinary()) + }) + const identity = { + scope: storage.admissionScope, + txid, + mode: 'historical' as const, + contextDigest: '00'.repeat(32), + topics: [{ topic, policyId: 'merkle-hydration-test' }] + } + const plan: AdmissionCommit = { + key: { + scope: storage.admissionScope, + operationId: `merkle-hydrate-${txid}`, + semanticDigest: admissionSemanticDigest(identity) + }, + identity, + payloads: [raw], + decisions: [ + { + topic, + expectedHistory: { chainEpoch: '0', topicHistoryGeneration: '0' }, + reads: [], + spends: [], + evictions: [], + outputs: [ + { + txid, + outputIndex: '0', + satoshis: String(proven.outputs[0].satoshis), + score: '1', + script: { payload: script, offset: '0', byteLength: script.byteLength } + } + ], + edges: [], + applied: { txid, proof: merkle, firstSeenHeight: '1' } + } + ], + outbox: [], + steak: JSON.stringify({ + [topic]: { outputsToAdmit: [0], coinsToRetain: [], coinsRemoved: [] } + }) + } + const committed = await storage.admission.commitAdmission(plan) + expect(committed.state).toBe('committed') + const found = await storage.findOutput(txid, 0, topic, false, true) + expect(found?.beef).toBeDefined() + const rebuilt = Transaction.fromBEEF(found?.beef ?? []) + expect(rebuilt.id('hex')).toBe(txid) + expect(rebuilt.merklePath).toBeDefined() + + // The merkle-path *reference* still names this transaction, but the + // payload it points at is no longer ready (GC'd, or never finished + // uploading) -- readBeef must fall back to the plain (non-atomic) BEEF + // rather than crashing or fabricating a merkle path. + await fixture.db + .collection(MongoCollectionNames.payloads) + .updateOne({ _id: storage.admission.payloadId(merkle) }, { $set: { state: 'deleted' } }) + const withoutMerkle = await storage.findOutput(txid, 0, topic, false, true) + expect(withoutMerkle?.beef).toBeDefined() + const rebuiltWithoutMerkle = Transaction.fromBEEF(withoutMerkle?.beef ?? []) + expect(rebuiltWithoutMerkle.id('hex')).toBe(txid) + expect(rebuiltWithoutMerkle.merklePath).toBeUndefined() + }) +}) diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoOverlayStorageHost.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoOverlayStorageHost.test.ts new file mode 100644 index 000000000..a5dff8c31 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoOverlayStorageHost.test.ts @@ -0,0 +1,38 @@ +import { createHash } from 'node:crypto' +import { getOverlayAdmissionHost } from '../../EngineAdmission.js' +import { MongoOverlayStorage } from '../../storage/mongo/MongoOverlayStorage.js' +import { bootstrapMongoOverlay } from '../../storage/mongo/MongoSchema.js' +import { createMongoReplicaFixture, type MongoReplicaFixture } from './MongoReplicaFixture.js' + +describe('MongoOverlayStorage as the Engine admission host', () => { + let fixture: MongoReplicaFixture + let storage: MongoOverlayStorage + + beforeAll(async () => { + fixture = await createMongoReplicaFixture() + await bootstrapMongoOverlay(fixture.db, fixture.scope) + storage = new MongoOverlayStorage(fixture.db, fixture.scope) + }, 120000) + + afterAll(async () => { + await fixture.close() + }, 60000) + + test('publishes a payload through the host object the Engine uses', async () => { + const host = getOverlayAdmissionHost(storage) + expect(host?.publishAdmissionPayload).toBeDefined() + const bytes = Uint8Array.of(9, 8, 7, 6) + const ref = await host?.publishAdmissionPayload?.({ kind: 'locking-script', bytes }) + expect(ref).toMatchObject({ + kind: 'locking-script', + digest: createHash('sha256').update(bytes).digest('hex'), + byteLength: '4' + }) + }, 30000) + + test('reads the history fence through the host object', async () => { + const host = getOverlayAdmissionHost(storage) + const fence = await host?.getHistoryFence?.('tm_host_binding') + expect(fence).toEqual(await storage.getHistoryFence('tm_host_binding')) + }, 30000) +}) diff --git a/packages/overlays/overlay/src/__tests/mongo/MongoPayloadStoreRead.test.ts b/packages/overlays/overlay/src/__tests/mongo/MongoPayloadStoreRead.test.ts new file mode 100644 index 000000000..b54fbf3c9 --- /dev/null +++ b/packages/overlays/overlay/src/__tests/mongo/MongoPayloadStoreRead.test.ts @@ -0,0 +1,193 @@ +import { createHash, randomUUID } from 'node:crypto' +import { ObjectId } from 'mongodb' +import { + bootstrapMongoOverlay, + encodeMongoUint64, + mongoChainKey, + mongoRecordKey, + MongoCollectionNames +} from '../../storage/mongo/MongoSchema.js' +import { MongoPayloadStore } from '../../storage/mongo/MongoPayloadStore.js' +import { createMongoReplicaFixture, type MongoReplicaFixture } from './MongoReplicaFixture.js' + +// Scoped to MongoPayloadStore.read()/readBytes() specifically (per the +// worklist for this PR). Every other MongoPayloadStore.ts line belongs to a +// different PR's worklist and is covered in MongoPayloadStore.test.ts on its +// own branch; this file stays separate so the two never collide on a merge. + +const bytes = async function* ( + value: Uint8Array, + split = value.byteLength +): AsyncIterable { + for (let index = 0; index < value.byteLength; index += split) + yield value.subarray(index, index + split) +} + +const digest = (value: Uint8Array): string => createHash('sha256').update(value).digest('hex') + +describe('MongoPayloadStore.read() and readBytes()', () => { + let fixture: MongoReplicaFixture + let store: MongoPayloadStore + + beforeAll(async () => { + fixture = await createMongoReplicaFixture() + await bootstrapMongoOverlay(fixture.db, fixture.scope) + store = new MongoPayloadStore(fixture.db, fixture.scope) + }, 120000) + + afterAll(async () => { + await fixture.close() + }, 30000) + + test('read() throws when no ready payload exists for the requested content', async () => { + await expect(store.read({ kind: 'locking-script', digest: 'aa'.repeat(32) })).rejects.toThrow( + 'Mongo payload is not ready' + ) + }) + + test('read() rejects a range whose offset or length is negative, or whose end exceeds the content', async () => { + const content = Buffer.from('range-guard-content') + const hash = digest(content) + await store.publish({ + kind: 'locking-script', + digest: hash, + byteLength: String(content.byteLength), + bytes: bytes(content) + }) + const ref = { kind: 'locking-script' as const, digest: hash } + + await expect( + store.read(ref, { offset: '-1', byteLength: String(content.byteLength) }) + ).rejects.toThrow('Mongo payload range is invalid') + await expect(store.read(ref, { offset: '0', byteLength: '-1' })).rejects.toThrow( + 'Mongo payload range is invalid' + ) + await expect( + store.read(ref, { offset: '1', byteLength: String(content.byteLength) }) + ).rejects.toThrow('Mongo payload range is invalid') + + // A range that fits exactly still succeeds, proving the guard is not + // simply rejecting every explicit range. + const whole = await store.read(ref, { offset: '0', byteLength: String(content.byteLength) }) + expect(Buffer.from(whole).toString()).toBe(content.toString()) + }) + + test('read() rejects a range whose end would exceed a safe JavaScript integer', async () => { + // Directly seed a "ready" payload record whose declared byteLength is + // enormous (but a legal uint64), without ever streaming that much real + // content -- read() must fail closed on the arithmetic before it ever + // attempts to fetch bytes. + const hugeLength = '18446744073709551615' + const id = mongoRecordKey(mongoChainKey(fixture.scope), 'locking-script', 'bb'.repeat(32)) + const now = new Date() + await fixture.db.collection(MongoCollectionNames.payloads).insertOne({ + _id: id, + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + kind: 'locking-script', + digest: 'bb'.repeat(32), + byteLength: encodeMongoUint64(hugeLength), + state: 'ready', + guard: randomUUID(), + ownerNodeId: fixture.scope.nodeId, + ownerId: randomUUID(), + fencingToken: encodeMongoUint64('1'), + leaseUntil: now, + createdAt: now, + updatedAt: now + }) + await expect( + store.read( + { kind: 'locking-script', digest: 'bb'.repeat(32) }, + { offset: '0', byteLength: hugeLength } + ) + ).rejects.toThrow('Mongo payload range exceeds safe integer') + }) + + test('readBytes() downloads content that spilled to GridFS across several chunks', async () => { + const tight = new MongoPayloadStore(fixture.db, fixture.scope, { + inlineCeilingBytes: 16, + gridFsChunkBytes: 8 * 1024 + }) + const content = Buffer.from( + 'this content is well over the sixteen byte inline ceiling for this test' + ) + const hash = digest(content) + const published = await tight.publish({ + kind: 'outbox-data', + digest: hash, + byteLength: String(content.byteLength), + bytes: bytes(content, 10) + }) + const row = await fixture.db + .collection(MongoCollectionNames.payloads) + .findOne({ digest: hash, kind: 'outbox-data' }) + expect(row?.fileId).toBeDefined() + expect(row?.inlineData).toBeUndefined() + + const roundTripped = await tight.read(published) + expect(Buffer.from(roundTripped).toString()).toBe(content.toString()) + + // A partial, mid-file range read must still line up byte-for-byte. + const slice = await tight.read(published, { offset: '5', byteLength: '10' }) + expect(Buffer.from(slice).toString()).toBe(content.subarray(5, 15).toString()) + }) + + test('readBytes() surfaces a download failure when the GridFS file behind a ready record is gone', async () => { + const now = new Date() + const missingFileId = new ObjectId() + const id = mongoRecordKey(mongoChainKey(fixture.scope), 'outbox-data', 'cc'.repeat(32)) + await fixture.db.collection(MongoCollectionNames.payloads).insertOne({ + _id: id, + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + kind: 'outbox-data', + digest: 'cc'.repeat(32), + byteLength: encodeMongoUint64('4'), + state: 'ready', + guard: randomUUID(), + ownerNodeId: fixture.scope.nodeId, + ownerId: randomUUID(), + fencingToken: encodeMongoUint64('1'), + leaseUntil: now, + createdAt: now, + updatedAt: now, + // Points at a GridFS file that was never uploaded (deleted/evicted), + // simulating a ready record left behind after its content was reclaimed. + fileId: missingFileId + }) + await expect(store.read({ kind: 'outbox-data', digest: 'cc'.repeat(32) })).rejects.toThrow( + /FileNotFound/ + ) + }) + + test('readBytes() fails closed when a ready record carries neither inline data nor a GridFS file', async () => { + const now = new Date() + const id = mongoRecordKey(mongoChainKey(fixture.scope), 'outbox-data', 'dd'.repeat(32)) + // A record can only reach `ready` through completeUpload(), which always + // sets exactly one of inlineData/fileId -- this simulates a corrupted or + // hand-edited row that skipped that invariant. + await fixture.db.collection(MongoCollectionNames.payloads).insertOne({ + _id: id, + schemaVersion: 1, + network: fixture.scope.network, + genesisHash: fixture.scope.genesisHash, + kind: 'outbox-data', + digest: 'dd'.repeat(32), + byteLength: encodeMongoUint64('4'), + state: 'ready', + guard: randomUUID(), + ownerNodeId: fixture.scope.nodeId, + ownerId: randomUUID(), + fencingToken: encodeMongoUint64('1'), + leaseUntil: now, + createdAt: now, + updatedAt: now + }) + await expect(store.read({ kind: 'outbox-data', digest: 'dd'.repeat(32) })).rejects.toThrow( + 'Mongo ready payload has no bytes' + ) + }) +}) diff --git a/packages/overlays/overlay/src/storage/AdmissionStorage.ts b/packages/overlays/overlay/src/storage/AdmissionStorage.ts index a7c34959a..653e19483 100644 --- a/packages/overlays/overlay/src/storage/AdmissionStorage.ts +++ b/packages/overlays/overlay/src/storage/AdmissionStorage.ts @@ -201,6 +201,11 @@ export function parseStorageUint64(value: string): bigint { return result } +export function asStorageUint64(value: string): StorageUint64 { + parseStorageUint64(value) + return value as StorageUint64 +} + /** Validate the wire outpoint domain before conversion to an exact STEAK number. */ export function parseStorageOutputIndex(value: string): number { const result = parseStorageUint64(value) diff --git a/packages/overlays/overlay/src/storage/mongo.ts b/packages/overlays/overlay/src/storage/mongo.ts index d73a111ef..a64a93056 100644 --- a/packages/overlays/overlay/src/storage/mongo.ts +++ b/packages/overlays/overlay/src/storage/mongo.ts @@ -3,3 +3,6 @@ export * from './mongo/MongoSchema.js' export * from './mongo/MongoPayloadStore.js' export * from './mongo/MongoReadGuards.js' export * from './mongo/MongoTransactionRunner.js' +export * from './mongo/MongoAdmissionPlan.js' +export * from './mongo/MongoAdmissionStorage.js' +export * from './mongo/MongoOverlayStorage.js' diff --git a/packages/overlays/overlay/src/storage/mongo/MongoAdmissionPlan.ts b/packages/overlays/overlay/src/storage/mongo/MongoAdmissionPlan.ts new file mode 100644 index 000000000..df3bc8f01 --- /dev/null +++ b/packages/overlays/overlay/src/storage/mongo/MongoAdmissionPlan.ts @@ -0,0 +1,268 @@ +import { + admissionSemanticDigest, + parseStorageOutputIndex, + parseStorageUint64, + type AdmissionCommit, + type AdmissionCommitResult, + type AdmissionOutboxIntent, + type AdmissionOutpoint, + type AdmissionOutput, + type AdmissionPayloadRef, + type AdmissionReceipt, + type AdmissionTopicDecision, + type StorageScope +} from '../AdmissionStorage.js' + +export type AdmissionRejectionCode = Extract['code'] + +export class AdmissionRejectedError extends Error { + readonly code: AdmissionRejectionCode + + constructor(code: AdmissionRejectionCode) { + super(code) + this.name = 'AdmissionRejectedError' + this.code = code + } +} + +export const rejectAdmission = (code: AdmissionRejectionCode): never => { + throw new AdmissionRejectedError(code) +} + +const isHash = (value: string): boolean => /^[0-9a-f]{64}$/.test(value) + +const isUint64 = (value: string): boolean => { + try { + parseStorageUint64(value) + return true + } catch { + return false + } +} + +const isWireOutpoint = (outpoint: AdmissionOutpoint): boolean => { + if (!isHash(outpoint.txid)) return false + try { + parseStorageOutputIndex(outpoint.outputIndex) + return true + } catch { + return false + } +} + +export const sameScope = (left: StorageScope, right: StorageScope): boolean => + left.network === right.network && + left.genesisHash === right.genesisHash && + left.nodeId === right.nodeId + +export const samePayload = ( + left: AdmissionPayloadRef | undefined, + right: AdmissionPayloadRef +): boolean => + left?.digest === right.digest && + left?.byteLength === right.byteLength && + left?.kind === right.kind + +export const admissionPlanPayloads = (plan: AdmissionCommit): AdmissionPayloadRef[] => { + const references = [...plan.payloads] + for (const decision of plan.decisions) { + for (const output of decision.outputs) references.push(output.script.payload) + if (decision.applied.proof !== undefined) references.push(decision.applied.proof) + } + for (const intent of plan.outbox) references.push(...intent.payloads) + return references +} + +const isSteakEntry = (entry: unknown): entry is { outputsToAdmit: number[] } => { + if (typeof entry !== 'object' || entry === null || Array.isArray(entry)) return false + const result = entry as Record + if (!Array.isArray(result.outputsToAdmit)) return false + if (result.coinsToRetain !== undefined && !Array.isArray(result.coinsToRetain)) return false + if (result.coinsRemoved !== undefined && !Array.isArray(result.coinsRemoved)) return false + return !result.outputsToAdmit.some( + value => + typeof value !== 'number' || !Number.isInteger(value) || value < 0 || value > 4294967295 + ) +} + +export const isBoundSteak = (plan: AdmissionCommit): boolean => { + if (typeof plan.steak !== 'string' || !plan.steak.isWellFormed()) return false + let steak: unknown + try { + steak = JSON.parse(plan.steak) + } catch { + return false + } + if (typeof steak !== 'object' || steak === null || Array.isArray(steak)) return false + const record = steak as Record + for (const entry of Object.values(record)) { + if (!isSteakEntry(entry)) return false + } + return plan.decisions.every(decision => { + const entry = record[decision.topic] + if (!isSteakEntry(entry)) return false + try { + const expected = decision.outputs.map(output => parseStorageOutputIndex(output.outputIndex)) + return JSON.stringify(entry.outputsToAdmit) === JSON.stringify(expected) + } catch { + return false + } + }) +} + +const hasInvalidTopicSet = (plan: AdmissionCommit): boolean => { + const topics = new Set(plan.identity.topics.map(item => item.topic)) + const decisionTopics = new Set(plan.decisions.map(decision => decision.topic)) + return ( + topics.size !== plan.identity.topics.length || + plan.decisions.length !== topics.size || + decisionTopics.size !== plan.decisions.length || + decisionTopics.size !== topics.size || + plan.decisions.some(decision => !topics.has(decision.topic)) || + new Set(plan.outbox.map(intent => intent.eventId)).size !== plan.outbox.length + ) +} + +const hasHistoricalPropagation = (plan: AdmissionCommit): boolean => + plan.identity.mode === 'historical' && plan.outbox.some(intent => intent.kind === 'propagation') + +const identityRejection = (plan: AdmissionCommit): AdmissionRejectionCode | undefined => { + let semanticDigest: string + try { + semanticDigest = admissionSemanticDigest(plan.identity) + } catch { + return 'digest-mismatch' + } + if ( + plan.key.semanticDigest !== semanticDigest || + !sameScope(plan.key.scope, plan.identity.scope) + ) { + return 'digest-mismatch' + } + if (hasInvalidTopicSet(plan) || hasHistoricalPropagation(plan)) return 'invalid-plan' + return undefined +} + +const hasUnreadyPayloads = (plan: AdmissionCommit): boolean => + admissionPlanPayloads(plan).some( + ref => !isHash(ref.digest) || !isUint64(ref.byteLength) || ref.digest.length === 0 + ) + +const isInvalidOutboxIntent = (intent: AdmissionOutboxIntent): boolean => + intent.eventId.length === 0 || + !intent.eventId.isWellFormed() || + intent.target.length === 0 || + !intent.target.isWellFormed() || + (intent.kind !== 'lookup' && intent.kind !== 'propagation') + +const isInvalidSpend = ( + spend: { outpoint: AdmissionOutpoint; spender: string }, + txid: string +): boolean => !isWireOutpoint(spend.outpoint) || spend.spender !== txid + +const isInvalidEdge = (edge: { + source: AdmissionOutpoint + consumer: AdmissionOutpoint +}): boolean => !isWireOutpoint(edge.source) || !isWireOutpoint(edge.consumer) + +const isInvalidOutput = (output: AdmissionOutput, txid: string): boolean => { + if (output.txid !== txid || !isWireOutpoint(output)) return true + if ( + ![output.satoshis, output.score, output.script.offset, output.script.byteLength].every(isUint64) + ) { + return true + } + if (!isHash(output.script.payload.digest) || !isUint64(output.script.payload.byteLength)) { + return true + } + return ( + parseStorageUint64(output.script.offset) + parseStorageUint64(output.script.byteLength) > + parseStorageUint64(output.script.payload.byteLength) + ) +} + +const isInvalidApplied = (applied: AdmissionTopicDecision['applied'], txid: string): boolean => { + if (applied.txid !== txid || !isHash(applied.txid)) return true + if (applied.firstSeenHeight !== undefined && !isUint64(applied.firstSeenHeight)) return true + if ( + applied.proof !== undefined && + (!isHash(applied.proof.digest) || !isUint64(applied.proof.byteLength)) + ) { + return true + } + return ( + applied.block !== undefined && + (![applied.block.height, applied.block.index].every(isUint64) || + !isHash(applied.block.hash) || + !isHash(applied.block.merkleRoot)) + ) +} + +const isInvalidHistoryUpdate = (decision: AdmissionTopicDecision): boolean => { + const update = decision.historyUpdate + if (update === undefined) return false + return ( + !isUint64(update.nextTopicHistoryGeneration) || + !isUint64(update.affectedFromHeight) || + parseStorageUint64(update.nextTopicHistoryGeneration) <= + parseStorageUint64(decision.expectedHistory.topicHistoryGeneration) + ) +} + +const isInvalidDecision = (decision: AdmissionTopicDecision, txid: string): boolean => { + if ( + !isUint64(decision.expectedHistory.chainEpoch) || + !isUint64(decision.expectedHistory.topicHistoryGeneration) + ) { + return true + } + if (decision.spends.some(spend => isInvalidSpend(spend, txid))) return true + if (decision.evictions.some(eviction => !isWireOutpoint(eviction))) return true + if (decision.outputs.some(output => isInvalidOutput(output, txid))) return true + if (decision.edges.some(isInvalidEdge)) return true + return isInvalidApplied(decision.applied, txid) || isInvalidHistoryUpdate(decision) +} + +export const validateAdmissionPlan = ( + plan: AdmissionCommit +): AdmissionRejectionCode | undefined => { + const rejected = identityRejection(plan) + if (rejected !== undefined) return rejected + if (hasUnreadyPayloads(plan)) return 'payload-not-ready' + if (!isBoundSteak(plan)) return 'invalid-plan' + if (plan.outbox.some(isInvalidOutboxIntent)) return 'invalid-plan' + if (plan.decisions.some(decision => isInvalidDecision(decision, plan.identity.txid))) { + return 'invalid-plan' + } + return undefined +} + +export const admissionReceiptFor = ( + plan: AdmissionCommit, + enlistedTargets: readonly string[] +): AdmissionReceipt => { + const enlisted = new Set(enlistedTargets) + const lookupTargets = [ + ...new Set(plan.outbox.filter(intent => intent.kind === 'lookup').map(intent => intent.target)) + ] + if (lookupTargets.some(target => enlisted.has(target))) rejectAdmission('invalid-plan') + return { + operationId: plan.key.operationId, + semanticDigest: plan.key.semanticDigest, + durability: 'atomic-local', + steak: plan.steak, + indexes: [ + ...enlistedTargets.map(target => ({ target, state: 'visible' as const })), + ...lookupTargets.map(target => ({ target, state: 'pending' as const })) + ], + propagation: plan.outbox.some(intent => intent.kind === 'propagation') + ? 'pending' + : 'not-requested' + } +} + +export const lookupOutboxIntents = (plan: AdmissionCommit): AdmissionOutboxIntent[] => + plan.outbox.filter(intent => intent.kind === 'lookup') + +export const propagationOutboxIntents = (plan: AdmissionCommit): AdmissionOutboxIntent[] => + plan.outbox.filter(intent => intent.kind === 'propagation') diff --git a/packages/overlays/overlay/src/storage/mongo/MongoAdmissionStorage.ts b/packages/overlays/overlay/src/storage/mongo/MongoAdmissionStorage.ts new file mode 100644 index 000000000..0895fa620 --- /dev/null +++ b/packages/overlays/overlay/src/storage/mongo/MongoAdmissionStorage.ts @@ -0,0 +1,1020 @@ +import { createHash } from 'node:crypto' +import { setTimeout as delay } from 'node:timers/promises' +import type { Collection, Db, Document } from 'mongodb' +import { + admissionSemanticDigest, + asStorageUint64, + isReplaySafeProjection, + type AdmissionCommit, + type AdmissionCommitResult, + type AdmissionOperationKey, + type AdmissionOutboxIntent, + type AdmissionOutpoint, + type AdmissionPayloadRef, + type AdmissionReconcileResult, + type AdmissionStorage, + type AdmissionTopicDecision, + type StorageScope +} from '../AdmissionStorage.js' +import type { RecoveryLease } from '../RecoveryContract.js' +import { + AdmissionRejectedError, + admissionPlanPayloads, + admissionReceiptFor, + lookupOutboxIntents, + propagationOutboxIntents, + rejectAdmission, + sameScope, + validateAdmissionPlan +} from './MongoAdmissionPlan.js' +import { MongoPayloadStore, type MongoPayloadKind } from './MongoPayloadStore.js' +import { MongoReadGuardConflictError, MongoReadGuards } from './MongoReadGuards.js' +import { + MongoCollectionNames, + encodeMongoOutputIndex, + encodeMongoUint64, + decodeMongoUint64, + mongoChainKey, + mongoNodeKey, + mongoRecordKey +} from './MongoSchema.js' +import { + MongoTransactionRunner, + type MongoTransactionContext, + type MongoTransactionOptions +} from './MongoTransactionRunner.js' + +function isMongoWriteConflict(error: unknown): boolean { + if (typeof error !== 'object' || error === null) return false + const code = 'code' in error ? error.code : undefined + if (code === 112 || code === '112') return true + if ('codeName' in error && error.codeName === 'WriteConflict') return true + const text = [ + error instanceof Error ? error.message : '', + 'errmsg' in error ? String(error.errmsg) : '' + ].join(' ') + return text.includes('Write conflict') +} + +export interface MongoEnlistedLookupIndex { + readonly protocol: 'overlay-mongo-index-v1' + readonly target: string + apply: (context: MongoTransactionContext, plan: AdmissionCommit) => Promise +} + +export interface MongoAdmissionStorageOptions { + runner?: MongoTransactionRunner + payloads?: MongoPayloadStore + readGuards?: MongoReadGuards + enlistedIndexes?: MongoEnlistedLookupIndex[] + projector?: unknown +} + +export interface MongoOutboxLease { + eventId: string + kind: 'lookup' | 'propagation' + target: string + payloads: AdmissionPayloadRef[] +} + +interface OutputDocument extends Document { + _id: string + topic: string + txid: string + outputIndex: string + satoshis: string + score: string + scriptPayloadId: string + scriptOffset: string + scriptByteLength: string + state: 'unspent' | 'spent' | 'evicted' + spender?: string + version: string +} + +interface AppliedDocument extends Document { + _id: string + topic: string + txid: string + state: 'active' | 'unproven' | 'evicted' + admissionId: string + firstSeenHeight?: string + proofPayloadId?: string +} + +interface GenerationDocument extends Document { + _id: string + topic: string + chainEpoch: string + topicHistoryGeneration: string + policyId: string +} + +interface JobDocument extends Document { + _id: string + topic: string + peerId: string + jobId: string + chainEpoch: string + topicHistoryGeneration: string + leaseToken: string + leaseUntil: Date + state: string + checkpoint: string +} + +interface OutboxDocument extends Document { + _id: string + eventId: string + target: string + state: string + nextAttemptAt?: Date + leaseUntil?: Date +} + +interface IdDocument extends Document { + _id: string + [key: string]: unknown +} + +const majority = { w: 'majority' as const, j: true } +const schemaVersion = 1 + +const duplicateKey = (error: unknown): boolean => + typeof error === 'object' && error !== null && 'code' in error && error.code === 11000 + +const isPayloadKind = (value: string): value is MongoPayloadKind => + value === 'raw-transaction' || + value === 'merkle-path' || + value === 'beef-manifest' || + value === 'locking-script' || + value === 'outbox-data' + +/** + * Opt-in Mongo AdmissionStorage. Engine submit uses this only when + * getAdmissionStorage(storage) observes overlay-admission-v1. + */ +export class MongoAdmissionStorage implements AdmissionStorage { + readonly protocol = 'overlay-admission-v1' as const + readonly scope: StorageScope + private readonly runner: MongoTransactionRunner + private readonly payloads: MongoPayloadStore + private readonly guards: MongoReadGuards + private readonly enlisted: MongoEnlistedLookupIndex[] + private readonly projector: unknown + private readonly peers = new Map() + + constructor( + private readonly db: Db, + scope: StorageScope, + options: MongoAdmissionStorageOptions = {} + ) { + this.scope = { ...scope } + this.runner = options.runner ?? new MongoTransactionRunner(db, this.scope) + this.payloads = options.payloads ?? new MongoPayloadStore(db, this.scope) + this.guards = options.readGuards ?? new MongoReadGuards(db) + this.enlisted = [...(options.enlistedIndexes ?? [])] + this.projector = options.projector + if ( + this.enlisted.some( + index => index.protocol !== 'overlay-mongo-index-v1' || index.target.length === 0 + ) + ) { + throw new Error('Invalid enlisted Mongo lookup index') + } + if (new Set(this.enlisted.map(index => index.target)).size !== this.enlisted.length) { + throw new Error('Duplicate enlisted Mongo lookup index target') + } + } + + enlistedTargets(): string[] { + return this.enlisted.map(index => index.target) + } + + async commitAdmission(plan: AdmissionCommit): Promise { + if (!sameScope(plan.key.scope, this.scope) || !sameScope(plan.identity.scope, this.scope)) { + return await this.peerFor(plan.key.scope).commitAdmission(plan) + } + let semanticDigest: string + try { + semanticDigest = admissionSemanticDigest(plan.identity) + } catch { + return { state: 'rejected', code: 'digest-mismatch' } + } + if ( + plan.key.semanticDigest !== semanticDigest || + !sameScope(plan.key.scope, plan.identity.scope) + ) { + return { state: 'rejected', code: 'digest-mismatch' } + } + if (this.projector !== undefined && !isReplaySafeProjection(this.projector)) { + return { state: 'rejected', code: 'unsupported-projection' } + } + const existing = await this.runner.reconcile(plan.key) + if (existing.state === 'committed' || existing.state === 'rejected') return existing + if (existing.state === 'pending' && existing.attemptId !== 'unlocated') { + const waited = await this.waitForPending(plan.key, existing.attemptId) + if (waited.state === 'committed' || waited.state === 'rejected') return waited + } + const rejected = validateAdmissionPlan(plan) + if (rejected !== undefined) return { state: 'rejected', code: rejected } + let receipt + try { + receipt = admissionReceiptFor(plan, this.enlistedTargets()) + } catch (error) { + return this.asResult(error) + } + await this.prepareReadGuards(plan) + await this.publishHistoryUpdatePayloads(plan) + return await this.runAdmissionWithWriteConflictRetry(plan, receipt) + } + + private async runAdmissionWithWriteConflictRetry( + plan: AdmissionCommit, + receipt: ReturnType + ): Promise { + for (let attempt = 0; attempt < 8; attempt += 1) { + try { + const result = await this.runner.run( + { key: plan.key, identity: plan.identity, receipt }, + async context => { + await this.applyPlan(context, plan) + } + ) + if (result.state !== 'pending') return result + return await this.waitForPending(plan.key, result.attemptId) + } catch (error) { + if (isMongoWriteConflict(error) && attempt < 7) continue + return this.asResult(error) + } + } + return { state: 'rejected', code: 'spend-conflict' } + } + + private async waitForPending( + key: AdmissionOperationKey, + attemptId: string + ): Promise { + let current: AdmissionCommitResult = { state: 'pending', attemptId } + for (let index = 0; index < 50 && current.state === 'pending'; index += 1) { + await delay(20) + const reconciled = await this.runner.reconcile( + key, + current.state === 'pending' ? current.attemptId : attemptId + ) + if (reconciled.state === 'committed' || reconciled.state === 'rejected') return reconciled + if (reconciled.state === 'aborted') { + return { state: 'pending', attemptId } + } + current = reconciled + } + return current + } + + async reconcileAdmission( + key: AdmissionOperationKey, + attemptId?: string, + options: MongoTransactionOptions = {} + ): Promise { + if (!sameScope(key.scope, this.scope)) + return await this.peerFor(key.scope).reconcileAdmission(key, attemptId, options) + return await this.runner.reconcile(key, attemptId, options) + } + + async close(): Promise { + await Promise.all([ + this.runner.close(), + ...[...this.peers.values()].map(async peer => await peer.close()) + ]) + } + + private peerFor(scope: StorageScope): MongoAdmissionStorage { + const id = mongoNodeKey(scope) + const existing = this.peers.get(id) + if (existing !== undefined) return existing + const peer = new MongoAdmissionStorage(this.db, scope, { + enlistedIndexes: this.enlisted, + projector: this.projector + }) + this.peers.set(id, peer) + return peer + } + + async claimOutbox( + kind: 'lookup' | 'propagation', + leaseMS = 15_000 + ): Promise { + if (!Number.isSafeInteger(leaseMS) || leaseMS < 1 || leaseMS > 60_000) { + throw new Error('Invalid Mongo outbox lease') + } + const claimed = await this.outbox(kind).findOneAndUpdate( + { + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + $or: [{ state: 'pending' }, { state: 'leased', $expr: { $lte: ['$leaseUntil', '$$NOW'] } }] + }, + [ + { + $set: { + state: 'leased', + leaseUntil: { $dateAdd: { startDate: '$$NOW', unit: 'millisecond', amount: leaseMS } }, + nextAttemptAt: { + $dateAdd: { startDate: '$$NOW', unit: 'millisecond', amount: leaseMS } + }, + updatedAt: '$$NOW' + } + } + ], + { returnDocument: 'after', writeConcern: majority } + ) + if (claimed === null) return null + return { + eventId: claimed.eventId, + kind, + target: claimed.target, + payloads: await this.outboxPayloads(kind, claimed.eventId) + } + } + + async acknowledgeOutbox(kind: 'lookup' | 'propagation', eventId: string): Promise { + const result = await this.outbox(kind).updateOne( + { + _id: this.outboxId(kind, eventId), + eventId, + state: { $in: ['leased', 'delivered'] } + }, + { $set: { state: 'delivered', updatedAt: new Date() } }, + { writeConcern: majority } + ) + if (result.matchedCount !== 1) throw new Error('Mongo outbox event is not leased') + } + + private asResult(error: unknown): AdmissionCommitResult { + if (error instanceof AdmissionRejectedError) return { state: 'rejected', code: error.code } + if (error instanceof MongoReadGuardConflictError) + return { state: 'rejected', code: 'read-conflict' } + if (error instanceof Error && error.message === 'Mongo payload is not ready for reference') { + return { state: 'rejected', code: 'payload-not-ready' } + } + throw error + } + + private async prepareReadGuards(plan: AdmissionCommit): Promise { + for (const decision of plan.decisions) { + for (const read of decision.reads) await this.guards.initialize(this.scope, read.key) + } + } + + private async publishHistoryUpdatePayloads(plan: AdmissionCommit): Promise { + for (const decision of plan.decisions) { + if (decision.historyUpdate === undefined) continue + const bytes = Buffer.from(JSON.stringify(this.historyUpdateRecord(decision)), 'utf8') + const digest = createHash('sha256').update(bytes).digest('hex') + await this.payloads.publish({ + kind: 'outbox-data', + digest, + byteLength: String(bytes.byteLength), + bytes: (async function* () { + yield bytes + })() + }) + } + } + + private historyUpdateRecord(decision: AdmissionTopicDecision): { + affectedFromHeight: string + checkpoint?: string + } { + if (decision.historyUpdate === undefined) return { affectedFromHeight: '0' } + return { + affectedFromHeight: decision.historyUpdate.affectedFromHeight, + ...(decision.historyUpdate.handoff === undefined + ? {} + : { checkpoint: decision.historyUpdate.handoff.checkpoint }) + } + } + + private async applyPlan(context: MongoTransactionContext, plan: AdmissionCommit): Promise { + await this.assertReadyPayloads(context, plan) + for (const decision of plan.decisions) { + await this.checkReads(context, decision) + await this.checkHistory(context, decision) + await this.assertAppliedAvailable(context, plan, decision) + } + for (const intent of plan.outbox) await this.assertOutboxAvailable(context, intent) + for (const decision of plan.decisions) { + for (const spend of decision.spends) await this.applySpend(context, decision.topic, spend) + for (const eviction of decision.evictions) + await this.applyEviction(context, decision.topic, eviction) + for (const output of decision.outputs) await this.insertOutput(context, decision, output) + for (const edge of decision.edges) await this.insertEdge(context, decision.topic, edge) + await this.insertApplied(context, plan, decision) + await this.applyHistoryUpdate( + context, + decision, + plan.identity.topics.find(item => item.topic === decision.topic)?.policyId ?? 'default' + ) + } + await this.upsertTransaction(context, plan) + for (const ref of admissionPlanPayloads(plan)) { + await this.pin(context, ref, 'transaction', plan.identity.txid, `${ref.kind}:${ref.digest}`) + } + for (const intent of lookupOutboxIntents(plan)) + await this.insertOutbox(context, 'lookup', intent) + for (const intent of propagationOutboxIntents(plan)) { + await this.insertOutbox(context, 'propagation', intent) + } + for (const index of this.enlisted) await index.apply(context, plan) + } + + private async assertReadyPayloads( + context: MongoTransactionContext, + plan: AdmissionCommit + ): Promise { + const seen = new Set() + for (const ref of admissionPlanPayloads(plan)) { + const id = this.payloadId(ref) + if (seen.has(id)) continue + seen.add(id) + const found = await this.db + .collection(MongoCollectionNames.payloads) + .findOne({ _id: id, state: 'ready', digest: ref.digest, kind: ref.kind }, context.options()) + if (found === null || decodeMongoUint64(String(found.byteLength)) !== ref.byteLength) { + rejectAdmission('payload-not-ready') + } + } + } + + private async checkReads( + context: MongoTransactionContext, + decision: AdmissionTopicDecision + ): Promise { + for (const read of decision.reads) { + const options = context.options() + await this.guards.check( + options.session, + { scope: this.scope, key: read.key, expectedVersion: read.expectedVersion }, + { timeoutMS: options.timeoutMS, signal: context.signal } + ) + } + } + + private async checkHistory( + context: MongoTransactionContext, + decision: AdmissionTopicDecision + ): Promise { + const document = await this.generations().findOne( + { _id: this.generationId(decision.topic) }, + context.options() + ) + const epoch = document === null ? '0' : decodeMongoUint64(document.chainEpoch) + const generation = document === null ? '0' : decodeMongoUint64(document.topicHistoryGeneration) + if ( + epoch !== decision.expectedHistory.chainEpoch || + generation !== decision.expectedHistory.topicHistoryGeneration + ) { + rejectAdmission('read-conflict') + } + } + + private async assertAppliedAvailable( + context: MongoTransactionContext, + plan: AdmissionCommit, + decision: AdmissionTopicDecision + ): Promise { + const existing = await this.applied().findOne( + { _id: this.appliedId(decision.topic, decision.applied.txid) }, + context.options() + ) + if (existing !== null && existing.admissionId !== plan.key.operationId) { + rejectAdmission('invalid-plan') + } + } + + private async assertOutboxAvailable( + context: MongoTransactionContext, + intent: AdmissionOutboxIntent + ): Promise { + const existing = await this.outbox(intent.kind).findOne( + { _id: this.outboxId(intent.kind, intent.eventId) }, + context.options() + ) + if (existing !== null) rejectAdmission('invalid-plan') + } + + private async applySpend( + context: MongoTransactionContext, + topic: string, + spend: { outpoint: AdmissionOutpoint; expectedVersion: string; spender: string } + ): Promise { + const id = this.outputId(topic, spend.outpoint) + const result = await this.outputs().findOneAndUpdate( + { + _id: id, + version: spend.expectedVersion, + state: 'unspent' + }, + { + $set: { + state: 'spent', + spender: spend.spender, + updatedAt: new Date() + } + }, + { ...context.options(), returnDocument: 'after' } + ) + if (result === null) { + const current = await this.outputs().findOne({ _id: id }, context.options()) + if (current?.spender === spend.spender && current.state === 'spent') return + rejectAdmission('spend-conflict') + } + } + + private async applyEviction( + context: MongoTransactionContext, + topic: string, + eviction: AdmissionOutpoint + ): Promise { + const id = this.outputId(topic, eviction) + const result = await this.outputs().updateOne( + { _id: id }, + { $set: { state: 'evicted', updatedAt: new Date() } }, + context.options() + ) + if (result.matchedCount !== 1) rejectAdmission('invalid-plan') + } + + private async insertOutput( + context: MongoTransactionContext, + decision: AdmissionTopicDecision, + output: AdmissionTopicDecision['outputs'][number] + ): Promise { + const now = new Date() + const payloadId = this.payloadId(output.script.payload) + try { + await this.outputs().insertOne( + { + _id: this.outputId(decision.topic, output), + schemaVersion, + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + topic: decision.topic, + txid: output.txid, + outputIndex: encodeMongoOutputIndex(output.outputIndex), + satoshis: encodeMongoUint64(output.satoshis), + score: encodeMongoUint64(output.score), + scriptPayloadId: payloadId, + scriptOffset: encodeMongoUint64(output.script.offset), + scriptByteLength: encodeMongoUint64(output.script.byteLength), + state: 'unspent', + version: '1', + createdAt: now, + updatedAt: now + }, + context.options() + ) + } catch (error) { + if (duplicateKey(error)) rejectAdmission('invalid-plan') + throw error + } + await this.pin( + context, + output.script.payload, + 'output', + this.outputId(decision.topic, output), + 'script' + ) + } + + private async insertEdge( + context: MongoTransactionContext, + topic: string, + edge: { source: AdmissionOutpoint; consumer: AdmissionOutpoint } + ): Promise { + const now = new Date() + const id = mongoRecordKey( + mongoNodeKey(this.scope), + 'edge', + topic, + edge.source.txid, + encodeMongoOutputIndex(edge.source.outputIndex), + edge.consumer.txid, + encodeMongoOutputIndex(edge.consumer.outputIndex) + ) + await this.db.collection(MongoCollectionNames.consumptionEdges).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion, + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + topic, + sourceTxid: edge.source.txid, + sourceOutputIndex: encodeMongoOutputIndex(edge.source.outputIndex), + consumerTxid: edge.consumer.txid, + consumerOutputIndex: encodeMongoOutputIndex(edge.consumer.outputIndex), + createdAt: now + }, + $set: { updatedAt: now } + }, + { upsert: true, ...context.options() } + ) + } + + private async insertApplied( + context: MongoTransactionContext, + plan: AdmissionCommit, + decision: AdmissionTopicDecision + ): Promise { + const now = new Date() + const proven = decision.applied.block !== undefined + try { + await this.applied().insertOne( + { + _id: this.appliedId(decision.topic, decision.applied.txid), + schemaVersion, + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + topic: decision.topic, + txid: decision.applied.txid, + state: proven ? 'active' : 'unproven', + admissionId: plan.key.operationId, + ...(decision.applied.firstSeenHeight === undefined + ? {} + : { firstSeenHeight: encodeMongoUint64(decision.applied.firstSeenHeight) }), + ...(decision.applied.proof === undefined + ? {} + : { proofPayloadId: this.payloadId(decision.applied.proof) }), + createdAt: now, + updatedAt: now + }, + context.options() + ) + } catch (error) { + if (duplicateKey(error)) rejectAdmission('invalid-plan') + throw error + } + if (decision.applied.proof !== undefined) { + await this.pin( + context, + decision.applied.proof, + 'applied-history', + this.appliedId(decision.topic, decision.applied.txid), + 'proof' + ) + } + } + + private async applyHistoryUpdate( + context: MongoTransactionContext, + decision: AdmissionTopicDecision, + policyId: string + ): Promise { + if (decision.historyUpdate === undefined) return + const now = new Date() + const next = decision.historyUpdate.nextTopicHistoryGeneration + const updated = await this.generations().findOneAndUpdate( + { + _id: this.generationId(decision.topic), + chainEpoch: encodeMongoUint64(decision.expectedHistory.chainEpoch), + topicHistoryGeneration: encodeMongoUint64(decision.expectedHistory.topicHistoryGeneration) + }, + { + $set: { + topicHistoryGeneration: encodeMongoUint64(next), + policyId, + updatedAt: now + } + }, + { ...context.options(), returnDocument: 'after' } + ) + if (updated === null) { + if ( + decision.expectedHistory.chainEpoch !== '0' || + decision.expectedHistory.topicHistoryGeneration !== '0' + ) { + rejectAdmission('read-conflict') + } + try { + await this.generations().insertOne( + { + _id: this.generationId(decision.topic), + schemaVersion, + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + topic: decision.topic, + chainEpoch: encodeMongoUint64(decision.expectedHistory.chainEpoch), + topicHistoryGeneration: encodeMongoUint64(next), + policyId, + createdAt: now, + updatedAt: now + }, + context.options() + ) + } catch (error) { + if (duplicateKey(error)) rejectAdmission('read-conflict') + throw error + } + } + if (decision.historyUpdate.handoff !== undefined) { + await this.applyHandoff(context, decision, decision.historyUpdate.handoff.expected, next) + } + const bytes = Buffer.from(JSON.stringify(this.historyUpdateRecord(decision)), 'utf8') + const digest = createHash('sha256').update(bytes).digest('hex') + const ref: AdmissionPayloadRef = { + kind: 'outbox-data', + digest, + byteLength: asStorageUint64(String(bytes.byteLength)) + } + const payload = await this.db + .collection(MongoCollectionNames.payloads) + .findOne({ _id: this.payloadId(ref), state: 'ready' }, context.options()) + if (payload === null) rejectAdmission('payload-not-ready') + await this.replaceHistoryPin( + context, + decision.topic, + decision.expectedHistory.topicHistoryGeneration, + next, + ref + ) + } + + private historyUpdateSlot(generation: string): string { + return `history-update:${generation}` + } + + private async replaceHistoryPin( + context: MongoTransactionContext, + topic: string, + previousGeneration: string, + nextGeneration: string, + ref: AdmissionPayloadRef + ): Promise { + const previousSlot = this.historyUpdateSlot(previousGeneration) + const previous = await this.db + .collection(MongoCollectionNames.payloadReferences) + .findOne( + { + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + ownerKind: 'basm-job', + ownerId: topic, + slot: previousSlot + }, + context.options() + ) + if (previous !== null) { + const previousPayload = await this.db + .collection(MongoCollectionNames.payloads) + .findOne({ _id: String(previous.payloadId) }, context.options()) + if ( + previousPayload !== null && + isPayloadKind(String(previousPayload.kind)) && + typeof previousPayload.digest === 'string' + ) { + const options = context.options() + await this.payloads.releaseReference( + options.session, + { + scope: this.scope, + payload: { + kind: previousPayload.kind as MongoPayloadKind, + digest: previousPayload.digest + }, + ownerKind: 'basm-job', + ownerId: topic, + slot: previousSlot + }, + { timeoutMS: options.timeoutMS, signal: context.signal } + ) + } + } + await this.pin(context, ref, 'basm-job', topic, this.historyUpdateSlot(nextGeneration)) + } + + private async applyHandoff( + context: MongoTransactionContext, + decision: AdmissionTopicDecision, + expected: RecoveryLease, + nextGeneration: string + ): Promise { + if ( + !sameScope(expected.scope, this.scope) || + expected.topic !== decision.topic || + expected.chainEpoch !== decision.expectedHistory.chainEpoch || + expected.topicHistoryGeneration !== decision.expectedHistory.topicHistoryGeneration + ) { + rejectAdmission('read-conflict') + } + const now = new Date() + const result = await this.jobs().findOneAndUpdate( + { + _id: this.jobId(expected), + topic: expected.topic, + peerId: expected.peerId, + jobId: expected.jobId, + chainEpoch: encodeMongoUint64(expected.chainEpoch), + topicHistoryGeneration: encodeMongoUint64(expected.topicHistoryGeneration), + leaseToken: encodeMongoUint64(expected.leaseToken), + $expr: { $gt: ['$leaseUntil', '$$NOW'] } + }, + { + $set: { + topicHistoryGeneration: encodeMongoUint64(nextGeneration), + checkpoint: decision.historyUpdate?.handoff?.checkpoint ?? expected.jobId, + updatedAt: now + } + }, + { ...context.options(), returnDocument: 'after' } + ) + if (result === null) rejectAdmission('read-conflict') + } + + private async upsertTransaction( + context: MongoTransactionContext, + plan: AdmissionCommit + ): Promise { + const raw = plan.payloads.find(ref => ref.kind === 'raw-transaction') + const manifest = plan.payloads.find(ref => ref.kind === 'beef-manifest') + const now = new Date() + const id = mongoRecordKey(mongoChainKey(this.scope), 'transaction', plan.identity.txid) + await this.db.collection(MongoCollectionNames.transactions).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion, + network: this.scope.network, + genesisHash: this.scope.genesisHash, + txid: plan.identity.txid, + ...(raw === undefined ? {} : { rawPayloadId: this.payloadId(raw) }), + ...(manifest === undefined ? {} : { manifestPayloadId: this.payloadId(manifest) }), + createdAt: now + }, + $set: { updatedAt: now } + }, + { upsert: true, ...context.options() } + ) + } + + private async insertOutbox( + context: MongoTransactionContext, + kind: 'lookup' | 'propagation', + intent: AdmissionOutboxIntent + ): Promise { + const now = new Date() + try { + await this.outbox(kind).insertOne( + { + _id: this.outboxId(kind, intent.eventId), + schemaVersion, + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + eventId: intent.eventId, + target: intent.target, + state: 'pending', + nextAttemptAt: now, + leaseUntil: new Date(0), + createdAt: now, + updatedAt: now + }, + context.options() + ) + } catch (error) { + if (duplicateKey(error)) rejectAdmission('invalid-plan') + throw error + } + for (const [index, payload] of intent.payloads.entries()) { + await this.pin(context, payload, `${kind}-outbox`, intent.eventId, String(index)) + } + } + + private async pin( + context: MongoTransactionContext, + ref: AdmissionPayloadRef, + ownerKind: + | 'transaction' + | 'applied-history' + | 'output' + | 'gasp-graph' + | 'gasp-node' + | 'basm-job' + | 'lookup-outbox' + | 'propagation-outbox' + | 'manifest' + | 'pin', + ownerId: string, + slot: string + ): Promise { + if (!isPayloadKind(ref.kind)) rejectAdmission('invalid-plan') + const options = context.options() + try { + await this.payloads.addReference( + options.session, + { + scope: this.scope, + payload: { kind: ref.kind, digest: ref.digest }, + ownerKind, + ownerId, + slot + }, + { timeoutMS: options.timeoutMS, signal: context.signal } + ) + } catch (error) { + if (error instanceof Error && error.message === 'Mongo payload is not ready for reference') { + rejectAdmission('payload-not-ready') + } + throw error + } + } + + private async outboxPayloads( + kind: 'lookup' | 'propagation', + eventId: string + ): Promise { + const refs = await this.db + .collection(MongoCollectionNames.payloadReferences) + .find({ + network: this.scope.network, + genesisHash: this.scope.genesisHash, + nodeId: this.scope.nodeId, + ownerKind: `${kind}-outbox`, + ownerId: eventId + }) + .sort({ slot: 1 }) + .toArray() + const result: AdmissionPayloadRef[] = [] + for (const reference of refs) { + const payload = await this.db + .collection(MongoCollectionNames.payloads) + .findOne({ _id: String(reference.payloadId) }) + if (payload === null || !isPayloadKind(payload.kind as string)) continue + result.push({ + kind: payload.kind as MongoPayloadKind, + digest: payload.digest as string, + byteLength: decodeMongoUint64(payload.byteLength as string) + }) + } + return result + } + + payloadId(ref: Pick): string { + return mongoRecordKey(mongoChainKey(this.scope), ref.kind, ref.digest) + } + + outputId(topic: string, outpoint: AdmissionOutpoint): string { + return mongoRecordKey( + mongoNodeKey(this.scope), + 'output', + topic, + outpoint.txid, + encodeMongoOutputIndex(outpoint.outputIndex) + ) + } + + private appliedId(topic: string, txid: string): string { + return mongoRecordKey(mongoNodeKey(this.scope), 'applied', topic, txid) + } + + generationId(topic: string): string { + return mongoRecordKey(mongoNodeKey(this.scope), 'generation', topic) + } + + jobId(lease: Pick): string { + return mongoRecordKey(mongoNodeKey(this.scope), 'job', lease.topic, lease.peerId, lease.jobId) + } + + outboxId(kind: 'lookup' | 'propagation', eventId: string): string { + return mongoRecordKey(mongoNodeKey(this.scope), `${kind}-outbox`, eventId) + } + + private outputs(): Collection { + return this.db.collection(MongoCollectionNames.outputs) + } + + private applied(): Collection { + return this.db.collection(MongoCollectionNames.appliedTransactions) + } + + generations(): Collection { + return this.db.collection(MongoCollectionNames.topicGenerations) + } + + jobs(): Collection { + return this.db.collection(MongoCollectionNames.basmRecoveryJobs) + } + + outbox(kind: 'lookup' | 'propagation'): Collection { + return this.db.collection( + kind === 'lookup' ? MongoCollectionNames.lookupOutbox : MongoCollectionNames.propagationOutbox + ) + } +} diff --git a/packages/overlays/overlay/src/storage/mongo/MongoOverlayStorage.ts b/packages/overlays/overlay/src/storage/mongo/MongoOverlayStorage.ts new file mode 100644 index 000000000..7c2e1cdbb --- /dev/null +++ b/packages/overlays/overlay/src/storage/mongo/MongoOverlayStorage.ts @@ -0,0 +1,563 @@ +import { createHash } from 'node:crypto' +import type { Db, Document } from 'mongodb' +import { MerklePath, Transaction } from '@bsv/sdk' +import type { Output } from '../../Output.js' +import type { AppliedTransaction, Storage } from '../Storage.js' +import { + asStorageUint64, + parseStorageOutputIndex, + parseStorageUint64, + type AdmissionPayloadRef, + type HistoryFence, + type StorageScope +} from '../AdmissionStorage.js' +import { + MongoAdmissionStorage, + type MongoAdmissionStorageOptions +} from './MongoAdmissionStorage.js' +import { MongoPayloadStore, type MongoPayloadKind } from './MongoPayloadStore.js' +import { + MongoCollectionNames, + decodeMongoUint64, + encodeMongoOutputIndex, + encodeMongoUint64, + mongoChainKey, + mongoNodeKey, + mongoRecordKey +} from './MongoSchema.js' + +type IdDocument = Document & { _id: string } + +const isPayloadKind = (value: string): value is MongoPayloadKind => + value === 'raw-transaction' || + value === 'merkle-path' || + value === 'beef-manifest' || + value === 'locking-script' || + value === 'outbox-data' + +/** + * Opt-in Mongo Storage adapter that advertises overlay-admission-v1 only because + * commitAdmission actually honors majority ack, spends, history, and outbox. + */ +export class MongoOverlayStorage implements Storage { + readonly admission: MongoAdmissionStorage + readonly admissionScope: StorageScope + private readonly payloads: MongoPayloadStore + + constructor( + private readonly db: Db, + scope: StorageScope, + options: MongoAdmissionStorageOptions = {} + ) { + this.admissionScope = { ...scope } + this.payloads = options.payloads ?? new MongoPayloadStore(db, this.admissionScope) + this.admission = new MongoAdmissionStorage(db, this.admissionScope, { + ...options, + payloads: this.payloads + }) + } + + enlistedIndexTargets(): readonly string[] { + return this.admission.enlistedTargets() + } + + async getHistoryFence(topic: string): Promise { + const document = await this.admission.generations().findOne({ + _id: this.admission.generationId(topic) + }) + if (document === null) { + return { chainEpoch: asStorageUint64('0'), topicHistoryGeneration: asStorageUint64('0') } + } + return { + chainEpoch: decodeMongoUint64(document.chainEpoch), + topicHistoryGeneration: decodeMongoUint64(document.topicHistoryGeneration) + } + } + + async publishAdmissionPayload(input: { + kind: AdmissionPayloadRef['kind'] + bytes: Uint8Array + txid?: string + }): Promise { + if (!isPayloadKind(input.kind)) throw new Error('Invalid Mongo admission payload kind') + const digest = createHash('sha256').update(input.bytes).digest('hex') + const published = await this.payloads.publish({ + kind: input.kind, + digest, + byteLength: asStorageUint64(String(input.bytes.byteLength)), + txid: input.txid, + bytes: (async function* () { + yield input.bytes + })() + }) + return { + kind: published.kind, + digest: published.digest, + byteLength: asStorageUint64(published.byteLength) + } + } + + async close(): Promise { + await this.admission.close() + } + + async insertOutput(utxo: Output): Promise { + const now = new Date() + const script = Buffer.from(utxo.outputScript) + const payload = await this.publishAdmissionPayload({ + kind: 'locking-script', + bytes: script + }) + await this.db.collection(MongoCollectionNames.outputs).updateOne( + { + _id: this.outputId(utxo.topic, utxo.txid, utxo.outputIndex), + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId + }, + { + $setOnInsert: { + _id: this.outputId(utxo.topic, utxo.txid, utxo.outputIndex), + schemaVersion: 1, + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + topic: utxo.topic, + txid: utxo.txid, + outputIndex: encodeMongoOutputIndex(String(utxo.outputIndex)), + satoshis: encodeMongoUint64(String(utxo.satoshis)), + score: encodeMongoUint64(String(utxo.score ?? 0)), + scriptPayloadId: this.admission.payloadId(payload), + scriptOffset: encodeMongoUint64('0'), + scriptByteLength: encodeMongoUint64(payload.byteLength), + state: utxo.spent ? 'spent' : 'unspent', + version: '1', + createdAt: now + }, + $set: { updatedAt: now } + }, + { upsert: true, writeConcern: { w: 'majority', j: true } } + ) + if (utxo.beef !== undefined) await this.persistTransactionBeef(utxo.txid, utxo.beef) + } + + private async persistTransactionBeef(txid: string, beef: number[]): Promise { + const tx = Transaction.fromBEEF(beef) + const published = await this.publishAdmissionPayload({ + kind: 'raw-transaction', + bytes: Buffer.from(tx.toBinary()), + txid + }) + const now = new Date() + const id = mongoRecordKey(mongoChainKey(this.admissionScope), 'transaction', txid) + await this.db.collection(MongoCollectionNames.transactions).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion: 1, + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + txid, + createdAt: now + }, + $set: { rawPayloadId: this.admission.payloadId(published), updatedAt: now } + }, + { upsert: true, writeConcern: { w: 'majority', j: true } } + ) + } + + async findOutput( + txid: string, + outputIndex: number, + topic?: string, + spent?: boolean, + includeBEEF = false + ): Promise { + const filter: Record = { + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + txid, + outputIndex: encodeMongoOutputIndex(String(outputIndex)) + } + // Evicted outputs remain for audit/history but must never be exposed as + // current overlay state. + filter.state = { $ne: 'evicted' } + if (topic !== undefined) filter.topic = topic + if (spent === true) filter.state = 'spent' + if (spent === false) filter.state = 'unspent' + const document = await this.db + .collection(MongoCollectionNames.outputs) + .findOne(filter, { + readConcern: { level: 'majority' }, + readPreference: 'primary' + }) + if (document === null) return null + return await this.toOutput(document, includeBEEF) + } + + async findOutputsForTransaction(txid: string, includeBEEF = false): Promise { + const documents = await this.db + .collection(MongoCollectionNames.outputs) + .find({ + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + txid, + state: { $ne: 'evicted' } + }) + .toArray() + return await this.toOutputs(documents, includeBEEF) + } + + async findUTXOsForTopic( + topic: string, + since?: number, + limit?: number, + includeBEEF = false + ): Promise { + const filter: Record = { + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + topic, + state: 'unspent' + } + if (since !== undefined && since > 0) filter.score = { $gte: encodeMongoUint64(String(since)) } + let query = this.db + .collection(MongoCollectionNames.outputs) + .find(filter) + .sort({ score: 1, _id: 1 }) + if (limit !== undefined && limit > 0) query = query.limit(limit) + return await this.toOutputs(await query.toArray(), includeBEEF) + } + + /** + * Maps a batch of raw output documents to `Output`s, dropping any that + * `toOutput` treats as non-serving (see its docstring). Every current + * caller's query already excludes `state: 'evicted'`, so this filter is + * defense in depth rather than the primary guard. + */ + private async toOutputs( + documents: Array>, + includeBEEF: boolean + ): Promise { + const mapped = await Promise.all( + documents.map(async document => await this.toOutput(document, includeBEEF)) + ) + return mapped.filter((output): output is Output => output !== null) + } + + async deleteOutput(txid: string, outputIndex: number, topic: string): Promise { + await this.db + .collection(MongoCollectionNames.outputs) + .updateOne( + { _id: this.outputId(topic, txid, outputIndex) }, + { $set: { state: 'evicted', updatedAt: new Date() } }, + { writeConcern: { w: 'majority', j: true } } + ) + } + + async markUTXOAsSpent(txid: string, outputIndex: number, topic: string): Promise { + await this.db + .collection(MongoCollectionNames.outputs) + .updateOne( + { _id: this.outputId(topic, txid, outputIndex), state: 'unspent' }, + { $set: { state: 'spent', updatedAt: new Date() } }, + { writeConcern: { w: 'majority', j: true } } + ) + } + + async updateConsumedBy( + txid: string, + outputIndex: number, + topic: string, + consumedBy: Array<{ txid: string; outputIndex: number }> + ): Promise { + const now = new Date() + for (const consumer of consumedBy) { + const id = mongoRecordKey( + mongoNodeKey(this.admissionScope), + 'edge', + topic, + txid, + encodeMongoOutputIndex(String(outputIndex)), + consumer.txid, + encodeMongoOutputIndex(String(consumer.outputIndex)) + ) + await this.db.collection(MongoCollectionNames.consumptionEdges).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion: 1, + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + topic, + sourceTxid: txid, + sourceOutputIndex: encodeMongoOutputIndex(String(outputIndex)), + consumerTxid: consumer.txid, + consumerOutputIndex: encodeMongoOutputIndex(String(consumer.outputIndex)), + createdAt: now + }, + $set: { updatedAt: now } + }, + { upsert: true, writeConcern: { w: 'majority', j: true } } + ) + } + } + + async updateTransactionBEEF(txid: string, _beef: number[]): Promise { + throw new Error(`Mongo overlay storage does not implement updateTransactionBEEF for ${txid}`) + } + + async insertAppliedTransaction(tx: AppliedTransaction): Promise { + const now = new Date() + const id = mongoRecordKey(mongoNodeKey(this.admissionScope), 'applied', tx.topic, tx.txid) + await this.db.collection(MongoCollectionNames.appliedTransactions).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion: 1, + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + topic: tx.topic, + txid: tx.txid, + state: tx.proven === true ? 'active' : 'unproven', + admissionId: tx.txid, + createdAt: now + }, + $set: { updatedAt: now } + }, + { upsert: true, writeConcern: { w: 'majority', j: true } } + ) + } + + async doesAppliedTransactionExist(tx: AppliedTransaction): Promise { + const found = await this.db + .collection(MongoCollectionNames.appliedTransactions) + .findOne( + { + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + topic: tx.topic, + txid: tx.txid + }, + { readConcern: { level: 'majority' }, readPreference: 'primary' } + ) + return found !== null + } + + async updateLastInteraction(host: string, topic: string, since: number): Promise { + const now = new Date() + const id = mongoRecordKey(mongoNodeKey(this.admissionScope), 'cursor', host, topic) + await this.db.collection(MongoCollectionNames.gaspCursors).updateOne( + { _id: id }, + { + $setOnInsert: { + _id: id, + schemaVersion: 1, + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + remoteHost: host, + topic, + createdAt: now + }, + $set: { + state: 'ready', + score: encodeMongoUint64(String(since)), + updatedAt: now + } + }, + { upsert: true, writeConcern: { w: 'majority', j: true } } + ) + } + + async getLastInteraction(host: string, topic: string): Promise { + const found = await this.db.collection(MongoCollectionNames.gaspCursors).findOne({ + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + remoteHost: host, + topic + }) + if (found?.score === undefined) return 0 + return this.toSafeNumber(decodeMongoUint64(found.score as string), 'GASP cursor') + } + + private outputId(topic: string, txid: string, outputIndex: number): string { + return mongoRecordKey( + mongoNodeKey(this.admissionScope), + 'output', + topic, + txid, + encodeMongoOutputIndex(String(outputIndex)) + ) + } + + private toSafeNumber(value: string, label: string): number { + const parsed = parseStorageUint64(value) + if (parsed > BigInt(Number.MAX_SAFE_INTEGER)) { + throw new Error(`Mongo ${label} exceeds a safe JavaScript integer`) + } + return Number(parsed) + } + + /** + * Maps a raw output document to an `Output`, or `null` when the document + * is not currently serving as overlay state. An admission eviction leaves + * its row in place (state: 'evicted') for audit/history, and it must never + * be reported as a live, unspent output — every caller that reaches here + * must treat a `null` result the same as "not found", regardless of + * whether its own query already excluded `evicted` rows. + */ + private async toOutput( + document: Record, + includeBEEF: boolean + ): Promise { + if (document.state === 'evicted') return null + const topic = document.topic as string + const txid = document.txid as string + const outputIndex = String(document.outputIndex) + const [outputsConsumed, consumedBy] = await Promise.all([ + this.readConsumptionEdges(topic, { consumerTxid: txid, consumerOutputIndex: outputIndex }, 'source'), + this.readConsumptionEdges(topic, { sourceTxid: txid, sourceOutputIndex: outputIndex }, 'consumer') + ]) + const output: Output = { + txid, + outputIndex: parseStorageOutputIndex(outputIndex), + outputScript: await this.readScript(document), + satoshis: this.toSafeNumber(decodeMongoUint64(document.satoshis as string), 'satoshis'), + topic, + spent: document.state === 'spent', + outputsConsumed, + consumedBy, + score: this.toSafeNumber(decodeMongoUint64(document.score as string), 'score') + } + if (includeBEEF) { + const beef = await this.readBeef(output.txid) + if (beef !== undefined) output.beef = beef + } + return output + } + + /** + * Reads consumption edges persisted for this output (by either + * MongoAdmissionStorage.insertEdge during a commit, or + * MongoOverlayStorage.updateConsumedBy on the classic storage path) and + * returns the opposite endpoint of each matching edge: querying by the + * consumer side returns `outputsConsumed` (what this output's transaction + * spent), and querying by the source side returns `consumedBy` (what has + * since spent this output). + */ + private async readConsumptionEdges( + topic: string, + match: { sourceTxid: string; sourceOutputIndex: string } | { consumerTxid: string; consumerOutputIndex: string }, + side: 'source' | 'consumer' + ): Promise> { + const edges = await this.db + .collection(MongoCollectionNames.consumptionEdges) + .find({ + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + topic, + ...match + }) + .toArray() + return edges.map(edge => ({ + txid: (side === 'source' ? edge.sourceTxid : edge.consumerTxid) as string, + outputIndex: parseStorageOutputIndex( + String(side === 'source' ? edge.sourceOutputIndex : edge.consumerOutputIndex) + ) + })) + } + + private async readScript(document: Record): Promise { + const payloadId = document.scriptPayloadId + if (typeof payloadId !== 'string') return [] + const payload = await this.db.collection(MongoCollectionNames.payloads).findOne({ + _id: payloadId, + state: 'ready' + }) + if ( + payload === null || + typeof payload.kind !== 'string' || + typeof payload.digest !== 'string' || + !isPayloadKind(payload.kind) + ) { + throw new Error('Mongo output script payload is not ready') + } + const bytes = await this.payloads.read( + { kind: payload.kind, digest: payload.digest }, + { + offset: decodeMongoUint64(document.scriptOffset as string), + byteLength: decodeMongoUint64(document.scriptByteLength as string) + } + ) + return Array.from(bytes) + } + + private async readBeef(txid: string): Promise { + const transaction = await this.db + .collection(MongoCollectionNames.transactions) + .findOne({ + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + txid + }) + const rawPayloadId = transaction?.rawPayloadId + if (typeof rawPayloadId !== 'string') return undefined + const rawPayload = await this.db.collection(MongoCollectionNames.payloads).findOne({ + _id: rawPayloadId, + state: 'ready' + }) + if ( + rawPayload === null || + typeof rawPayload.kind !== 'string' || + typeof rawPayload.digest !== 'string' || + !isPayloadKind(rawPayload.kind) + ) { + return undefined + } + const raw = await this.payloads.read({ kind: rawPayload.kind, digest: rawPayload.digest }) + const tx = Transaction.fromBinary(Array.from(raw)) + const merkle = await this.db + .collection(MongoCollectionNames.payloadReferences) + .findOne({ + network: this.admissionScope.network, + genesisHash: this.admissionScope.genesisHash, + nodeId: this.admissionScope.nodeId, + ownerKind: 'transaction', + ownerId: txid, + slot: { $regex: '^merkle-path:' } + }) + if (merkle !== null && typeof merkle.payloadId === 'string') { + const merklePayload = await this.db + .collection(MongoCollectionNames.payloads) + .findOne({ + _id: merkle.payloadId, + state: 'ready' + }) + if ( + merklePayload !== null && + typeof merklePayload.kind === 'string' && + typeof merklePayload.digest === 'string' && + isPayloadKind(merklePayload.kind) + ) { + const path = await this.payloads.read({ + kind: merklePayload.kind, + digest: merklePayload.digest + }) + tx.merklePath = MerklePath.fromBinary(Array.from(path)) + } + } + return tx.merklePath === undefined ? tx.toBEEF() : tx.toAtomicBEEF() + } +} diff --git a/packages/overlays/overlay/src/storage/mongo/MongoPayloadStore.ts b/packages/overlays/overlay/src/storage/mongo/MongoPayloadStore.ts index ae1d8e0fe..83ea5537e 100644 --- a/packages/overlays/overlay/src/storage/mongo/MongoPayloadStore.ts +++ b/packages/overlays/overlay/src/storage/mongo/MongoPayloadStore.ts @@ -194,6 +194,29 @@ export class MongoPayloadStore { }) } + public async read( + payload: Pick, + range?: { offset: string; byteLength: string } + ): Promise { + const payloadId = this.payloadId(this.scope, payload) + const record = await this.db + .collection(MongoCollectionNames.payloads) + .findOne({ + _id: payloadId, + state: 'ready' + }) + if (record === null) throw new Error('Mongo payload is not ready') + const total = BigInt(decodeMongoUint64(record.byteLength)) + const offset = range === undefined ? BigInt(0) : BigInt(range.offset) + const length = range === undefined ? total : BigInt(range.byteLength) + if (offset < BigInt(0) || length < BigInt(0) || offset + length > total) + throw new Error('Mongo payload range is invalid') + if (offset + length > BigInt(Number.MAX_SAFE_INTEGER)) + throw new Error('Mongo payload range exceeds safe integer') + const bytes = await this.readBytes(record) + return bytes.subarray(Number(offset), Number(offset + length)) + } + public async publish(input: MongoPayloadInput): Promise { input = { ...input } this.validateInput(input) @@ -898,6 +921,28 @@ export class MongoPayloadStore { if (file !== null) await this.bucket.delete(fileId).catch(() => undefined) } + private async readBytes(record: PayloadDocument): Promise { + if (record.inlineData !== undefined) return Buffer.from(record.inlineData.buffer) + if (record.fileId === undefined) throw new Error('Mongo ready payload has no bytes') + const chunks: Buffer[] = [] + const download = this.bucket.openDownloadStream(record.fileId) + const iterator = download[Symbol.asyncIterator]() + try { + while (true) { + const next = await iterator.next() + if (next.done) break + const chunk = next.value + if (!(chunk instanceof Uint8Array)) + throw new Error('Mongo GridFS returned a non-byte chunk') + chunks.push(Buffer.from(chunk)) + } + } catch (error) { + download.destroy(error instanceof Error ? error : new Error('Mongo payload read failed')) + throw error + } + return Buffer.concat(chunks) + } + private async verifyGridFs( fileId: ObjectId, input: Pick, diff --git a/packages/overlays/overlay/src/storage/mongo/MongoSchema.ts b/packages/overlays/overlay/src/storage/mongo/MongoSchema.ts index a7810dd8b..36d0a4411 100644 --- a/packages/overlays/overlay/src/storage/mongo/MongoSchema.ts +++ b/packages/overlays/overlay/src/storage/mongo/MongoSchema.ts @@ -1,9 +1,11 @@ import { createHash } from 'node:crypto' import type { CollectionInfo, Db, Document, IndexDescription } from 'mongodb' import { + asStorageUint64, parseStorageOutputIndex, parseStorageUint64, - type StorageScope + type StorageScope, + type StorageUint64 } from '../AdmissionStorage.js' /** The immutable chain namespace shared by all Overlay nodes. */ @@ -110,13 +112,13 @@ export function encodeMongoUint64(value: string): string { } /** Strict inverse of encodeMongoUint64; rejects non-canonical padded values. */ -export function decodeMongoUint64(value: string): string { +export function decodeMongoUint64(value: string): StorageUint64 { if (typeof value !== 'string' || !new RegExp(paddedUint64Pattern).test(value)) throw new Error('Invalid Mongo uint64') if (value > maxUint64) throw new Error('Invalid Mongo uint64') const decoded = BigInt(value).toString(10) if (encodeMongoUint64(decoded) !== value) throw new Error('Invalid Mongo uint64') - return decoded + return asStorageUint64(decoded) } /** Validates the wire uint32 domain and returns its canonical decimal spelling. */ diff --git a/packages/overlays/overlay/src/storage/mongo/MongoTransactionRunner.ts b/packages/overlays/overlay/src/storage/mongo/MongoTransactionRunner.ts index 7464ac30c..439128967 100644 --- a/packages/overlays/overlay/src/storage/mongo/MongoTransactionRunner.ts +++ b/packages/overlays/overlay/src/storage/mongo/MongoTransactionRunner.ts @@ -83,6 +83,20 @@ function hasLabel(error: unknown, label: string): boolean { ) } +function isTransientTransactionError(error: unknown): boolean { + if (hasLabel(error, 'TransientTransactionError')) return true + if (typeof error !== 'object' || error === null) return false + const code = 'code' in error ? error.code : undefined + if (code === 112 || code === '112') return true + if ('codeName' in error && error.codeName === 'WriteConflict') return true + const text = [ + error instanceof Error ? error.message : '', + 'errmsg' in error ? String(error.errmsg) : '', + 'cause' in error && error.cause instanceof Error ? error.cause.message : '' + ].join(' ') + return text.includes('Write conflict') +} + function duplicateKey(error: unknown): boolean { return typeof error === 'object' && error !== null && 'code' in error && error.code === 11000 } @@ -286,8 +300,12 @@ export class MongoTransactionRunner { this.calls += 1 try { for (let bodyIndex = 0; bodyIndex < this.maxBodyAttempts; bodyIndex += 1) { - const completed = await this.runBodyAttempt(id, request, receipt, body, budget, bodyIndex) - if (completed !== undefined) return completed + try { + const completed = await this.runBodyAttempt(id, request, receipt, body, budget, bodyIndex) + if (completed !== undefined) return completed + } catch (error) { + if (!isTransientTransactionError(error) || bodyIndex + 1 >= this.maxBodyAttempts) throw error + } } throw new Error('Mongo transaction attempt limit reached') } finally { @@ -366,7 +384,7 @@ export class MongoTransactionRunner { } catch (error) { bodyActive = false await this.abort(attempt) - if (!hasLabel(error, 'TransientTransactionError') || bodyIndex + 1 >= this.maxBodyAttempts) throw error + if (!isTransientTransactionError(error) || bodyIndex + 1 >= this.maxBodyAttempts) throw error return undefined } finally { bodyActive = false diff --git a/packages/overlays/topics/package.json b/packages/overlays/topics/package.json index bf49900ff..c4d620f1f 100644 --- a/packages/overlays/topics/package.json +++ b/packages/overlays/topics/package.json @@ -1,6 +1,6 @@ { "name": "@bsv/overlay-topics", - "version": "1.8.2", + "version": "1.8.3", "sideEffects": false, "engines": { "node": ">=22" diff --git a/specs/overlay/mongo-v1.md b/specs/overlay/mongo-v1.md index ebeea19f0..98f1c90e5 100644 --- a/specs/overlay/mongo-v1.md +++ b/specs/overlay/mongo-v1.md @@ -5,9 +5,9 @@ This is an additive, opt-in MongoDB storage foundation for `@bsv/overlay`. It supplies schema bootstrap, scoped keys, content-addressed payload primitives, and durable read-guard primitives. It does **not** select MongoDB by default, -connect MongoDB to `Engine`, implement `AdmissionStorage`, or make an admission -receipt claim. The existing Engine, Knex, and injected-storage paths retain -their current behavior. +change the SQL default, or claim production activation. `MongoOverlayStorage` +may be injected explicitly; Knex and injected-storage callers without that +adapter retain their current behavior. MongoDB is an optional peer dependency. An application that imports a Mongo entry point must install a compatible driver explicitly: @@ -99,29 +99,30 @@ controls where they operate in a caller's admission body. Neither payload publication nor collection runs verifier logic, network activity, uploads, or plugin callbacks inside an admission transaction. -## Admission transaction integration is pending - -The foundation is deliberately smaller than the v1 admission contract. A later -Engine/adapter integration must compose the guarded payload primitives with the -whole admission operation in one short transaction. Until that integration and -its conformance evidence land, no Mongo method is an `AdmissionStorage` -implementation and no saved STEAK/receipt is exposed as a durable admission -acknowledgment. - -The pending transaction core must use a finite retry budget and distinguish -MongoDB's labels on the **same session and operation identity**: rerun a -`TransientTransactionError` body only under that identity, and reconcile an -`UnknownTransactionCommitResult` before a new body may run. Its completion -record must be majority+journal persisted with the exact receipt. Expired -orphan cleanup must use a compare-and-set proof rather than treating absence or -an ambiguous commit as an abort. MongoDB documents the retry-label behavior in -its [transaction API guidance](https://www.mongodb.com/docs/manual/core/transactions-in-applications/). - -That integration must keep external indexes and propagation asynchronous: -persist an idempotent outbox intent in the admission transaction, then let a -leased worker deliver it. A callback may participate only after it declares -replay safety, reconciliation, cancellation, and bounded work. It must never -be invoked in the Mongo admission transaction. +## Admission transaction integration + +`MongoAdmissionStorage` implements `overlay-admission-v1` using the schema, +payload, read-guard, and transaction-runner primitives. `MongoOverlayStorage` +is the Engine `Storage` adapter that advertises that capability only because +`commitAdmission` actually honors majority ack, conditional spends, applied +history, and outbox publication. `getAdmissionStorage(storage)` remains a +declaration check: Knex and incomplete Mongo helpers still return undefined. + +Engine `submit` / historical submit call `commitAdmission` when that capability +is present. Parse, SPV, topic decision, and broadcast stay outside the Mongo +transaction. Broadcast-before-mutation ordering is unchanged for live mode; +historical mode still skips broadcast/propagation. The returned STEAK is the +exact UTF-8 JSON saved with the majority receipt. A crash retry of the same +operation identity returns that saved STEAK. `TransientTransactionError` +reruns the body under the same operation id; `UnknownTransactionCommitResult` +reconciles the same commit identity before a new body. + +Native Mongo lookup indexes may enlist writes on the admission session and +are the only indexes that may be `visible` at commit. External plugins receive +a durable lookup outbox event with leased delivery; they are not assumed +replay-safe, and STEAK is overlay admission, not remote delivery. Propagation +uses the same outbox protocol. Ban/serving eviction must not erase historical +applied/BASM records. Mongo is not the overlay default. ## Compatibility and operations diff --git a/specs/overlay/persistence-v1.md b/specs/overlay/persistence-v1.md index aa4bae016..272a07210 100644 --- a/specs/overlay/persistence-v1.md +++ b/specs/overlay/persistence-v1.md @@ -21,7 +21,8 @@ declarations only. Go uses a separate optional provider/interface, preserving it existing `engine.Storage` implementers. Shape detection cannot certify durability. The existing engines, SQL adapter, injected Go stores and default selections are unchanged. Their existing STEAK callback must not be described as this receipt. -S03/S04 must integrate and verify the capability before activating durable ACKs. +S03 integrates the TS Engine/Mongo capability behind an explicit adapter; +durable ACKs are not a production default. S04 owns the Go equivalent. ## Identity and exact values