feat(bdk_electrum_streaming)!: Verify proof-of-work against trusted headers

[evanlinjin]

The client took the server's word for chain data. Headers were spliced into the checkpoint chain unchecked, and a merkle proof was validated against whatever header the server returned for that height — so an anchor only ever meant "this server says so". A server could invent a height, hand back a matching fake header, and the proof would pass.

Rebased directly onto current main (already includes #13, #17, #21, #24) — the branch's own prior commits for the cache/anchor/chain-ownership refactors turned out to already be superseded by more-evolved equivalents merged there independently, so this PR now carries only what's new on top: PoW verification, a follow-up fix, and a small cache cleanup.

HeaderChain

A CheckPoint<Header> anchored to a set of user-provided trusted headers. Syncing starts at highest_trusted + 1, so the first header's prev_blockhash pins the run to a block the user vouched for. From there up, every header must match the trusted header at its height if one was given, link to the block below it via prev_blockhash, claim the difficulty consensus requires (recomputed at each retarget boundary), and hash below that difficulty's target.

A reorg on top of that is accepted only if it brings more work than the blocks it replaces. Everything below the run's start is untouched, so comparing the two chains from there is the same as comparing their totals — an O(reorg depth) sum. Re-applying blocks we already have is not a reorg and is exempt. An eviction that would displace a trusted block is refused outright.

On networks where difficulty moves, the highest trusted block must sit on a difficulty-adjustment boundary, so every retarget above it is recomputed from a header already in hand rather than taken on faith. Trusted blocks below the sync start are exempt: a backfilled run is pinned by a trusted block at the bottom and the verified chain at the top.

ProvenAnchor

Transactions were already merkle-proved, but against an unverified header. The proof is now checked against a header in the verified chain, and kept rather than discarded: ProvenAnchor { block_id, pos, merkle }. Update<K> becomes FullScanResponse<K, ProvenAnchor, Header>. No block time on the anchor — the header it was proved against travels with every update in chain_update.

ConfirmationJob (the job that moves the chain and anchors what the scripts found) now fetches contiguous runs of headers rather than scattered heights, since a header is only verifiable as part of a chain reaching a block we trust, and gained a backfill: a transaction confirmed below where the chain starts grows it downwards to the nearest trusted block. Cache::headers is gone — only the verified chain may hold a header.

Do not replan a run that can never be fetched

FetchAnchors read a missing header as history the chain had not backfilled yet and went back to Init to plan a run for it. But a run only ever reaches the tip the server has announced, so an anchor above that tip has no run to plan: Init produced the same runs, FetchHeaders completed with nothing to do, and the anchor stage asked again — a loop with no exit inside a single State::poll.

A history can name a height above the announced tip in the ordinary course of things: electrs notifies the script hash before the header. The fix distinguishes the two misses — below the chain base is a backfill, which Init can plan; above the verified tip is simply pending, and the announcement that carries the tip is what re-polls it.

Cache no longer persists

Everything in it can be rebuilt from what a wallet already knows, so Cache doesn't need to round-trip through serde any more. Cache::from_wallet_txs takes (tx, status, relevant_scripts) — one entry per transaction, not one per script it pays, since that is how a wallet holds them — no bdk_chain dependency, just a TxConfirmationStatus enum (Confirmed(ProvenAnchor) or Mempool { confirmed_inputs }). It rebuilds all four fields: subscriptions and spk_txids from computing the same status hash a server would, txs from the transaction itself, and anchors for anything already confirmed. A reconnect need not redownload a script's history, refetch its transactions, or reprove its anchors — only whatever actually changed.

With nothing left to persist, TxCache's fields (spk_txids, txs, anchors) move directly onto Cache, and the serde-only persist module (HistoryTx, the anchors-as-seq codec, Subscriptions' hand-written Serialize/Deserialize) is gone.

The status computation needs care: the Electrum status hash mixes in just txid:height:, but the protocol orders entries by height and then block position — so two transactions confirmed in the same block hash to a status a server would not recognise unless their order matches the block's. ProvenAnchor::pos (a wallet already has it from the merkle proof) is what orders them correctly, and mempool entries are ordered by the protocol's (-height, tx_hash) rule rather than left arbitrary. same_block_transactions_are_ordered_by_block_position covers it.

Behaviour changes worth reviewing

An equal-work fork is refused. A same-height reorg carries the same work as what it replaces, so the verified chain keeps what it has — which is what a full node does. A server that genuinely reorged will also have extended, and that tip announcement is what triggers the switch. a_fork_without_more_work_is_refused covers the refusal; anchor_is_refetched_when_tx_moves_to_another_block_of_same_height covers the switch when the fork does out-work us.

A trusted block the server disagrees with stops the connection. Every header above it descends from a block we do not accept, so there is nothing to reconcile. The old client asked again forever.

A reorg deeper than the re-download window errors rather than walking further back. Marked with a ponytail: comment.

trusted-headers-gen

Trusting a header means asserting it's canonical, which has to come from somewhere the user actually checked — ideally their own node. trusted-headers-gen is a new workspace crate: cargo run -p trusted-headers-gen -- --url ... --network bitcoin fetches a header from a Bitcoin Core node and writes it out as reviewable, diffable Rust source rather than anything fetched at runtime. Each run only touches the network it's pointed at (a node only ever serves one), merging into whatever heights are already recorded for the others.

The default height is the second highest difficulty-adjustment boundary at or below the tip: a boundary because that's what HeaderChain::new requires of the highest trusted header, and the second one because the highest can be the tip itself — a block a few confirmations deep is no basis for trust. That puts the anchor 2016..=4032 blocks back.

--network is checked against what the node reports for chain: a header filed under the wrong network is a poisoned anchor, and the node is the one thing that knows which chain it serves. That field is read out of the raw response rather than a typed one — the rest of getblockchaininfo has changed shape across Core releases, and this deliberately uses the v17 client surface to stay version-agnostic (the typed version does in fact fail against v28).

bdk_electrum_streaming ships none of this data, and gains no API from it. The output lands in the current directory and placing it takes an explicit --out, so trusted headers live in the tree of whoever reviewed them as the diff they are. Shipping them from a library would put its release process in the trust path instead of the operator's own node, which is a strictly larger trust surface.

Header can't be built in a const context (its component types have private fields and non-const constructors), so the generated module keeps each network's data as a const array of (height, hex) — the auditable source of truth — decoded once into pub static ..._TRUSTED_HEADERS: LazyLock<[(u32, Header); N]>, with a trusted_headers() map over whichever networks are present.

It's a separate crate rather than an example because it needs corepc-client, which has nothing to do with the library and shouldn't reach downstream consumers' dependency trees; it isn't bdk_electrum_streaming-specific either, only touching bdk_core's Header.

Testing

cargo test — 62 tests. bdk_electrum_streaming: 23 unit (15 of them header_chain, mining real headers and grinding nonces over three parameter sets so each rule actually runs), 29 deterministic (tests/state.rs), 5 live (tests/env.rs).

trusted-headers-gen: 5, including one that spins up a real bitcoind and checks the header fetched is the one that node has at that height, that a height above the tip is refused, that a regtest node cannot be talked into answering for mainnet, and that the result survives the round trip out into generated source and back. Verified against Bitcoin Core v28.1, and the generated module was confirmed to compile.

The deterministic tests grind nonces: regtest's target is easy but not free, and a little under half of all nonces miss it.

backfills_history_below_the_sync_start mines 101 blocks to the wallet's spk before the client connects and trusts only the tip, so the whole history sits below the sync start. anchor_below_the_trusted_block_is_backfilled is the deterministic version.

reorg_to_same_height_block_refetches_anchor_live (issue #12, end to end) mines one extra block, so the fork out-works the chain it replaces — which is what makes a real node switch too.

From review

Two holes worth naming, both now fixed with a regression test that fails without the fix:

• HeaderChain::apply could accept an equal-work fork. It took the backfill path for any run starting below the base, requiring only that the run reach at least the base. A run that also carried on past the tip took that path while replacing verified blocks — and the backfill path rebuilds rather than comparing work, so the check the extend path exists for was bypassed. Not reachable through ConfirmationJob (its backfills always stop at base - 1), but apply is public and its contract read as a lower bound. A backfill must now fill the gap exactly.
• A lagging server was silently accommodated. For a target more than REORG_WINDOW below the verified tip, no run was planned to it, so the job found its remaining runs complete, applied nothing, and handed back the chain it already had as though the server had confirmed it — then asked that server for proofs at heights it does not have. The REORG_WINDOW doc already claimed the connection errors out here; now it does.

Also: FetchHeaders re-queues its gaps when a run is incomplete, since a server may return fewer headers than asked for and nothing else would ask again; git dependencies are pinned to a rev rather than tracking master; and the Subscriptions doc no longer claims statuses are computed "the same way Electrum does" — that holds for confirmed history, but a server may hash its mempool entries in its own index's order.

One raised point I did not act on: that chain.tip() == None reporting Done should be an error. Two attempts at a guard each broke a legitimate flow — anchor_below_the_trusted_block_is_backfilled passes through exactly that state, because a backfill is planned from heights the histories have yet to name. The condition isn't cleanly separable from healthy operation, and I'd rather leave it than ship a guard I've already seen reject valid syncs twice.

Note for reviewers

LocalChain::canonicalize is still LocalChain<BlockHash>-only upstream, so tests/env.rs converts the header chain to a blockhash one to compute balance and chain positions. Nothing to fix in this crate — flagging it as the friction point for CheckPoint<Header> consumers.

These crates depend on bdk_core/bdk_chain from git (pinned) because CheckPoint<D> and FullScanResponse<K, A, D> are unreleased. cargo publish rejects git dependencies, so a release has to wait for them upstream.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Ga32WdTURWnpMDxAckM2K