diff --git a/CHANGELOG.md b/CHANGELOG.md index 101bfff6b..0ca9a9910 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,15 @@ before 1.0). ### Added +- **Confirmed-tx chain view:** Electrum/Esplora responses for confirmed + txs are built at one published tip and stamp that tip for the client + (`X-Bitcoin-Chain-Tip` on Esplora; JSON-RPC `chain_tip` on Electrum). + Same-height reorgs invalidate the SH join cache and change Electrum + status (block hash in the preimage). Yuval raised the A-B-A hole; + shape from [mempool#6584](https://github.com/mempool/mempool/issues/6584) + and [electrum-protocol#2](https://github.com/spesmilo/electrum-protocol/pull/2). + [`COMPAT.md`](COMPAT.md), [`docs/concurrency.md`](docs/concurrency.md). + - **Road to 1.0:** [`docs/road-to-1.0.md`](docs/road-to-1.0.md) owns 1.0 product gates (claimed Core functional, Core-parity fuzz, selected crates.io libraries, SH/RSS, eclipse/DoS, fee validation, schema freeze). diff --git a/COMPAT.md b/COMPAT.md index 8df5e0290..5c6fd896e 100644 --- a/COMPAT.md +++ b/COMPAT.md @@ -77,9 +77,9 @@ Full method list, auth, and shindex matrix: **[`docs/rpc.md`](./docs/rpc.md)**. | server.version / banner / features | done | Banner: libre-relay-class. `server.version[0]` is `rbitcoin-electrs ` — **not electrs**; see below | | blockchain.tweaks.subscribe | done | Cake stream (first height as result, then notifies + `done`). Naive walk, or `--sptweaks` thin index (`len:tweak` only; one `txout` span per wave). Pre-taproot: empty maps in ≤1024-height writes. Isolate may still hardcode `electrs.cakewallet.com` | | headers / block headers | done | Tip push on subscribe | -| scripthash history / balance / listunspent | done | Unconf when mempool attached; `get_history` optional BCH-style `from_height` / exclusive `to_height` (`-1` = tip + mempool); 1-arg = full history; **subscribe status always full**; `listunspent` loads `txid.body` only for unspent creates; one TCP connection reuses the last SH outs+spent join until tip height changes | -| scripthash.get_mempool / subscribe | done | Status on mempool announce **and** on confirming tip when that block creates or spends the hash (posting-list probe; no Class A expand on a miss) | -| transaction.get / get_merkle | done | get falls back to mempool | +| scripthash history / balance / listunspent | done | Unconf when mempool attached; `get_history` optional BCH-style `from_height` / exclusive `to_height` (`-1` = tip + mempool); 1-arg = full history; **subscribe status always full**; `listunspent` loads `txid.body` only for unspent creates; one TCP connection reuses the last SH outs+spent join until tip **hash** changes. Confirmed methods stamp `chain_tip` / `chain_tip_height` on the JSON-RPC object (not inside `result`). `server.features.chain_tip = true`. | +| scripthash.get_mempool / subscribe | done | Status on mempool announce **and** on confirming tip when that block creates or spends the hash (posting-list probe; no Class A expand on a miss). Reorg (`TipNotify.reorg_from_height`) restatuses every watch even if the new block misses the script. Status preimage is `txid:height:blockhash:` for confirmed rows (mempool rows stay `txid:height:`). | +| transaction.get / get_merkle | done | get falls back to mempool; confirmed responses stamp `chain_tip` | | transaction.broadcast | done | Mempool accept + P2P inv | | relayfee / estimatefee / histogram | done | Libre min + live median | | TLS | external | terminate at reverse proxy; node is plain TCP | @@ -94,6 +94,31 @@ will probe tweaks. Isolate may still hardcode `electrs.cakewallet.com` after a passing probe. | DoS floor | always on | max conn / line / idle / subs / broadcast hex (`ServeLimits`); public bind OK behind proxy | +### Chain view (confirmed-tx snapshot token) + +Yuval pointed out that Electrum status and Esplora list envelopes are +**A-B-A**: a same-height reorg can leave `txid:height` (and a height-keyed +join cache) unchanged while merkle proofs and confirming block hashes +moved. We researched +[mempool/mempool#6584](https://github.com/mempool/mempool/issues/6584) +(tnull: stamp chain tip **hash** on every API response header so sequential +fetches detect tip movement, including A-B-A) and +[spesmilo/electrum-protocol#2](https://github.com/spesmilo/electrum-protocol/pull/2) +(1.7 `chaintip` on `scriptpubkey.*`, reverted in +[#17](https://github.com/spesmilo/electrum-protocol/pull/17) because ElectrumX +is bitcoind middleware and cannot pin). rbitcoin owns Query+store, so we pin +the published tip and retry if it disconnects +([`docs/concurrency.md`](docs/concurrency.md#confirmed-tx-readers-pin--retry-not-a-lock)). + +| Surface | Token | Body | +|---------|--------|------| +| Esplora HTTP | `X-Bitcoin-Chain-Tip` + `X-Bitcoin-Chain-Tip-Height` | Unchanged JSON. Client: if two sequential fetches disagree on the hash, drop the batch and restart. | +| Electrum TCP | JSON-RPC extra members `chain_tip` / `chain_tip_height` next to `result` (ping/version omit). `server.features.chain_tip`. | `result` shape unchanged. Status preimage includes confirming `blockhash` so subscribe clients refetch on same-height replace. Notification `params` stay `[scripthash, status]`. | + +We stamp **tip**, not only the last relevant history tx hash (empty history +and list envelopes still need a token). We do **not** serve “as of hash H” +after H is disconnected. + ## Esplora REST surface Plain HTTP via `--esplora-listen` / conf `esplora_listen` (default **off**). TLS @@ -101,11 +126,11 @@ via reverse proxy; app `ServeLimits` always on (same model as Electrum). | Endpoint group | Status | Notes | |----------------|--------|-------| -| Tip | done | `/blocks/tip/height`, `/blocks/tip/hash` | +| Tip | done | `/blocks/tip/height`, `/blocks/tip/hash`. Every REST response with a published tip also stamps `X-Bitcoin-Chain-Tip` (display-order hex, same as `/blocks/tip/hash`) and `X-Bitcoin-Chain-Tip-Height`, CORS-exposed. Empty chain omits them (existing 503). If the pin dies mid-request: **503** `chain view moved`. | | Blocks list | done | `/blocks`, `/blocks/:start_height` (10 summaries, newest-first) | | Block | done | `/block/:hash` JSON, `/raw`, `/status`, `/header`, `/txids`, `/txid/:i`, `/txs[/:start]` | | Tx | done | `/tx/:txid` full JSON, `/hex`, `/raw`, `/status`, Electrum `/merkle-proof`, BIP37 `/merkleblock-proof`, `/outspend(s)` | -| Address / scripthash | done | stats + `/utxo` + `/txs` + `/txs/mempool` + `/txs/chain[/:last_seen_txid]`; `/utxo` matches Electrum listunspent (mempool funding + drop mempool-spent confirmed); `/txs` from SH join fks; last SH join reused across sequential REST calls until tip height changes; needs SH finalize | +| Address / scripthash | done | stats + `/utxo` + `/txs` + `/txs/mempool` + `/txs/chain[/:last_seen_txid]`; `/utxo` matches Electrum listunspent (mempool funding + drop mempool-spent confirmed); `/txs` from SH join fks; last SH join reused across sequential REST calls until tip **hash** changes; needs SH finalize | | Mempool / fees | done | `/mempool`, `/mempool/txids`, `/mempool/recent` (accept-order ring), `/fee-estimates` | | `POST /tx` | done | broadcast via mempool hub; **503** if hub absent | | `POST /txs/package` | done | JSON array of hex txs → `accept_package`; **503** without hub; max 25 txs | diff --git a/crates/rbitcoin-electrum/src/server.rs b/crates/rbitcoin-electrum/src/server.rs index 28f003c90..b283b4f2e 100644 --- a/crates/rbitcoin-electrum/src/server.rs +++ b/crates/rbitcoin-electrum/src/server.rs @@ -8,7 +8,7 @@ use bitcoin::hashes::Hash; use rbitcoin_consensus::ChainParams; use rbitcoin_net::MempoolHub; use rbitcoin_primitives::{Fk, Height}; -use rbitcoin_query::{HistoryFilter, Query, ShJoinSlot}; +use rbitcoin_query::{ChainView, HistoryFilter, Query, ShJoinSlot}; use rbitcoin_store::{script_hash, StoreError}; use serde_json::{json, Value}; use std::collections::HashSet; @@ -182,6 +182,9 @@ impl ElectrumHandle { pub struct TipNotify { pub height: u32, pub header_hex: String, + /// Set when this tip replaced a previous prefix (reorg / same-height). + /// Subscribe restatuses every watched scripthash, not only the new block. + pub reorg_from_height: Option, } /// Start Electrum **plain TCP** listener. @@ -370,20 +373,24 @@ where let mp = mempool.clone(); let subs: Vec<[u8; 32]> = sh_subs.iter().copied().collect(); let height = t.height; + let restatus_all = t.reorg_from_height.is_some(); let notes = tokio::task::spawn_blocking(move || { let mut out = Vec::new(); for sh in subs { - let hit = q - .scripthash_touched_at_height(&sh, Height(height)) - .ok() - .unwrap_or(false); + let hit = restatus_all + || q + .scripthash_touched_at_height(&sh, Height(height)) + .ok() + .unwrap_or(false); if !hit { continue; } let status = if let Some(mp) = &mp { scripthash_status_full(&q, mp, &sh).ok() } else { - q.scripthash_history(&sh).ok().map(|h| scripthash_status(&h)) + q.scripthash_history(&sh).ok().map(|h| { + scripthash_status(Some(&q), &h) + }) }; if let Some(status) = status { out.push((sh, status)); @@ -488,6 +495,7 @@ where continue; } let t0 = Instant::now(); + let mut stamped: Option = None; let result = if method_stays_on_worker(method) { dispatch_with_join( method, @@ -510,26 +518,47 @@ where let mut hs = header_sub; let mut shs = sh_subs.clone(); let mut slot = sh_join.take(); + let stamp = method_stamps_chain_tip(&method_owned); match tokio::task::spawn_blocking(move || { - let r = dispatch_with_join( - &method_owned, - ¶ms_owned, - &q, - &cfg, - &p, - mp.as_deref(), - &mut hs, - &mut shs, - &mut slot, - ); - (r, hs, shs, slot) + let (r, view) = if stamp { + electrum_at_chain_view(&q, |q| { + dispatch_with_join( + &method_owned, + ¶ms_owned, + q, + &cfg, + &p, + mp.as_deref(), + &mut hs, + &mut shs, + &mut slot, + ) + }) + } else { + ( + dispatch_with_join( + &method_owned, + ¶ms_owned, + &q, + &cfg, + &p, + mp.as_deref(), + &mut hs, + &mut shs, + &mut slot, + ), + None, + ) + }; + (r, hs, shs, slot, view) }) .await { - Ok((r, hs, shs, slot)) => { + Ok((r, hs, shs, slot, view)) => { header_sub = hs; sh_subs = shs; sh_join = slot; + stamped = view; r } Err(e) => { @@ -552,7 +581,7 @@ where wall_ms, None, ); - json!({"jsonrpc":"2.0","id": id, "result": v}) + rpc_result(&id, &v, stamped.as_ref()) } Err(e) => { rbitcoin_log::api_call( @@ -800,6 +829,52 @@ fn method_stays_on_worker(method: &str) -> bool { ) } +fn method_stamps_chain_tip(method: &str) -> bool { + matches!( + method, + "blockchain.scripthash.get_history" + | "blockchain.scripthash.get_balance" + | "blockchain.scripthash.listunspent" + | "blockchain.transaction.get" + | "blockchain.transaction.get_merkle" + ) +} + +fn rpc_result(id: &Value, result: &Value, view: Option<&ChainView>) -> Value { + let mut obj = json!({"jsonrpc":"2.0","id": id, "result": result}); + if let Some(v) = view { + obj["chain_tip"] = json!(hash_hex_rev(&v.hash)); + obj["chain_tip_height"] = json!(v.height.0); + } + obj +} + +fn electrum_at_chain_view(query: &Query, mut f: F) -> (Result, Option) +where + F: FnMut(&Query) -> Result, +{ + const BOUND: u32 = 8; + for _ in 0..BOUND { + let view = match query.pin_chain_view() { + Ok(v) => v, + Err(e) => return (Err(e.to_string()), None), + }; + let Some(view) = view else { + return (f(query), None); + }; + let out = f(query); + if out.is_err() { + return (out, None); + } + match view.still_live(query) { + Ok(true) => return (out, Some(view)), + Ok(false) => continue, + Err(e) => return (Err(e.to_string()), None), + } + } + (Err("chain view moved".into()), None) +} + #[cfg(test)] fn dispatch( method: &str, @@ -847,6 +922,7 @@ fn dispatch_with_join( // implement server.features. "silent_payments": [0], "tweaks": true, + "chain_tip": true, })), "blockchain.headers.subscribe" => { *header_sub = true; @@ -948,7 +1024,7 @@ fn dispatch_with_join( let hist = query .scripthash_history_slot(&sh, sh_join) .map_err(|e| e.to_string())?; - scripthash_status(&hist) + scripthash_status(Some(query), &hist) }; Ok(json!(status)) } @@ -1197,14 +1273,26 @@ fn hash_hex_rev(h: &[u8; 32]) -> String { rbitcoin_primitives::hex_encode(r) } -fn scripthash_status(hist: &[rbitcoin_query::ScriptHashHistoryItem]) -> String { +fn scripthash_status( + query: Option<&Query>, + hist: &[rbitcoin_query::ScriptHashHistoryItem], +) -> String { if hist.is_empty() { return String::new(); } use bitcoin::hashes::{sha256, Hash as _}; let mut s = String::new(); for i in hist { - s.push_str(&format!("{}:{}:", txid_hex(&i.txid), i.height)); + if i.height > 0 { + let bh = query + .and_then(|q| q.header_at_height(Height(i.height as u32)).ok()) + .flatten() + .map(|(_, rec)| hash_hex_rev(&rec.hash)) + .unwrap_or_default(); + s.push_str(&format!("{}:{}:{}:", txid_hex(&i.txid), i.height, bh)); + } else { + s.push_str(&format!("{}:{}:", txid_hex(&i.txid), i.height)); + } } let hash = sha256::Hash::hash(s.as_bytes()); rbitcoin_primitives::hex_encode(hash.to_byte_array()) @@ -1232,7 +1320,7 @@ fn scripthash_status_full_slot( }); } hist.sort_by_key(|i| i.height); - Ok(scripthash_status(&hist)) + Ok(scripthash_status(Some(query), &hist)) } /// Helper to compute electrum scripthash hex (reversed) from script bytes. @@ -1315,13 +1403,16 @@ mod tests { .unwrap_err() .contains("to_height")); - let empty_status = scripthash_status(&[]); + let empty_status = scripthash_status(None, &[]); assert!(empty_status.is_empty()); - let status = scripthash_status(&[rbitcoin_query::ScriptHashHistoryItem { - height: 1, - txid: [1u8; 32], - tx_fk: Fk::NULL, - }]); + let status = scripthash_status( + None, + &[rbitcoin_query::ScriptHashHistoryItem { + height: 1, + txid: [1u8; 32], + tx_fk: Fk::NULL, + }], + ); assert_eq!(status.len(), 64); use bitcoin::hashes::Hash; @@ -1418,6 +1509,7 @@ mod tests { assert_eq!(features["server_version"], v[0]); assert_eq!(features["silent_payments"], json!([0])); assert_eq!(features["tweaks"], json!(true)); + assert_eq!(features["chain_tip"], json!(true)); let probe = dispatch( "blockchain.tweaks.subscribe", @@ -1651,6 +1743,10 @@ mod tests { let v: Value = serde_json::from_str(&resp).unwrap(); assert_eq!(v["id"], 1); assert!(v.get("result").is_some()); + assert!( + v.get("chain_tip").is_none(), + "ping must not grow chain_tip: {v}" + ); // Empty line ignored; malformed JSON ignored; then version. let stream = reader.into_inner(); @@ -1678,6 +1774,184 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + #[tokio::test] + async fn chain_view_get_history_stamps_tip_and_changes_on_replace() { + use rbitcoin_primitives::{Fk, Height}; + use rbitcoin_query::TxApply; + use rbitcoin_store::{HeaderRecord, InputRecord, OutputRecord, TxRecord}; + + let (dir, q) = tmp_store(); + let (h0, t0) = { + let merkle = [0xab; 32]; + let header = HeaderRecord { + prev_fk: Fk::NULL, + version: 1, + timestamp: 1, + bits: 0x207fffff, + nonce: 0, + merkle_root: merkle, + hash: merkle, + }; + let mut txid = [0xcb; 32]; + txid[31] = 0; + let ta = TxApply { + tx: TxRecord { + txid, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![0], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(50_0000_0000, vec![0x51])], + }; + (header, ta) + }; + let hash0 = h0.hash; + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + let mut h1 = h0.clone(); + h1.prev_fk = prev_fk; + h1.timestamp = 2; + h1.nonce = 1; + h1.hash = rbitcoin_store::block_header_hash( + h1.version, + &hash0, + &h1.merkle_root, + h1.timestamp, + h1.bits, + h1.nonce, + ); + let mut t1 = { + let mut txid = [0xcb; 32]; + txid[5] = 0xaa; + TxApply { + tx: TxRecord { + txid, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![1], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(50_0000_0000, vec![0x51])], + } + }; + t1.tx.txid[0] = 1; + q.connect_block(Height(1), &h1, &[t1]).unwrap(); + let tip_a = hash_hex_rev(&h1.hash); + + let params = ChainParams::regtest(); + let q = std::sync::Arc::new(q); + let (tip_tx, _) = broadcast::channel(4); + let cfg = ElectrumConfig::for_params("127.0.0.1:0".parse().unwrap(), ¶ms); + let handle = run_electrum(cfg, Arc::clone(&q), params, tip_tx, None) + .await + .expect("listen"); + let sh = electrum_scripthash_hex(&[0x51]); + + let mut stream = TcpStream::connect(handle.local_addr).await.unwrap(); + let req = json!({ + "jsonrpc":"2.0","id":1, + "method":"blockchain.scripthash.get_history","params":[sh] + }); + let mut line = serde_json::to_string(&req).unwrap(); + line.push('\n'); + stream.write_all(line.as_bytes()).await.unwrap(); + let mut reader = BufReader::new(&mut stream); + let mut resp = String::new(); + tokio::time::timeout( + std::time::Duration::from_secs(3), + reader.read_line(&mut resp), + ) + .await + .expect("timeout") + .unwrap(); + let v: Value = serde_json::from_str(&resp).unwrap(); + assert_eq!(v["id"], 1, "{v}"); + assert!(v["result"].as_array().is_some(), "{v}"); + assert_eq!(v["chain_tip"], tip_a, "{v}"); + assert_eq!(v["chain_tip_height"], 1, "{v}"); + + q.disconnect_tip().unwrap(); + let mut h1b = h1.clone(); + h1b.nonce = 9; + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &hash0, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + let mut t1b = { + let mut txid = [0xcb; 32]; + txid[5] = 0xbb; + TxApply { + tx: TxRecord { + txid, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![2], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(50_0000_0000, vec![0x51])], + } + }; + t1b.tx.txid[0] = 2; + q.connect_block(Height(1), &h1b, &[t1b]).unwrap(); + let tip_b = hash_hex_rev(&h1b.hash); + assert_ne!(tip_a, tip_b); + + let stream = reader.into_inner(); + let mut line = serde_json::to_string(&req).unwrap(); + line.push('\n'); + stream.write_all(line.as_bytes()).await.unwrap(); + let mut reader = BufReader::new(stream); + resp.clear(); + tokio::time::timeout( + std::time::Duration::from_secs(3), + reader.read_line(&mut resp), + ) + .await + .expect("timeout") + .unwrap(); + let v: Value = serde_json::from_str(&resp).unwrap(); + assert_eq!(v["chain_tip"], tip_b, "{v}"); + assert_eq!(v["chain_tip_height"], 1, "{v}"); + + handle.shutdown().await; + let _ = std::fs::remove_dir_all(&dir); + } + /// On a 1-worker runtime, ping must complete while another socket is inside /// a real blocking store query (`blockchain.block.headers`). #[tokio::test(flavor = "current_thread")] @@ -2317,7 +2591,7 @@ mod tests { }; let full_hist = q.scripthash_history(&sh_bytes).unwrap(); assert_eq!(full_hist.len(), 4); - assert_eq!(scripthash_status(&full_hist), status_s); + assert_eq!(scripthash_status(Some(&q), &full_hist), status_s); let _ = prev; let _ = std::fs::remove_dir_all(&dir); @@ -2393,6 +2667,7 @@ mod tests { .send(TipNotify { height: 1, header_hex: "aa".repeat(80), + reorg_from_height: None, }) .unwrap(); resp.clear(); @@ -2517,6 +2792,7 @@ mod tests { .send(TipNotify { height: 1, header_hex: "aa".repeat(80), + reorg_from_height: None, }) .unwrap(); resp.clear(); @@ -2641,6 +2917,7 @@ mod tests { .send(TipNotify { height: 1, header_hex: "aa".repeat(80), + reorg_from_height: None, }) .unwrap(); resp.clear(); @@ -2655,6 +2932,246 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + #[test] + fn chain_view_status_includes_blockhash() { + use rbitcoin_primitives::{Fk, Height}; + use rbitcoin_query::TxApply; + use rbitcoin_store::{HeaderRecord, InputRecord, OutputRecord, TxRecord}; + + let (dir, q) = tmp_store(); + let merkle = [0xab; 32]; + let h0 = HeaderRecord { + prev_fk: Fk::NULL, + version: 1, + timestamp: 1, + bits: 0x207fffff, + nonce: 0, + merkle_root: merkle, + hash: merkle, + }; + let mut txid = [0xcb; 32]; + txid[31] = 0; + let t0 = TxApply { + tx: TxRecord { + txid, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![0], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(1, vec![0x51])], + }; + q.connect_block(Height(0), &h0, &[t0.clone()]).unwrap(); + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + let mut h1 = h0.clone(); + h1.prev_fk = prev_fk; + h1.timestamp = 2; + h1.nonce = 1; + h1.hash = rbitcoin_store::block_header_hash( + h1.version, + &merkle, + &h1.merkle_root, + h1.timestamp, + h1.bits, + h1.nonce, + ); + let mut t1 = t0; + t1.tx.txid[5] = 0xaa; + q.connect_block(Height(1), &h1, &[t1.clone()]).unwrap(); + let sh = script_hash(&[0x51]); + let hist_a = q.scripthash_history(&sh).unwrap(); + let status_a = scripthash_status(Some(&q), &hist_a); + let legacy = { + use bitcoin::hashes::{sha256, Hash as _}; + let mut s = String::new(); + for i in &hist_a { + s.push_str(&format!("{}:{}:", txid_hex(&i.txid), i.height)); + } + rbitcoin_primitives::hex_encode(sha256::Hash::hash(s.as_bytes()).to_byte_array()) + }; + assert_ne!( + status_a, legacy, + "status preimage must include confirming block hash" + ); + + q.disconnect_tip().unwrap(); + let mut h1b = h1.clone(); + h1b.nonce = 9; + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &merkle, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + q.connect_block(Height(1), &h1b, &[t1]).unwrap(); + let hist_b = q.scripthash_history(&sh).unwrap(); + let status_b = scripthash_status(Some(&q), &hist_b); + assert_eq!( + hist_a + .iter() + .map(|i| (i.txid, i.height)) + .collect::>(), + hist_b + .iter() + .map(|i| (i.txid, i.height)) + .collect::>(), + "same txs at the same heights" + ); + assert_ne!( + status_a, status_b, + "same-height replace must change status via blockhash" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[tokio::test] + async fn chain_view_reorg_notifies_dropped_scripthash() { + use rbitcoin_query::TxApply; + use rbitcoin_store::{HeaderRecord, InputRecord, OutputRecord, TxRecord}; + + let (dir, q) = tmp_store(); + let mut hash = [0u8; 32]; + hash[0] = 0x42; + let header = HeaderRecord { + prev_fk: Fk::NULL, + version: 1, + timestamp: 1, + bits: 0x207fffff, + nonce: 0, + merkle_root: hash, + hash, + }; + let mut txid = [0u8; 32]; + txid[31] = 0xcb; + let ta = TxApply { + tx: TxRecord { + txid, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![0], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(1, vec![0x51])], + }; + let hfk0 = q.connect_block(Height(0), &header, &[ta]).unwrap(); + let sh = electrum_scripthash_hex(&[0x51]); + + let params = ChainParams::regtest(); + let q = std::sync::Arc::new(q); + let (tip_tx, _) = broadcast::channel(2); + let cfg = ElectrumConfig::for_params("127.0.0.1:0".parse().unwrap(), ¶ms); + let handle = run_electrum(cfg, std::sync::Arc::clone(&q), params, tip_tx.clone(), None) + .await + .unwrap(); + let mut stream = TcpStream::connect(handle.local_addr).await.unwrap(); + let mut line = serde_json::to_string(&json!({ + "jsonrpc":"2.0","id":1,"method":"blockchain.scripthash.subscribe","params":[sh] + })) + .unwrap(); + line.push('\n'); + stream.write_all(line.as_bytes()).await.unwrap(); + let mut reader = BufReader::new(&mut stream); + let mut resp = String::new(); + tokio::time::timeout( + std::time::Duration::from_secs(3), + reader.read_line(&mut resp), + ) + .await + .unwrap() + .unwrap(); + let first: Value = serde_json::from_str(&resp).unwrap(); + let status0 = first["result"].as_str().unwrap().to_string(); + assert!(!status0.is_empty()); + let _ = hfk0; + + q.disconnect_tip().unwrap(); + let mut hash_b = [0u8; 32]; + hash_b[0] = 0x43; + let header_b = HeaderRecord { + prev_fk: Fk::NULL, + version: 1, + timestamp: 1, + bits: 0x207fffff, + nonce: 1, + merkle_root: hash_b, + hash: hash_b, + }; + let mut txid_b = [0u8; 32]; + txid_b[0] = 0x99; + txid_b[31] = 0xcd; + let ta_b = TxApply { + tx: TxRecord { + txid: txid_b, + version: 1, + locktime: 0, + input_start_fk: Fk::NULL, + input_count: 1, + output_start_fk: Fk::NULL, + output_count: 1, + }, + inputs: vec![InputRecord { + prev_txid: [0u8; 32], + create_fk: Fk::NULL, + prev_index: u32::MAX, + sequence: u32::MAX, + script_sig: vec![1], + witness: vec![], + }], + outputs: vec![OutputRecord::unspent(1, vec![0x00])], + }; + q.connect_block(Height(0), &header_b, &[ta_b]).unwrap(); + tip_tx + .send(TipNotify { + height: 0, + header_hex: "aa".repeat(80), + reorg_from_height: Some(0), + }) + .unwrap(); + resp.clear(); + tokio::time::timeout( + std::time::Duration::from_secs(3), + reader.read_line(&mut resp), + ) + .await + .expect("reorg must restatus even when the new block misses the scripthash") + .unwrap(); + let push: Value = serde_json::from_str(&resp).unwrap(); + assert_eq!( + push["method"].as_str(), + Some("blockchain.scripthash.subscribe") + ); + assert_ne!( + push["params"][1].as_str().unwrap_or("missing"), + status0, + "dropped history must change status" + ); + + handle.shutdown().await; + let _ = std::fs::remove_dir_all(&dir); + } + #[test] fn dispatch_with_mempool_and_param_errors() { use rbitcoin_net::MempoolHub; diff --git a/crates/rbitcoin-esplora/src/server.rs b/crates/rbitcoin-esplora/src/server.rs index 153e68533..c0caaeee4 100644 --- a/crates/rbitcoin-esplora/src/server.rs +++ b/crates/rbitcoin-esplora/src/server.rs @@ -4,7 +4,7 @@ use crate::handlers; use crate::tx_json::{build_tx_json, tx_status_json}; use crate::ws; use axum::extract::{Path, Request, State}; -use axum::http::{header, StatusCode}; +use axum::http::{header, HeaderValue, StatusCode}; use axum::middleware::{self, Next}; use axum::response::{IntoResponse, Response}; use axum::routing::{get, post}; @@ -15,7 +15,7 @@ use bitcoin::Network; use rbitcoin_electrum::ServeLimits; use rbitcoin_net::{MempoolHub, TipEvent}; use rbitcoin_primitives::Height; -use rbitcoin_query::{Query, ShJoinSlot}; +use rbitcoin_query::{ChainView, Query, ShJoinSlot}; use rbitcoin_store::StoreError; use std::net::SocketAddr; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; @@ -81,6 +81,43 @@ async fn meter_rest(req: Request, next: Next) -> Response { resp } +pub(crate) const HDR_CHAIN_TIP: &str = "x-bitcoin-chain-tip"; +pub(crate) const HDR_CHAIN_TIP_HEIGHT: &str = "x-bitcoin-chain-tip-height"; + +fn stamp_chain_view_headers(resp: &mut Response, view: &ChainView) { + let hash = block_hash_hex(&view.hash); + let height = view.height.0.to_string(); + if let Ok(v) = HeaderValue::from_str(&hash) { + resp.headers_mut().insert(HDR_CHAIN_TIP, v); + } + if let Ok(v) = HeaderValue::from_str(&height) { + resp.headers_mut().insert(HDR_CHAIN_TIP_HEIGHT, v); + } + resp.headers_mut().insert( + header::ACCESS_CONTROL_EXPOSE_HEADERS, + HeaderValue::from_static("X-Bitcoin-Chain-Tip, X-Bitcoin-Chain-Tip-Height"), + ); +} + +async fn stamp_chain_view_mw(State(st): State, req: Request, next: Next) -> Response { + let view = match st.query.pin_chain_view() { + Ok(v) => v, + Err(e) => return store_err(e), + }; + let mut resp = next.run(req).await; + let Some(view) = view else { + return resp; + }; + match view.still_live(&st.query) { + Ok(true) => { + stamp_chain_view_headers(&mut resp, &view); + resp + } + Ok(false) => (StatusCode::SERVICE_UNAVAILABLE, "chain view moved").into_response(), + Err(e) => store_err(e), + } +} + /// Default concurrent upgraded WebSocket sockets (separate from REST concurrency). pub const DEFAULT_MAX_WS_CONNECTIONS: usize = 64; /// Default max inbound client WebSocket text frame size. @@ -274,7 +311,11 @@ pub async fn run_esplora( .route("/mempool/recent", get(handlers::mempool_recent)) .route("/fee-estimates", get(handlers::fee_estimates)) .fallback(fallback_404) - // Outer → inner: concurrency → body → timeout → meter (around handler). + // Outer → inner: concurrency → body → timeout → meter → chain-view stamp. + .layer(middleware::from_fn_with_state( + state.clone(), + stamp_chain_view_mw, + )) .layer(middleware::from_fn(meter_rest)) .layer(TimeoutLayer::with_status_code( StatusCode::REQUEST_TIMEOUT, @@ -428,6 +469,7 @@ pub(crate) fn not_found() -> Response { pub(crate) fn store_err(e: rbitcoin_query::QueryError) -> Response { match e { StoreError::NotFound => not_found(), + StoreError::Stale(m) => (StatusCode::SERVICE_UNAVAILABLE, m).into_response(), other => (StatusCode::INTERNAL_SERVER_ERROR, other.to_string()).into_response(), } } @@ -529,13 +571,30 @@ mod tests { (header, ta) } + fn header_value(text: &str, name: &str) -> Option { + let want = name.to_ascii_lowercase(); + text.split("\r\n\r\n") + .next() + .unwrap_or("") + .lines() + .skip(1) + .filter_map(|l| l.split_once(':')) + .find(|(k, _)| k.trim().eq_ignore_ascii_case(&want)) + .map(|(_, v)| v.trim().to_string()) + } + async fn http_get(addr: SocketAddr, path: &str) -> (u16, String) { + let (status, _hdrs, body) = http_get_raw(addr, path).await; + (status, body) + } + + async fn http_get_raw(addr: SocketAddr, path: &str) -> (u16, String, String) { let mut stream = TcpStream::connect(addr).await.expect("connect"); let req = format!("GET {path} HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n"); stream.write_all(req.as_bytes()).await.unwrap(); let mut buf = Vec::new(); stream.read_to_end(&mut buf).await.unwrap(); - let text = String::from_utf8_lossy(&buf); + let text = String::from_utf8_lossy(&buf).into_owned(); let status = text .lines() .next() @@ -548,7 +607,7 @@ mod tests { .unwrap_or("") .trim() .to_string(); - (status, body) + (status, text, body) } #[tokio::test] @@ -587,14 +646,98 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + #[tokio::test] + async fn chain_view_tip_header_matches_hash_body() { + let (dir, q) = temp_query("chain-view-hdr"); + let (h0, t0) = coinbase(0, Fk::NULL, None); + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let q = Arc::new(q); + let cfg = EsploraConfig::new("127.0.0.1:0".parse().unwrap()); + let handle = run_esplora(cfg, q, None, None).await.expect("listen"); + let addr = handle.local_addr; + + let (st, raw, body) = http_get_raw(addr, "/blocks/tip/hash").await; + assert_eq!(st, 200, "hash body={body}"); + let tip = header_value(&raw, HDR_CHAIN_TIP).expect("X-Bitcoin-Chain-Tip"); + let height = header_value(&raw, HDR_CHAIN_TIP_HEIGHT).expect("height"); + assert_eq!(tip, body); + assert_eq!(height, "0"); + let expose = header_value(&raw, "access-control-expose-headers").unwrap_or_default(); + assert!( + expose.to_ascii_lowercase().contains("x-bitcoin-chain-tip"), + "CORS must expose the tip header: {expose}" + ); + + let (st, raw, _) = http_get_raw(addr, "/blocks/tip/height").await; + assert_eq!(st, 200); + assert_eq!( + header_value(&raw, HDR_CHAIN_TIP).as_deref(), + Some(tip.as_str()) + ); + + handle.shutdown().await; + let _ = std::fs::remove_dir_all(&dir); + } + + #[tokio::test] + async fn chain_view_header_changes_after_same_height_replace() { + let (dir, q) = temp_query("chain-view-reorg-hdr"); + let (h0, t0) = coinbase(0, Fk::NULL, None); + let hash0 = h0.hash; + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + let (h1, t1) = coinbase(1, prev_fk, Some(hash0)); + q.connect_block(Height(1), &h1, &[t1]).unwrap(); + let q = Arc::new(q); + let cfg = EsploraConfig::new("127.0.0.1:0".parse().unwrap()); + let handle = run_esplora(cfg, Arc::clone(&q), None, None) + .await + .expect("listen"); + let addr = handle.local_addr; + let sh = rbitcoin_store::script_hash(&[0x51]); + let sh_hex = block_hash_hex(&sh); + + let (st, raw_a, _) = http_get_raw(addr, &format!("/scripthash/{sh_hex}/utxo")).await; + assert_eq!(st, 200, "utxo A"); + let tip_a = header_value(&raw_a, HDR_CHAIN_TIP).expect("tip A"); + assert_eq!(tip_a, block_hash_hex(&h1.hash)); + + q.disconnect_tip().unwrap(); + let mut h1b = coinbase(1, prev_fk, Some(hash0)).0; + h1b.nonce = h1.nonce.wrapping_add(1); + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &hash0, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + let t1b = coinbase(1, prev_fk, Some(hash0)).1; + q.connect_block(Height(1), &h1b, &[t1b]).unwrap(); + + let (st, raw_b, _) = http_get_raw(addr, &format!("/scripthash/{sh_hex}/utxo")).await; + assert_eq!(st, 200, "utxo B"); + let tip_b = header_value(&raw_b, HDR_CHAIN_TIP).expect("tip B"); + assert_eq!(tip_b, block_hash_hex(&h1b.hash)); + assert_ne!(tip_a, tip_b); + + handle.shutdown().await; + let _ = std::fs::remove_dir_all(&dir); + } + #[tokio::test] async fn empty_chain_tip_is_unavailable() { let (dir, q) = temp_query("empty"); let q = Arc::new(q); let cfg = EsploraConfig::new("127.0.0.1:0".parse().unwrap()); let handle = run_esplora(cfg, q, None, None).await.expect("listen"); - let (st, _) = http_get(handle.local_addr, "/blocks/tip/height").await; + let (st, raw, _) = http_get_raw(handle.local_addr, "/blocks/tip/height").await; assert_eq!(st, 503); + assert!( + header_value(&raw, HDR_CHAIN_TIP).is_none(), + "empty chain must omit the tip header" + ); let (st, _) = http_get(handle.local_addr, "/blocks/tip/hash").await; assert_eq!(st, 503); handle.shutdown().await; diff --git a/crates/rbitcoin-esplora/src/tx_json.rs b/crates/rbitcoin-esplora/src/tx_json.rs index 0e2e47f9f..1751fad34 100644 --- a/crates/rbitcoin-esplora/src/tx_json.rs +++ b/crates/rbitcoin-esplora/src/tx_json.rs @@ -13,7 +13,8 @@ use std::collections::HashMap; /// Esplora `status` object for a Class A tx fk (confirmed or not). pub fn tx_status_json(query: &Query, tx_fk: Fk) -> Result { - let confirmed = query.store().is_confirmed_strong(tx_fk)?; + let tip = query.pin_chain_view()?.map(|v| v.height.0); + let confirmed = query.store().is_confirmed_strong_at(tx_fk, tip)?; if !confirmed { return Ok(json!({ "confirmed": false })); } diff --git a/crates/rbitcoin-node/src/run.rs b/crates/rbitcoin-node/src/run.rs index 11c3e88db..84831c2e1 100644 --- a/crates/rbitcoin-node/src/run.rs +++ b/crates/rbitcoin-node/src/run.rs @@ -567,6 +567,11 @@ pub async fn run_p2p(config: NodeConfig) -> Result<(), NodeError> { let _ = bridge_tx.send(TipNotify { height: ev.height, header_hex: rbitcoin_primitives::hex_encode(buf), + reorg_from_height: if ev.reorg_branch_len > 0 { + Some(ev.height.saturating_sub(ev.reorg_branch_len)) + } else { + None + }, }); } Err(broadcast::error::RecvError::Lagged(_)) => continue, diff --git a/crates/rbitcoin-query/src/chain_view.rs b/crates/rbitcoin-query/src/chain_view.rs index 05edf62e3..33f290618 100644 --- a/crates/rbitcoin-query/src/chain_view.rs +++ b/crates/rbitcoin-query/src/chain_view.rs @@ -1,8 +1,66 @@ -//! Header navigation, locators, height lookup. +//! Header navigation, locators, height lookup, confirmed-read pin. use super::*; +/// Published best-chain prefix a confirmed-tx read was built against. +/// +/// `still_live` is true while `confirmed[height]` is still this `header_fk`. +/// Tip **extension** leaves a shorter pin live (prefix-stable). Disconnect or +/// same-height replace of that height does not. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct ChainView { + pub height: Height, + pub hash: [u8; 32], + pub header_fk: Fk, +} + +impl ChainView { + pub fn still_live(&self, query: &Query) -> Result { + match query.store.confirmed.get(self.height)? { + Some(fk) if fk == self.header_fk => Ok(true), + _ => Ok(false), + } + } +} + impl Query { + /// Capture the published tip as a [`ChainView`], or `None` if the chain is empty. + pub fn pin_chain_view(&self) -> Result, QueryError> { + let Some(height) = self.tip_height() else { + return Ok(None); + }; + let Some(header_fk) = self.store.confirmed.get(height)? else { + return Ok(None); + }; + let rec = self.store.get_header(header_fk)?; + Ok(Some(ChainView { + height, + hash: rec.hash, + header_fk, + })) + } + + /// Pin, run `f`, return the body if that pin is still published; else retry. + /// + /// Bound is 8. Empty chain is [`StoreError::NotFound`]. A pin that never + /// stays live is [`StoreError::Stale`], not corruption. + pub fn run_at_chain_view(&self, mut f: F) -> Result<(ChainView, T), QueryError> + where + F: FnMut(&ChainView) -> Result, + { + const BOUND: u32 = 8; + for _ in 0..BOUND { + let Some(view) = self.pin_chain_view()? else { + return Err(StoreError::NotFound); + }; + let out = f(&view)?; + if view.still_live(self)? { + return Ok((view, out)); + } + } + Err(StoreError::Stale("chain view moved")) + } + pub fn header_at_height( &self, height: Height, diff --git a/crates/rbitcoin-query/src/lib.rs b/crates/rbitcoin-query/src/lib.rs index 48c18c766..4d794fffe 100644 --- a/crates/rbitcoin-query/src/lib.rs +++ b/crates/rbitcoin-query/src/lib.rs @@ -150,6 +150,7 @@ pub use batch_parents::{ SharedParentPin, U32Map, U64IdentityHasher, U64Map, U64Set, SPENDER_REL_UNKNOWN, }; pub use catchup::IndexMode; +pub use chain_view::ChainView; pub use confirm_load::BatchThin; pub use confirm_load::ConfirmLoadStats; pub use connect::{format_disconnect_tip_line, ConfirmPrepared}; @@ -2650,6 +2651,225 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + #[test] + fn chain_view_pin_none_on_empty_store() { + let (dir, q) = temp_query("chain-view-empty"); + assert!(q.pin_chain_view().unwrap().is_none()); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn chain_view_pin_live_across_extension_dead_after_same_height_replace() { + let (dir, q) = temp_query("chain-view-pin"); + let (h0, t0) = coinbase_block(0, Fk::NULL, None); + let hash0 = h0.hash; + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + + let genesis = q.pin_chain_view().unwrap().expect("genesis tip"); + assert_eq!(genesis.height, Height(0)); + assert_eq!(genesis.hash, hash0); + assert!(genesis.still_live(&q).unwrap()); + + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + let (h1, t1) = coinbase_block(1, prev_fk, Some(hash0)); + q.connect_block(Height(1), &h1, &[t1]).unwrap(); + assert!( + genesis.still_live(&q).unwrap(), + "prefix pin stays live across tip extension" + ); + + let tip1 = q.pin_chain_view().unwrap().expect("height 1"); + assert_eq!(tip1.height, Height(1)); + assert_eq!(tip1.hash, h1.hash); + assert!(tip1.still_live(&q).unwrap()); + + q.disconnect_tip().unwrap(); + assert!( + !tip1.still_live(&q).unwrap(), + "disconnect of pinned height kills the view" + ); + assert!(genesis.still_live(&q).unwrap()); + + let mut h1b = coinbase_block(1, prev_fk, Some(hash0)).0; + h1b.nonce = h1.nonce.wrapping_add(1); + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &hash0, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + let t1b = coinbase_block(1, prev_fk, Some(hash0)).1; + q.connect_block(Height(1), &h1b, &[t1b]).unwrap(); + assert_ne!(h1b.hash, tip1.hash); + assert!( + !tip1.still_live(&q).unwrap(), + "same-height replace must not keep the old pin live" + ); + let tip1b = q.pin_chain_view().unwrap().expect("replacement tip"); + assert_eq!(tip1b.height, Height(1)); + assert_eq!(tip1b.hash, h1b.hash); + assert!(tip1b.still_live(&q).unwrap()); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn chain_view_sh_join_slot_miss_on_same_height_replace() { + let (dir, q) = temp_query("chain-view-sh-slot"); + let (h0, t0) = coinbase_block(0, Fk::NULL, None); + let hash0 = h0.hash; + let genesis_txid = t0.tx.txid; + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + + let (h1, mut t1) = coinbase_block(1, prev_fk, Some(hash0)); + t1.tx.txid[5] = 0xaa; + let txid_a = t1.tx.txid; + q.connect_block(Height(1), &h1, &[t1]).unwrap(); + let view_a = q.pin_chain_view().unwrap().unwrap(); + + let sh = script_hash(&[0x51]); + let mut slot = None; + let hist_a = q.scripthash_history_slot(&sh, &mut slot).unwrap(); + let ids_a: Vec<_> = hist_a.iter().map(|i| i.txid).collect(); + assert!(ids_a.contains(&txid_a), "height-1 A must be in history"); + assert!(ids_a.contains(&genesis_txid)); + + let live = q.scripthash_history_in(&sh, &view_a).unwrap(); + assert!(live.iter().any(|i| i.txid == txid_a)); + let genesis_view = ChainView { + height: Height(0), + hash: hash0, + header_fk: prev_fk, + }; + let only_g = q.scripthash_history_in(&sh, &genesis_view).unwrap(); + let g_ids: Vec<_> = only_g.iter().map(|i| i.txid).collect(); + assert!(g_ids.contains(&genesis_txid)); + assert!( + !g_ids.contains(&txid_a), + "history under a height-0 pin must omit the height-1 create" + ); + + q.disconnect_tip().unwrap(); + let mut h1b = coinbase_block(1, prev_fk, Some(hash0)).0; + h1b.nonce = h1.nonce.wrapping_add(1); + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &hash0, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + let mut t1b = coinbase_block(1, prev_fk, Some(hash0)).1; + t1b.tx.txid[5] = 0xbb; + let txid_b = t1b.tx.txid; + q.connect_block(Height(1), &h1b, &[t1b]).unwrap(); + assert_ne!(txid_a, txid_b); + assert_eq!(q.tip_height(), Some(Height(1))); + + let hist_b = q.scripthash_history_slot(&sh, &mut slot).unwrap(); + let ids_b: Vec<_> = hist_b.iter().map(|i| i.txid).collect(); + assert!( + ids_b.contains(&txid_b), + "same-height replace must miss the slot and emit B, got {ids_b:?}" + ); + assert!( + !ids_b.contains(&txid_a), + "stale slot would still show A after same-height replace: {ids_b:?}" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn chain_view_run_retries_after_same_height_replace() { + let (dir, q) = temp_query("chain-view-retry"); + let (h0, t0) = coinbase_block(0, Fk::NULL, None); + let hash0 = h0.hash; + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + let (h1, t1) = coinbase_block(1, prev_fk, Some(hash0)); + q.connect_block(Height(1), &h1, &[t1]).unwrap(); + + let mut calls = 0u32; + let (view, n) = q + .run_at_chain_view(|view| { + calls += 1; + if calls == 1 { + q.disconnect_tip().unwrap(); + let mut h1b = coinbase_block(1, prev_fk, Some(hash0)).0; + h1b.nonce = h1.nonce.wrapping_add(7); + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &hash0, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + let t1b = coinbase_block(1, prev_fk, Some(hash0)).1; + q.connect_block(Height(1), &h1b, &[t1b]).unwrap(); + assert!(!view.still_live(&q).unwrap()); + } + Ok(calls) + }) + .unwrap(); + assert!(calls >= 2, "must retry after the pin died, calls={calls}"); + assert_eq!(n, calls); + assert!(view.still_live(&q).unwrap()); + assert_eq!(view.hash, q.pin_chain_view().unwrap().unwrap().hash); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn chain_view_run_errors_when_always_stale() { + let (dir, q) = temp_query("chain-view-stale"); + let (h0, t0) = coinbase_block(0, Fk::NULL, None); + let hash0 = h0.hash; + q.connect_block(Height(0), &h0, &[t0]).unwrap(); + let prev_fk = q.tip_header_fk().unwrap().unwrap(); + let (h1, t1) = coinbase_block(1, prev_fk, Some(hash0)); + q.connect_block(Height(1), &h1, &[t1]).unwrap(); + let mut nonce = h1.nonce; + let err = q + .run_at_chain_view(|_view| { + q.disconnect_tip().unwrap(); + nonce = nonce.wrapping_add(1); + let mut h1b = coinbase_block(1, prev_fk, Some(hash0)).0; + h1b.nonce = nonce; + h1b.hash = rbitcoin_store::block_header_hash( + h1b.version, + &hash0, + &h1b.merkle_root, + h1b.timestamp, + h1b.bits, + h1b.nonce, + ); + let t1b = coinbase_block(1, prev_fk, Some(hash0)).1; + q.connect_block(Height(1), &h1b, &[t1b]).unwrap(); + Ok(()) + }) + .unwrap_err(); + assert!( + err.to_string().contains("chain view moved"), + "stale bound must name the move, got {err}" + ); + assert!( + !err.to_string().contains("corrupt"), + "a moved view is not corruption: {err}" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn chain_view_run_not_found_on_empty() { + let (dir, q) = temp_query("chain-view-retry-empty"); + let err = q.run_at_chain_view(|_v| Ok(())).unwrap_err(); + assert!(matches!(err, StoreError::NotFound)); + let _ = std::fs::remove_dir_all(&dir); + } + /// Finish-path stamp-only notes must not wipe leftover_n for the fail pack. #[test] fn leftover_last_plan_batch_survives_stamp_only_note() { @@ -2865,8 +3085,9 @@ mod tests { assert_eq!(utxos[0].tx_pos, 1); assert_eq!(utxos[0].value, 20_0000_0000); + let view = q.pin_chain_view().unwrap().unwrap(); let list_join = q - .join_creates_and_spends(&sh, crate::scripthash::ShJoinNeed::LISTUNSPENT, None) + .join_creates_and_spends(&sh, crate::scripthash::ShJoinNeed::LISTUNSPENT, None, &view) .unwrap(); assert!( list_join.iter().any(|r| r.spent && r.spenders.is_empty()), @@ -2874,7 +3095,7 @@ mod tests { ); assert!(list_join.iter().any(|r| !r.spent)); let hist_join = q - .join_creates_and_spends(&sh, crate::scripthash::ShJoinNeed::HISTORY, None) + .join_creates_and_spends(&sh, crate::scripthash::ShJoinNeed::HISTORY, None, &view) .unwrap(); assert!( hist_join.iter().any(|r| r.spent && !r.spenders.is_empty()), @@ -2894,7 +3115,7 @@ mod tests { assert_eq!(stats.spent_txo_sum, 10_0000_0000); assert_eq!(body_ok_reads(), 1); let stats_join = q - .join_creates_and_spends(&sh, crate::scripthash::ShJoinNeed::CHAIN_STATS, None) + .join_creates_and_spends(&sh, crate::scripthash::ShJoinNeed::CHAIN_STATS, None, &view) .unwrap(); assert!( stats_join.iter().all(|r| r.spenders.is_empty()), diff --git a/crates/rbitcoin-query/src/scripthash.rs b/crates/rbitcoin-query/src/scripthash.rs index 2374d8328..3ff4c6512 100644 --- a/crates/rbitcoin-query/src/scripthash.rs +++ b/crates/rbitcoin-query/src/scripthash.rs @@ -257,10 +257,11 @@ pub(crate) struct ShJoinedOut { /// Last scripthash expand+spend join for one Electrum connection. /// /// Holds BALANCE-level outs + spentness. Identity is filled in place on -/// history / listunspent. Invalid once [`Query::tip_height`] moves. +/// history / listunspent. Invalid once the published tip **hash** moves +/// (including a same-height replace). pub struct ShJoinSlot { scripthash: [u8; 32], - tip_height: u32, + tip_hash: [u8; 32], joined: Vec, } @@ -379,22 +380,28 @@ impl Query { } } + fn confirmed_strong_in(&self, fk: Fk, view: &ChainView) -> Result { + self.store.is_confirmed_strong_at(fk, Some(view.height.0)) + } + /// Confirmed-strong create outpoints plus confirmed spenders (create_fk join). /// /// When `to_height` is set, creates with Class C height `>= to_height` are not - /// expanded (their spends cannot fall in the window). + /// expanded (their spends cannot fall in the window). Visibility is + /// [`Store::is_confirmed_strong_at`] against `view.height`. pub(crate) fn join_creates_and_spends( &self, scripthash: &[u8; 32], need: ShJoinNeed, to_height: Option, + view: &ChainView, ) -> Result, QueryError> { let t_pages = std::time::Instant::now(); let entries = self.store.scripthash.create_fks(scripthash)?; let pages_us = t_pages.elapsed().as_micros(); let mut fks = Vec::new(); for fk in entries { - if self.store.is_confirmed_strong(fk)? { + if self.confirmed_strong_in(fk, view)? { fks.push(fk); } } @@ -425,7 +432,7 @@ impl Query { let creates = self.expand_create_fks_wave(scripthash, wave, need)?; class_a_us = class_a_us.saturating_add(t_a.elapsed().as_micros()); let t_s = std::time::Instant::now(); - out.extend(self.join_spends_wave(&creates, need)?); + out.extend(self.join_spends_wave(&creates, need, view)?); spends_us = spends_us.saturating_add(t_s.elapsed().as_micros()); } let total_us = pages_us @@ -445,30 +452,38 @@ impl Query { Ok(out) } - fn tip_height_u32(&self) -> u32 { - self.tip_height().map(|h| h.0).unwrap_or(0) + fn sh_join_slot_hit(slot: &ShJoinSlot, scripthash: &[u8; 32], view: &ChainView) -> bool { + slot.scripthash == *scripthash && slot.tip_hash == view.hash } - fn sh_join_slot_hit(slot: &ShJoinSlot, scripthash: &[u8; 32], tip: u32) -> bool { - slot.scripthash == *scripthash && slot.tip_height == tip + fn ensure_sh_join_slot( + &self, + scripthash: &[u8; 32], + slot: &mut Option, + ) -> Result<(), QueryError> { + let Some(view) = self.pin_chain_view()? else { + *slot = None; + return Ok(()); + }; + self.ensure_sh_join_slot_in(scripthash, slot, &view) } - fn ensure_sh_join_slot( + fn ensure_sh_join_slot_in( &self, scripthash: &[u8; 32], slot: &mut Option, + view: &ChainView, ) -> Result<(), QueryError> { - let tip = self.tip_height_u32(); if slot .as_ref() - .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, tip)) + .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, view)) { return Ok(()); } - let joined = self.join_creates_and_spends(scripthash, ShJoinNeed::BALANCE, None)?; + let joined = self.join_creates_and_spends(scripthash, ShJoinNeed::BALANCE, None, view)?; *slot = Some(ShJoinSlot { scripthash: *scripthash, - tip_height: tip, + tip_hash: view.hash, joined, }); Ok(()) @@ -604,6 +619,7 @@ impl Query { &self, creates: &[ScriptHashOutpoint], need: ShJoinNeed, + view: &ChainView, ) -> Result, QueryError> { if creates.is_empty() { return Ok(Vec::new()); @@ -658,12 +674,12 @@ impl Query { } if flags & output_flags::MULTI_SPENDER != 0 { for fk in self.store.spenders_create(c.create_tx_fk, c.vout)? { - if self.store.is_confirmed_strong(fk)? { + if self.confirmed_strong_in(fk, view)? { per_out[i].push(fk); spender_fks.push(fk); } } - } else if self.store.is_confirmed_strong(field)? { + } else if self.confirmed_strong_in(field, view)? { per_out[i].push(field); spender_fks.push(field); } @@ -722,20 +738,41 @@ impl Query { self.scripthash_history_filtered(scripthash, &HistoryFilter::open()) } + /// Confirmed history for `scripthash` as of `view` (open filter). + pub fn scripthash_history_in( + &self, + scripthash: &[u8; 32], + view: &ChainView, + ) -> Result, QueryError> { + self.scripthash_history_filtered_in(scripthash, &HistoryFilter::open(), view) + } + /// Confirmed history for a scripthash, filtered by height window / limit / cursor. /// /// Assembles the full confirmed history (creates + confirmed spenders), then /// applies [`apply_history_filter`]. When `filter.to_height` is set, create /// outpoints with `create_height >= to_height` are skipped during expand /// (spends of those creates are also ≥ create height, so they cannot fall - /// inside the window). + /// inside the window). Pins the live tip. pub fn scripthash_history_filtered( &self, scripthash: &[u8; 32], filter: &HistoryFilter, + ) -> Result, QueryError> { + let Some(view) = self.pin_chain_view()? else { + return Ok(Vec::new()); + }; + self.scripthash_history_filtered_in(scripthash, filter, &view) + } + + pub fn scripthash_history_filtered_in( + &self, + scripthash: &[u8; 32], + filter: &HistoryFilter, + view: &ChainView, ) -> Result, QueryError> { let joined = - self.join_creates_and_spends(scripthash, ShJoinNeed::HISTORY, filter.to_height)?; + self.join_creates_and_spends(scripthash, ShJoinNeed::HISTORY, filter.to_height, view)?; Ok(history_items_from_joined(&joined, filter)) } @@ -756,14 +793,17 @@ impl Query { filter: &HistoryFilter, slot: &mut Option, ) -> Result, QueryError> { - let tip = self.tip_height_u32(); + let Some(view) = self.pin_chain_view()? else { + *slot = None; + return Ok(Vec::new()); + }; let hit = slot .as_ref() - .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, tip)); + .is_some_and(|s| Self::sh_join_slot_hit(s, scripthash, &view)); if !hit && filter.to_height.is_some() { - return self.scripthash_history_filtered(scripthash, filter); + return self.scripthash_history_filtered_in(scripthash, filter, &view); } - self.ensure_sh_join_slot(scripthash, slot)?; + self.ensure_sh_join_slot_in(scripthash, slot, &view)?; let recs = &mut slot .as_mut() .ok_or(StoreError::Corrupt("invariant: SH join slot missing"))? @@ -831,7 +871,13 @@ impl Query { &self, scripthash: &[u8; 32], ) -> Result { - let joined = self.join_creates_and_spends(scripthash, ShJoinNeed::BALANCE, None)?; + let Some(view) = self.pin_chain_view()? else { + return Ok(ScriptHashBalance { + confirmed: 0, + unconfirmed: 0, + }); + }; + let joined = self.join_creates_and_spends(scripthash, ShJoinNeed::BALANCE, None, &view)?; self.balance_from_joined(&joined) } @@ -842,11 +888,13 @@ impl Query { slot: &mut Option, ) -> Result { self.ensure_sh_join_slot(scripthash, slot)?; - let recs = &slot - .as_ref() - .ok_or(StoreError::Corrupt("invariant: SH join slot missing"))? - .joined; - self.balance_from_joined(recs) + let Some(recs) = slot.as_ref() else { + return Ok(ScriptHashBalance { + confirmed: 0, + unconfirmed: 0, + }); + }; + self.balance_from_joined(&recs.joined) } fn fill_create_txids( @@ -905,7 +953,11 @@ impl Query { &self, scripthash: &[u8; 32], ) -> Result, QueryError> { - let mut joined = self.join_creates_and_spends(scripthash, ShJoinNeed::LISTUNSPENT, None)?; + let Some(view) = self.pin_chain_view()? else { + return Ok(Vec::new()); + }; + let mut joined = + self.join_creates_and_spends(scripthash, ShJoinNeed::LISTUNSPENT, None, &view)?; self.fill_create_txids(&mut joined, true)?; self.listunspent_from_joined(&joined) } @@ -917,12 +969,11 @@ impl Query { slot: &mut Option, ) -> Result, QueryError> { self.ensure_sh_join_slot(scripthash, slot)?; - let recs = &mut slot - .as_mut() - .ok_or(StoreError::Corrupt("invariant: SH join slot missing"))? - .joined; - self.fill_create_txids(recs, true)?; - self.listunspent_from_joined(recs) + let Some(recs) = slot.as_mut() else { + return Ok(Vec::new()); + }; + self.fill_create_txids(&mut recs.joined, true)?; + self.listunspent_from_joined(&recs.joined) } /// Confirmed unspents whose `scriptPubKey` is in `scripts`. @@ -1004,7 +1055,17 @@ impl Query { &self, scripthash: &[u8; 32], ) -> Result { - let joined = self.join_creates_and_spends(scripthash, ShJoinNeed::CHAIN_STATS, None)?; + let Some(view) = self.pin_chain_view()? else { + return Ok(ScriptHashChainStats { + tx_count: 0, + funded_txo_count: 0, + funded_txo_sum: 0, + spent_txo_count: 0, + spent_txo_sum: 0, + }); + }; + let joined = + self.join_creates_and_spends(scripthash, ShJoinNeed::CHAIN_STATS, None, &view)?; self.chain_stats_from_joined(&joined) } @@ -1015,11 +1076,16 @@ impl Query { slot: &mut Option, ) -> Result { self.ensure_sh_join_slot(scripthash, slot)?; - let recs = &slot - .as_ref() - .ok_or(StoreError::Corrupt("invariant: SH join slot missing"))? - .joined; - self.chain_stats_from_joined(recs) + let Some(recs) = slot.as_ref() else { + return Ok(ScriptHashChainStats { + tx_count: 0, + funded_txo_count: 0, + funded_txo_sum: 0, + spent_txo_count: 0, + spent_txo_sum: 0, + }); + }; + self.chain_stats_from_joined(&recs.joined) } } diff --git a/crates/rbitcoin-store/src/error.rs b/crates/rbitcoin-store/src/error.rs index feadf4a5b..6e20d68c2 100644 --- a/crates/rbitcoin-store/src/error.rs +++ b/crates/rbitcoin-store/src/error.rs @@ -25,6 +25,8 @@ pub enum StoreError { Cancelled(&'static str), /// Operator layout / open-option error (not on-disk corruption). Layout(String), + /// Published chain prefix moved (reorg) during a confirmed-tx read. Retry. + Stale(&'static str), } impl StoreError { @@ -56,6 +58,7 @@ impl fmt::Display for StoreError { StoreError::BudgetFull(m) => write!(f, "budget full: {m}"), StoreError::Cancelled(m) => write!(f, "cancelled: {m}"), StoreError::Layout(m) => write!(f, "{m}"), + StoreError::Stale(m) => write!(f, "{m}"), } } } @@ -96,6 +99,7 @@ mod tests { StoreError::BudgetFull("block_queue"), StoreError::Cancelled("stop"), StoreError::Layout("inwit is on a cold datadir".into()), + StoreError::Stale("chain view moved"), ]; let texts: Vec = arms.iter().map(|e| e.to_string()).collect(); assert_eq!(texts[0], "invalid store magic"); @@ -109,6 +113,7 @@ mod tests { assert!(texts[7].contains("budget full: block_queue")); assert!(texts[8].contains("cancelled: stop")); assert_eq!(texts[9], "inwit is on a cold datadir"); + assert_eq!(texts[10], "chain view moved"); for e in &arms { assert!(e.source().is_none()); } diff --git a/crates/rbitcoin-test/tests/electrum_protocol.rs b/crates/rbitcoin-test/tests/electrum_protocol.rs index d70bbb952..353691a1f 100644 --- a/crates/rbitcoin-test/tests/electrum_protocol.rs +++ b/crates/rbitcoin-test/tests/electrum_protocol.rs @@ -126,6 +126,7 @@ async fn electrum_server_version_history_balance() { .send(rbitcoin_electrum::TipNotify { height: tip_h + 1, header_hex: tip_hex.clone(), + reorg_from_height: None, }) .expect("tip push"); // Notification has no id — wait for one line. diff --git a/docs/concurrency.md b/docs/concurrency.md index b3dbdeb3f..0d4d0fd71 100644 --- a/docs/concurrency.md +++ b/docs/concurrency.md @@ -79,6 +79,31 @@ spend annotations. There is **no** global “pause queries during confirm write.” Tip-as-commit + `is_confirmed_strong` define query visibility ([`crash-recovery.md`](./crash-recovery.md)). +### Confirmed-tx readers: pin + retry (not a lock) + +Wallet APIs (Electrum / Esplora) must not mix two chain prefixes in one +response, and they must tell the client **which tip hash** the body belongs +to. Yuval brought the A-B-A hole to our attention (same height, different +block, silent status). The shape we ship follows +[mempool/mempool#6584](https://github.com/mempool/mempool/issues/6584) +(HTTP tip-hash header on every response) and the Electrum 1.7 +`chaintip` discussion in +[spesmilo/electrum-protocol#2](https://github.com/spesmilo/electrum-protocol/pull/2) +(added, then reverted in +[#17](https://github.com/spesmilo/electrum-protocol/pull/17) because ElectrumX +cannot pin bitcoind RPC — we can). + +| Rule | Detail | +|------|--------| +| Pin | `Query::pin_chain_view` captures `{height, hash, header_fk}` of published tip | +| Filter | SH join uses `is_confirmed_strong_at(fk, view.height)`; slot keys on **hash** | +| Live-check | `ChainView::still_live` ⇔ `confirmed[height] == header_fk` | +| Extension | Prefix pin stays live; creates above the pin are filtered | +| Disconnect / same-height replace | Pin dies; `run_at_chain_view` retries (bound 8) then `StoreError::Stale` | +| Not OK | Pause queries during write, MVCC Class C, serving a disconnected hash | + +API tokens: [`COMPAT.md`](../COMPAT.md) (Esplora headers, Electrum JSON-RPC extra members, status preimage). + ## Practical rules 1. Do **not** spawn a second Class A writer while IBD confirm write is running. diff --git a/docs/crash-recovery.md b/docs/crash-recovery.md index 770123bd1..a6d59b188 100644 --- a/docs/crash-recovery.md +++ b/docs/crash-recovery.md @@ -16,7 +16,7 @@ Best-chain views ignore uncommitted Class C state: | 4. Body-queue dequeue for those heights | Only after confirm-write returns Ok | | 5. Spend annotations (Direct) | After tip; spentness filters use strong+fence | -`is_confirmed_strong(tx)` ⇔ strong ∧ height fence contains the fk. Queries that mean “on best chain” use this (or equivalent). +`is_confirmed_strong(tx)` ⇔ strong ∧ height fence contains the fk. Queries that mean “on best chain” use this (or equivalent). Confirmed-tx **API** readers pin that tip (`Query::pin_chain_view` / `still_live`) and retry on disconnect rather than pausing the writer — [`concurrency.md`](./concurrency.md#confirmed-tx-readers-pin--retry-not-a-lock). On open (in order):