Utreexo, State Commitments, and Lite Wallet support

[akumaigorodski]

This is a second, more structured attempt to address three related areas:

• Full-node Utreexo accumulator handling.
• A consensus-defined block-header commitment to compact chain state, including state not derivable from UTXOs, such as active AMMs and auctions, open for future extensions, intended for fast full node sync.
• Full-node support for lite wallets.

This began as lite-wallet work, but all three depend on the same consensus-critical state commitment, which should be defined first.

Plan

1. Define storage formats suited to Utreexo and lite-wallet queries.
2. Define an extensible compact-state bundle for fast, trust-less initial sync; commit to it in block headers and add network transfer.
3. Add lite-wallet data formats and protocol support.

[akumaigorodski]

So far this is the first point of the plan and it only touches proposed storage formats such as:

• UTXOs searchable by address and creation height, so lite wallets can request delta updates (from genesis block for first sync, from block N since last time a wallet has been launched etc.).
• Recent transactions archive, also searchable by address and height and along with Merkle proofs, so lite wallets can request these.
• Utreexo db formats for storing infrequent memforest snapshots and less heavy per-block utreexo delta updates.

The general rationale is that lite wallet sync should not be a computational/traffic hit for a full node and it should be able to serve everything with minimal effort.

When it comes to utreexo storage formats, this is a tradeoff between storing a lot of heavy memforest snapshots and taking a lot of computation/time to reconstruct current memforest from per-block diffs.

All proposed databases are designed to be prunable (except UTXO database).

[akumaigorodski]

I propose to move though the whole plan by first reaching an agreement on what is already done, and then the further work will be based on that, point by point.