Skip to content

Feat/pipeline parallelize fastsync - #59

Open
secretnamebasis wants to merge 6 commits into
DEROFDN:community-devfrom
secretnamebasis:feat/pipeline-parallelize-fastsync
Open

Feat/pipeline parallelize fastsync#59
secretnamebasis wants to merge 6 commits into
DEROFDN:community-devfrom
secretnamebasis:feat/pipeline-parallelize-fastsync

Conversation

@secretnamebasis

Copy link
Copy Markdown

Description

bootstrap_chain()'s --fastsync state-tree fetch (both the balance-tree phase and the smart-contract-tree phase) fetched one Peer.TreeSection chunk at a time, waiting for each round-trip before requesting the next. This PR pipelines both phases: up to 16 concurrent Peer.TreeSection requests are in flight at once, with results drained in completion order rather than strict index order.

Draining in completion order also fixes a real bug in a naive "just add concurrency" approach: waiting on requests in strict index order means one slow chunk head-of-line-blocks visibility into other chunks that already finished, which mostly cancels out the benefit of firing them concurrently in the first place. Chunk writes are commutative (each writes a disjoint key range into the same tree), so there's no ordering requirement — only the resume checkpoint needs to stay correct, which is now tracked via a low-water-mark instead of the last-fired index.

Only the network round-trips (connection.Client.Call) run inside worker goroutines. All graviton Tree/Snapshot reads and writes (GetTree, Put, Commit) stay on the single calling goroutine as results are drained — there is no concurrent access to graviton objects at any point, since that thread-safety was never something I could verify.

Live-tested against a real mainnet peer:

  • Balance-tree phase: previously serial at ~11-12s/chunk; now completes in a small fraction of a second per test run (effectively bound by the peer's own response time under a full concurrent window, not by client-side waiting).
  • Smart-contract-tree phase: ~2.9x faster (0.159%/sec vs ~0.056%/sec baseline), measured over 98 seconds of continuous progress against a real peer.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • This will require a HardFork to be enabled

Which part is impacted?

  • Wallet
  • Daemon
  • Miner
  • Explorer
  • Simulator
  • Misc (documentation, comments, text...)

Checklist:

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings

License

I am contributing & releasing the code under DERO Research License (which can be found here).

@moralpriest

Copy link
Copy Markdown

Great work — I've been running this pipelined fastsync against mainnet: ~26–28 MB/min sustained vs ~2–3 MB/min serial on the same link.

I built a wallet-TUI progress consumer on top of it and needed visibility into the pipeline state, plus better window scaling on many-core machines. Opened a small follow-up PR stacked on this branch (+21/−2, p2p/chain_bootstrap.go):

  • GetSyncProgress() / IsBootstrapActive() — atomic reads of the existing sync state so external tools can render real per-chunk progress
  • pipeline_window / sc_pipeline_window: hardcoded 16runtime.GOMAXPROCS(0) clamped [1..32] — scales up on big rigs, less memory pressure on small boxes

Branch also at moralpriest/derohe@fastsync-progress-api for direct cherry-pick. Happy to rework either way.

@secretnamebasis

secretnamebasis commented Aug 25, 2026

Copy link
Copy Markdown
Author

@moralpriest thank you for the feedback and the pr on the base branch.

@moralpriest

Copy link
Copy Markdown

Opened a separate PR for the progress accessors per this discussion: #130

The balance-tree loop and the nested per-SC data-tree fetches were
already pipelined, but the SC-meta tree's own outer loop still issued
one blocking Peer.TreeSection call per chunk - caught live by watching
a real bootstrap run, where step 2 progressed noticeably slower than
step 1 per unit of work.

Fire outer chunk requests ahead using the same fire()/drain-by-
completion pattern as the balance tree, but cap the outer window at 4
(a separate, smaller budget than the nested per-SC pipeline) so the
two concurrency layers don't compound against the single peer being
used - each outer chunk still spins up its own nested per-SC pipeline
below, so an unbounded outer window would multiply, not just add to,
total in-flight requests.
Same cleanup as applied to feat/torrent-and-pipeline-merge: drops
internal planning-narrative framing ("caught live", "flagged early
on") from the SC-meta pipelining comment, keeping the technical why.
No behavior change.
Same fix as on feat/torrent-and-pipeline-merge: bootstrap_chain()
writes topo-store placeholder entries with BLOCK_ID=zerohash but a
non-zero State_Version, which TopoRecord.IsClean() treats as
"not clean" - so while bootstrap is mid-write, Get_Top_ID() can return
the zero hash literally, and fill_common's ReadBlockSnapshotVersion
lookup on that hash panics (maps to a shard directory that
legitimately never exists). Treat a zero Get_Top_ID() as "chain not
ready yet" and skip the lookup instead of panicking. Same signature,
no call-site changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants