feat(bench): add feeder-sim, a feeder gateway simulator for sync benchmarks

[infrmtcs]

PR Type

Enhancement, Documentation

---

Description

• Introduce feeder-sim, a tool to simulate the Starknet feeder gateway for sync benchmarks.

• Add capabilities to capture, compress (gzip), and cache data from the real feeder gateway.

• Implement an HTTP server to mock Starknet's feeder gateway, capable of advancing the chain tip sequentially.

• Provide configurable simulation features via CLI, including latency, replay speed, and fixed tip intervals.

---

File Walkthrough

	Relevant files
Enhancement	11 files clock.go Simulation logic for advancing chain block tip                      +70/-0    dataset.go File system operations and gzip handling for datasets        +98/-0    decode.go Decoders for Starknet JSON structures and resources            +90/-0    endpoint.go Feeder gateway endpoint resource representations and URLs +86/-0    flags.go CLI flags handling and validation for simulation configuration +235/-0  main.go Application entrypoint and root command declaration            +88/-0    query.go Data structures for HTTP query arguments encoding                +53/-0    server.go HTTP server that serves dataset responses as feeder endpoints +195/-0  source.go Dataset downloader to capture live feeder gateway responses +178/-0  store.go In-memory storage and concurrency control for cached datasets +73/-0    walk.go Concurrent walker utility to pre-fetch blocks and classes +161/-0
Configuration changes	1 files Makefile Add makefile target to build `feeder-sim`                                +4/-0
Documentation	1 files README.md Usage instructions and dataset specification for `feeder-sim` +69/-0

---

[github-actions]

Claude finished @infrmtcs's task in 9m 6s —— View job

---

Reviewing PR #4071

• Gather context (read changed files, repo conventions)
• Review for correctness/concurrency/resource-leak issues
• Review error handling
• Review tests (if any)
• Post inline comments for findings
• Post summary

Summary

Reviewed all 13 files (bench/sync/cmd/feeder-sim/*, Makefile, README.md). This is a standalone dev/benchmarking CLI, not on the consensus/sync/p2p hot paths.

The PR is clean — no blockers, no important issues. Specifically verified and found no problems in:

• Concurrency: store map access is correctly guarded by sync.RWMutex (store.go); the worker pool writes to disjoint slice indices in walker.walkBlocks, which is safe; clock's tip is accessed via atomic.Uint64.
• Resource handling: HTTP response bodies are closed via defer in source.go; temp files in dataset.write are properly closed/removed on error and renamed atomically on success.
• Context/shutdown: errgroup-based shutdown in server.go/main.go correctly cancels and waits, with a bounded shutdown timeout.
• Bounds/validation: block range indices used in clock.nextDelay are validated (from <= to) before any indexing occurs, so no out-of-range panics.
• Error handling: retry logic in source.go correctly distinguishes retryable vs. permanent (400) errors and wraps errors with context.

No tests are included, but this matches the precedent set by the sibling tool bench/rpc/cmd/corpus-gen, so not flagged.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

⚠️ Risk level: Low

📂 Priority files bench/sync/cmd/feeder-sim/server.go bench/sync/cmd/feeder-sim/clock.go

🏅 Score: 85

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Uint64 underflow In checkTip, server.config.from-1 is computed to build the error message when a requested block is below --from. If --from is 0, this underflows to math.MaxUint64, producing a nonsensical, extremely large number in the error/log output instead of a meaningful value. if key.BlockNumber < server.config.from { return fmt.Errorf( "block %d is below --%s %d; Juno's DB is probably not at %d", key.BlockNumber, fromFlag, server.config.from, server.config.from-1, ) }

[codecov]

Codecov Report

❌ Patch coverage is 0% with 588 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.19%. Comparing base (71d25f5) to head (59a8139).

Files with missing lines	Patch %	Lines
bench/sync/cmd/feeder-sim/flags.go	0.00%	115 Missing ⚠️
bench/sync/cmd/feeder-sim/source.go	0.00%	88 Missing ⚠️
bench/sync/cmd/feeder-sim/server.go	0.00%	87 Missing ⚠️
bench/sync/cmd/feeder-sim/walk.go	0.00%	71 Missing ⚠️
bench/sync/cmd/feeder-sim/main.go	0.00%	45 Missing ⚠️
bench/sync/cmd/feeder-sim/dataset.go	0.00%	42 Missing ⚠️
bench/sync/cmd/feeder-sim/decode.go	0.00%	37 Missing ⚠️
bench/sync/cmd/feeder-sim/store.go	0.00%	36 Missing ⚠️
bench/sync/cmd/feeder-sim/endpoint.go	0.00%	27 Missing ⚠️
bench/sync/cmd/feeder-sim/clock.go	0.00%	26 Missing ⚠️
... and 1 more

❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4071      +/-   ##
==========================================
- Coverage   79.48%   78.19%   -1.29%     
==========================================
  Files         466      477      +11     
  Lines       36009    36597     +588     
==========================================
- Hits        28620    28617       -3     
- Misses       7380     7971     +591     
  Partials        9        9              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Claude finished @infrmtcs's task in 18m 34s —— View job

---

Reviewing PR #4071

• Gather context (read changed files, repo conventions)
• Review for correctness/concurrency/resource-leak issues
• Review error handling
• Review tests (if any)
• Post inline comments for findings
• Post summary

Summary

Reviewed all 13 files (bench/sync/cmd/feeder-sim/*, Makefile, bench/sync/README.md). This is a standalone dev/benchmarking CLI, not on the consensus/sync/p2p hot paths. Verified the package builds cleanly (go build ./bench/sync/...) and go vet is clean.

Nit

• server.checkTip in server.go:176-180: server.config.from-1 underflows to math.MaxUint64 when --from 0, producing a nonsensical number in the error/log message. Inline suggestion posted.

No blockers or important issues. Specifically checked and found no problems in:

• Concurrency: store.bodies map access is correctly guarded by sync.RWMutex; walker.walkBlocks/walkClasses workers write to disjoint slice indices; clock.current is atomic.Uint64; walker.each's progress goroutine is cleanly stopped via context.WithCancel + defer.
• Resource handling: HTTP response bodies closed via defer in source.go; temp files in dataset.write/writeTemporary are closed/removed on error and renamed atomically on success (fsync before rename).
• Path safety: request-time lookups in server.lookup/store.get are pure in-memory map lookups keyed by strings built from the request — untrusted classHash/blockNumber query values can't cause a filesystem read (the map is only ever populated from real captured files at startup), so no path-traversal concern despite resource.file being built from user-controlled query params.
• Context/shutdown: errgroup-based shutdown in server.run/main.go correctly cancels and waits, with a bounded shutdown timeout.
• Bounds/validation: clock.nextDelay's block-range indexing (clock.blocks[tip-from]/[tip+1-from]) stays in bounds because the loop condition (tip < to) guarantees tip+1-from <= to-from, matching len(blocks).
• Error handling: retry logic in source.go correctly distinguishes retryable vs. permanent (400) errors and wraps errors with context; errors.Join used correctly for cleanup-then-report patterns in dataset.go.

No tests included, consistent with the precedent set by the sibling tool bench/rpc/cmd/corpus-gen.
· branch dat/feedersim

[github-actions]

Standalone PR Review

PR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review.

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

⚠️ Risk level: Low

📂 Priority files bench/sync/cmd/feeder-sim/clock.go bench/sync/cmd/feeder-sim/server.go bench/sync/cmd/feeder-sim/source.go

🏅 Score: 82

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Out-of-bounds panic nextDelay indexes clock.blocks[tip+1-clock.from] without bounds checking. When clock.tip() equals clock.to (the loop condition in run only checks before entering, but nextDelay is called via time.After(clock.nextDelay()) each iteration up to to-1), this is fine as long as to == from + len(blocks) - 1, but if blocks is shorter than expected (e.g., capture was interrupted or --to doesn't match dataset), this indexes out of range and panics the whole simulator. func (clock *clock) nextDelay() time.Duration { if clock.speed == 0 { return clock.interval } tip := clock.tip() current := clock.blocks[tip-clock.from].timestamp next := clock.blocks[tip+1-clock.from].timestamp if next <= current { return 0 } return time.Duration(float64(next-current) * float64(time.Second) / clock.speed) } Retry loop swallows context error In download, when waitRetry returns a non-nil ctx.Err(), the function returns nil, err directly, but the earlier err from downloadOnce is discarded and replaced silently by the context error without wrapping context, making it slightly harder to diagnose partial retry state (minor, but worth noting since it differs from the "after N attempts" wrapping path used elsewhere). func (source *source) download(ctx context.Context, requestURL *url.URL) ([]byte, error) { body, err := source.downloadOnce(ctx, requestURL) for attempt := 1; attempt <= source.retries && retryable(err); attempt++ { source.logger.Debug( "retrying capture request", zap.Stringer("url", requestURL), zap.Int("attempt", attempt), zap.Error(err), ) if err := source.waitRetry(ctx); err != nil { return nil, err } body, err = source.downloadOnce(ctx, requestURL) } if retryable(err) { return nil, fmt.Errorf("after %d attempts: %w", source.retries+1, err) } return body, err }

[github-actions]

Nit: server.config.from-1 underflows to math.MaxUint64 when --from 0, producing a nonsensical number in the error/log message instead of a meaningful one.

Suggested change

	if key.BlockNumber < server.config.from {
	return fmt.Errorf(
	"block %d is below --%s %d; Juno's DB is probably not at %d",
	key.BlockNumber, fromFlag, server.config.from, server.config.from-1,
	)
	if key.BlockNumber < server.config.from {
	message := fmt.Sprintf("block %d is below --%s %d", key.BlockNumber, fromFlag, server.config.from)
	if server.config.from > 0 {
	message += fmt.Sprintf("; Juno's DB is probably not at %d", server.config.from-1)
	}
	return errors.New(message)

Fix this →