[DO NOT MERGE] bench: add ingest benchmark for execute_concurrent - #832
Draft
mykaul wants to merge 1 commit into
Draft
[DO NOT MERGE] bench: add ingest benchmark for execute_concurrent#832mykaul wants to merge 1 commit into
mykaul wants to merge 1 commit into
Conversation
mykaul
changed the title
mykaul
marked this pull request as draft
April 24, 2026 09:42
- bench_ingest.py: benchmark for execute_concurrent ingestion throughput - run_bench.sh: harness to run across multiple venvs and compare results - Warmup phase, multiple timed runs with best/avg/worst reporting - Pre-loads data before timing to separate data prep from ingestion - Variants for stock master, numpy, and decoupled executor
mykaul
force-pushed
the
bench/ingest-benchmark
branch
from
July 29, 2026 17:32
85e8795 to
fc19643
Compare
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
There was a problem hiding this comment.
Pull request overview
Adds a benchmark harness for comparing vector ingestion throughput across execute_concurrent, decoupled, NumPy, free-threaded, and Rust-backed driver variants.
Changes:
- Adds dataset loading, schema setup, warmup, timed runs, and JSON throughput reporting.
- Adds a shell runner for variant-specific environments, CPU pinning, and comparison summaries.
- Introduces benchmark variants intended to evaluate the concurrent submitter work from PR #827.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
bench/bench_ingest.py |
Implements ingestion variants, timing, schema management, and result reporting. |
bench/run_bench.sh |
Orchestrates environments, prerequisites, benchmark execution, and summary output. |
Comments suppressed due to low confidence (1)
bench/bench_ingest.py:267
- Every
_DecoupledExecutoris constructed with at mostconcurrencyrows. Thereforebatch == self.total,run()always takes the early return at lines 195-199, and the submitter thread is never started. Variants A3/E currently benchmark only initial synchronous submission rather than the advertised decoupled executor.
for start in range(0, n, concurrency):
chunk = params[start:start + concurrency]
executor = _DecoupledExecutor(session, prepared, chunk, concurrency)
executor.run()
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # run_bench.sh -- Run all four ingestion benchmark variants and print comparison. | ||
| # | ||
| # Prerequisites: | ||
| # 1. Run bench/setup_bench.sh first (container, venvs, dataset) |
| # 2. ScyllaDB container "scylla-bench" must be running on port 9042 | ||
| # | ||
| # Usage: | ||
| # bash bench/run_bench.sh [--variants=A,B,C,D] [--batch-size=100] [--host=127.0.0.1] |
Comment on lines
+88
to
+90
| if ! $RUNTIME exec scylla-bench cqlsh -e "SELECT now() FROM system.local" &>/dev/null; then | ||
| echo "ERROR: ScyllaDB container 'scylla-bench' is not responding" | ||
| echo " Run: bash bench/setup_bench.sh" |
Comment on lines
+130
to
+147
| n = len(params) | ||
| t0 = time.perf_counter() | ||
|
|
||
| # Feed params in chunks matching concurrency to avoid materialising | ||
| # a huge intermediate list inside execute_concurrent_with_args. | ||
| for start in range(0, n, concurrency): | ||
| chunk = params[start:start + concurrency] | ||
| execute_concurrent_with_args(session, prepared, chunk, | ||
| concurrency=concurrency, | ||
| raise_on_first_error=True) | ||
| total = start + len(chunk) | ||
| if total % PROGRESS_INTERVAL < concurrency: | ||
| elapsed = time.perf_counter() - t0 | ||
| print(f" [{total:>8,} rows] {total/elapsed:,.0f} rows/sec", | ||
| file=sys.stderr) | ||
|
|
||
| elapsed = time.perf_counter() - t0 | ||
| return n, elapsed |
Comment on lines
+413
to
+418
| # TODO: add warmup + multi-run support for rust variant | ||
| total, elapsed = ingest_fn((args.host, args.port), CQL_INSERT, params, concurrency) | ||
| rows_per_sec = total / elapsed if elapsed > 0 else 0 | ||
| result = { | ||
| "variant": args.variant, "label": label, | ||
| "rows": total, "runs": 1, |
Comment on lines
+296
to
+297
| await s.execute(cql_create_keyspace()) | ||
| await s.execute(f"USE {KEYSPACE}") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bench/bench_ingest.pyandbench/run_bench.shfor benchmarkingexecute_concurrentingestion throughputSplit out from #827 to keep that PR focused on the concurrent submitter feature itself.