Gpu opt 5090#863
Open
ColoCarletti wants to merge 12 commits into
Open
Conversation
- run_profile.sh: nsys export needs --force-overwrite (nsys stats already materializes the sqlite); tolerate runs that produce no timeline JSON - flamegraphs.sh: fixed off-CPU capture window sized from the on-CPU run (SIGINT through sudo is unreliable and produced 0-byte captures); find offcputime-bpfcc in /usr/sbin (Debian) - bench_mode.sh: set the CPU governor via sysfs when cpupower is absent - setup_machine.sh: Debian-aware perf install (linux-perf); extract libnvToolsExt from the cuda-nvtx-12-8 deb into ~/nvtx (CUDA >= 12.9 removed NVTX v2 from the toolkit) with LAMBDA_VM_NVTX_LIB override - docs: benchmark/profiling examples use ethrex 5tx/10tx fixtures only (team convention: never fibonacci); plan status updated
Adds the pieces needed to use the tooling without reading the scripts: column-by-column semantics for phase_table.md and phase_busy.md (including NVML gpu% vs nsys busy% and launch-site attribution), a reference table of every script with its flags, the environment variables the tooling understands plus the pre-existing prover knobs for A/B experiments, and a troubleshooting section (missing NVTX ranges, silent CPU fallback, empty off-CPU captures, jitter, concurrent-thread span nesting).
…ee cache The optimization half of the original campaign commit, without its profiling layer (this branch keeps gpu-profiling-tooling's toolkit as the only instrumentation): - async_dtoh_via/PendingD2H: big D2H copies go through per-worker pinned slabs via raw cuMemcpyDtoHAsync + a reusable completion event, instead of cudarc's memcpy_dtoh whose pageable path blocks the calling thread for all prior stream work (host DtoH blocking 12.4s -> 6.7s on the original ethrex A/B). - GpuLdeBase/GpuLdeExt3 carry a 'ready' event; consumers wait device-side (cuStreamWaitEvent) instead of producers host-synchronizing. - Events are pre-created at backend init plus a reusable pool: a mid-prove cuEventCreate convoys the driver lock (~30ms/call measured under load). - Precomputed-column Merkle trees are cached process-wide keyed by their commitment root, so preprocessed tables (DECODE/BITWISE/range) stop rebuilding identical trees on every prove; only the multiplicity columns are recommitted.
Producer thread executes and builds epoch i+1's traces while epoch i proves; K epoch provers (LAMBDA_VM_EPOCH_CONCURRENCY, default 3) consume prepared epochs concurrently — epoch proofs are mutually independent (label-domain-separated transcripts), results re-ordered by index so proof bytes match the sequential schedule. The DECODE commitment is computed once per continuation prove instead of per epoch. Same as the original campaign commit minus its epoch-timeline instrumentation (this branch keeps the profiling toolkit's spans as the only instrumentation; they are re-homed onto this pipelined flow at the end of the series).
…e slots The constraint interp/composition kernels evaluated every IR node as ext3 and kept one global-memory scratch slot per node, so scratch size and traffic scaled with program length (KECCAK_RND/ECSM/ECDAS at full thread count needed 26-39 GB, failing the alloc and silently falling back to CPU via result.ok()). Lowering (constraint_ir/device.rs) now assigns dim-split slots: - Base-dim nodes compute in the base field (1 mul vs 9 for ext3) and live in u64 slots (8B vs 24B); mixed base*ext ops use mul_base / componentwise shortcuts that are bit-identical to the full ext op on the embedded operand (SUB components keep the literal sub(0, x) form, which is NOT bitwise neg on non-canonical limbs). - Slots are liveness-reused (linear scan, freed at last use, roots pinned), so per-thread scratch is the max-live-set, not the node count: 8-35x smaller across the 26 tables (CPU 14.4KB -> 1.3KB, ECDAS 596KB -> 17KB per thread). Scratch allocs drop the memset. - Row-invariant leaves (constants, RAP challenges, alpha powers, table offset) are propagated into operand encodings (kind<<29|payload) and never touch scratch; they only materialize when a root needs them. The CPU walker eval_device_program mirrors the new walk and stays the pre-GPU parity oracle; the 26-table differential vs the production folder and the on-GPU parity tests (synthetic + all real programs) pass bit-for-bit. ir_stats_dump (ignored) prints per-table node/slot stats to size scratch when tuning. Measured on RTX 5090 (nsys, ethrex): constraint_composition_kernel 814ms -> 267ms (-67%) over the same 29 launches; ethrex 10tx continuations ABBA 15.16s -> 14.77s.
Preprocessed tables (DECODE/BITWISE: precomputed + multiplicity column split) skipped the fused GPU commit entirely — commit_main_trace only tried the GPU when precomputed.is_none() — so they paid the CPU row-major LDE plus two CPU subset Merkle trees (~2.2s thread-time of R1 'Main commit Merkle CPU' on ethrex). - keccak256_leaves_base_row_major_row_pair_range: column-range variant of the row-pair leaf kernel, byte-identical to the CPU commit_rows_bit_reversed_subset layout. - coset_lde_row_major_split_trees: one row-major GPU LDE of all columns plus the two subset trees built on device; both node buffers download to host and rebuild full host trees via from_precomputed_nodes, so the preprocessed opening path, the process-wide precomputed-tree cache and disk-spill work unchanged. The shared expansion stage is factored into expand_row_major_on_stream (same code path as the existing fused commit). - The table now gets a GpuLdeBase handle (column-major LDE + trace snapshot, no device tree), so its rounds 2-4 (composition, DEEP, barycentric) run on GPU too. Preprocessed openings short-circuit to the host trees via is_preprocessed, as before. - REGISTER stays on CPU (LDE below the dispatch threshold). Parity: split_tree_tests pins roots and opening paths against the CPU subset commits on device; cross-binary verification of full ethrex bundles passes both ways. Measured on RTX 5090: ethrex 10tx continuations interleaved 3-way 14.77s -> 14.23s (cumulative -6.1% vs the pre-kernel baseline).
prove_global consumes only execution artifacts — the per-epoch cell boundaries built by the producer, the ELF and the genesis pages — never an epoch proof, yet it ran serially after every epoch prove finished (~0.9s of pure tail on ethrex 10tx). The producer now publishes each epoch's boundary (an Arc share of the one already flowing to the epoch provers — no data copy) on a dedicated channel, in epoch order. A scoped thread drains that channel until the producer hangs up (last epoch prepared) and proves the global memory argument while the tail epochs are still proving. On an epoch failure first_err still wins and the global result is discarded; proof bytes and bundle content are unchanged — only the schedule moves. The epoch timeline confirms the tail is gone: the global prove runs fully inside the window of the last three in-flight epoch proves. Measured on RTX 5090: ethrex 10tx continuations ABBA 14.16s -> 13.66s (-3.5%); cross-binary verification passes both ways. Day cumulative across the three optimizations: -9.4% (15.16s -> 13.66s).
Every epoch's trace build re-parsed the ELF and regenerated the pristine DECODE trace (~1M rows) inside the serial producer chain, plus moved a ~900K-entry pc->row map by value per epoch. DecodeArtifacts (instruction map + pristine DECODE trace + pc->row index) is a pure function of the ELF: prove_continuation builds it once and every epoch's build clones the pristine trace (a memcpy) and fills its own multiplicities; build_traces now borrows the pc->row map. The monolithic entry point delegates and is unchanged. Net work removal with identical trace bytes (cross-binary verification passes). Wall-neutral within noise on a 32-core box; groundwork for pipelining the epoch trace build out of the producer chain, where parallel builders would otherwise each redo the ELF parse.
The continuation producer built every epoch's full trace tables inline, so the serial chain feeding the provers was execute + collect + BUILD per epoch (~95% of it table generation) — 7.2s of a ~18s wall on a 32-core box, with the last epochs' proves gated on it. The epoch trace build is now split at its real sequential boundary: - Traces::collect_epoch (Phases 1-2): op collection over the advancing memory image — stays on the producer, in epoch order. - Traces::build_from_collected (Phases 3-5): table generation — pure epoch-local work, runs on a small builder pool (LAMBDA_VM_TRACE_BUILDERS, default 2) between the producer and the epoch provers, bounded channels capping peak memory. The cross-epoch chain no longer touches traces: the boundary derives from CollectedEpoch::touched_memory_cells (same function, same immutable memory_state as the build) and the next epoch's register init from register::fini_from_final_state — a trace-free mirror of the REGISTER FINI column, pinned by fini_from_final_state_matches_trace. PAGE tables are the build's only image consumers and continuation mode skips them, so builders need no image snapshot. Measured on a 32-core RTX 5090 box (ethrex 10tx continuations): the producer chain drops 7.2s -> 2.9s and the first three proves start ~1s earlier, but the wall ties (~18s) — the box is bound by total CPU work, which this change conserves (proves and the global dilate to absorb the freed schedule). A K/builders sweep confirms K=3/B=2 stays optimal. Expected to pay on wider boxes where idle cores can absorb the parallelism; groundwork for cutting per-epoch CPU work (AIR/capture caching), which is the binding constraint on narrow boxes.
Constructing an AirWithBuses runs every constraint body through a MetaBuilder, and the first constraint_program() runs them again for the IR capture — for ECDAS/ECSM/KECCAK_RND (16-25K IR nodes) that dominates AIR construction (0.78s per VmAirs::new on ethrex). Continuation epochs rebuild the full AIR set per epoch and shard tables build one instance per shard, so the same walks re-ran dozens of times per prove. build_air now keeps a process-wide prototype cache keyed by (table name, proof options): the prototype is built and pre-captured once, and every later request clones it — Clone on AirWithBuses copies the derived meta, LogUp layout and the captured IR inside the OnceLock, never re-running the bodies. PAGE stays correct because its page base is part of its name. with_name/with_preprocessed apply to the caller's clone; the cached prototype stays pristine. Wall-neutral within noise on the 32-core box (the removed work is a few core-seconds against a ~580 core-second prove); cross-binary verification passes both ways. Also cuts AIR construction out of the monolithic path and the test suites.
…flow The toolkit's continuation instrumentation assumed the sequential epoch loop. With the producer/builder/prover pipeline the stages run on different threads, so the spans move to where the work actually happens: - prove_continuation_total root span + timeline reset at entry, drained at the end exactly like the monolithic path (stdout tree + LAMBDA_VM_TIMELINE_JSON for phase_table.py). - epoch_execute / epoch_collect on the producer, epoch_trace_build on the builder pool, epoch_prove on the prove workers — each prove/build/ collect also opens an NVTX range with per-epoch identity (epoch_*[i=N]) for Nsight timelines. - Spans close BEFORE blocking channel sends, so backpressure waits are never booked as work. - prove_global span on the overlapped global-prove thread.
Collaborator
Author
|
/bench-gpu |
GPU Benchmark (ABBA) —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.