Skip to content

CUDA backend (up to active_width=11) - #219

Open
rafaelha wants to merge 39 commits into
mainfrom
cuda-statevector-executor
Open

rafaelha wants to merge 39 commits into
mainfrom
cuda-statevector-executor

Conversation

@rafaelha

@rafaelha rafaelha commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

With strategy="active_statevector" (the default), the samplers run on an NVIDIA GPU through CuPy whenever every statevector component has at most 11 active qubits. The limit comes from the kernels' design: a shot's state vector lives in the registers of one warp, 32 threads of 64 complex64 amplitudes (2^11). Wider programs sample on the CPU under device="auto".

The noise sampler and every component's tape run as CUDA kernels, in float32. The kernels are generated at runtime: the tapes are emitted as CUDA C++ source specialized to the circuit (and to its postselection mask), which CuPy compiles with NVRTC at the first sample. On the RTX 5090 that takes 0.7 s for cultivation d3 and 2.1 s for end-to-end d5. CuPy caches the compiled kernels on disk, so a later process sampling the same circuit compiles in 0.01–0.16 s.

With a postselection_mask, only the shots that pass are copied off the GPU.

The host side is shared by Tsim's GPU backends (statevector/gpu/: the tape generator, the noise tables, the row selection); this PR adds the CUDA dialect, kernels and sampler. It is up to date with main.

User-facing changes

  • device="cuda" samples on an NVIDIA GPU or raises with the reason: no CuPy or no device, a component wider than 11, a rotation float32 cannot resolve, noise beyond the GPU tables, out of GPU memory, a kernel that does not compile, or a process forked after its parent used the GPU.
  • device="auto" (the default) tries CUDA before any other GPU backend, then the CPU, silently. sampler.stats.device and the repr (samples on cuda) say where it ended up.
  • Install: the new cuda extra (pip install "bloqade-tsim[cuda]": cupy-cuda13x[ctk], CUDA 13, driver 580 or later). The JAX extras are now jax-cuda12 and jax-cuda13; Tsim 1.0's cuda12 and cuda13 still install them.
  • Seeds: on the GPU a seed's samples differ from the CPU's and depend on batch_size (documented). On the CPU they are unchanged: the GPU draws from a fourth child of the seed's SeedSequence.
  • Docs: install.mdx, index.mdx, the README, the changelog, the user skill and the developer skills cover CUDA. The skill warns that booleans cost an NVIDIA GPU about 8x against bit-packed rows (RTX 5090: surface_d7_r7 1.2B against 148M shots/s, end-to-end cultivation d=3 340M against 42M).

What's in it

  • statevector/cuda/codegen.py: the CUDA dialect of gpu.tape (__shfl_xor_sync among a shot's lanes, goto past a pruned shot) and the prelude; the outcome draw falls back to double precision near the boundary.
  • statevector/cuda/noise.py: the noise and noise_kept kernels on gpu.noise's tables (40-bit keep thresholds, a hashed stream per block, 24 extra bits for the skips of tiny probabilities). Rows too wide for shared memory keep their words in device memory (NOISE_DEVICE).
  • statevector/cuda/kernels.py: the tape kernel, the bitmap scan that lists the accepted shots, and the gather_rows and select_rows kernels, which pack the rows and pick the requested outputs on the GPU.
  • statevector/cuda/runtime.py: loads CuPy (preloading a pip CUDA 13 NVRTC when present) and detects forks and out-of-memory errors.
  • statevector/cuda/sampler.py: CudaShotSampler. A call runs as chunks that alternate between two sets of streams; each chunk's rows are copied to pinned host memory while the next chunk's kernels run. Noise that postselection cannot see is drawn on the accepted shots only. Out of GPU memory, a call retries in smaller batches, then samples on the CPU (or raises with device="cuda"). When NVRTC fails to compile or to load, the sampler samples on the CPU from then on.
  • sampler.py: _GPUS and the device="auto" loop. gpu/tape.unsupported, which says why a program cannot run on a GPU, is now shared by the GPU backends.

Benchmark

Shots per second at d140f3d, from tsim-dev bench --suite quick plus end2end d5 (tsim-bench 19804f26): bit-packed rows, tsim-bench's postselection, copy to the host included. The CPU columns are Tsim's CPU executor on one thread and on all 24 threads of the AMD Ryzen 9 9900X that hosts the RTX 5090 (--device cpu --threads 1 and --threads 24). The H100 SXM and A100 SXM ran on RunPod (driver 595, CUDA 13).

circuit k kept CPU, 1 thread CPU, 24 threads RTX 5090 H100 SXM A100 SXM
inject_cultivate_d3_p0.001 4 0.687 27.8M 173M 14.7B 10B 4.9B
inject_cultivate_d5_p0.001 6 0.144 14.7M 145M 6.6B 3B 2.1B
end2end_d3_dsurface15_p0.001 4 0.539 1.1M 8.2M 340M 339M 129M
end2end_d5_dsurface15_p0.001 6 0.0519 848k 8.8M 1.8B 1.6B 825M
hook_inject_T_d15_pd5_p0.001 1 0.667 1.1M 7M 170M 174M 68.7M
distillation_color_d5_Z 3 0.0798 12.8M 158M 16B 17.6B 6.9B
coherent_d3_r3 4 1 19.6M 117M 9.8B 6.3B 2.9B
surface_d7_r7 0 1 13.1M 166M 1.2B 1.2B 486M
qv_q10 10 1 10.8M 9.2M 181M 113M 46.3M
  • Against the CPU: the RTX 5090 is 17x (qv_q10) to 2100x (end2end d5) faster than one CPU thread, and 7x (surface_d7_r7) to 205x (end2end d5) faster than all 24. qv_q10 does not gain from threads.
  • Across GPUs: the kernels are float32, so the RTX 5090 leads on most circuits. The H100 matches it within 3% on the circuits with the widest rows (surface_d7_r7, end2end d3, hook injection) and beats it by 10% on distillation_color_d5_Z. The A100 reaches 0.26–0.46x of the RTX 5090.
  • Profiles (nsys): the GPU is 96–100% busy on all three machines, and the copies to the host overlap the kernels. end2end d5 spends most of its time in the noise kernel, and qv_q10 in the tape. Details below.
Profiling with Nsight Systems on the RTX 5090, H100 and A100

nsys profile --trace=cuda around tsim-bench's worker in --mode profile, at each circuit's benchmark settings (bit-packed rows, tsim-bench's postselection). The numbers cover the last 2.5 s of sampling. Shares are each kernel's summed time over wall time; they can exceed 100% because a call's chunks alternate between two sets of streams, whose kernels overlap.

RTX 5090 H100 SXM A100 SXM
end2end d5: GPU busy 97% 96% 98%
noise / noise_kept / tape share 98 / 14 / 9% 101 / 16 / 16% 98 / 17 / 19%
noise / noise_kept / tape, µs per launch 291 / 40 / 27 340 / 55 / 55 618 / 107 / 117
copy to the host, busy 61% 53% 60%
cultivation d3: GPU busy 96% 96% 98%
tape / noise / list_bits / gather_rows share 51 / 41 / 22 / 14% 68 / 37 / 20 / 20% 75 / 22 / 23 / 17%
tape / noise, µs per launch 73 / 59 143 / 77 322 / 94
qv_q10: GPU busy 100% 100% 100%
tape, ms per launch 16.0 23.1 55.6

What they show:

  • The host is not the limit. The GPU is busy 96–100% of the time on every machine, and the copies of the rows to pinned host memory run on their own streams, overlapping the kernels.
  • Each circuit has one dominant kernel. end2end d5 (2,745 outputs, 5% kept) is limited by the noise kernel, which draws the faults and applies them to every shot's row of outputs. qv_q10 (width 10, 1,000 T gates) is all tape. cultivation d3 splits its time across the tape, the noise and the postselection scan (count_bits and list_bits) that lists the accepted shots.
  • The tape is relatively slower on the data-center cards. On end2end d5 the tape takes 2x as long per launch on the H100 as on the RTX 5090, while the noise kernel takes 1.2x as long. The tape is float32 arithmetic and warp shuffles on registers, where the RTX 5090 has more SMs at higher clocks; the noise kernel probably leans more on memory, where the H100 is faster (not confirmed with hardware counters).
  • The tape's occupancy is set by registers and a fixed grid. The tape launches 4 blocks of 128 threads per SM, 16 warps. At 128 registers per thread (end2end d5) that fills the 64K-register file exactly, so more blocks would not fit. qv_q10's tape uses 188–248 registers per thread, so only 2 of its blocks fit per SM. cultivation d3's uses 72–80 registers, so its occupancy is capped by the fixed 4 blocks per SM. The H100 and A100 could hold up to 64 warps per SM, so their tapes run at a lower occupancy than the hardware allows. Tuning the grid and the register budget per GPU is an open item.

Testing

  • uv run tsim-dev check on the RTX 5090: lint passes and 2779 tests pass, the CUDA tests included.
  • test/integration/test_cuda_sampler.py (95 tests, skipped without CuPy and a GPU) compares detector, measurement and postselected samples with cat5 and the CPU executor. It covers every output layout, widths up to 11, looping tapes, tiny noise and outcome probabilities, every CPU fallback (including a kernel that does not compile and an NVRTC that does not load), out-of-memory retries, device memory per call, fork, pickle and threads. It passes on the RTX 5090, H100 SXM and A100 SXM.
  • A 5-minute GPU fuzz run at d140f3d (4 workers), against an exact density-matrix oracle and, for circuits too wide for it, against the CPU executor: 668 cases clean, with all 8,886 samplers on the GPU.
  • CI has no GPU, so the CUDA tests run on a GPU machine only; codecov.yml ignores statevector/cuda.

Known limits, not fixed here

  • Active width above 11: a component wider than 11 active qubits does not fit in one warp's registers, so its programs sample on the CPU (or raise with device="cuda"). Going wider means splitting a shot's state vector across warps, through shared or device memory, as a separate tape layout.
  • Pinned host memory: results come from CuPy's pinned pool, which rounds each up to a power of two and keeps it after the array is freed (a 1.1 GB result holds 2 GB of RAM).
  • CERTAIN: outcomes of weight at most 1e-11 of the total count as impossible.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.2

QR code for preview link

🚀 View preview at
https://queracomputing.github.io/tsim/pr-preview/pr-219/

Built to branch gh-pages at 2026-11-05 23:15 UTC.
Preview will be ready when the GitHub Pages deployment is complete.

@rafaelha
rafaelha force-pushed the fast-amplitude-kernels branch from e1d3453 to 73656a2 Compare October 27, 2026 04:30
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

☂️ Code Coverage

current status: ✅

Overall Coverage

Statements Covered Coverage Threshold Status
6735 5409 80% 0% 🟢

New Files

File Coverage Status
src/tsim/statevector/cuda/_init_.py 55% 🟢
src/tsim/statevector/cuda/codegen.py 100% 🟢
src/tsim/statevector/cuda/kernels.py 82% 🟢
src/tsim/statevector/cuda/noise.py 69% 🟢
src/tsim/statevector/cuda/runtime.py 57% 🟢
src/tsim/statevector/cuda/sampler.py 33% 🟢
TOTAL 66% 🟢

Modified Files

File Coverage Status
src/tsim/circuit.py 95% 🟢
src/tsim/core/types.py 100% 🟢
src/tsim/sampler.py 95% 🟢
src/tsim/statevector/gpu/tape.py 99% 🟢
src/tsim/statevector/metal/_init_.py 59% 🟢
TOTAL 90% 🟢

updated for commit: d140f3d by action🐍

@rafaelha
rafaelha force-pushed the fast-amplitude-kernels branch from 73656a2 to ab4a19d Compare October 27, 2026 05:13
Base automatically changed from fast-amplitude-kernels to main October 27, 2026 17:33

@rafaelha rafaelha left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the CUDA backend. The first three comments change sampled statistics relative to device="cpu"; the next two are memory-scaling failures; the last is test coverage.

Comment thread src/tsim/statevector/cuda/noise.py Resolved
Comment thread src/tsim/statevector/cuda/sampler.py Resolved
Comment thread src/tsim/statevector/cuda/sampler.py Resolved
Comment thread src/tsim/statevector/cuda/sampler.py Resolved
Comment thread src/tsim/statevector/cuda/sampler.py Resolved
Comment thread test/integration/test_cuda_sampler.py Resolved
@rafaelha
rafaelha force-pushed the cuda-statevector-executor branch from 211fed3 to 840be38 Compare October 28, 2026 04:46
rafaelha added a commit that referenced this pull request Nov 3, 2026
…223)

Stacked on #222 (stack #245). This PR's diff is against that branch.

## Summary
This PR changes no behaviour. It is cleanup from the statevector review,
made of one new test and three refactors. Samples are bit-identical
after every commit. No feature or fix depends on it, so it can wait
behind #222 and #219.

- **Brute-force test for `_Relations`** (`test_pbc.py`). `_Relations` is
the GF(2) elimination that turns classical spiders into output parities.
Until now it was only covered through whole-circuit distributions. The
test builds 300 random small systems and sums the sign over the
eliminated variables by enumeration. It then checks the reduced rows,
and that the reduction raises exactly when no such rows exist.
- **One symbol encoding** (`program.py`, `shared.py`, `shots.py`; 13
lines added, 32 removed). The tape used to pack symbol sets into uint32
words, which `lane_program` unpacked again before packing them into
uint64 lane words. The tape now keeps a boolean column per symbol.
`_pack`, `unpack_symbols` and `StatevectorComponent.num_words` are gone,
and `num_symbols` replaces `num_words`. The tape grows slightly: its
symbol rows take 1.3 MB on d=5 cultivation, up from about 0.16 MB.
- **No graves in `run_lanes`** (`shared.py`; 161 lines added, 100
removed). A pruning now finishes the dying lanes' shots right away, with
their outputs, coins and hidden bits, and drops them from `live`.
Before, they waited in a grave buffer until the next random measurement
or the end of the run. This removes `graves`, `grave_at`, `dead_shot`,
`dead_grave`, the per-entry hidden masks and the `-1 - grave` codes. The
per-shot work moves into `_finish_shot`, and `lane_bytes` counts two
rows of lane symbols per lane instead of four.
- **One sampling entry point** (`sampler.py`).
`CompiledMeasurementSampler.sample` and `CompiledDetectorSampler.sample`
both call `_sample_outputs(shots, batch_size, columns, bit_packed, mask,
referenced)`. That method picks the shot-word or stabilizer-rank path
once. The stabilizer-rank path's mask screening and reference XOR move
into `_sample_stabilizer_rank_outputs`, so
`CompiledDetectorSampler.sample` no longer duplicates them.

## Open before merge
- [x] **Single-thread slowdown on the shared, unpruned path.** Fixed in
`ad8d042`: the end-of-run loop is inline again and has no `hidden` pass,
and `_finish_shot` serves only pruned shots. On one thread, a width-7
component with 12 detectors, 2M shots and sharing on takes 0.031 s,
against 0.033 s on #222 and 0.045 s before this fix. Sample hashes are
identical.
- [x] **The `_Relations` test checks the reduced echelon form**
(`ad8d042`, `04b7ae3`): pivots are unique and in `kept` order, every row
term is in `kept`, and no other row contains a pivot. Dropping the
back-substitution and reversing the pivot priority both fail it now.
- [x] **Docstring nit in `_sample_outputs`**, fixed in `ad8d042`.
- [x] **Stacked on #222 in stack #245.** Both branches are rebased
linearly onto `main` at `6ee9e1b` (#240), and the stack merges them
together.
- [ ] **Overlap with #219 (CUDA).** It merges cleanly with this branch.
Its new `sample_postselected` builds `referenced`/`flip` the same way
`_sample_outputs` now does, so whichever PR lands second can reuse that
code.

## Verification
- After the rebase onto `main` at `6ee9e1b`, the full suite passes
locally (2367 passed) and pyright is clean. `CX_d1=5_cult`, with every
detector postselected, gives the same hash as #222 with sharing on and
off. Before the rebase, CI was green on `c221fea`.
- **Executor commits.** A snapshot hashed exact samples and lane
statistics for 48 circuits, with sharing on and off and three masks,
plus two prune-heavy workloads. The hash is identical before and after.
- **Sampler commit.** A second snapshot covered both strategies and
every combination of mask, both reference flags, `bit_packed` and
observable layout. The hash is identical before and after.
- **Review harness.** 74,304 `active_statevector` cases and 4,788 `cat5`
cases produced identical hashes against the base branch. So did lane
statistics on 62 more circuits.
- **Timing.** The table below uses cultivation, with every detector
postselected, best of 5. Sample hashes are identical.

  | Workload | Base | This PR |
  | --- | --- | --- |
  | `CX_d1=5_cult`, 2M shots, 1 thread, sharing on | 1.38 s | 1.31 s |
  | `CX_d1=5_cult`, 2M shots, 1 thread, sharing off | 1.87 s | 1.82 s |
| `CX_d1=5_cult`, 2M shots, all threads, sharing on | 0.18 s | 0.19 s |
| `CX_d1=5_cult`, 2M shots, all threads, sharing off | 0.25 s | 0.26 s |
| Width-7 component, 12 detectors with half postselected, 400k shots,
sharing on | 0.05 s | 0.05 s |

Pruning now scans the live shots each time a lane dies, instead of
waiting for the next random measurement. That cost doesn't show up even
on this prune-heavy workload. The only slowdown measured was the
single-thread case without pruning, now fixed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@rafaelha
rafaelha force-pushed the cuda-statevector-executor branch from 2092c79 to a2af975 Compare November 4, 2026 13:16
@rafaelha rafaelha changed the title Run the statevector sampler on CUDA GPUs (device="cuda") Sample on CUDA GPUs automatically with the statevector strategy Nov 4, 2026
rafaelha added a commit that referenced this pull request Nov 4, 2026
Defines what tsim 1.0.0 commits to under semantic versioning.

## Public API
- `tsim.__all__ = ["Circuit", "CompiledDetectorSampler",
"CompiledMeasurementSampler"]`. These are the only objects semver
covers. `tsim.__init__`'s docstring and the docs index ("API stability")
say that every other module is internal, even where it is importable.
- New `tsim.experimental` re-exports `CompiledStateProbs`,
`SteaneEncoder` and `ColorEncoder5`, which the tutorials use. They may
change in any release. The old import paths still work.
- `docs/scripts/gen_ref_nav.py` no longer generates a page per module.
The API reference has two pages, "Tsim API" (the three public objects)
and "Experimental".
- `overview.ipynb` and `magic_state_distillation.ipynb` import from
`tsim.experimental`.

## Sampling strategies
Only `strategy="active_statevector"` is fully stable. The stabilizer
rank strategies (`"cat5"`, `"bss"`, `"cutting"`) are provisional: they
may be removed in a minor release, after at least one minor release in
which they emit a `DeprecationWarning`. The docstrings say only
"provisional"; the policy is in the "API stability" section of
`docs/index.md` and in the CHANGELOG.

## Seed reproducibility
The `seed` docstrings of `compile_sampler`, `compile_detector_sampler`,
the sampler classes and `CompiledStateProbs` now say that a seed
reproduces its samples only with the same Tsim version and backend (CPU
or GPU). Without this, changing a random stream (as #263 did), or adding
a float32 GPU executor (#219), would technically be a breaking change.

## Test plan
- [x] `uv run pytest test/unit`: 2072 passed. Includes a new
`test/unit/test_public_api.py`.
- [x] pre-commit (black, isort, ruff, pyright, codespell)
- [x] `mkdocs build` from a clean checkout. The tutorials run, and the
reference contains only `tsim.Circuit`, the two samplers and the
`tsim.experimental` objects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@rafaelha
rafaelha force-pushed the cuda-statevector-executor branch 2 times, most recently from a99fb38 to 20b3eef Compare November 5, 2026 23:14
rafaelha and others added 3 commits November 10, 2026 20:17
The CUDA and Metal backends of the statevector executor share their host
side: `statevector/gpu/` holds the noise kernels' tables (`noise`), a tape
generator that writes a component's tape in either language through a small
`Dialect` (`tape`), and the packing of returned rows (`rows`).

Two changes to the statevector plans come with it. `shared.common_prefix`
returns the leading actions of a tape that every shot runs alike, which a GPU
backend runs once. `plan_component` also plans each component with its
measurements deferred (`schedule(defer=True)`) and keeps that plan when it is
no wider and leaves less shot-dependent work.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rk model

- noise_tables raises when an outcome flips more than 255 words (the record's
  8-bit count would wrap) and when the events alone do not fit shared_bytes
  without shared words (MIN_EVENTS is now a named constant).
- Dialect rejects an amplitude_chunk that is not a positive power of two, which
  would declare fewer chunk arrays than the tape indexes.
- _shot_work ends the shared prefix where shared.common_prefix does: an unread
  dormant measurement is a fair coin and no longer counts as shot work.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Below p ~ 2^-24, a kernel's 24-bit geometric skip reaches only some trials of a
segment, so a group of many such channels fired them at biased per-channel
rates (0.93 to 1.17 of their rates at p = 1e-9 on 3200 channels; the total
rate was right). Such groups are now cut into segments of fewer than 2^-13
expected faults, which the kernels draw from 48 bits (their fine path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rafaelha added a commit that referenced this pull request Nov 10, 2026
Rebuild #219's CUDA backend on #284's tsim.statevector.gpu: the tape comes
from gpu.tape with a CUDA dialect (byte-identical to #219's codegen), the
noise tables from gpu.noise and the row packing from gpu.rows. cuda/ keeps
the CUDA prelude, the noise kernels' source, the runtime and the sampler.

The dispatch follows the Metal backend's shape (_gpu_shots): TSIM_DEVICE=cpu
keeps the samplers on the CPU, sampler.stats.device and the repr say where a
sampler samples, programs without shot words stay on the CPU, and noise
beyond the GPU tables' limits falls back to the CPU with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rafaelha
rafaelha force-pushed the cuda-statevector-executor branch from 20b3eef to d11d4d6 Compare November 10, 2026 21:26
@rafaelha
rafaelha changed the base branch from main to feat/gpu-shared November 10, 2026 21:26
rafaelha and others added 6 commits November 11, 2026 03:56
Both dialects are internal constants, so a ValueError and its tests guard
nothing a user can reach.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both backends return every output, in order, so each packed word is a whole
shot word and the piece tables only encoded the identity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both GPU backends built the read mask of a postselected batch, chose the
deferred channels and estimated the screen's pass rate with the same code;
gpu.noise now has split, screen_pass_rate and words_fit (the shared-memory
fit test that Metal repeated with literals).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both GPU backends wrote the same component calls, seeds, word stores and
postselection hits around the tape functions, and repeated MAX_WIDTH, the
threads per shot, the deviation tolerance and the layout check. gpu.tape now
has tape_kernel, threads_per_shot, deviation and supports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebuild #219's CUDA backend on #284's tsim.statevector.gpu: the tape comes
from gpu.tape with a CUDA dialect (byte-identical to #219's codegen), the
noise tables from gpu.noise and the row packing from gpu.rows. cuda/ keeps
the CUDA prelude, the noise kernels' source, the runtime and the sampler.

The dispatch follows the Metal backend's shape (_gpu_shots): TSIM_DEVICE=cpu
keeps the samplers on the CPU, sampler.stats.device and the repr say where a
sampler samples, programs without shot words stay on the CPU, and noise
beyond the GPU tables' limits falls back to the CPU with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rafaelha added a commit that referenced this pull request Nov 12, 2026
## Problem

On an Apple GPU, detector samples that return a subset or reordering of
the outputs split each call's rows on the host:

- **Bit-packed:** columns that do not lead the rows (the observables of
`separate_observables=True`, every column of `prepend_observables=True`)
went through `gpu.rows.select`, which unpacks every row to booleans,
indexes the columns and repacks them.
- **Booleans:** the GPU unpacked every output, then the host copied the
requested columns out of them (`_columns`).

The GPU idled meanwhile. On distance-3 cultivation postselected on every
detector, `sample(2**26, separate_observables=True, bit_packed=True)`
spent 0.48 s of its 0.53 s in `rows.select`, and the GPU work took 0.04
s. The most Stim-like call, `detectors, observables =
sampler.sample(..., separate_observables=True)`, was 10 to 15 times
slower than appended observables. On end-to-end cultivation d=5 (2745
outputs), several layouts sampled 0 to 3 M shots/s.

## Fix

A new Metal kernel, `select_rows`, writes each requested column list
into its own array, bit-packed (zero bits past the columns) or as one
boolean per output. It runs one thread per output byte, and the column
lists of a call share one command buffer.

- All outputs in order are still the rows themselves (bit-packed) or
`unpack_rows` (booleans), and leading whole bytes stay a view of the
rows.
- Calls past 32-bit indices keep the host path.
- Boolean arrays stay C-contiguous.

`_columns` is gone, and `MetalShotSampler.sample_accepted` takes the
column lists (`columns=`) and returns one array per list.

## Numbers

Apple M4 Pro, 20-core GPU, Metal. The work is on the GPU, so the numba
thread count does not change these. `main` (`7ec136e6`) → this branch,
in M shots/s, postselected as tsim-bench records:

| cultivation d3 (21 outputs) | bit-packed | booleans |
|---|---|---|
| separate observables | 147 → **1316** | 97 → **1104** |
| prepended observables | 55 → **1508** | 89 → **1249** |
| detectors only | 1312 → 1468 | 101 → **1241** |
| appended observables | 1835 → 1863 | 1424 → 1391 |

| end2end d5 (2745 outputs) | bit-packed | booleans |
|---|---|---|
| separate observables | 59 → **291** | 3 → **156** |
| prepended observables | 1 → **221** | 3 → **171** |
| detectors only | 305 → 306 | 3 → **170** |
| appended observables | 308 → 307 | 214 → 213 |

Without postselection the same layouts gain alike: cultivation goes from
21–102 to 888–1187, and end2end from 0–7 to 14–65. Every layout of
end2end is then bound near 68 M shots/s by its 344-byte rows.

These grids come from a timing script, because `tsim-dev bench` only
times appended observables. That bench is unchanged: `tsim-dev bench
--device gpu --ref origin/main`, M4 Pro, 1 numba thread, quick suite:

```
circuit                                        k device  shots/s   kept  compile   shots/s vs ref (min, max)      compile vs ref
inject_cultivate_d3_p0.001                     4  metal    1.84G  0.687     0.18   0.99x [0.98, 0.99] same    1.14x same
inject_cultivate_d5_p0.001                     6  metal     722M  0.144     0.30   1.02x [1.00, 1.06] same    1.09x same
end2end_d3_dsurface15_p0.001                   4  metal     166M  0.539     2.13   1.00x [0.99, 1.00] same    1.02x same
hook_inject_T_d15_pd5_p0.001                   1  metal    52.7M  0.667     2.47   0.98x [0.94, 1.01] same    0.99x same
distillation_color_d5_Z                        3  metal    2.38G 0.0798     0.18   1.00x [0.99, 1.01] same    0.81x same
coherent_d3_r3                                 4  metal    1.09G      -     0.17   1.00x [0.99, 1.00] same    1.16x same
surface_d7_r7                                  0  metal     727M      -     0.28   1.00x [1.00, 1.00] same    1.14x same
qv_q10                                        10  metal      11M      -     0.37   1.00x [1.00, 1.00] same    1.12x same
SUMMARY: 8 circuits, 0 errors, 0 slower, 0 faster, 0 noisy (tolerance 5%).
```

## Tests

- `test_metal_selects_every_layout_of_columns` checks separate,
prepended and detectors-only output, bit-packed and as booleans, with
and without postselection, against appended booleans (the unchanged
`unpack_rows` path). Its five shapes have observables that start inside
a byte, end inside one, span two bytes, or align with bytes. It also
checks that booleans are C-contiguous.
- `test_metal_samples_no_shots` now also checks
`separate_observables=True` with zero shots, which an early version of
this change returned as one array instead of two.
- I also checked every layout against appended booleans for six shapes,
three postselection masks and the measurement sampler (not committed).

## Verification

- `uv run tsim-dev check` passes, apart from
`test_dev_skills_have_valid_frontmatter`. That one fails only because my
checkout has untracked local skills in `.claude/skills`, so CI will not
see it.
- The Metal tests run only on Apple silicon, so CI does not run the new
kernel.
- Not verified: CUDA (#219) has its own sampler and is unaffected; the
shared `gpu/rows.py` is unchanged.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@rafaelha rafaelha changed the title Sample on CUDA GPUs automatically with the statevector strategy CUDA backend Nov 12, 2026
rafaelha and others added 14 commits November 12, 2026 14:55
Takes main's shared GPU code (40-bit keep thresholds, RESOLVABLE), the device
argument, and Metal's docs; the CUDA backend is wired back in by the next
commits.
CUDA now behaves as the Metal backend does on main:

- device="cuda" samples on an NVIDIA GPU or raises with the reason, and
  device="auto" tries Metal, then CUDA, then the CPU, without warnings.
  Kernel compile failures (runtime.CompileError) move a sampler to the CPU;
  out of memory, forked processes and pickles follow 'required'.
- The noise kernels draw outcomes from 40-bit keep thresholds, hash each
  block's stream, and keep rows too wide for shared memory in device memory;
  gather_bytes packs rows too wide to stage, so no output count falls back.
- Requested output columns (separate, prepended, detectors only) and
  booleans are selected on the GPU (select_rows) before the copy to the host.
- The reason a layout cannot run on a GPU is shared (gpu.tape.unsupported).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t of line

Bit-packed outputs in order that end on a byte boundary (detectors only, the
detectors of separate_observables) are a view of the copied rows again, as on
Metal, and the 40-bit keep threshold's rare tie check is __noinline__.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TSIM_DEVICE is gone, as on main; the rotation-angle message moves into the
shared gpu.tape.unsupported, which CUDA uses too; CUDA's CHANGELOG entry goes
under Unreleased.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Postselected rows whose block of packed rows does not fit in shared memory
beside its output words (li_inject_T_d15, zz_inject_T_d15: 6577 outputs)
launched noise_kept with more than 48 KB of shared memory and failed with
CUDA_ERROR_INVALID_VALUE; they now draw all their noise in the screened pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A chunk that kept more rows than the room left grew the pinned output to
room for every remaining shot: a 2^27-shot call of star_d=13 (357-byte rows,
kept 9e-5) asked for 48 GB and was killed. The room now follows the rate of
the shots so far, and at least doubles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deferral reused gather_columns' bound, whose blocks may fill the 48 KB
of shared memory exactly. noise_kept counts its events there too, so with
6137 to 6144 outputs a postselected call failed at launch with
CUDA_ERROR_INVALID_VALUE.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A call kept every chunk's rows on the device until it returned, so its
device memory grew with its result: a result larger than the free device
memory ran out of it at every batch size, and then sampled on the CPU
(or raised, with device="cuda"). At most MAX_IN_FLIGHT chunks' rows now
wait for their copies.

A call that no batch size fits no longer leaves the sampler at the
smallest batch for its later calls.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
CI has no NVIDIA GPU or CuPy, so the CUDA tests skip, and codecov/patch
counted the whole package as uncovered (31.68% of the diff). The shared
code that the PR moved out of the ignored Metal package is now tested on
the CPU: the GPU support checks' messages and the order in which
device="auto" tries CUDA and Metal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user skill now names the cuda extra and CUDA's errors, says that
booleans cost an NVIDIA GPU 7-8x (RTX 5090: surface_d7_r7 1,168M packed
against 159M boolean shots/s), so large GPU runs should pass
bit_packed=True, and notes CUDA's pinned host memory. The developer
skills cover benchmarking on CUDA (the --ref env has no CuPy), its
profilers, and the GPU tests that CI cannot run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in Tsim asked the GPU samplers for expectations: they served only
tests. Remove them from the shared tape and noise split, the CUDA and Metal
kernels and samplers, and merge sample_accepted into the private _accepted,
which always takes columns. Also drop CUDA's buffer view cache, which grew
with every distinct shot count. Seed-fixed CUDA samples are unchanged, and so
are Metal's traced dispatches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gather_columns staged rows in shared memory before packing them, and
gather_bytes read them from device memory where 32 rows did not fit. On the
RTX 5090 the second alone is as fast (within 1% on cultivation, surface,
end2end, hook injection and 800/3000-detector rows, packed and booleans), so
it is now the only one, as gather_rows. Seed-fixed samples are unchanged.
The publish kernel stays: reading the counts with a copy instead costs 3-7%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The removal of expectations from the shared tape, the noise split and the
Metal sampler moves to its own PR (#301). CUDA keeps its own simplification:
it asks the shared tape for no observables, so its tapes compute none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rafaelha added a commit that referenced this pull request Nov 13, 2026
Nothing in Tsim asks the Metal sampler for observable expectations:
`sample_accepted(expectations=True)` served only tests. This drops the
feature and the plumbing around it (about 310 lines).

## Changes

- `gpu/tape.py`: tapes no longer compute expectations: no `ev` argument,
`observables` map or weight bookkeeping. `tape_kernel` loses
`num_detectors`, and `tape_function` loses `observables`.
- `gpu/noise.py`: `split` loses `observables_from`.
- `metal/kernels.py`: the tape kernel loses its `expectations` buffer,
so `deviation` moves from buffer 7 to 6. The `gather_float` kernel is
gone.
- `metal/sampler.py`:
- `sample_accepted` and `_sample_call` merge into the private
`_accepted`, which always takes columns. `direct_bools` passes every
output.
- Slots, shares and the noise split are no longer keyed by
`expectations`. The split is computed when the module compiles.
- `_select` no longer commits an empty command buffer when it has
nothing to dispatch.
- Tests: the expectation tests are gone.
`test_metal_unmasked_expectations_after_a_masked_call` becomes a
rows-only check of the same buffer reuse.

## Testing

- `pytest test/unit test/integration/test_metal_sampler.py` on Linux:
2254 passed, 103 skipped (the Metal tests skip without a Mac).
- **Not run on Apple hardware.** On Linux the sampler ran against a fake
Metal runtime. Every kernel dispatch (kernel, arguments, grid) for 30
circuits, with and without masks, packed and boolean, matches main,
apart from the dropped `values` argument and the empty command buffers
that are no longer committed.
- The changed MSL has not been compiled. **Please run the Metal tests on
a Mac before merging.**

PR #219 (CUDA) leaves this shared code alone and will drop its own use
of the old signatures once this lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
rafaelha and others added 5 commits November 17, 2026 10:40
CUDA takes the shared tape's new signatures (#301): tape_kernel has no
num_detectors, and the tape kernel no expectations pointer.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
_sample_rows was 145 lines of three closures over a state dict. A call's
chunks are now a _Chunk each, launched by _start (which takes in _run_batch,
its only caller) and finished by _finish, with the call's shared state in a
_Call, as in the Metal sampler.

Also:
- drop the tape blocks from an occupancy query that CuPy does not have (the
  branch was dead: 4 blocks per multiprocessor always), and _Module's unused
  plan and psm_words
- merge _HostRows.allocate and grow into reserve
- a call of one chunk copies on its kernel stream, and every call ends by
  waiting for all of its streams, which replaces the copied flag and the set
  of used streams
- move kept_fits next to noise_kept, and drop unused template arguments

Seed-fixed samples are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The kernel, noise and copy streams and the pinned counts were four lists
indexed by a slot number; a _Slot holds one slot's. A call now ends by
waiting for its copies in flight (_Call.land), which follow all of its
kernels, instead of for every stream, and _select returns the rows of a whole
output itself, so _finish selects every output alike.

Seed-fixed samples are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
_start made three views of a chunk's counts before it launched the noise
kernel. A small call waits for its counts with a blocking read that polls
more coarsely the longer it waits, so time before the noise launch costs it
about twice: a 1000-shot call of cultivation d3 took 82 us instead of the 75
us before the refactor. The kernels now take the array (the tape kernel
derives its two other counters), and both seeds come from one draw, with the
same values: 79 us.

Seed-fixed samples are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
CuPy raises a plain RuntimeError when it cannot load NVRTC, which
_compile did not turn into CompileError, so device="auto" raised at the
first sample instead of sampling on the CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rafaelha rafaelha changed the title CUDA backend CUDA backend (up to active_width=11) Nov 17, 2026
@rafaelha
rafaelha marked this pull request as ready for review November 17, 2026 20:48
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T20:51:51.545387Z d140f3d Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d140f3d98a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +95 to +98
@functools.cache
def arch() -> str:
"""Return the NVRTC architecture option of the current device, e.g. ``-arch=sm_120``."""
return "-arch=sm_" + cupy().cuda.Device().compute_capability

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Key cached CUDA properties by device

In a process using heterogeneous NVIDIA GPUs, after arch() is first called on one device, switching CuPy's current device and constructing another sampler reuses the first device's cached compute capability. The second sampler therefore asks NVRTC to compile for the wrong architecture, which can make an otherwise supported GPU fall back to the CPU under device="auto" or raise under device="cuda"; cache these properties by device ID or store them on each sampler.

Useful? React with 👍 / 👎.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant