feat: reader-thread affinity clamp + distributed-load hardening & public API - #31
feat: reader-thread affinity clamp + distributed-load hardening & public API#31AlperenKonukbay wants to merge 3 commits into
Conversation
Two independent fleet-facing improvements, both measured on real registry apps and multi-GPU runners (receipts in the PR body): Reader-thread affinity clamp - effective_read_threads(): reader-pool sizes clamp to the process's CPU affinity mask. Prod runners execute in dedicated cpusets (measured: 12 CPUs exposed where the default requested 16 threads); oversubscribed byte-copy threads measurably hurt (-18% read wall on a 38GB pack read, 1.70s vs 2.07s on a 12-cpu H200 runner; app-level: id-lora request 4.9% faster with byte-identical output). FLASHPACK_NO_THREAD_CLAMP=1 restores the raw request. Distributed (rank0-read + broadcast) load hardening + public API - _broadcast_storage now broadcasts uint8 VIEWS of the macroblocks: the collective moves bits, and torch's NCCL dtype map lacks float8_e8m0fnu (the mxfp8 scale dtype; gloo lacks all float8s) — native-dtype broadcast of quantized packs crashes. Byte views are dtype-agnostic and copy-free (verified byte-exact on real NCCL across 8 GPUs). - read_flashpack_file_distributed(): the rank0-read + broadcast load as a public storage-level API (assign_from_file's use_distributed_loading branch now delegates to it), so low-level callers — falcon's quantized loaders are the motivating case — can opt in with one line. Measured on 8x H200: all-ranks-read 25.8s (page-cold) -> broadcast 2.44s on all eight ranks; clear errors for no-process-group and NCCL+CPU. - first tests for the distributed path (2-rank gloo, CPU-only): byte-view broadcast incl. float8 blocks, end-to-end assign_from_file, and the new public API. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@fal-9000 review |
|
Review by fal-9000 (GPT-5.6 Sol, xhigh) — reviewed at head No issues found. Checked:
Checks were green at review time. |
…d reads regress when clamped below it Pre-merge regression gate (same-node interleaved A/B, 12-CPU H200, 38 GB pack, parity-checked): clamping the raw file->GPU path to the affinity mask starved the IO queue at every tier -- cold 11.3-11.9 GB/s at 12 threads vs 13.2-14.2 at 16 (R=1.18), 2x on a 6-CPU cpuset, and 1.19x even page-hot. Reader threads on that path are mostly blocked in pread: they are the IO queue depth, not CPU consumers, so the affinity budget does not apply. Oversubscribed requests still get capped (64 threads on 12 CPUs measured unstable: 1.9-4.6 s swings vs stable 3.1). effective_read_threads gains a floor kwarg: the GPU path floors the clamp at the 16-thread default (cap = max(affinity, default)); the CPU-destination eager path keeps the pure affinity clamp -- its work is minflt/memcpy-bound where oversubscription genuinely never helps. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Post-review change in 949412b: a pre-merge regression gate showed the affinity clamp starving the IO-bound raw path (12T 11.3-11.9 GB/s vs 16T 13.2-14.2 cold, 2x on a 6-CPU cpuset). The raw path now floors the clamp at the default and only caps oversubscribed requests (t64-capped measured 27% faster than unclamped); CPU-destination reads keep the pure affinity clamp. Re-receipt same node: R=1.04/1.00/1.00. |
|
@fal-9000 review |
|
Closing as subsumed: this work is on #36 was stacked on this branch, so its head contained these three commits (
Nothing here was reverted on the way: the sharded rewrite in #36 replaced its own GitHub shows this PR as Two things worth flagging, since the squash cost you some credit:
Full context on what #36 became, including the measurements: #36. |
2 participants
Two independent improvements, both measured on real registry apps and multi-GPU runners.
Reader-thread cap (oversubscription only)
Reader-thread requests on the raw file->GPU path are capped at
max(cpu affinity, 16); the CPU-destination eager path keeps a pure affinity clamp.FLASHPACK_NO_THREAD_CLAMP=1restores the raw request.This section originally shipped a pure affinity clamp on all paths. A pre-merge regression gate (same-node interleaved A/B, 12-CPU H200, 38 GB pack, parity-checked) showed that clamping the raw path below the default starves the IO queue — reader threads there are mostly blocked in
pread, so they are queue depth, not CPU consumers: 12 threads read 11.3–11.9 GB/s cold vs 13.2–14.2 at 16; on a 6-CPU cpuset the gap was 2x. The clamp now floors at the default and only caps oversubscribed requests, which is the case it measurably helps: 64 threads on 12 CPUs ran unstable (1.9–4.6 s swings) while capped-to-16 was 27% faster than unclamped-64 and steady. Re-receipt on the fixed head, same node: R = 1.04/1.00/1.00 across cold-native, 6-CPU, and hot cells (pass lines 1.05/1.10).Distributed load: hardening + public API
The rank0-read + NCCL-broadcast path runs in several production apps but had no tests and two latent issues, and was unreachable for low-level callers.
_broadcast_storagebroadcastsuint8views. Torch's NCCL dtype map lacksfloat8_e8m0fnu(the mxfp8 scale dtype), so native-dtype broadcast of an mxfp8 pack cannot work (latent failure, found by inspection rather than a production crash); byte views are dtype-agnostic and copy-free. Verified byte-exact over NCCL on 8 GPUs, including the float8 dtypes.read_flashpack_file_distributed();assign_from_filedelegates to it. This lets low-level callers (falcon's quantized loaders) opt in — today every rank of a world-size-N app re-reads the full pack, since the O_DIRECT reader shares nothing through the page cache.assign_from_file.Measured on the real 38 GB pack, 8× H200: all-ranks-read 25.8 s max/rank page-cold (4.89 s page-hot) → rank0+broadcast 2.44 s on all eight ranks (2.22 s hot); at world=2 the cold win is ~−25%. Cross-rank parity verified on every run. Cold-tier magnitude is node-dependent (rank self-contention on one node's disk); the mechanism — N duplicate O_DIRECT reads collapsed to one — is scale-invariant.
Review notes