perf(mpsc): separate bounded waiting and unbounded reclamation - #290
Merged
tisonkun merged 10 commits intoSep 8, 2026
Conversation
Member
Author
|
Merging for benchmark cases. I have a new idea for mpsc implementation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Keep both custom MPSC implementations while making their module and ownership boundaries explicit. Follow-up to #247. Public channel APIs and cancellation semantics are preserved.
f2ce322bounded/mod.rs: endpoints and lifecycle;ring.rs: storage, publication, and receiver waiting;waiters.rs: sender retries.unbounded/mod.rs: endpoints, liveness, and wake registration;buffer.rs: segmentation, batch handoff, and reclamation.Both channels use a module directory with
mod.rs. The bounded path no longer usesinternal::AtomicWaker: this PR's changes to that helper have been reverted, preserving its main-branch implementation and tests. With no production callers remaining, the module is compiled only for tests.All PR now measurements below identify
f2ce32294cf94c0d98bff61a11abdaa90113b0c5; PR before identifies95ef9eb, which used the custom ring with a hinted generic AtomicWaker. main identifies27c3067, also a custom ring, with semaphore-based send retries. None of these three columns uses a std channel backend.Measurements: Apple M4 Max, macOS 26.6.2, Rust 1.99 nightly, Tokio 1.53.1, async-channel 2.5.0, Flume 0.12.0. The same benchmark harness was used for all implementations. Results are medians of five round medians, with implementation order shuffled each round. Each round uses 35 samples except capacity-one OS-thread cases (10), inline concurrent bursts (50), and ready paths (100 samples of 1,000 operations). Lower is better. Throughput tables use ms per complete sample; ready-path tables use ns per send/receive pair. These are workload comparisons, not latency percentiles or a claim of universal superiority; no x86 performance measurements were taken.
Bounded
Design
The ring alone accounts for capacity, including reserved but unpublished slots. A send future owns its unsent value and joins
SendWaitersonly when it must wait. A notification grants a retry, not a capacity permit; cancelling a notified sender passes the retry to the next waiter. The receiver is the only consumer.Receiver waiting uses one cache-padded atomic flag and
Mutex<Option<Waker>>inside the ring. After publishing a slot, a producer checks the flag and atomically claims a notification before taking the waker lock. Registration installs the waker, arms the flag, and then the receiver retries the queue. Paired sequentially consistent fences prevent registration and publication from both being missed. Taking the waker clears the flag under its lock, so a delayed notification may wake a newer registration but cannot silently erase it. Clone, drop, and wake callbacks run outside the lock. This protocol has no unsafe waker storage; unsafe code remains confined to slot ownership and consumption in the ring.The receiver registration path does not need a retry loop: only the receiver can install a waker; producers can only take it. Queue positions, the receiver wait flag, and the waker mutex do not share a cache line.
Persistent OS threads
16,384
usizemessages per sample, concurrent receiving, reused producer threads and channel. C = capacity, P = producer count. ms.27c306795ef9ebf2ce322Tokio tasks
16,384
usizemessages per sample, reused tasks and channel. The receiver runs on the thread callingblock_on; W0 is a current-thread runtime, W4 has four worker threads. ms.27c306795ef9ebf2ce322Ready paths
Capacity 64, one thread, no actual async suspension. ns per pair.
27c306795ef9ebf2ce322Alternatives and tradeoffs
A local mutex makes receiver-waker ownership and callback reentrancy easier to audit than the modified generic AtomicWaker, but does not win every workload. In particular, inspect C64/P8 OS threads and C4096/P1/W4 tasks against PR before; the latter loses a substantial part of the earlier PR's advantage. Compare those same rows with main and the ecosystem columns before treating this as a general channel regression.
Three five-round local ablations informed the design. These are separate experiments, not extra implementations shipped by this PR:
STD-A (std sync_channel + SendWaiters + hinted AtomicWaker) and STD-H (std sync_channel + SendWaiters + mutex/flag notification) were earlier local candidates. They are not included in this PR. For historical context, C64/P8 OS threads measured 3.886/9.064 ms for STD-A/STD-H; C4096/P8 OS threads measured 3.562/2.252 ms; C64/P8/W4 tasks measured 1.288/1.811 ms. Those earlier campaign results are not paired with the new tables. They show why neither backend nor notification strategy should be selected from one case alone. The retained ring preserves explicit publication/capacity behavior and the stronger buffered-value cleanup described below.
Unbounded
Design
The inbox mutex serializes enqueueing, receiver liveness, and wake registration. The receiver consumes a private batch between refills. Buffer owns the writable segment, sealed segments, and one spare; it has no knowledge of wakers or futures. Segments target at most 32 KiB of inline storage, and consumed segments are released incrementally. At most two empty payload buffers are retained, independently of peak occupancy; small channels grow lazily and oversized inline allocations are released after their last value. Message and waker callbacks run outside the inbox lock.
This cleanup preserves the previous PR's storage/reclamation policy. Allocation-boundary tests now exercise Buffer directly; public FIFO, cancellation, lifecycle, and concurrent delivery remain covered at the channel boundary.
Persistent OS threads
16,384
usizemessages per sample, concurrent receiving and reused workers/channel. ms.27c306795ef9ebf2ce322Tokio tasks
16,384
usizemessages per sample, reused tasks/channel, with the same W0/W4 convention as bounded. ms.27c306795ef9ebf2ce322Repeated bursts and storage reuse
One thread sends all 65,536 messages, then receives them, reusing the channel across samples. The backlog case keeps half a burst queued. These cases include movement/allocation/reclamation and do not measure asynchronous wakeup scheduling. ms.
27c306795ef9ebf2ce322Concurrent inline messages and receiver wakeups
16,384 inline 1 KiB messages per sample, four Tokio workers, 1/4/8 producers, and aggregate bursts of 64/1,024 messages. Each burst first polls an empty receiver to Pending, then releases producers. Per-producer sequence checks verify delivery/order. Results include producer coordination, scheduling, movement, and reclamation; they are not isolated wake latency. ms.
27c306795ef9ebf2ce322Ready paths
One thread, no actual async suspension. ns per pair.
27c306795ef9ebf2ce322Tradeoffs
The batched design remains competitive for small messages and many repeated bursts. It does not eliminate the earlier large-inline-message contention: at P4/P8 with burst 1,024, compare the Tokio and async-channel columns, and at P1/burst 64 compare main. These limitations remain visible rather than being hidden by the single-thread burst result.
The earlier 20-pair inline-burst comparison with Flume found only about 0.26% geometric-mean difference, with its 95% interval crossing zero; treat that result as parity. Flume's warmed VecDeque retained the 64 MiB burst allocation, whereas this PR's segmented policy kept roughly 64 KiB of empty payload buffers. That retention comparison comes from the earlier allocation campaign; the current Buffer boundary tests verify the same incremental-reclamation and bounded-retention policy.
An earlier STD-A unbounded candidate measured approximately 1.612/1.614 ms for P4/P8 with burst 1,024, showing a real alternative for that workload. It was weaker in the earlier small-message multiproducer tests and is not selected here.
Tests and fixes
The integration entry file starts with FIFO, capacity, value ownership, receive cancellation, and disconnection contracts. Separate suites cover sender backpressure/cancellation, reentrant callbacks and panic cleanup, and concurrent/executor operation. Tests that assert wakeup behavior now count actual notifications before repolling, rather than relying on noop wakers. Callback-clone fixtures share one RawWaker implementation. Duplicate trivial executor tests were consolidated; real executor and selection coverage remains.
The release comparison was checked against
v0.7.1andv0.7.2-rc.1(both used the std-backed bounded implementation):Validation on
f2ce322:cargo x test,cargo x check, andcargo x lintpass.cargo x miripasses with seed 0, including the unchanged generic AtomicWaker tests. The MPSC unit and integration suites also pass with seeds 1, 2, and 3: 10 storage tests and 21 integration tests per seed; only the three OS-backed Tokio tests are skipped by Miri and pass natively. Benchmark binaries and source trees were fingerprinted, and all recorded runs explicitly enabled Divan benchmarking with--bench.