Add a ring-native wait: SubmitTimeout in core, RingTimer in ioxide.timer - #213
Merged
Conversation
The reactor already submits IORING_OP_TIMEOUT for its own 250ms ticker, so the kernel-side timer was there; there was just no way for a caller to ask for one. Without it, code that needs a deadline has to build its own out of a timerfd and SubmitRead, which costs a timerfd_create per waiter, a timerfd_settime per wait and a read to consume the expiration. SubmitTimeout(nanoseconds, completion) goes through the same path as every other client op, so it inherits the slot allocation, the off-reactor marshalling and the O(1) completion routing, and it needs no new dispatch case. The duration rides in the offset argument rather than adding a signature, and the timespec lives in a block parallel to the op slots: the kernel reads it while the op is in flight, so tying it to the slot makes its lifetime exactly the operation's and costs no allocation per wait. Expiry arrives as -ETIME, which is io_uring's normal report for a timeout rather than a failure. That is documented on the interface so callers do not treat it as an error. Motivated by an HTTP benchmark that answers after a delay named in the request: at 32,000 connections that is 32,000 live deadlines, and building them out of timerfds cost a syscall per request.
SubmitTimeout gives the verb; this gives it the shape the other ring clients have. RingSocket wraps SubmitSend and SubmitRecv so a protocol client never touches an SQE, and a deadline deserves the same: RingTimer holds one RingOpSource and hands it to SubmitTimeout, so a wait is Prepare, submit, await, with nothing allocated per wait and the completion arriving inline on the reactor that owns the caller. It follows RingSocket's conventions rather than inventing new ones - one op in flight per instance, results returned as the ring reports them rather than translated. That last point needs saying out loud, so it is on the type: a wait that runs its course completes with -ETIME, which is io_uring reporting the timeout expired and is the success case. Expired() exists so callers do not have to know that to read their own result. Four tests, on a handler that answers after the milliseconds named in the path while holding one timer for the connection - the shape a caller is meant to copy: that the wait actually elapses, that expiry arrives as expiry rather than an error, that one timer serves a whole connection through repeated re-arming, and that three concurrent connections asking for 80ms, 10ms and 40ms each get their own deadline and their own value back rather than whichever finished parsing last. 158 passed, 0 failed.
…ents It went into core next to RingSocket, which was the wrong read of where the line is. RingSocket is in core because it is what the protocol clients are built from, not because it wraps a core verb - ioxide.file already wraps one, SubmitRead, and ships as its own package to do it. RingFile is 59 lines against RingTimer's 60. Same shape, same dependency, so the same packaging. SubmitTimeout stays in core, where it has to be: it is an IRingHost method and its implementation is inside the reactor's op slots, its SQE submission and its teardown. That is the verb. ioxide.timer is the client over it, and depends on nothing but ioxide. 158 passed, 0 failed.
MDA2AV
changed the title
ioxide.timer is new and cannot ship on its own: it needs the SubmitTimeout that lands in core with it, so core has to move too. The rest follow because these are released together and a set where some packages say 0.7.210 and others 0.7.211 is harder to reason about than one where they all agree, whether or not each one changed. 12 packages, all at 0.7.211.
…he fd path SubmitTimeout borrowed SubmitClientOp by passing fd -1 and a null buffer, and SubmitClientOpCore then had to ask every op whether it was a timeout before filling the SQE. That put a comparison on the path every send, recv, read and write takes, to serve the one op that is none of them, and it read as though a timeout were a degenerate fd op rather than a different thing. It is a different thing: it carries a deadline where they carry a buffer. So it fills its own SQE now, in SubmitTimeoutCore, and SubmitClientOpCore is back to exactly what it was - this change no longer edits a single line of it. What the two still share is the part worth sharing: the slot table, the completion routing and the off-reactor handover. The one place they have to be told apart is DrainRemoteOps, which is the cross-thread path and runs once per handover rather than once per op. 158 passed, 0 failed.
Both submissions had their own copy of the same six lines - opcode, fd, addr, len, off, then the client tag - which is the part that genuinely is common and the part that would quietly drift. Emit takes it. What is left in each caller is only what it knows: a buffer and a length for the fd ops, a deadline and a count of one for a timeout. They differ in what goes in the fields, not in how it is submitted. This is as far as the two can be merged. The timeout's addr is the timespec belonging to its own slot, so the deadline cannot be resolved until the slot is allocated, which is inside. Folding it back into SubmitClientOpCore would mean handing that function a branch, on the path every send, recv, read and write takes, for the one op that is none of them. 158 passed, 0 failed.
SubmitTimeout had its own copy of the thread check, the enqueue and the wake - the same four lines SubmitClientOp already had, differing only in which core they fell through to. HandedOff takes them, and each verb is now a handover attempt followed by its own submit. That is as much as the two can share, and the reason is in the comment on it: the fd ops put a buffer and a length in the SQE, a timeout puts a deadline that lives with its own slot, and the slot is not known until the reactor thread allocates it. Calling SubmitClientOp outright would mean asking every send, recv, read and write which of the two it is, on the path they all take, for the one op that is neither. So they now share what is common at both ends - the handover on the way in, Emit on the way out - and differ only in the middle, where they actually differ. 158 passed, 0 failed.
HandedOff put the whole handover behind a call, which meant every send, recv, read and write reached it through six argument pushes to ask a question that is two comparisons. Whether the JIT inlined that back is a guess, and a guess is not worth making on the path every op takes. The test goes back where it was, character for character with what is on main. What is shared is only the part that runs when the answer is yes - the enqueue and the wake - and it is marked NoInlining so it cannot climb back into a caller that will not execute it. So the hot path is what it always was: two comparisons and a call to its own core. The duplication that is gone is the part that only runs off-reactor. 158 passed, 0 failed.
MDA2AV
added a commit
to MDA2AV/HttpArena
that referenced
this pull request
Aug 28, 2026
All seven references go to 0.7.211, the 140 KB build in localfeed/ and the NuGet.config that pointed at it are deleted, and the Dockerfile restores the way it always did. That scaffolding existed to measure IORING_OP_TIMEOUT before the API shipped, the measurement is done and the API has shipped, so it goes. ReactorDelay is gone with it. It had grown four mechanisms - a userspace tick posting drains across threads, a per-reactor timerfd with a priority queue, a per-request timerfd, and the ring timeout - which existed to find out which one belonged in the entry. The bench hardware answered that, so the other three are history rather than code, and what is left is 60 lines against one submission. What replaces it is the RingTimer from MDA2AV/ioxide#213, kept local for now: ioxide.timer missed the 0.7.211 release even though the SubmitTimeout it needs did ship, so the class lives here and becomes a package reference when it is published. It is the same code either way. The wait is per connection and built on first use, so a connection that never waits never makes one - which is every connection on every profile but this one. One op in flight is all a connection needs, because the handler awaits before it reads again. Verified against the released packages: /delay/10 answers in 10.02ms and /delay/200 in 200.09ms, 120 sequential waits on one connection are none early with a mean overshoot of 0.05ms, 32 concurrent distinct delays complete in 69.4ms against 1088ms serialised, and baseline is 3.75M.
Merged
MDA2AV
added a commit
to MDA2AV/HttpArena
that referenced
this pull request
Aug 28, 2026
All seven references go to 0.7.211, the 140 KB build in localfeed/ and the NuGet.config that pointed at it are deleted, and the Dockerfile restores the way it always did. That scaffolding existed to measure IORING_OP_TIMEOUT before the API shipped, the measurement is done and the API has shipped, so it goes. ReactorDelay is gone with it. It had grown four mechanisms - a userspace tick posting drains across threads, a per-reactor timerfd with a priority queue, a per-request timerfd, and the ring timeout - which existed to find out which one belonged in the entry. The bench hardware answered that, so the other three are history rather than code, and what is left is 60 lines against one submission. What replaces it is the RingTimer from MDA2AV/ioxide#213, kept local for now: ioxide.timer missed the 0.7.211 release even though the SubmitTimeout it needs did ship, so the class lives here and becomes a package reference when it is published. It is the same code either way. The wait is per connection and built on first use, so a connection that never waits never makes one - which is every connection on every profile but this one. One op in flight is all a connection needs, because the handler awaits before it reads again. Verified against the released packages: /delay/10 answers in 10.02ms and /delay/200 in 200.09ms, 120 sequential waits on one connection are none early with a mean overshoot of 0.05ms, 32 concurrent distinct delays complete in 69.4ms against 1088ms serialised, and baseline is 3.75M.
MDA2AV
added a commit
to MDA2AV/HttpArena
that referenced
this pull request
Aug 28, 2026
* ioxide: subscribe to async, with a delay that stays on the reactor
GET /delay/{ms}, answered after the milliseconds named in the path.
Task.Delay is the wrong tool twice over here: its continuation resumes on
the thread pool, which drags a connection off the reactor that owns its ring
and buffers, and it allocates a timer per call when the profile holds 64,000
of them at once.
ioxide gives us Reactor.ScheduleOnReactor, which runs a callback on a
specific reactor thread. That solves affinity but not timing, and posting
one callback per expiry would be ~4M cross-thread posts a second at this
profile's load.
So the posts are batched. One process thread ticks every 250us and asks each
reactor to drain; the reactor pops everything now due in one pass. ~64 posts
per tick rather than one per request. The pending timers live in
[ThreadStatic] state touched only by the reactor that owns them - the
handler parks a request from that thread and the drain runs on that thread -
so the hot path takes no lock at all.
The request itself parks the way /async-db already does: PendingDelay stops
the parser, the handler waits, writes the response and resumes the carry, so
pipelined requests behind it are still answered in order.
## Measured
c=4096 265.28K req/s ceiling 273K 97.2% avg 15.48ms
c=16384 1.02M req/s ceiling 1092K 93.4% avg 15.82ms
against a 15ms request, which is the same 93-96% band tokio sits in. All
eight async validation checks pass, including the 32-overlapping-requests
check that catches per-connection state being shared.
## What this is not
A kernel timer. IORING_OP_TIMEOUT is the right primitive - the kernel holds
the deadline and completes it on the ring, on the reactor thread, with no
second thread and no per-timer allocation - but ioxide exposes no way to
submit one, and no route onto the ring at all for anything that is not
socket-shaped. Asked for upstream in MDA2AV/ioxide#212; the code says so
where the tick is defined rather than leaving it to be discovered.
Validation now 64 passed, 31 failed, every remaining failure still TLS.
* ioxide: enable the entry
Everything except TLS passes: 64 checks, and the 31 that fail are all
json-tls, static-tls and the two 8443 profiles.
Those were never shown to be broken, only untestable here. The entry does
the TLS handshake in userspace and relies on kTLS for TX, and the tls kernel
module is not loaded on this machine -- the .ko is present but unloaded, and
containers see no tcp_available_ulp -- so outbound writes leave as plaintext
on a socket the client believes is encrypted. Loading it needs root I do not
have locally. The handshake itself passes every check: mounted certificate,
TLS 1.3, AEAD cipher, ALPN, and the whole TLS quality suite.
The bench host is the only place that can answer it, and CI validation runs
there. Enabling is how we find out rather than a claim that it works.
Its tests keep json-tls, static-tls, baseline-h2, static-h2 and both h3
profiles, so if kTLS is unavailable there too the validate job will say so
against those five and nothing else.
* Benchmark results: ioxide [skip ci]
* ioxide: take the delay registration lock once per thread, not once per request
ReactorDelay.Delay called Register on every request, and Register takes a
process-global lock and scans a list under it. At the async profile's load
that is one lock acquired several million times a second, contended by every
reactor thread at once, and it was the profile's ceiling rather than anything
about the delay itself.
The signature was visible in the published run: 1.06M req/s at 2411% CPU on a
64-core box, so two thirds of the machine idle while throughput sat flat. Two
local measurements confirm it. Shortening the delay from 15ms to 5ms to 2ms,
which raises the theoretical ceiling from 1.1M to 3.3M to 8.2M, moved the
result 1.89M to 1.93M and no further: a limit that ignores the workload. And
the cost scales with reactor count, which is why the 64-reactor bench box was
hurt roughly twice as hard as a 32-reactor box.
Registration is permanent and per thread, so it now runs on the first call
from each thread. The pending queue is already thread-static and is empty
exactly once per thread, so it gates the registration with no extra state.
Also surfaces the ServerConfig and TcpOptions knobs the entry never set
(RingEntries, ListenBacklog, PoolMax, RecvQueueEntries) as environment
overrides that default to the library's own values, and logs both the
defaults and the effective set at startup. Measured at 64,000 connections
over four interleaved repetitions, none of them shifts the result beyond the
run-to-run spread, so nothing here changes a default:
stock 1.73M (1.52-2.01) 1965% cpu
recvQueue 1024 + sqEntries 32768 1.68M (1.49-1.85) 1972% cpu
tokio, same harness 2.00M (1.76-2.25) 1845% cpu
Delay accuracy is unchanged: /delay/15 answers in 15.5ms and /delay/50 in
50.2ms, the 0.5ms being the 250us tick.
* ioxide: drain delays on the reactor that owns them, and stop spinning a core
Three changes to the async profile's wait, all of them removing work that was
never needed rather than adding machinery.
Drains move onto the reactor's own hot path. The tick thread used to be the
only thing that completed a timer, so every expiry cost a cross-thread
ScheduleOnReactor post: 4,000 ticks a second times one post per reactor, which
is 128,000 posts/s on a 32-reactor box and 256,000 on the 64-reactor bench box.
The handler loop is already running on the right thread with the right
thread-static queue in hand, so it drains there instead - about 50,000 passes
per reactor per second at this load, for a thread-static read and a queue peek.
The tick now skips any reactor that has drained itself within the last tick,
which under load is all of them.
The tick stops spinning. It held its old 250us schedule with SpinOnce(-1),
which never yields, so the thread burned 87% of a core whether it had anything
to post or not - measured per-thread, it was the busiest thread in the process,
busier than any reactor at 62-64%. On a box where reactors are pinned that was
most of a core taken away from one of them. As a fallback for parked reactors it
can sleep: reactor-delay now falls below 1% of a core. Validation asserts only a
lower bound on the wait, so the coarser granularity cannot fail an entry, and
measured overshoot is 0.22-0.82ms.
The completion is reused instead of allocated. A TaskCompletionSource and its
Task per request was 354 MB/s of garbage at this rate. A connection only ever
has one delay outstanding, because the handler awaits it before reading again,
so one ManualResetValueTaskSourceCore per session covers every request that
connection makes. Measured over three interleaved repetitions at 64,000
connections:
per-request 1.63M rps p99 70.1ms p99.9 277.7ms 364 MB/s
pooled 1.59M rps p99 69.4ms p99.9 142.0ms 272 MB/s
Throughput is unchanged within the run-to-run spread; p99.9 halves. Allocation
does not fall to zero, so something outside this file still allocates on the
async path - the rest of the request path does not, since 89.3M baseline
requests moved the heap by 0 MB.
Correctness, on a single socket: 60 sequential delays, every body correct, none
answered early, mean overshoot 0.25ms. Concurrent requests each get their own
wait. Baseline is unaffected at 3.53M.
None of this can be measured for what it is on a 32-reactor box that is already
at 0.8% idle with the load generator on it; the numbers that matter come from
the bench box, where the entry sat at 3000% CPU of an available 6400%.
* ioxide: give each reactor 1024 recv slots instead of 256
Recv slots are what lets a connection have its next request picked up, so the
count has to be read against connections per reactor rather than against the
box. The async profile holds 64,000 connections open, about 1,000 per reactor
on the bench hardware, and at 256 slots most of them have nothing posted and
wait for one to free.
The measured breakdown says that is where the time goes. A request asking for a
15ms wait comes back in 37.8ms, and instrumenting the timer against its own
deadline shows it completes 3.2ms late on average - so the delay accounts for
about 13% of the overhead and the remaining 20ms is the read and write path.
That is also consistent with the entry sitting at 3000% CPU with 6400%
available: reactors idle rather than saturated.
256 was not chosen for this profile. The comment above it is about upload,
where the buffer size drives how many slices a 20MB body takes, and the slot
count came along with it. ioxide's own default is 4096.
This is a trade, not a free win: 1024 reserves 16 MB of recv buffers per
reactor, so 1 GB across 64 of them against 256 MB before, and memory is a
scored factor in the composite. 4096 would be 4 GB, which is why this stops at
1024. Whether the throughput pays for the memory is what the run decides.
Checked locally at this setting: upload returns the right length for both
fixed-length and chunked 20MB bodies, /json/1 serves its payload, /delay/15
answers in 15.98ms, and baseline is 3.80M.
* ioxide: add a ring-native delay on a timerfd, off by default
The rest of this entry never waits in userspace: ioxide.pg parks on the
Postgres socket through the reactor's ring and the completion arrives on the
reactor thread. The delay was the one wait that did not, so this makes it
possible to do the same and measures what it costs.
Reactor.SubmitRead takes a raw fd, and a timerfd is pollable, so io_uring arms
a poll on it internally rather than handing the read to a worker thread. That
gets to the same place as IORING_OP_TIMEOUT, which ioxide does not expose
(MDA2AV/ioxide#212). One timerfd per connection, created on its first delay and
re-armed after, released with the connection.
It is correct and it is more precise than the tick: /delay/1 answers in 1.06ms,
/delay/200 in 200.23ms, and 30 re-arms on one connection overshoot by 0.18ms on
average against the tick's 0.25-0.82ms.
It is off by default because it measured worse, and the reason is worth keeping
in the file. A socket read is I/O the connection has to do anyway, so the ring
costs nothing extra for it. A timer is not: this is a timerfd_settime syscall
plus an SQE and a CQE per request, roughly 1.5M syscalls a second at this load,
where the tick completes about 13 timers per drain and makes no syscall at all.
Four interleaved repetitions at 64,000 connections:
ring 1.50M rps 2007% cpu p99 104.2ms p99.9 429.6ms
tick 1.58M rps 1765% cpu p99 68.4ms p99.9 373.4ms
14% more CPU for 5% less throughput. The tick's batching is what wins, so the
precision is not worth buying here. IORING_OP_TIMEOUT would remove the syscall
but not the SQE and CQE, so it is worth re-measuring if that ever lands.
IOXIDE_DELAY_MODE=ring selects it. Both paths verified after the default flip.
* Revert "ioxide: give each reactor 1024 recv slots instead of 256"
This reverts commit 7a2727d.
* ioxide: default the delay to the ring timer, to read it on the bench hardware
The two mechanisms cost differently and the difference scales with reactor
count, so the 32-reactor result does not settle it.
The ring timer costs a timerfd_settime syscall plus an SQE and a CQE per
request. That is fixed per request whatever the box. The tick costs one
cross-thread post per reactor per tick, so 128,000 posts a second across 32
reactors and 256,000 across the bench box's 64, against the same per-request
cost on the other side.
Locally the tick wins, over four interleaved repetitions at 64,000 connections:
ring 1.50M rps 2007% cpu p99 104.2ms overshoot 0.18ms
tick 1.58M rps 1765% cpu p99 68.4ms overshoot 0.25-0.82ms
Twice the reactors doubles one side of that and not the other, so this defaults
to the ring to get a reading from the hardware the profile is scored on. If the
ordering holds there, this flips back.
IOXIDE_DELAY_MODE=tick selects the other path. Both verified after the flip:
ring answers /delay/15 in 15.17ms and /delay/200 in 200.32ms, tick in 16.17ms.
* Revert "ioxide: default the delay to the ring timer, to read it on the bench hardware"
This reverts commit f18dad0.
* ioxide: default the delay to the ring timer
Measured on the bench hardware at 16,000 connections and 5ms:
tick 1,842,424 rps 3435% cpu
ring 2,462,698 rps 5831% cpu avg 6.48ms p99 9.91ms p99.9 16.50ms
34% more throughput than the tick and 20% more than tokio's 2,054,680, with
1.48ms of overhead on a 5ms wait.
The earlier reading that said the opposite was taken on a 32-reactor box sitting
at 0.8% idle with the load generator on it. The ring costs a timerfd_settime
syscall plus an SQE and a CQE per request, and on a machine with no CPU to spare
that can only come out of throughput, so it lost there. The bench box has 6400%
available and the tick was using 3435%, so the same trade buys something instead
of costing something. Nothing about the mechanism changed between those two
measurements, only whether there was CPU free to spend on it.
That also explains the shape of the tick's result rather than just its size.
The tick completes a timer only when a drain runs, so work arrives in batches
and the reactors idle between them - which is why CPU sat at half the box while
throughput flattened. Handing each deadline to the kernel keeps it flowing.
The trade needs spare CPU to be worth making, so the comment records both
readings and says to re-read it if the profile or the hardware changes.
IOXIDE_DELAY_MODE=tick still selects the other path.
* ioxide: do not resume a ring delay before its deadline
The ring completion was treated as proof the wait was over. It is not, and
under load it demonstrably is not: instrumenting the completion against its own
deadline showed about 0.1% of waits resuming early, the worst of them 14.9ms
into a 15ms delay, which is the entire wait skipped.
The cause is not the one the code was written to expect. The fd is non-blocking
so a read the kernel does not arm a poll for returns -EAGAIN, but that never
appeared in the measurement: every early completion carried a valid 8-byte
expiration count. The fd is reused for every request on a connection, so a read
can pick up an expiration left behind by an earlier one and report a timer that
really did fire, only not this request's.
So the deadline decides rather than the completion. An early one goes back on
the ring instead of resuming, which costs one comparison on a path that is
already making a syscall. Measured: earlyCompletions equalled resubmitted on
every sample, and completions still tracked armings, so it converges rather
than spinning.
The effect on the published 2,462,698 is negligible at 0.1%, but a request that
answers before it waited is wrong regardless of what it does to the number, and
a sampled one would fail validation outright.
120 sequential delays on one connection: none early, none wrong-bodied,
overshoot 0.05-0.46ms.
* async: 16,000 connections at 5ms, down from 64,000 at 15ms
Experiment. At 64,000 connections the profile has been reporting a gap between
entries that does not survive inspection: instrumenting the wait shows the timer
completing 3.2ms late against 23ms of end-to-end overhead, so the delay is about
13% of what is being measured and the rest is the read and write path. Fewer
connections and a shorter wait puts the load somewhere different, and whether
the ordering between entries holds is the thing worth knowing.
The ceiling is connections over delay, so this moves it from 4.27M to 3.20M. The
best entry so far is 2.24M, which leaves headroom but less than before; the docs
now say so rather than claiming the ceiling does not bind.
Existing results are all at async-64000 and are orphaned by this, so the column
is empty until the profile is re-run. That is accepted for now.
Generator runs clean and badge parity holds at 579 ranks.
* async: regenerate board data for the 16k/5ms conn set
The async column drops out until the profile is re-run, since every existing
result is at async-64000.
* async: 32,000 connections at 10ms, holding the ceiling at 3.20M
This is a control rather than a retune. The ceiling is connections over delay,
so 32,000 at 10ms and 16,000 at 5ms both come to 3.20M. Doubling both leaves
connection count as the only thing that changed, which is the question the
profile has been circling.
What is known so far, all on the bench hardware:
64,000 / 15ms ceiling 4.27M ioxide 1,568,519 tokio 2,263,962
16,000 / 5ms ceiling 3.20M ioxide 2,360,000 tokio 2,054,680
The ordering reverses between those two, but they differ in ceiling as well as
in connections, so the comparison does not isolate anything. This one does: a
drop against the 16k/5ms numbers is connection scaling and cannot be the
arithmetic, because the arithmetic is identical.
Docs carry the new figures - a 64-thread blocking server tops out near 6,400
rps here, and 1-2ms of timer overshoot now costs a tenth to a fifth of the wait
rather than a fifth to a third. The best-entry reference moves to 2.36M.
Generator runs clean and badge parity holds at 579 ranks.
* async: 64,000 connections at 20ms, third point on the same 3.20M ceiling
Continues the controlled series. Connections and delay double together so the
ceiling does not move, which leaves connection count as the only variable:
16,000 / 5ms ioxide 2,360,000 (74% of ceiling) tokio 2,054,680 (64%)
32,000 / 10ms ioxide 2,390,257 (75%) tokio 2,359,835 (74%)
64,000 / 20ms ceiling 3.20M
What the first two points already show is that tokio scales on this axis and
ioxide does not. Doubling the connections took tokio from 2,054,680 to
2,359,835 while ioxide went 2,360,000 to 2,390,257, flat, and it needed 6066%
CPU against tokio's 4079% to stay level - 39,402 requests per CPU percent
against 57,856.
This point matters for a different reason. The original 64,000 result was
1,568,519 against a 4.27M ceiling, 37% of it, and that was read as connections
being the problem. Here the connections are the same and only the arithmetic
differs. If this lands near 2.4M then 64,000 connections were never the
difficulty and the earlier collapse belongs to something else in that
configuration.
Read the numbers with the ceiling in mind: both entries are already at about
75% of 3.20M, and queueing grows quickly that close to the limit, so some of
the convergence between them may be the ceiling rather than the frameworks.
Generator runs clean and badge parity holds at 579 ranks.
* async: settle at 32,000 connections and 10ms
The profile was walked along a line where connections and delay double together,
so the ceiling stays 3.20M and connection count is the only thing that moves:
16,000 / 5ms ioxide 2,360,000 (74%) tokio 2,054,680 (64%)
32,000 / 10ms ioxide 2,390,257 (75%) tokio 2,359,835 (74%)
64,000 / 20ms ioxide 1,641,196 (51%) tokio 1,887,483 (59%)
64,000 costs both entries about a third of their throughput against the same
ceiling, and that is the reason to stop at 32,000. Both losing together points
at something they share rather than at either server: the load generator holding
64,000 sockets, and the kernel's own cost for that many. A profile whose
headroom is being spent by its own harness is not measuring what it says it is.
The servers do not argue for stopping here. Measured locally against this build:
one reactor serves 1000 connections at 97% of its own ceiling on a fifth of a
core, and pinning 1000 per reactor while going from 1 to 16 reactors holds
94-97% throughout with latency flat at 20.1ms. Neither connections per reactor
nor reactor count is what bends, so the limit is in the shape of the test.
32,000 is also where the two entries agree, 2,390,257 against 2,359,835, which
is what a profile measuring concurrency rather than harness capacity should look
like. The gap that does remain is in CPU, 6066% against 4079%, and that is a
real difference worth reporting rather than one manufactured by the load.
The docs now record why the connection count is what it is, so the next person
to reach for a bigger number has the measurement instead of the intuition.
* async: drop every stored result from a connection count the profile no longer runs
13 rows across 5 entries, none of which the board can show. async-64000 is from
the shape the profile had before it settled at 32,000, and async-32768 and
async-49152 are older still, left behind while the connection count was being
tuned in the first place.
A stale row is not inert here. rebuild_site_data.py keys results by profile and
connection count and never prunes, so anything the CATALOG stops asking for
simply accumulates, and the next person reading these files cannot tell which
numbers the profile actually produces.
aspnet-minimal, express, fastapi, tokio async-32768, async-49152, async-64000
ioxide async-64000
async-db is a different profile and is untouched. The async column is empty
until the profile is re-run for its subscribers.
* Benchmark results: 7 frameworks (async) [skip ci]
* ioxide: hold one timer per reactor instead of one per request
The delay now keeps deadlines in the reactor's own queue and arms a single
timerfd at the front of it. This is what an event loop normally does - Node and
Bun put the next deadline into the poll they were making anyway - and it is the
only shape that neither syscalls per request nor leaves the reactor asleep past
a deadline that has already passed.
The two it replaces each gave up one of those. The per-request timerfd woke the
reactor exactly on time but cost a timerfd_settime, an SQE and a CQE every
single wait. The tick cost nothing per wait but only completed a timer when
something else happened to wake the reactor, which is what left it idle with
work already due. Arming only when a wait is due before whatever the timer
already holds means most waits cost nothing at all: with waits coming due in
batches of about thirteen, that is one arming per thirteen requests.
Measured on this box, 16,000 connections at 2ms, three interleaved repetitions:
queue 1.99M rps 1852% cpu p99 19.93ms p99.9 34.23ms
tick 1.98M rps 1861% cpu p99 18.47ms p99.9 31.07ms
ring 1.81M rps 1959% cpu p99 27.80ms p99.9 56.03ms
The box is saturated with the load generator on it, so the three compress
together here; the bench box is where ring's per-request syscall started to pay
for itself by keeping reactors fed, and this gets the same wake-up behaviour
without buying it.
IORING_OP_TIMEOUT was tried too, with the deadline handed to the kernel on the
ring. It needs a Reactor.SubmitTimeout that no published ioxide has, and it
measured no better than this: 1.98M and a worse tail, because it is still one
SQE and one CQE per request where this is one per batch. Left out until the
library exposes it.
IOXIDE_DELAY_MODE=ring or =tick still select the other two. Verified on the
default path: 80 sequential delays with none early, mean overshoot 0.18ms, and
32 concurrent distinct delays served in 69.6ms against 1088ms serialised.
* bun: answer GET /delay/{ms} and subscribe to async
Bun.sleep returns a promise the scheduler resolves on a timer, so awaiting it
parks the request and hands the thread back rather than holding it.
Measured on this box: the wait costs bun 22%, 2.09M on the baseline mix against
1.63M on the delay path. That is worth having next to the other JS entry, where
the same wait is free - fulmine goes 1.59M to 1.68M - so the profile has two
runtimes with the same concurrency model and visibly different timer costs.
Verified before subscribing: /delay/10 answers in 10.16ms and /delay/50 in
50.37ms, and 32 overlapping requests carrying delays from 3ms to 65ms complete
in 69.6ms against the 1088ms they would take one at a time, none of them early
and every body its own value.
* ioxide: complete due waits from the reactor's hot path, not only from the timer
The per-reactor timer was the only thing completing a wait, and deadlines do not
arrive in groups: with one wait per request they are spread continuously, so
each got its own arming and its own read. That is a syscall per request, which
is precisely what holding one timer per reactor was supposed to avoid, and the
published run shows it - 2,179,118 rps at 4251% CPU with 2,150% of the box
sitting idle.
The reactor already passes through the handler loop tens of thousands of times a
second under load. Draining there costs a thread-static read and a peek, so most
waits now finish for nothing at all and the timer is left covering the only case
that needs it: the reactor about to sleep with work already due.
Both drains take the same re-entrancy guard. Completing a wait runs its
continuation inline and that continuation goes straight back round the loop,
which drains again, so without it one connection's resume nests inside another's
for as deep as the queue happens to be.
Locally this is inside the noise, 2.00M at 1877% against 1.99M at 1852%, because
the box is saturated with the load generator on it and has no idle CPU to
recover. The bench box has 2,150% of it.
Accuracy is the best of any mode so far: /delay/10 answers in 10.02ms and
/delay/200 in 200.09ms, 100 sequential waits on one connection are none early
with a mean overshoot of 0.15ms, and 32 concurrent distinct delays complete in
69.8ms against 1088ms serialised.
* async: let the generator's thread count be set on its own, and try 128
The profile is closed-loop: every connection holds exactly one request for the
length of its wait, so what comes out is connections divided by latency, and the
generator is part of that latency. It follows that the number is not the
server's alone, and measurement says so - holding the server fixed on a 32-core
box and changing only the generator's threads took the result from 1.42M at 8 to
2.16M at 16, a 52% move with nothing on the server side touched.
Both entries also sit well under the hardware they are given, 4251% and 4068% of
6400%, and they sit there together despite being unrelated implementations. Two
different servers agreeing on how much of the box to leave idle points at
something they share.
ASYNC_THREADS defaults to 128 here, against the 64 the profile has been running,
to find out which side of the curve the bench hardware is on. If throughput does
not move, the generator was not the constraint and the remaining 3.5ms of
queueing above the 10ms wait belongs elsewhere. If it does move, the profile has
been reporting the generator as much as the server and the number needs revising
before anyone reads a ranking off it.
* Revert the hot-path drain and put the generator back to 64 threads
Both were tested on the bench hardware and both were wrong.
The generator was not the constraint. tokio is the control - unchanged code, only
the thread count moved - and it went 2,366,578 to 2,364,326 across 64 to 128
threads. A tenth of a percent. So the ~3.5ms of queueing above the 10ms wait does
not belong to the harness, and the earlier 52% swing on a 32-core box was that
box being oversubscribed rather than anything about the profile.
That control also isolates the other change, and it cost throughput: ioxide went
2,179,118 to 2,089,696, down 4.1%, with the thread count accounted for by tokio
sitting still.
The reason is a hole in the idea rather than a tuning matter. Draining on the hot
path never cancelled the armed timerfd, so the timer still fired and still cost
its arm and its read; every pass of the handler loop simply added a peek on top.
It could only ever be more work. Cancelling the timer to make the drain worth
having would cost a syscall of its own, which is the thing being avoided, so
there is nothing to salvage here.
Back to the per-reactor timer alone, which measured 2,179,118 at 4251% CPU -
31% better throughput per unit CPU than the per-request timerfd it replaced,
and level with tokio on CPU where that was 40% above it.
ASYNC_THREADS stays as a knob, documented and defaulted to THREADS' 64, since
knowing the generator is not the constraint is worth being able to re-check.
* async: halve the server's cores, to tell "out of work" from "contended"
Both entries leave a third of the box unused and they leave it together, which
has looked like a cap all along. The arithmetic says otherwise: the profile is
closed-loop, so work is capped at connections over delay, and at the ~19.5us of
CPU a request costs, running the 3.20M ceiling would take about 6240% of the
6400% available. ioxide sits at 2.18M, 68% of the ceiling, using 4251% - which
is 68% of that. On those numbers nothing is blocked; there is simply no more
work to be had.
That is a prediction rather than a proof, and halving the cores tests it. 16
cores with their SMT siblings is 32 logical CPUs and a 3200% budget, which at
19.5us per request buys about 1.64M.
drops to ~1.6M and pins near 3200% the server was never blocked, only idle
holds ~2.18M at ~3200% per-request cost halved, so the wider
set was losing CPU to contention
The second outcome is the interesting one and would mean the extra cores were
costing more than they returned - cross-reactor traffic, memory bandwidth or
SMT siblings fighting over a core. The first says the remaining throughput is
bought by cutting the latency tail rather than by adding CPU, which is where the
zrk numbers already point: p50 sits at 2.21ms against a 2ms wait while p99 is
152ms, so the mean that divides into the connection count is made almost
entirely of tail.
Experiment on the branch, not a proposal for the profile.
* ioxide: put the per-request ring timer back as the default, on 16 cores
Halving the server's cores answered the question the idle CPU had been posing,
and not the way the arithmetic predicted: tokio pinned its 3200% while ioxide
did not. Two entries on identical hardware and an identical ceiling, one able to
spend everything it is given and one not. So ioxide is blocked rather than out
of work, and the model that said otherwise - work capped at connections over
delay, everyone equally short of it - is wrong about ioxide specifically.
The per-request timerfd is the mode that has ever driven its CPU up: 5988% at
the full cpuset against the per-reactor timer's 4251%, for 2,336,243 against
2,179,118. That looked like waste at the time and was reverted on those grounds,
since it bought 7% more throughput for 41% more CPU. On 16 cores the reading is
different: a mode that can occupy the machine is worth more than a mode that
cannot, and whether it can is exactly what is being asked.
What separates them is where the wake-up comes from. The per-request timer hands
every deadline to its own submission, so each one wakes the reactor on its own
account. The per-reactor timer holds one deadline at a time, so a reactor with
work due behind the front of its queue stays asleep until the front fires. That
is the shape that would show up as a reactor unable to fill its core.
IOXIDE_DELAY_MODE=queue and =tick still select the other two.
* ioxide: wait on IORING_OP_TIMEOUT, with the library build vendored to measure it
Sixteen cores separated the two existing modes cleanly, and neither is good
enough:
tokio 1,769,456 3236% of 3200% 18.3us/req
ioxide queue 1,335,794 2678% of 3200% 20.0us/req
ioxide ring 1,211,556 3192% of 3200% 26.3us/req
Per request, the per-reactor timer is nearly level with tokio. It loses on
occupancy: it holds one deadline at a time, so wake-ups arrive in bursts and the
reactor sleeps between them with work already due, leaving 522% of the box
unused. The per-request timerfd has the opposite problem - every deadline wakes
the reactor on its own account, which fills the box, but a timerfd_settime per
wait means the CPU goes into syscalls rather than into serving, and it delivers
less while using more.
Neither can have both, and IORING_OP_TIMEOUT is the shape that can: the deadline
rides in the SQE, submitted with the batch the reactor was sending anyway. Every
wait wakes the reactor on its own account, and nothing is spent to arrange it.
ioxide has submitted IORING_OP_TIMEOUT for its own ticker since long before this,
but never exposed it; Reactor.SubmitTimeout does, in 68 lines that reuse the
existing slot allocation and completion routing. That is committed separately on
the library's own branch as MDA2AV/ioxide#212.
The package is vendored rather than published because this is a measurement, not
a release. localfeed/ holds one 140 KB build and NuGet.config adds it beside
nuget.org, so the runner restores it out of the Docker build context with no feed
access; every other package resolves normally. Both go away with the answer -
either the API ships and this becomes an ordinary version bump, or the mode loses
and all of it comes out.
Verified on the default path: /delay/10 answers in 10.11ms and /delay/200 in
200.20ms, 100 sequential waits on one connection are none early with a mean
overshoot of 0.28ms, and 32 concurrent distinct delays complete in 69.3ms against
1088ms serialised. IOXIDE_DELAY_MODE=ring, =queue and =tick still select the
other three.
* async: back to the profile's own 32 cores, with the ring timeout in place
Sixteen cores were a diagnostic and they answered it. Every mode, CPU per
request and how much of the 3200% each could actually occupy:
tokio 1,751,948 3228% 18.4us
ioxide native 1,422,729 3205% 22.5us
ioxide queue 1,335,794 2678% 20.0us
ioxide ring 1,211,556 3192% 26.3us
The per-reactor timer is the cheapest of the three and still loses, because
cheap per request is worth nothing when 522% of the box goes unused. The
per-request timerfd occupies the machine and loses harder, because what it puts
the CPU to is timerfd_settime. IORING_OP_TIMEOUT is the only one that occupies
the machine and spends it on serving, and it wins ioxide's side by 6.5%.
Restoring the cpuset because the constrained box has now said what it had to
say, and because the mode that was chosen on it has more room to show on the
one the profile actually uses. At 32 cores the per-reactor timer was never CPU
limited - 4251% of 6400% - so its throughput was bounded by the idle rather
than by the hardware, and that is the bound this lifts. At 22.5us a request,
6400% is 2.84M against tokio's 2,366,578, under a ceiling of 3.20M.
That is arithmetic, not a result. The per-request cost may not hold once there
are twice the reactors to keep fed.
* ioxide: move onto the released 0.7.211 and drop the vendored package
All seven references go to 0.7.211, the 140 KB build in localfeed/ and the
NuGet.config that pointed at it are deleted, and the Dockerfile restores the way
it always did. That scaffolding existed to measure IORING_OP_TIMEOUT before the
API shipped, the measurement is done and the API has shipped, so it goes.
ReactorDelay is gone with it. It had grown four mechanisms - a userspace tick
posting drains across threads, a per-reactor timerfd with a priority queue, a
per-request timerfd, and the ring timeout - which existed to find out which one
belonged in the entry. The bench hardware answered that, so the other three are
history rather than code, and what is left is 60 lines against one submission.
What replaces it is the RingTimer from MDA2AV/ioxide#213, kept local for now:
ioxide.timer missed the 0.7.211 release even though the SubmitTimeout it needs
did ship, so the class lives here and becomes a package reference when it is
published. It is the same code either way.
The wait is per connection and built on first use, so a connection that never
waits never makes one - which is every connection on every profile but this one.
One op in flight is all a connection needs, because the handler awaits before it
reads again.
Verified against the released packages: /delay/10 answers in 10.02ms and
/delay/200 in 200.09ms, 120 sequential waits on one connection are none early
with a mean overshoot of 0.05ms, 32 concurrent distinct delays complete in
69.4ms against 1088ms serialised, and baseline is 3.75M.
* ioxide: take the wait from ioxide.timer instead of a copy of it
ioxide.timer 0.7.211 is on nuget now, so the RingTimer that was carried here
while the package was missing from the release becomes a package reference and
the file goes. Same code, one fewer thing to keep in step.
ReactorDelay.cs is deleted rather than shrunk. It was 552 lines at its largest,
holding four mechanisms - a userspace tick posting drains across threads, a
per-reactor timerfd with a priority queue, a per-request timerfd, and the ring
timeout - which were there to find out which belonged in the entry. The bench
hardware answered that, so the answer is a package and the other three are
history. The entry now says what it wants and nothing about how:
await (httpSession.Timer ??= new RingTimer(reactor)).DelayAsync(ms);
One timer per connection, built on first use, so a connection that never waits
never makes one - which is every connection on every profile except this one.
Verified against the published packages: /delay/10 answers in 10.17ms and
/delay/200 in 200.28ms, 120 sequential waits on one connection are none early
with a mean overshoot of 0.11ms, 32 concurrent distinct delays complete in
68.9ms against 1088ms serialised, /json/1 and a 1 MB upload are unaffected, and
baseline is 3.85M.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #212.
The reactor has been submitting
IORING_OP_TIMEOUTfor its own 250ms ticker since long before this, so the kernel-side timer was already here. There was just no way for a caller to ask for one, and without it anything needing a deadline has to build it out of a timerfd: atimerfd_createper waiter, atimerfd_settimeper wait, and a read to consume the expiration.Two pieces, split the way
ioxide.filealready splits: the verb in core, the client in its own package.ioxide—Reactor.SubmitTimeout(nanoseconds, completion)Has to be in core: it is an
IRingHostmethod and its implementation lives inside the reactor's op slots, its SQE submission and its teardown.It goes through the same path as every other client op, so it inherits the slot allocation, the off-reactor marshalling and the O(1) completion routing, and needs no new dispatch case. The duration rides in the offset argument rather than widening a signature. The timespec sits in a block parallel to the op slots — the kernel reads it while the op is in flight, so tying it to the slot makes its lifetime exactly the operation's, with nothing allocated per wait. Freed in
Teardownalongside the ticker's.ioxide.timer—RingTimerioxide.filewrapsSubmitReadwith aRingOpSourceand ships as its own package to do it;RingFileis 59 lines againstRingTimer's 60. Same shape, same single dependency on core, so the same packaging.One
RingOpSource, one op in flight per instance, nothing allocated per wait, completion inline on the reactor that owns the caller. It followsRingSocket's conventions rather than inventing new ones, including returning results as the ring reports them rather than translating: a wait that runs its course completes with-ETIME, which is io_uring reporting expiry and is the success case here.Expired()exists so callers need not know that to read their own result.Tests
Four, in
Ioxide.Tests.E2E/Core/TimerTests.cs, on a handler that answers after the milliseconds named in the path while holding one timer per connection — the shape a caller is meant to copy:158 passed, 0 failed (the 7 pending are pre-existing and unrelated).
Why it came up
An HTTP benchmark that answers after a delay named in the request, at 32,000 connections — 32,000 live deadlines. Three mechanisms measured against each other on 16 cores, and the two that avoid this API both lose:
SubmitTimeoutThe per-reactor timer is the cheapest per request and still loses, because holding one deadline at a time means wake-ups arrive in bursts and the reactor sleeps with work already due — 522% of the box unused. The per-request timerfd occupies the machine and loses harder, because what it puts the CPU to is
timerfd_settime. Only the ring-native op does both: every wait wakes the reactor on its own account, and nothing is spent to arrange it.On the full 32-core cpuset that difference is worth 2,496,818 rps against 2,179,118 for the same entry.