diff --git a/docs/10-fps-limit.md b/docs/10-fps-limit.md index 703f65d..25701a6 100644 --- a/docs/10-fps-limit.md +++ b/docs/10-fps-limit.md @@ -104,139 +104,172 @@ divisor cap reads its target exactly. took the slot; a subscriber whose peer is gone is dropped there, and the connection currently being served stays its own loop's to close. This is what makes a cap change made mid-session reach an already-connected shim at all. -- **Who enforces it follows the present mode, and the mode follows the container.** - On gamescope's override route the shim declares to gamescope, per commit - (`gamescope_swapchain.set_present_mode`), the container's - `MESA_VK_WSI_PRESENT_MODE` (mailbox is the container default; the user may set - fifo, relaxed or immediate), or the guest's own mode when the env is absent - (`icd_shim/src/proxy.rs` `effective_present_mode`, `live_present_mode`). - gamescope's limiter holds only commits declared FIFO (`wlserver_surface_is_fifo`): - it latches one per limited vblank (`steamcompmgr.cpp` `handle_done_commits_xwayland`) - and the client is paced by `wl_buffer.release`. Any other mode is taken at the - next composite, so there the shim's acquire gate is the cap: - - | container mode | cap | declared | what limits the rate | - |---|---|---|---| - | mailbox (default) | none, guest asked for FIFO (in-game VSync on) | fifo (`source=guest`) | the display (FIFO) | - | mailbox (default) | none, guest asked for mailbox or immediate | mailbox | nobody | - | mailbox | at the display's rate, guest asked for FIFO (in-game VSync on) | mailbox (`source=cap-rate-vsync`) | the shim's acquire gate, every vblank; the present wait is answered when the guest queues the present | - | mailbox | at the display's rate, guest asked for mailbox or immediate | fifo (`source=cap-rate`) | gamescope's limiter (divisor 1), one FIFO commit per vblank; no gate, as on master | - | mailbox | below the display's rate | mailbox | the shim's acquire gate, on the display's vblank grid | - | fifo | none | fifo | the display (FIFO) | - | fifo | set | fifo | gamescope's divisor limiter | - | any, frame generation on | any | mailbox | the source-rate gate (`docs/17`) | - - FIFO-relaxed (`relaxed`) and immediate behave as mailbox in this table. No env: the guest's - own mode takes the env's place. "At the display's rate" is a cap within 0.5 % - of the measured rate (60 on a 59.97 Hz panel); a cap above it runs on the - gate's own clock under the container's mode. - - **With no cap a guest's FIFO request wins over the container's default.** Every - container carries `MESA_VK_WSI_PRESENT_MODE=mailbox` as a stamped default the user - did not choose. With no cap, honouring it turned a game's own VSync (a FIFO - request) into an uncapped mailbox session, which is neither master's behaviour nor - what the user asked for. A FIFO request (FIFO-relaxed too) is a specific wish and - is declared as asked (`source=guest`); a mailbox or immediate request means "do not - make me wait", and the container's mailbox is the substitute, as on the X11 route - (an uncapped immediate guest gets mailbox, not tearing). An env of `fifo` is an - explicit wish and wins in every case. Under a cap the table above applies whatever - the guest asked for. - - **The user's control is `MESA_VK_WSI_PRESENT_MODE`.** It sits in the container's - environment, written from the driver's present-mode setting, and takes Mesa's - names: `fifo`, `relaxed` (FIFO-relaxed), `mailbox`, `immediate`. Anything else, - `fifo_relaxed` included, is rejected as Mesa rejects it, the guest's own mode is - used, and the shim logs `Invalid MESA_VK_WSI_PRESENT_MODE value!` whether or not - `GNDRM_LOG_VKSHIM` is set, so a typo shows in a shipped session. It is - overridden in one case: a non-FIFO value under a cap at the display's rate is - declared FIFO (`source=cap-rate`, below). Mesa reads the same variable, so it - also sets the mode a zink guest's own swapchain asks for below the shim. - - **Why mailbox is not declared at the display's rate.** At that cap the panel - shows one frame per vblank under either mode, and under mailbox the shim has to - keep two commits out of one gamescope composite. It cannot see when gamescope - latches: gamescope takes commits at its wakeup, the target vblank minus its - rolling draw time and red zone (`vblankmanager.cpp` 142-199), and that draw time - is a peak hold that jumps on any slow draw (164-165) and decays per wakeup - (167); draw time runs to the page flip queued (`DRMBackend.cpp` 4465), on this - stack through the daemon's commit. The client sees only `presented` (stamped - with the wakeup's target vblank, `steamcompmgr.cpp` 8140, 8168) and `discarded`, - one inequality per frame. Measured on a Retroid Pocket 6 (DTD, cap 60, in-game - VSync off, a guest pacing itself at ~59.3 fps on a 59.98 Hz panel): mailbox with - the gate gave 58.3 fps on `SurfaceFlinger` with ~1 commit a second replaced in - gamescope's mailbox; a one-commit-per-window queue in the shim at a guessed - 2.5 ms boundary gave 59.0 with ~0.8 a second still replaced; FIFO on master - gives 59.22 on the same scene. Under FIFO gamescope latches one commit per - vblank at its own wakeup (`steamcompmgr.cpp` 8028-8049) and its limiter's - divisor at this cap is 1, which is master's behaviour in this cell. That holds - for a guest that asked for mailbox or immediate, which paces itself. A guest - that asked for FIFO (in-game VSync on) waits on each frame's presentation - before starting the next; declared FIFO, that wait is gamescope's `presented` - at its latch, so the whole frame has to fit between two latches, and with - ~16 ms of work and three images it fell to 40 fps at 60 Hz (a 1280x960 - device, DTD in game: 400 commits per 10 s, 400/600 publishes). That guest is - declared the container's mode with the gate every vblank instead - (`source=cap-rate-vsync`): its wait is answered when it queues the present - (`docs/15`), the gate paces it, - and the count stays the guest's own ask — the declared mode moves pacing, - not images (`proxy::swapchain_slots`; Retroid Pocket 6, DTD, - VSync on, cap 60: 59.983 fps, no doubles; with the wait answered at queue, - 42 fps before and 60.0 after in a GPU-heavy 1080p scene). This is the only - case answered at queue; every other mailbox or immediate presentation is - answered at the commit, as on master. A container set to - `MESA_VK_WSI_PRESENT_MODE=fifo` never reaches it, so setting a game's - container to fifo restores master's behaviour for that game. The - vblank-grid gate is not kept on top of it: FIFO's back-pressure already holds - the guest, and the two stacked turned a guest whose acquire-to-commit grew - past a period into a 40 fps session on a 60 Hz panel (DTD on a 1280x960 - device after a live 120 to 60 Hz switch: `gate_holds` 0, `gate_late` 400 and - `skipped` 200 per 10 s, commit latency 24 ms, 400/600 publishes). Below the - display's rate the gate's releases - are at least two vblanks apart and two frames in one composite are rare, so - mailbox stays (FIFO with gamescope's limiter quantized a 32 fps game to 30 at a - 60 cap on 120 Hz; mailbox with the gate did not). - - The gate (`icd_shim/src/vsync_gate.rs`) releases acquires on the vblank grid - gamescope composites from, not on a clock of its own: the daemon's `VsyncNote` - latch grid (measured period and phase) moved earlier by the lead the daemon - stamps flip events with (`stamp_lead_ns`). A cap at the display's rate releases - every vblank (60 on a 59.97 Hz panel is 59.97, one frame per latch); a cap that - divides the rate, every `refresh / cap`-th vblank (30 on 60, 60 on 120); any - other cap below the rate, the vblank nearest each instant of an ideal - `1e9 / cap` timeline, which averages the cap on vblank boundaries (45 on 60 - releases 3 of 4 vblanks); a cap above the rate, the cap's own clock. A frame - that arrives after its release point starts at once, and the release after it - is never more than one grid instant after it, so a game below the cap keeps its - own rate. A - gate on the shim's own `1e9 / N` clock, as before 7bdcbae, beat against the - panel: 57-59 fps on `SurfaceFlinger` at a 60 cap on a 59.97 Hz panel. - - The gate needs the stamp lead. Until a note carrying one has arrived (no note - yet, or a daemon that predates the field and sends 0), a capped swapchain whose - mode is not FIFO is declared FIFO and gamescope's limiter is the cap, as before - the container's mode was honoured here; the census reads `source=cap-compat`. - Released on the latch points themselves, frames start ~2 ms after gamescope's - wakeup and straddle its composite (Retroid Pocket 6, cap 60: 58.3 fps and 102 - frames replaced in gamescope's mailbox per minute at 60 Hz, 13 % of publishes - late at 120 Hz). A rollout that ships only the payload (the shim) therefore keeps - today's pacing until the app ships the daemon that sends the lead. A swapchain - created before the lead is known keeps the slot count of the guest's own request - until it is recreated. +- **The request is never rewritten; a cap is a rate.** On gamescope's override + route the shim declares, per commit (`gamescope_swapchain.set_present_mode`), + the mode the game requested (`icd_shim/src/proxy.rs` `guest_present_mode`: its + `VkSwapchainCreateInfoKHR.presentMode`, FIFO when it names an unknown one). + Nothing above it changes that: no environment variable, and neither the cap nor + the frame-generation multiplier. What those decide is a rate — + `icd_shim/src/limits.rs` turns the guest's present mode, the cap, the + multiplier and the panel's refresh into the frames per second the guest may + acquire and the frames per second the generator may add: + + | present | cap | LSFG | acquire | generated | total | hold | show in HUD | + |---|---|---|---|---|---|---|---| + | synced | on | off | cap | — | `cap` | note | `cap` | + | synced | off | off | refresh | — | `refresh` | note | `refresh` | + | synced | on | on | cap | `cap × (m−1)`, at most `refresh − cap` | `cap + min(cap × (m−1), refresh − cap)` | note | total | + | synced | off | on | `refresh / m` | `acquire × (m−1)` | `acquire × m` | note | total | + | unsynced | on | off | cap | — | `cap` | note | `cap` | + | unsynced | off | off | none | — | none | note | measured | + | unsynced | on | on | cap | `cap × (m−1)` | `cap × m` | note | total | + | unsynced | off | on | `refresh / m` | `acquire × (m−1)` | `acquire × m` | note | total | + + **Synced, the source frames are part of the panel's budget.** The generated + ceiling on that row is `refresh − acquire`, not `refresh`: a 60 Hz panel can show + 60 frames whichever way they were made, so a cap 40 session leaves 20 for the + generator and no more. The consequence is deliberate and it is not a rounding + detail — at cap 40, ×2, that is 20 generated against 40 source frames, half a + frame each, which floors to none, so frame generation does nothing at any cap + above half the refresh. Lower the cap or the multiplier to make room. + + `hold` is when an acquired image may be drawn into again. It follows the + **served** mode — the same one `on_buffer_release` keys on — and on this route + every declaration is served as FIFO or MAILBOX: FIFO for a guest that asked for + it, MAILBOX for a generation session and for a declared `IMMEDIATE` or + `FIFO_RELAXED` (`choose_live_mode`). Both are synced, so the hold is `note` on every row: the + image comes back only when the compositor reports its release *and* the display + reports its scan-out release after that — measured at 1.5-2.5 display periods + (`sf hold`). [24-release-notes.md](24-release-notes.md) describes that channel: + the note names the commit's ticket, so the pairing is exact where it once needed + an inode, an order window and a bound, and a park whose note never arrives is + released on its fence after four tries (`channel=[unanswered:… dead:…]`). + + `FIFO_RELAXED` is the second substitution, and a different bargain: served as + itself it costs a whole vblank per frame that lands after the compositor's latch + point, because the next frame cannot be taken until the following vblank. Doom + 2016 measured 28.8 fps against a 60 cap with the GPU at 41 % and the CPU at 26 % + — `2T` per frame, nothing busy. Served as mailbox the newest frame supersedes + the pending one, so a missed latch is dropped rather than deferred. That is what + `MAILBOX` promises and not what `FIFO_RELAXED` promises (it permits tearing, not + loss), which is why the substitution is a judgement: at a render time below the + panel period nothing is superseded and the two are indistinguishable. + + That substitution is what makes a single `note` column possible. Serving a + declared `IMMEDIATE` as itself would return the image the moment it was handed + over, and the client would draw into a buffer the display was still scanning — + the tear Portal 2 showed under direct scan-out. Serving it as `MAILBOX` keeps + the guest's intent (no wait on the client, and a frame that cannot be shown is + dropped for it) while giving the compositor a buffer it can latch at a vblank + instead of one it might replace mid-scan. + + What the hold costs is the rate whenever the pool is shallow: + `min(cap, depth × refresh / hold)`. At 3 images against a 2.5-period hold that + is ~54 on a 60 Hz panel, which is why the image count follows the served mode + as well (`count_present_mode`). + + Below the fold, `depth` has to exceed the hold for the rate column to be the + binding one. At 3 images against a 3.4-period hold the pool caps the session + instead, and the cap moves in whole periods as the hold crosses + `depth / refresh` — the 100<->120 oscillation measured on an uncapped 120 Hz + session, where the hold sat either side of 25 ms. + + `total` is what the rule lets the session produce: acquired plus generated. It + is not always what reaches the screen — the synced row caps generation at what + the panel has left after the source frames, so cap 40 at ×4 on a 60 Hz panel is + 40 + 20, and at ×2 it is 40 + 20 as well. The `show in HUD` + column is what the widget *should* read, `total`, because that is the number the + user asked the multiplier for; the cap in force is a separate figure and governs + acquires only. What it reads today is the bullet below. + + "Synced" is FIFO and FIFO-relaxed, where the panel is the ceiling in both + directions; "unsynced" is immediate and mailbox, where the guest asked to run + past the panel and the compositor drops what it cannot show. The generated + ceiling is a rate and not a mode: 40 acquired ×2 on a 60 Hz panel is 80 + presented frames, of which the compositor shows 60. An unreported panel imposes + no limit rather than a wrong one, and a multiplier of 1 is not generation. + + The shim's acquire gate holds the rate for every mode the guest declares. A + FIFO declaration is not held by gamescope's limiter: that limiter withholds the + window's frame callback, which the shim never requests, and its divisor only + acts when the cap is below the refresh and divides it — so `60` on a 60 Hz + panel and `40` on 60 Hz had no cap at all. gamescope's divisor remains the + pacer for frames the shim does not present. With generation on, the gate holds + the source rate on its own clock and the generator places the batches on the + latch grid (`docs/17`). + + **What the code does today, which is narrower than the table.** The table is + the rule `limits.rs` encodes; the sessions that run are a subset of it: + + - The app disarms the cap when frame generation is armed (`XServerScreen.kt`: + `setFpsLimit(if (lsfg) 0 else limit)`), so a gamescope session at cap 60 with + ×2 reaches the shim as `cap=0` and takes the *no cap* row: `refresh / m` + acquires. The `cap + cap × (m−1)` rows are unreachable from the UI today. + - Generation runs through `flush_due` and reports in `GEN_COMMITS`: measured on + Witcher 3 at cap 30 with ×2, `generated 3103` over a session against + `acquire=30 generate=30`. The guest-side layer (`VkrLsfg`) is also loaded and + presents its own frames on the same swapchain, where they are neither an + acquire nor a `GEN_COMMITS` entry — its counters read `guest=28.0 needs=55.9Hz` + while the shim's own generator is doing the work, so the layer's copy is not + what reaches the panel. + - `note_produced` counts `Δacquires + ΔGEN_COMMITS`, and since a generated frame + reports itself at its own commit (`commit_generated`) the ring now receives both + streams, one message each, so it holds `total`. The widget strides the ring by + the multiplier (`frameSampleStride = m`, `XServerScreen.kt`), which divides that + stream back down: it reads the source rate where the column above says `total`. + The stride is the app's and is the remaining gap between the table and the + HUD. + + `generate_per_s` is consumed as a bound on how many frames one source frame may + yield, not as a fourth limiter: `Limits::generate_per_source(m)` caps the batch + the generator plans (`proxy.rs`, `lsfg_generate`), and the batch's frames are + placed on the latch grid (`docs/17`). The rate in the table is therefore the + generator's ceiling; what actually lands is that ceiling, floored to whole frames + per source. + + The gate (`icd_shim/src/vsync_gate.rs`) is one regulator and one rule: a frame + may start `1e9 / limit` after the previous frame started, and not before. + + ``` + gate: while now < next_ns { wait until next_ns } + book: next_ns += period (from the release point, not from now) + late by >= one period: next_ns = now + period, and the skip is counted + ``` + + So the limit is a maximum by construction — two releases are never closer than + one period. A frame that overshoots its release point by less than a period does + not move the grid: the following frame gets a proportionally shorter hold, which + is what keeps the average *at* the limit instead of below it (`next_ns = now + + period` lost 7 % to accumulated lateness). A stall of a whole period or more is a + lost frame, not a debt: the schedule re-anchors at `now + period` and counts the + skip, so nothing repays itself with back-to-back releases. A game slower than the + limit is never held, because its next acquire is already past `next_ns`. + + Releases are *not* snapped to the display's vblank grid. A grid-snapped release + holds the phase but bounds only the average rate: a release a slot wait delayed + lands between vblanks, and the next booking on the following vblank could be a + fraction of a period later. Measured on this path at a 60 limit: 74.7 frames/s + with the gate holding 8 % of acquires. The daemon's latch grid is still used + where a phase is what is wanted — frame generation places its batches on it + (`docs/17`) — not to limit a rate. + + `gate=clock` is every gated session now; `gate=off` means no limit. The census says which path ran. Ungated, once when frames first flow and again - whenever the declared mode, its source, the cap or the gate changes: - `present mode: declared=… requested=… source=env|guest|cap-rate|cap-compat - route=gamescope cap=… gate=vsync|clock|off lsfg=…` (`source=guest`: the guest's - own mode, no env or a FIFO request with no cap; `source=cap-rate`: declared FIFO - because the cap is the display's rate and the guest paces itself, with - `gate=off`; `source=cap-rate-vsync`: the container's mode with `gate=vsync` - because the cap is the display's rate and the guest asked for FIFO; `source=cap-compat`: - declared FIFO for want of the stamp lead, with `gate=off`; `gate=clock` is frame - generation's source rate or a cap above the display's rate). With + whenever the declaration, its source, the cap, the limits or the gate changes: + `present mode: declared=… source=guest|generating route=gamescope cap=… gate=… + lsfg=… acquire=… generate=…` (`source=generating`: a generated frame, declared + mailbox; `acquire` and `generate` are the limits in frames per second, + `acquire=0` for none and `generate=0` for generation off; `gate=clock` is + every session with a cap, whatever the display's rate — the gate books its own + grid at `1e9 / acquire_per_s` and does not defer to the compositor below a + refresh divisor). With `GNDRM_LOG_VKSHIM=1` the same line repeats every 10 s with the counters a device run checks the pacing against: `acquires= commits= declared_fifo= - declared_other= releases= feedback= presented= discarded= fallback= - gate_holds= gate_late= skipped= slot_waits= commit_lat_us= spacing=[…]`. They + declared_other= releases= feedback= presented= discarded= fallback= at_queue= + gate_holds= gate_late= skipped= slot_waits= commit_lat_us= spacing=[…]`, and on + the release path `slots=[…] dests=[in_flight/total parked=n] gendrel=`, which is + the generation ring's own state. They are cumulative except the latency and the spacing: `gate_holds`, acquires the gate held; `gate_late`, acquires that started after their release point (not the daemon's `late=` on its `vsync: check` line, which counts publishes that @@ -256,8 +289,9 @@ Open points: - **Divisor rounding bites the FIFO route only.** `nVblankDivisor = nRefreshHz / nTargetFPS` (`steamcompmgr.cpp:7040`, and `:10621` for the window's limited cycle) lands on the nearest divisor of the grid, so with the container - set to fifo a 40 or 45 cap on 60 Hz runs at 60. Under mailbox the shim's gate - averages a non-divisor cap on vblanks instead. + set to fifo a 40 or 45 cap on 60 Hz runs at 60 — for frames gamescope's limiter + holds. The shim's own gate is unaffected: it releases on its own grid, so a + non-divisor cap is held exactly whatever mode is served. - **The client-visible refresh is the mode's, and the mode is the display's.** `-r` reaches `g_nOutputRefresh` only transiently (`drm_unset_mode`, `:3680`), before `drm_set_mode` (`:3728`) overwrites it with the mode's rate — @@ -276,8 +310,15 @@ Open points: frames than the panel shows — which is the point, and why the publish count is kept only as the session's own statistic. - **Engagement feedback**: `gamescope_limiter.state` covers the gamescope route; - the shim logs `scanout frame cap: fps=… effective=…` (`scanout.rs:148-158`), + the shim logs `scanout frame cap: fps=… effective=…` (`scanout.rs:132`), `effective` being the target in force once the note is adopted, and reports what - the cap delivered in its own census (`cap=[fps= notes= gate_holds= acq_fps=]`, - `GNDRM_LOG_VKSHIM=1`): `acq_fps` is the rate at the instant the cap acts, so it - — not the app's display-side frame count — is what a target is checked against. + the cap delivered in its own census (`cap=[fps= notes= acq_fps= books= + gate_holds= gate_us= gate_avg_us= gate_max_us=]`): `acq_fps` is grants per + second over the + whole window — grants divided by the clock time between the first and the last — + so it, not the app's display-side frame count, is what a target is checked + against. `books=` counts schedule bookings against the grants: equal to the + grant count means every granted frame booked the grid, and short of it means + some grants skipped the book. It read as the reciprocal of the mean acquire + duration until that was corrected; a rate has to divide frames by elapsed time, + not by the sum of their own waits. diff --git a/docs/12-present-latency-budget.md b/docs/12-present-latency-budget.md index de2335b..960db79 100644 --- a/docs/12-present-latency-budget.md +++ b/docs/12-present-latency-budget.md @@ -21,6 +21,55 @@ emitted with the publish, so a paced frame's present does not wait it out. Buffer reuse still rides it — the next publish onto a slot waits on the previous release fence the completion stored. +## SurfaceFlinger's hold on a buffer (which release rule is safe) + +The page-flip event says the frame was *published*, not that SurfaceFlinger has +finished with it. A producer handed its buffer at that event may still be +writing into a buffer on the panel — a scan-out tear, and one no screen +recording can show, because the buffer is repainted before anything reads it +again. Deferring the release by one flip removes the tear by removing a buffer +from every producer: a 3-image swapchain keeps one in flight instead of two, so +a producer can no longer run ahead of the flip and a late frame has no cushion. + +SurfaceFlinger's own release fence is the exact answer, and the daemon already +receives it. `ASurfaceTransactionStats_getPreviousReleaseFenceFd` reports, in the +completion of a transaction, the fence of the buffer that transaction +**replaced** — not the one it published: + +``` +publish X, which replaces W ──▶ completion reports W's fence + ▼ + fence signals: W is free +``` + +So the fence is filed on W's mailbox (`ReleaseCtx::replaced_ctx`) rather than on +the publisher's — that attribution is what makes it usable at all — and it is +waited on off the completion thread. Two things use it: + +- the publish that reuses W waits on it before applying, so the pool never + republishes a buffer the panel is still reading; +- a watcher thread waits on it and pushes `BufferReleased` (`CAP_RELEASE_NOTE`) + carrying W's scan-out inode. A shim that declares the cap keeps that slot busy + until the note arrives: the compositor's `wl_buffer.release` only parks it. + +A producer the shim does not render for — the guest's own GL blits through +Xwayland — reads no notes, and for that class the deferral above is the only +lever. + +`sf hold:` measures the hold where it ends: the watcher samples the gap from the +released buffer's latch to the instant its fence signals. + +``` +gndrm-daemon: sf hold: n= half-one= one-1.5= 1.5-2.5= over-2.5= max=us period=us timeouts= +``` + +Buckets are halves of one display period, and the period is the latch grid's own +(median of the last nine accepted gaps; a gap outside 3–50 ms is a burst or a +stall, not a period). `one-1.5` and above is the share a one-flip deferral does +**not** cover. `timeouts` counts fences that reached `RELEASE_WATCH_MAX` (500 ms) +without signalling — the release is reported anyway, so a stuck compositor +delays a producer instead of hanging it. One line per 10 s. + ## Why the event is emitted with the publish `commit_worker_loop` emits the page-flip event immediately after the publish @@ -44,9 +93,9 @@ into every paced frame. Buffer-reuse safety is unchanged: the scan-out pool still caches a buffer for reuse only once the presenter reports SurfaceFlinger's completion (the slot's -`latched` flag), and the next publish still blocks on the previous release -fence. Only the *event* — a pacing/visibility signal to the client — stops -carrying the round trip. +`latched` flag), and the publish that reuses a buffer still waits on that +buffer's own release fence. Only the *event* — a pacing/visibility signal to the +client — stops carrying the round trip. ## Measured effect @@ -61,11 +110,18 @@ The paced path meets its target, and the uncapped rate is bounded upstream — the client's own present lap through the ICD shim to gamescope — not by the flip-event path. -### The FIFO declaration is not the throughput cost +### The FIFO declaration is not the throughput cost, and not a limiter The shim's per-commit `swapchain_set_present_mode(FIFO)` declaration does not -bound the uncapped rate — removing both sites left it unchanged. It stays in the -tree: the request/divisor cap needs it. +bound the uncapped rate — removing both sites left it unchanged. + +It is not a limiter either, and the gate no longer stands down for it. +gamescope's `steamcompmgr_should_vblank_window` withholds the window's frame +callback, and this shim never requests one; the divisor also only acts when the +cap is below the refresh AND divides it. A FIFO guest at 60 on a 60 Hz panel, or +at 40 on 60 Hz, therefore had no cap at all — measured as a 26 fps Dishonored +session whose Target FPS did nothing. Every declaration is gated now; gamescope's +divisor is left as the pacer for content the shim does not present. ## The X11 arm as a control — pacing is deadline/timer-based, not deferral @@ -96,6 +152,8 @@ number into the arm asymmetry from this. ## What is NOT proven / open +- `sf hold:` has no device capture yet: the buckets say which release rule is + sufficient, and nothing has read them on hardware. - The uncapped ceiling (~259 /s on this scene) is **not** the flip-event path; it is the client↔gamescope lap (`vkAcquireNextImageKHR` in the ICD shim blocking on the compositor's `wl_buffer.release`). Not measured end to end; diff --git a/docs/15-fps-limiter-pacing-and-vsync.md b/docs/15-fps-limiter-pacing-and-vsync.md index 062f441..f9004ee 100644 --- a/docs/15-fps-limiter-pacing-and-vsync.md +++ b/docs/15-fps-limiter-pacing-and-vsync.md @@ -11,25 +11,22 @@ cursor, scan-out and mode patches. | place | what it holds | set by | |---|---|---| -| the shim's declared present mode | the container's `MESA_VK_WSI_PRESENT_MODE` (else the guest's mode); with no cap, a guest's FIFO request over a non-FIFO env; FIFO under a cap at the display's rate; mailbox under frame generation | the container, the guest and the cap (`docs/10`) | -| gamescope's own limiter | `nRefreshHz / nTargetFPS`, a divisor of the mode's rate; holds FIFO commits only | `debug_set_fps_limit`, sent only when generation is off | -| the shim's acquire gate | the cap, on the display's vblank grid, unless the container's mode is FIFO | `FrameCapNote{fps}` and `VsyncNote` from the daemon | +| the shim's declared present mode | the mode the guest asked for, except `IMMEDIATE` which is served as `MAILBOX`; FIFO under a cap at the display's rate when the guest asked for mailbox; mailbox under frame generation | the guest, the served mode and the cap (`docs/10`) | +| gamescope's own limiter | `nRefreshHz / nTargetFPS`, a divisor of the mode's rate; acts only when the cap is below the refresh and divides it, and only for clients that wait on a frame callback | `debug_set_fps_limit`, sent only when generation is off | +| the shim's acquire gate | the cap, one release per `1e9 / cap`, for every declared mode | `FrameCapNote{fps}` | | the presenter's latch grid | the display's latch points, as measured | `VsyncNote` from the daemon (`docs/17`) | -Which one holds the cap follows the declared mode. Declared FIFO (the container set -fifo, or no env and a FIFO guest), gamescope's limiter passes every -`nRefreshHz / target`-th vblank (`nVblankDivisor = nRefreshHz / min(target, nRefreshHz)`) -and holds the guest's commits to it. Declared mailbox, immediate or FIFO-relaxed (mailbox -is the container default), the limiter does not hold them and the shim's acquire gate -releases the guest's frames on the display's vblank grid. A container in one of those -modes under a cap at the display's rate is declared FIFO with no gate when the guest -asked for mailbox or immediate (`source=cap-rate`), as on master, and keeps its mode with -the gate every vblank when the guest asked for FIFO (`source=cap-rate-vsync`, `docs/10`). -For the former: gamescope latches one commit per vblank at its own -wakeup, which the shim cannot see under mailbox, and its limiter (divisor 1) paces the -guest (`docs/10`). Under frame generation the app -disarms gamescope's limiter and the shim's time gate holds the source rate instead -(`docs/17`). +The shim's gate holds the acquire limit whatever the declared mode, and it is the +only regulator: one release per `1e9 / cap`, booked from the release that just +happened. gamescope's limiter is not a fallback for it — it passes every +`nRefreshHz / limit`-th vblank (`nVblankDivisor = nRefreshHz / min(limit, +nRefreshHz)`) to a client that waits on its frame callback, which the shim never +does, and the truncating divisor means `40` on 60 Hz passes every one of them. It +stays as the pacer for content the shim does not present. What the limit *is* comes from one place — the guest's present mode, the cap, +the multiplier and the panel's refresh (`docs/10`, `icd_shim/src/limits.rs`) — and +the declaration is the guest's own in every case, so the cap never moves it. Under +frame generation the app disarms gamescope's limiter and the shim's gate holds the +source rate instead (`docs/17`). ## 2. The mode is the display's rate @@ -50,8 +47,8 @@ that rate is the display's. - **FIFO.** A present is requeued until the display releases a buffer, so the guest is paced to the panel whether or not it asked to be. On gamescope's override route the - mode declared per commit is the container's `MESA_VK_WSI_PRESENT_MODE` (the guest's - own without it), cap or no cap, and mailbox under frame generation (`docs/10`). On + mode declared per commit is the one the guest asked for, cap or no cap, and mailbox + under frame generation (`docs/10`). On that route the shim answers `vkWaitForPresentKHR` from gamescope's presentation feedback, as Mesa's Wayland WSI does: every commit that carries a wait requests a `wp_presentation_feedback`, and the wait is answered on @@ -71,13 +68,13 @@ that rate is the display's. session takes (the `fallback=` counter of the periodic `present mode:` line counts it; that line repeats with `GNDRM_LOG_VKSHIM=1`, while the ungated one prints at session start and on a change of the declared mode, its source, the cap or the gate, `docs/10`). Without the global (or off the override route) the wait is answered on the release, or two refresh periods after the commit when no release report arrives first. -- **Mailbox and immediate.** A guest that asked for either does not wait, and neither +- **Mailbox, and immediate (served as mailbox).** A guest that asked for either does not wait, and neither would one whose frames go out as mailbox or immediate (frame generation, or the container's env while the guest believes it has FIFO), so those waits are not held for the compositor: they are answered at the commit, after the frame's render fence (`present_wait_answer`). One case is answered earlier, when the guest queues the - present: `source=cap-rate-vsync` (a cap at the display's rate, a container mode other - than FIFO, a guest that asked for FIFO, declared mailbox with the vblank-grid gate). + present: a FIFO-relaxed guest on gamescope's route whose rate the acquire gate holds + (one that waits on every frame). That guest waits on every frame, and answered after the fence a latency-1 guest alternated CPU and GPU: Retroid Pocket 6, DTD at 1080p, cap 60, VSync on, 60 Hz, 42 fps with 23 ms from acquire to commit, the GPU at 65-69 % busy and 550 MHz because its @@ -137,11 +134,9 @@ that rate is the display's. commit per limited vblank, on the display's vblank stream above, and hands the previous buffer back with `wl_buffer.release`, which is what frees the guest's next acquire; a cap that is not the display's rate over a whole number runs at gamescope's - next divisor rate above it. Declared FIFO because the cap is the display's rate - (`source=cap-rate`), no gate runs either: gamescope's FIFO latch alone paces it. - Declared anything else, - gamescope takes the newest commit at each composite and the shim's acquire gate is - the cap (`icd_shim/src/vsync_gate.rs`). + next divisor rate above it. Declared anything else, + gamescope takes the newest commit at each composite and the shim's acquire gate holds + the acquire limit (`icd_shim/src/vsync_gate.rs`). Its release points are the instants the flip events are stamped at: the daemon's latch grid (`VsyncNote` anchor and measured period) minus the stamp lead (`stamp_lead_ns`). gamescope wakes to composite just before each (wakeup = stamp + @@ -161,9 +156,9 @@ that rate is the display's. own clock (exactly 1e9 / N ns) and beat against the display's 59.97-59.99 Hz (57-59 fps at a 60 cap). A cap above the rate runs on the cap's own clock (`gate=clock`, also frame generation's source rate). Without the stamp lead (no - note yet, or a daemon that predates it) the gate does not run: a capped swapchain is - declared FIFO and gamescope's limiter holds it (`source=cap-compat`), so a - payload-only rollout keeps today's pacing until the app ships the daemon. + note yet, or a daemon that predates it) the gate cannot run on the grid and paces + on its own clock (`gate=clock`), which beats against the panel: a payload-only + rollout keeps that pacing until the app ships the daemon. - **Fallbacks.** On API 29-32 Choreographer reports the vsync time only (`AChoreographer_postFrameCallback64`): its measured period is used with the latch grid's phase (`vsync.rs`, from `ASurfaceTransactionStats_getLatchTime`), or the vsync @@ -177,15 +172,10 @@ that rate is the display's. ## Behaviour changes in gamescope sessions -- **The container's present mode is declared, except under a cap at the display's - rate.** `MESA_VK_WSI_PRESENT_MODE` decides the mode on the gamescope route as on the - X11 Present route; without it the guest's own mode is declared. A container left at - the mailbox default runs, with no cap, FIFO for a guest that asked for FIFO (its - own VSync, `source=guest`) and mailbox for any other guest, and mailbox capped by the shim's - vblank-grid gate under a cap below the display's rate. Under a cap at the display's - rate it is declared FIFO with no gate (`source=cap-rate`), because the shim - cannot keep two mailbox commits out of one gamescope composite (`docs/10`). One set - to fifo runs FIFO under gamescope's limiter. +- **The guest's present mode is what gets declared.** There is no environment + override above it, so a game's own VSync request is never turned into a mailbox + session — and no cap or multiplier changes it either: they set the acquire and generation limits (`docs/10`, + `icd_shim/src/limits.rs`). - **A game below the cap keeps its own rate under mailbox** (caps below the display's rate). Under FIFO and gamescope's limiter each frame is held to whole limited vblanks, so a game below the cap judders @@ -207,6 +197,13 @@ that rate is the display's. separately; between the two, for a moment, the swapchain's mode is declared with no cap, so a mailbox/immediate swapchain can run uncapped for a few frames. +- **`IMMEDIATE` is served as `MAILBOX`.** The one substitution the layer makes + (`choose_live_mode`): a frame that cannot be shown is dropped for the client + instead of being scanned out mid-flip, and the declaration follows the served + mode. Its hold is the served mode's too (`note`), so the buffer still waits out + its own scan-out — a mode override that changed only the present path would + leave the reuse tear open. + ## 4. What the daemon owns - The latch grid: `ASurfaceTransactionStats_getLatchTime` on every publish, the phase @@ -216,7 +213,8 @@ that rate is the display's. flip events' lead (below), so `anchor_ns - stamp_lead_ns` is gamescope's vblank grid, the one the shim's acquire gate releases on. The daemon sends at least 1 ns while the flip events follow a grid, so 0 means "not known" (no grid, or a daemon that - predates the field), and the shim then declares FIFO under a cap instead of gating. Its period is the one measured from Choreographer while that is locked + predates the field), and the shim then cannot gate on the grid and paces on its own + clock (`gate=clock`). Its period is the one measured from Choreographer while that is locked (the mode's advertised rate otherwise), so frame generation extrapolates the display's real rate. Its anchor is also the flip timeline's phase where Choreographer gives no deadline, and its fallback. @@ -244,9 +242,10 @@ measures — the latch grid to the presenter, the vsync timeline in the flip eve Retroid Pocket 6 (59.97-59.98 Hz), DTD title scene, cap 60, against master's FIFO declaration on the same scene (59.22 fps, 1.3 % doubles): -- **A cap at the display's rate, declared FIFO** (measured with the vblank-grid gate - kept, `source=cap-rate gate=vsync`, before the gate was dropped from this cell; - without it the cell is master's FIFO path): in-game VSync off, 59.199 fps on `SurfaceFlinger` +- **A limit at the display's rate** (measured while the shim still rewrote that one + cell's declaration to FIFO with the vblank-grid gate kept, `source=cap-rate + gate=vsync`; the rewrite is gone — the declaration is the guest's and the gate + holds the limit): in-game VSync off, 59.199 fps on `SurfaceFlinger` and 1.3 % doubles over 60 s, shim commits equal to the daemon's publishes within 0-2 per 10 s (the guest paces itself at ~59.3); VSync on, 59.983 fps and no doubles. diff --git a/docs/17-lsfg-frame-generation.md b/docs/17-lsfg-frame-generation.md index 714a8c7..eba0e17 100644 --- a/docs/17-lsfg-frame-generation.md +++ b/docs/17-lsfg-frame-generation.md @@ -79,22 +79,18 @@ two commits fall inside one period and the first is replaced before it is shown, and a rate above the panel's (60 × 3 on a 120 Hz panel) drops frames in a drifting pattern. Three rules keep one frame per period, in order: -- **Multiplier.** A batch generates `multiplier - 1` frames and the source frame - after them, one batch per acquired source frame, so the session produces - `cap × multiplier` frames whatever the display can latch: a 40 cap at x4 on a - 120 Hz panel produces 160 and the panel shows 120 of them. Generation is not - capped by the panel: the mode is the display's rate and never moves for a cap or - a multiplier, so capping the output by it would only make x4 at 40 read 120 and - say nothing about what the session was doing. The rate is the user's cap, or - the measured source interval without one (a source hovering at the threshold, - 64 fps on 120 Hz, flips between x2 and none from batch to batch), and the - display never reshapes it: a 40 cap runs 40 acquires a second on a 60 Hz panel - as much as on a 120 Hz one, and the multiplier decides what is generated on - top. The source rate is held by the acquire gate on the cap's own clock - (`scanout::next_acquire_after`, `gate=clock` in the census), not on the - display's vblank grid: the vblank-grid gate of `docs/10` covers a cap without - frame generation only, and generated frames are placed on the latch grid - below. The panel's rate is the one the app reported (`LsfgConfig.refresh_mhz`), +- **Multiplier.** A batch generates up to `multiplier - 1` frames and the source + frame after them, one batch per acquired source frame, so the session produces + `acquire × multiplier` frames: a 40 limit at x4 on a 120 Hz panel produces 160 + and the panel shows 120 of them. Both numbers are `icd_shim/src/limits.rs`'s + answer (`docs/10`): the acquire rate is the user's cap, or `refresh / multiplier` + without one, and the generated ceiling is `acquire × (m−1)` — clamped to the + panel's rate on a synced present mode, unclamped when the guest asked for + immediate or mailbox. A ceiling that is not a whole number of frames per source + frame is floored: 40 × 4 on a 60 Hz panel allows 60 generated a second, which is + one per source frame. The source rate is held by the acquire gate on its own + clock (`gate=clock` in the census), because the batches are placed on the latch + grid below. The panel's rate is the one the app reported (`LsfgConfig.refresh_mhz`), not a measurement: while the daemon is the only thing publishing, the latch samples are spaced by its own rate, so a panel SurfaceFlinger runs below that rate (an LTPO panel that dropped to 60 Hz) is still paced at the reported rate @@ -171,3 +167,12 @@ being created — the feature has to be on the device the pipelines are built on - Frame generation for X11 sessions: `legacy` and `native` keep that path. - Sharing one engine copy between GameNative and this repo: the copies are independent, so an algorithm fix has to be applied in both. + +## The served mode with generation on + +Generation has exactly two permitted declares: `MAILBOX` and `FIFO`, and which of +them is served follows the user's QuickMenu setting rather than a hardcoded +choice. `MAILBOX` is the default because the generated frame must replace the +pending one rather than queue behind it; `FIFO` is the option for a session that +wants the output paced by the panel with no drops. `IMMEDIATE` is never served +here — generation cannot be told apart from a tear if frames are flipped mid-scan. diff --git a/docs/20-swapchain-maintenance1.md b/docs/20-swapchain-maintenance1.md index 3d90d4f..3ab242b 100644 --- a/docs/20-swapchain-maintenance1.md +++ b/docs/20-swapchain-maintenance1.md @@ -50,9 +50,8 @@ present wait and the cap gate, which were already served. compatibility answer names one mode, so a conforming guest never asks; a request for another mode is logged once and the swapchain's mode kept (`MODE_SWITCH_NOTED`). Serving it means the requested mode enters - `effective_present_mode` / `live_present_mode` and the cap gate's - FIFO-versus-mailbox rule per present, with the container override still winning - (`docs/15` §"present mode"). That is the pacing-sensitive part: it must be + `guest_present_mode` / `live_present_mode` and the cap gate's + FIFO-versus-mailbox rule per present (`docs/15` §"present mode"). That is the pacing-sensitive part: it must be measured in a scene whose frame time is near the period, with the cap at the panel rate, before it ships. - **`VK_KHR_present_id2` / `VK_KHR_present_wait2`.** Same semantics as v1 with the diff --git a/docs/24-release-notes.md b/docs/24-release-notes.md new file mode 100644 index 0000000..d6f560d --- /dev/null +++ b/docs/24-release-notes.md @@ -0,0 +1,228 @@ +# Release notes: how a buffer is reported free + +The compositor's release and the display's release are different events. A +compositor hands a buffer back when it has finished *compositing* with it; the +display keeps scanning it out for a while after that — measured as 1.5-2.5 display +periods on Nova and 1-1.5 on Odin 3 (`sf hold`), so the size of the gap is a +property of the device and not a constant. +A producer that reuses the buffer on the compositor's release writes into memory +the panel is still reading, and the panel shows both frames at once for as long as +the two writes overlap. That is the tear Portal 2 showed under direct scan-out. + +The shim therefore needs a second, later signal, and only the process holding +SurfaceFlinger's release fence can produce it. This document describes that +channel, the rule that pairs a note with the frame it describes, and what the +three devices actually do. + +## The three messages + +All three ride the scan-out channel (`SOCK_SEQPACKET`, magic-dispatched). + +| message | direction | layout | carries | +|---|---|---|---| +| `PresentMark` | shim → daemon | `magic, reserved, ino, id, ticket` (32 B) | the scan-out inode, the guest's present id (0 when it chained none), and **the shim's own ticket** | +| `PresentDone` | daemon → shim | `magic, reserved, id, latch_ns` (24 B) | the panel's latch time for the marked frame | +| `BufferReleased` | daemon → shim | `magic, reserved, ticket, ino, latch_ns, source` (40 B) | **the ticket of the frame the buffer carried**, and what the release rests on | + +`source` is one of `RELEASE_FENCE` (the fence signalled), `RELEASE_FENCE_TIMEOUT` +(it did not, within the watcher's 500 ms budget, so the buffer was reported +anyway), `RELEASE_POOL_COPY` (the pool copied in, so nothing outside holds it) or +`RELEASE_UNWATCHED` (no subscription was watching). + +`BufferReleased` was the only one of the three without the ticket for most of its +life, and six mechanisms existed to compensate — inode matching, an order +fallback with a 5 ms reorder window, a one-display-period floor, +`since = commit_ns.min(parked)`, a bound derived from the interval it governed, and +a liveness heuristic. One missing field, six fixes. + +## Ownership is inverted + +On API 33 an `AHardwareBuffer` cannot be obtained from a bare dma-buf fd, and +`ASurfaceTransaction_setBuffer` must be called by whoever allocated it. So the +daemon allocates and owns the scan-out buffers; the shim requests one over a Unix +socket (`serve_scanout`) and receives a dma-buf fd back, which gamescope renders +into exactly as it would into a dma-heap allocation. The registration key is the +fd's `stat.st_ino`, the same identity the GEM store dedups `PRIME_FD_TO_HANDLE` +imports by. + +## The publish path + +Shim sends a mark → daemon applies the ASC transaction → `replaced_ctx` points at +the buffer **this transaction replaced** → its SF release fence goes to +`watch_release()` → the `gn-release` thread waits (max 500 ms, at most 32 fences) +→ `report_release()` pushes one `BufferReleased` per fence. Every retirement path +funnels through `report_release`, so a note is never duplicated. + +`watch_release` returns early, closing the fd, when `ino == 0` (a pooled copy, +which nothing outside holds) or when `!pushes::wants(CAP_RELEASE_NOTE)` — the +channel is opt-in per process and re-checked per fence, because the subscriber may +connect after the first buffers are already in flight. + +## The pairing rule + +A park is taken at commit for a synced, non-hidden slot, and is answered by the +first of: + +1. **The ticket.** `take_released_ticket()` looks the note up by the commit id the + shim minted. Exact, O(1), no window. This is the rule the protocol is for: the + key is a number the process that owns the slot made up, so it cannot be + ambiguous the way a dma-buf inode can, and it cannot be recycled into a live + slot the way a freed buffer's inode can. +2. **The inode.** `take_released(ino, since)` — where the shim's slot inode and the + daemon's registration inode are the same object. +3. **The order.** The oldest park by `commit_ns`, with a floor one display period + before the commit, because the note that answers a park is emitted when the + *next* frame latches and so legitimately predates the park. + + Neither fallback takes a note whose ticket is older than the park's own: that + is a late note for an earlier use of the buffer, or the note of a commit that + was never parked (a hidden slot, a generation destination), and the frame it + describes says nothing about the parked one. A newer ticket still pays — a + note can arrive before its own slot is parked, and the ticket arm only looks + at parked slots. +4. **The bound.** `min(2 × commit spacing, 4 × panel period)`, floored at two + periods. +5. **The fail-open.** `PARK_UNANSWERED` counts parks that expired without a note; + at four in a row the slot is freed on its fence instead, which is the behaviour + from before the park existed. A `BufferReleased` resets it, so a producer that + comes back needs no probation timer. Liveness is keyed on `BufferReleased` + alone — keying it on the socket let RP6 keep parking, because that device sends + ~5 000 `VsyncNote`s while releasing nothing. + +## What each device actually does + +Measured with Dave the Diver at 1 280 × 720, 60 Hz panel, cap 60, `fifo`. + +| | Nova (Adreno 740) | Odin 3 (Adreno 830) | RP6 (Adreno 740) | +|---|---|---|---| +| notes per session | ~1 per frame (9 528) | ~1 per frame | 17-198 per session | +| `sf mark:` | **hit=4 736, order=1, miss=2, skew=0** — 1:1 with the notes | **hit=5 440, order=1, miss=2, skew=0** | not re-read after the buffer fix; the pre-fix session printed one line in 115 s (`hit=0 order=1 miss=2 last=1`) — see the open question below | +| `tick=` (notes carrying the ticket) | **5 168** of 4 627 | **1 760** of 1 793 (in game) | **15 of 16** after the buffer fix, 0 before | +| pairing | **ticket, with `by_order` frozen at 1 211 across the last 240 frames** — identity on Portal 2 took 6 % order, Dave took 84 % before the ticket | **`by_order` rising (971 at 54 % of acquires) but each answered in ~110 us** — the fallback is load-bearing and cheap | none | +| `channel=` | `unanswered:0 dead:false` | `unanswered:0 dead:false` | `unanswered:6 dead:true` | +| fps, menu / in game | 60 / **60.0** | 60 / **59.7** (`p95=17.2 ms`, `slow=0/118`, `repeated=0`) | 60 / **59.8** | +| `sf hold:` | prints, 1.5-2.5 periods | **76 % at 1-1.5 periods, 22 % at 1.5-2.5**, `timeouts=0` (n=7 442, in game) | **silent** — `latch_ns` is 0 | +| `park_timeouts` / acquires | 114 / 4 739 | — | 7 / 5 186 | + +All three rendered the menu at 60 fps, and the row above is menu / in game. For +context against earlier work: Nova held 59.5 before the park was fixed, Odin 3 sat +at 3 fps while the park's bound fed on the interval it governed, and RP6 was 35.9 +with no notes to answer a park. + +Where the shim actually runs, the mark resolves by identity almost exclusively — +`order=1` on Nova and `order=2` on Odin 3 against thousands of hits — so the mark +path is sound and the ticket it carries is the frame's own. + +Three failure points, and they are independent: + +- **the watcher may not be armed** — RP6 sends notes but `latch_ns` is 0, so + `note_hold` never runs and `sf hold:` never prints; `replaced_ctx` is reached + sometimes (198 notes) but without a latch time; +- **the inode may name another buffer** — Odin 3 measured `slot ino=11411` against + `released=[11366 11368 11370]`; +- **the note may predate the park** — the order arm's floor exists for this, and + Nova-on-Dave needed it for 84 % of parks *before* the ticket; after it, `by_order` + stopped moving altogether. + +A fourth is silent and looks like all of them: no subscriber +(`wants(CAP_RELEASE_NOTE)` false). + +The inode is therefore a property of the *title's* allocation pattern, not of the +device: the same Nova paired 94 % of Portal 2's parks by identity and 16 % of +Dave's. Any rule that depends on it is a rule that works until a different game +runs. + +## Two defects found by measuring this + +**`park_timeouts` counted loop turns, not parks.** The bound was re-evaluated on +every pass of the acquire loop over a slot it had already decided, so RP6 read +`park_timeouts=14 544` against 14 679 acquires while the total slot wait was 14.5 s +— under a millisecond per park it claimed to have timed out. `GnSlot.park_expired` +now answers each park once. + +**A 24-byte read buffer truncated a 32-byte mark.** The daemon read the channel +into `[0u8; ScanoutAllocReq::LEN]` (24). `SOCK_SEQPACKET` truncates a frame to the +receive buffer and discards the rest, so after `PresentMark` grew to carry the +ticket every mark arrived truncated, `decode` returned `None`, and the marks never +reached `note_present_mark` — `sf mark: hit=0 order=0 miss=3 pending=0` with +`tick=[n:0]` on every line, while both ends looked correct. The reader now uses +`wire::INBOUND_MAX`, derived from the inbound types, and a wire test fails the +build if a new frame does not fit. + +## Reading the census + +**It needs `GNDRM_LOG_VKSHIM=1`, and a session without it looks broken.** +`present mode:` is printed twice: `logi!` emits the header unconditionally, and +`logf!` emits the census — `acquires=`, `tick=`, `channel=`, `by_order=`, every +figure in this table — only when that variable is set. A device without it produces +a short `present mode:` line and no `proxy commit:` lines at all, which is +indistinguishable from a shim built before the census existed. Odin 3's first two +sessions were misdiagnosed that way, including once as "the payload landed after +launch" on timestamps that did not support it: the process was new, the `.so` was +mapped, and only the variable was missing. + +Odin 3 also runs a driver pack rather than the system driver +(`files/contents/adrenotools/Turnip Gen8 V30/libvulkan_freedreno.so`), which is +worth knowing when comparing its numbers with the other two. + +### A rate comes from two adjacent lines, not from a counter + +Odin 3's first in-game reading looked like a failure — `PowerMetrics fps=36.6`, +`p95=101 ms`, `repeated=20`, and `by_order` climbing through 54 % of acquires. It +was the loading transition: the same ten-second window contains `presiv +max=6 496 260`, a 6.5 s stall while the scene loaded, and `acq_fps=36.03` is a +lifetime average that the stall drags down for the rest of the session. + +The steady state, one loading transition later: + + PowerMetrics: fps=59.7 p95=17.2ms slow=0/118 repeated=0 cpu=21% gpu=67% + intervals: 120 acquires / 2.003 s = 59.9 + slot growth: 13 224 us per 120 frames = 110 us/frame + tick=1 760 of rnotes=1 793 channel=[unanswered:0 dead:false] + +So `by_order` climbing is **not** a defect signal by itself: those parks are +answered in about a tenth of a millisecond, where Nova's ticket-answered parks cost +63 us. The number that looked alarming — `slot_avg_us=1171` over 89 % of frames — +belongs to the loading phase and decays as the window fills. + +`acq_fps`, the `acqus` means and the `presiv` means are all lifetime figures, so +any session with a startup stall reads low long after the stall is over. Take the +rate from two adjacent lines. + +### What the ticket bought, measured + +Nova, Dave, same payload and scene, before and after: + + before by_order=3 640 of rnotes=4 328 (84 % of parks) slot_avg_us=37 + after by_order=1 211 and frozen (0 in steady state) slot_avg_us=63 + sf mark: hit=4 736 order=1 miss=2 skew=0 + tick=5 168 of rnotes=4 627, off=0 + fps 60.0, park_timeouts 114 of 4 739 (2.4 %) + +`by_order` not moving across 240 consecutive frames is the result that matters on +that device: the order arm is cold there, so for Nova the identity and order paths +are already dead code. + +## What is still open + +- **RP6 has no latch time.** It runs 60 fps, but no session has produced a + `sf hold:` sample, so how long SurfaceFlinger holds a buffer there is unknown. + Whether its output is clean or merely fast has not been measured; the fail-open + hides it. This is the only gap that is a defect rather than a tuning question. +- **The order arm is still load-bearing on Odin 3** — `by_order` at 54 % of + acquires, each answered in ~110 us. It cannot be deleted with Nova's evidence: + on Odin the ticket lookup misses often enough that the fallback carries half the + parks, which is a timing relationship between park and release that Nova does + not have. Deleting the arms removes a cost this device does not show. +- **The daemon's counters do not reconcile on RP6**, though they do on Nova + (4 736 lookups against 4 627 notes at the same instant). RP6 printed one + `sf mark:` line in 115 s — a single ticket ever assigned — while 15 notes arrived + carrying tickets. Both cannot be true with `ctx.ticket` written only from + `take_present_mark`. The cause is likelier to be that RP6 publishes almost + nothing through the ASC path than a defect in the counter, but that is a + hypothesis, not a measurement, and a publish counter would settle it. +- **A pooled copy produces no note at all** (`ino == 0` returns early). That is a + known gap: `RELEASE_POOL_COPY` exists in the wire and nothing sends it. +- **The shim's `by_order` conflates two causes** — a note naming another buffer, + and a note older than the anchor. They need opposite fixes, and one counter + hides which is happening. diff --git a/docs/README.md b/docs/README.md index be6d40b..2225d08 100644 --- a/docs/README.md +++ b/docs/README.md @@ -53,6 +53,7 @@ winlator X11 path (which remains the default for normal launches). 21. [21-wsi-extension-policy.md](21-wsi-extension-policy.md) — which surface, swapchain and present extensions the guest sees under the WSI proxy: the registry-derived WSI scope, the served/hidden table in `wsi_policy.rs`, the per-process log line, and how to classify a new extension. 22. [22-unresolvable-extensions.md](22-unresolvable-extensions.md) — the resolve probe: the reported apiVersion is clamped below the first core version whose commands do not resolve by exact name on the guest's instance (its Vulkan1xFeatures read false); the Honor GT 1.4 case, the probe line with its cost, the 1.0-instance skip, `GN_VKSHIM_NO_RESOLVE_PROBE=1`; extension hiding removed. 23. [23-strict-image-count.md](23-strict-image-count.md) — the games that fail unless the swapchain has exactly the image count they asked for: Mesa's strict list carried by the shim, what changes for them, the hidden slots behind their images and how to keep the list in step. +24. [24-release-notes.md](24-release-notes.md) — the scan-out release channel: the three messages and the ticket that pairs them, why the compositor's release is not the display's, the five-step rule that answers a park, what Nova, Odin 3 and RP6 each do with it, and the two counting/truncation defects that hid it. ### Performance diff --git a/docs/performance/01-gamescope-vs-x11-cpu.md b/docs/performance/01-gamescope-vs-x11-cpu.md index c220e33..5d64748 100644 --- a/docs/performance/01-gamescope-vs-x11-cpu.md +++ b/docs/performance/01-gamescope-vs-x11-cpu.md @@ -273,13 +273,14 @@ time (`tools/gamescope-cpu-arm.sh`, full numbers in | is `vkWaitForPresentKHR` called? | **0 lines** | same | | do acquires ever wait for a slot? | **0.0**, dwell **0.013 ms**, `starved=0` | cause C dead, item 3 closed | | what depth does the guest actually get? | **4** (mean in flight 1.66, free 2.08) | item 3's "add a 4th slot" was already true | -| is a FIFO declaration ever made? | **no** — `declared=mailbox requested=immediate source=env`, `declared_fifo=0` over 25 009 presents | item 5 unreachable as well as semantic | +| is a FIFO declaration ever made? | **no** — `declared=mailbox requested=immediate source=env`, `declared_fifo=0` over 25 009 presents | item 5 unreachable as well as semantic (the `source=env` form of the census no longer exists; the shim declares the guest's mode now) | | does the X11 arm exercise the shim? | **no** — `gn-vkshim: ICD loaded` = 0 | the +5.1 pp guest term *is* the shim's existence, not its behaviour | -The container's `MESA_VK_WSI_PRESENT_MODE=mailbox` overrides the title's own -`immediate` request, which is why no FIFO path is ever taken. That is a -deliberate container default, and it is also why three of the plan items cannot -fire on this title. +At the time of this run the container's `MESA_VK_WSI_PRESENT_MODE=mailbox` +overrode the title's own `immediate` request, which is why no FIFO path was ever +taken, and it is also why three of the plan items could not fire on this title. +The shim reads no such override any more (`refactor/vsync`): the declared mode is +the guest's request, so a re-run of this arm would declare `immediate`. ~~One asymmetry is left unexplained: 241.9 atomic commits/s against 357.3 presents/s.~~ **Withdrawn — the two numbers count different events and were never diff --git a/docs/performance/02-instrumentation.md b/docs/performance/02-instrumentation.md index 3c4635f..4f1ab2e 100644 --- a/docs/performance/02-instrumentation.md +++ b/docs/performance/02-instrumentation.md @@ -405,8 +405,9 @@ Read it like this: | `fallback` | `Δfallback` — waits the daemon's latch did **not** answer: the buffer release (`present.rs:316`–`:324`) **or** the 2-refresh grace (`:405`–`:420`). Both increment the same counter; only the per-wait µs in G6 separates them (grace ≈ `2/refresh` = 16.7 ms at 120 Hz, release ≈ one frame + a vblank) | | `semwait_rate` | G11 count / Δt — `vkWaitSemaphores`, the DXVK latency-sleep shape | -The verdict must be **mode-aware**, because the container sets -`MESA_VK_WSI_PRESENT_MODE=mailbox`: +The verdict must be **mode-aware**, because at the time of these runs the +container set `MESA_VK_WSI_PRESENT_MODE=mailbox` (the shim no longer reads it; +these are runs from before `refactor/vsync`): * `present_wait_answer` returns `AtQueue` only when the route is gamescope **and** `why == CapRateVsync`; `AtCommit` when either the guest's requested mode or the @@ -766,8 +767,8 @@ the shim exists in the guest, and in X11 mode it does not exist at all. 2.08 free, `starved=0`. The acquire is not stalling. Plan item 3 → **closed**, including its "add a 4th slot" sub-item: **the depth is already 4**, not 3. - **Cause F cannot even be reached.** `declared_fifo=0` over 25 009 presents: the - container's `MESA_VK_WSI_PRESENT_MODE=mailbox` overrides the title's own - `immediate` request and the shim never enters a FIFO path. Plan item 5 stays + container's `MESA_VK_WSI_PRESENT_MODE=mailbox` then overrode the title's own + `immediate` request and the shim never entered a FIFO path. Plan item 5 stays closed for a stronger reason than the semantic one in `06-present-mode-declaration.md`. - **A new, unexplained number worth one look:** the daemon serves **241.9** diff --git a/docs/performance/03-present-wait.md b/docs/performance/03-present-wait.md index 12be617..62c9b2a 100644 --- a/docs/performance/03-present-wait.md +++ b/docs/performance/03-present-wait.md @@ -231,8 +231,8 @@ fn present_wait_answer(route, why, requested, live) -> WaitAnswer ``` with `wait_policy()` reading `GN_VKSHIM_PRESENT_WAIT ∈ {queue, commit, release}`, default -`queue`, parsed once at first present (same shape as `parse_present_mode_override`, -`proxy.rs:7181`). `route`, `why`, `requested`, `live` stay in the signature so the +`queue`, parsed once at first present (the shape `parse_present_mode_override` used +before `refactor/vsync` deleted it). `route`, `why`, `requested`, `live` stay in the signature so the `commit`/`release` arms remain exactly today's behaviour and the existing table test (`proxy.rs:7124`–`:7167`) keeps its meaning by being run under `Commit`. Reason to keep three arms: §4 risks 1, 3 and 5 all need a per-container revert without a diff --git a/docs/performance/06-present-mode-declaration.md b/docs/performance/06-present-mode-declaration.md index 575cfd7..5c7a50e 100644 --- a/docs/performance/06-present-mode-declaration.md +++ b/docs/performance/06-present-mode-declaration.md @@ -120,7 +120,7 @@ one: | change | effect on the computed value | re-sent? | |---|---|---| | initial attach (first commit on a surface) | `decl_pm = NO_DECL` ≠ any wire value | yes, invariant 1 | -| present mode change (`effective_present_mode`, `proxy.rs:2302`) | value changes | yes | +| present mode change (the guest's `presentMode` at swapchain creation) | value changes | yes | | fps cap set/cleared (`scanout::cap_fps_note`, `:173`) | `capped` flips ⇒ `Declared::Mode` ↔ `CapRate`/`CapCompat` | yes | | **display type / refresh change** (`vsync_gate::current_grid` period, `vsync_gate.rs:73-85`) | `at_display_rate(cap, grid)` (`vsync_gate.rs:218`) flips ⇒ FIFO ↔ the swapchain's mode. 60-cap on 60 Hz = FIFO; same 60-cap on 120 Hz = mailbox ⇒ 2→1 | yes, because the value moved. If the value does *not* move (same refresh, same cap) there is nothing to tell gamescope | | frame generation on/off (`lsfg::live()`) | `Declared::Generating` ⇒ mailbox, and back | yes (§5) | diff --git a/icd_shim/src/image_count.rs b/icd_shim/src/image_count.rs index 40b6566..408b139 100644 --- a/icd_shim/src/image_count.rs +++ b/icd_shim/src/image_count.rs @@ -46,10 +46,19 @@ static STRICT_GAMES: &[Match] = &[ Match::AppName("DetroitBecomeHuman"), ]; -/// The hidden slots behind a strict game's swapchain. A swapchain of 3 or more -/// already has an image to spare, so it gets none. Below that it gets the frame on -/// screen, the one committed behind it, and one for each of the two frames the guest -/// can have between its present and the commit. +/// The hidden slots behind a swapchain that asked for fewer than three images. +/// +/// A swapchain of 3 or more already has an image to spare, so it gets none. Below +/// that it gets the frame on screen, the one committed behind it, and one for each +/// of the two frames the guest can have between its present and the commit. +/// +/// The rule is the ask, not the game. It began as the strict list's exception, but +/// the need is a property of the count: a guest with two images cannot pipeline +/// past the hold the display imposes (`min(cap, depth x refresh / hold)`), and Doom +/// 2016 measured 24 fps against a cap of 60 with the GPU idle before it had any. +/// The caller must pass the guest's *requested* count, never the served one: a +/// mailbox floor can raise the served count past the ask, and a guest that asked +/// for two has no image to spare however many it was given. pub fn hidden_slot_count(guest_slots: u32) -> u32 { if guest_slots < 3 { 4 @@ -164,7 +173,10 @@ mod tests { #[test] fn only_a_swapchain_with_no_image_to_spare_gets_hidden_slots() { + // Doom 2016 asks for two and is served mailbox, which floors the count at + // four; the ring is sized from the ask, so this is the case that matters. assert_eq!(hidden_slot_count(2), 4); + // Three already has one to spare, and a larger ask needs nothing. assert_eq!(hidden_slot_count(3), 0); } diff --git a/icd_shim/src/lib.rs b/icd_shim/src/lib.rs index d3e812c..01b4d3e 100644 --- a/icd_shim/src/lib.rs +++ b/icd_shim/src/lib.rs @@ -113,6 +113,7 @@ pub mod advertise; pub mod divisor; pub mod export; pub mod image_count; +pub(crate) mod limits; pub mod placed; pub mod present; pub mod lsfg; diff --git a/icd_shim/src/limits.rs b/icd_shim/src/limits.rs new file mode 100644 index 0000000..e86e99c --- /dev/null +++ b/icd_shim/src/limits.rs @@ -0,0 +1,250 @@ +//! The acquire and generation limits, in frames per second. +//! +//! One rule, four inputs: whether the guest asked for a synced present mode, +//! its cap, frame generation's multiplier, and the panel's refresh. gamescope +//! is told the guest's own mode and nothing else — the cap and the multiplier +//! never rewrite it — so what a cap or a multiplier changes is *how often the +//! guest may acquire* and *how many frames the generator may add*. +//! +//! Synced (FIFO / FIFO-relaxed), the panel is the ceiling in both directions: a +//! cap below it is the cap, no cap is the refresh, and generation may fill up +//! to the refresh but not past it. Unsynced (immediate / mailbox) the guest is +//! free to run past the panel when it asks to — the compositor drops what it +//! cannot show — so only a cap or generation bounds it. +//! +//! | present | cap | LSFG | acquire | generated | +//! |---|---|---|---|---| +//! | synced | on | off | cap | — | +//! | synced | off | off | refresh | — | +//! | synced | on | on | cap | `cap × (m−1)`, at most `refresh − cap` | +//! | synced | off | on | `refresh / m` | `acquire × (m−1)` | +//! | unsynced | on | off | cap | — | +//! | unsynced | off | off | none | — | +//! | unsynced | on | on | cap | `cap × (m−1)` | +//! | unsynced | off | on | `refresh / m` | `acquire × (m−1)` | + +/// What one session allows, in frames per second. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Limits { + /// Frames/s the guest may acquire. `None` = no limit. + pub acquire_per_s: Option, + /// Frames/s the generator may insert. `None` = generation off. + pub generate_per_s: Option, +} + +impl Limits { + /// How many frames the generator may add per source frame, `None` when it + /// may not add any. + /// + /// The ceiling is a rate, so a panel that cannot show `acquire × multiplier` + /// frames leaves the generator fewer per source frame; it never leaves it + /// more. `multiplier` is the configured one, and the return value never + /// exceeds `multiplier - 1`. + pub(crate) fn generate_per_source(&self, multiplier: u32) -> Option { + let generate = self.generate_per_s?; + let acquire = self.acquire_per_s?; + let asked = multiplier.saturating_sub(1); + Some(asked.min(generate / acquire.max(1))) + } +} + +/// The limits for one session. `refresh_hz` is the panel's, and 0 when the +/// daemon has not reported it yet: rules that need it then impose no limit +/// rather than a wrong one. +pub(crate) fn limits( + vsync: bool, + cap_per_s: Option, + multiplier: Option, + refresh_hz: u32, +) -> Limits { + let cap = cap_per_s.filter(|c| *c > 0); + let mult = multiplier.filter(|m| *m >= 2); + + let acquire_per_s = match (cap, mult) { + // A cap the user asked for is the cap, whatever else is on. + (Some(c), _) => Some(c), + // Uncounted, generation derives the source rate from the panel so its + // output lands on the refresh exactly. + (None, Some(m)) => (refresh_hz > 0).then(|| (refresh_hz / m).max(1)), + // No cap: a synced present mode is already held by the panel. + (None, None) if vsync => (refresh_hz > 0).then_some(refresh_hz), + (None, None) => None, + }; + + let generate_per_s = match (mult, acquire_per_s) { + (Some(m), Some(a)) => { + let raw = a.saturating_mul(m - 1); + // Synced, the panel bounds what it can show *and the source frames + // are part of that budget*: acquire + generated <= refresh, so what + // is left for the generator is `refresh - acquire`. Clamping at + // `refresh` alone let a cap 40 ×2 session on a 60 Hz panel produce + // 80 commits a second, of which the compositor dropped 20 — the + // budget is the panel's, so it belongs in the same subtraction. + // + // The consequence is deliberate and worth knowing: at 60 Hz, cap 40, + // ×2, this leaves 20 generated against 40 source frames, i.e. 0.5 + // per source frame, which `generate_per_source` floors to 0. Frame + // generation therefore does nothing at any cap above half the + // refresh, where before it over-subscribed the panel instead. + // + // Unsynced, only the cap does, and the compositor drops the + // remainder by design. + Some(if vsync && refresh_hz > 0 { + raw.min(refresh_hz.saturating_sub(a)) + } else { + raw + }) + } + _ => None, + }; + + Limits { + acquire_per_s, + generate_per_s, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The table in the module notes, row by row. + #[test] + fn the_table_holds() { + // synced, cap, no generation + assert_eq!( + limits(true, Some(40), None, 60), + Limits { + acquire_per_s: Some(40), + generate_per_s: None + } + ); + // synced, no cap, no generation: the panel is the limit + assert_eq!( + limits(true, None, None, 60), + Limits { + acquire_per_s: Some(60), + generate_per_s: None + } + ); + // synced, cap, generation: generation fills what the panel has left + // after the source frames + assert_eq!( + limits(true, Some(40), Some(2), 60), + Limits { + acquire_per_s: Some(40), + generate_per_s: Some(20) + } + ); + // synced, cap, generation, panel too slow for both: the panel still + // leaves its 20 after the source frames whatever the multiplier asks + assert_eq!( + limits(true, Some(40), Some(4), 60), + Limits { + acquire_per_s: Some(40), + generate_per_s: Some(20) + } + ); + // synced, no cap, generation: source derives from the panel + assert_eq!( + limits(true, None, Some(2), 60), + Limits { + acquire_per_s: Some(30), + generate_per_s: Some(30) + } + ); + // unsynced, cap, no generation + assert_eq!( + limits(false, Some(40), None, 60), + Limits { + acquire_per_s: Some(40), + generate_per_s: None + } + ); + // unsynced, no cap: nothing bounds it + assert_eq!( + limits(false, None, None, 60), + Limits { + acquire_per_s: None, + generate_per_s: None + } + ); + // unsynced, cap, generation: the cap alone bounds it + assert_eq!( + limits(false, Some(40), Some(2), 60), + Limits { + acquire_per_s: Some(40), + generate_per_s: Some(40) + } + ); + // unsynced, no cap, generation: source still derives from the panel + assert_eq!( + limits(false, None, Some(2), 60), + Limits { + acquire_per_s: Some(30), + generate_per_s: Some(30) + } + ); + } + + #[test] + fn an_unreported_panel_imposes_no_limit() { + assert_eq!( + limits(true, None, None, 0), + Limits { + acquire_per_s: None, + generate_per_s: None + } + ); + // A cap still stands: it does not need the panel. + assert_eq!( + limits(true, Some(40), None, 0), + Limits { + acquire_per_s: Some(40), + generate_per_s: None + } + ); + } + + #[test] + fn a_multiplier_of_one_is_not_generation() { + assert_eq!( + limits(true, Some(40), Some(1), 60), + Limits { + acquire_per_s: Some(40), + generate_per_s: None + } + ); + } + + #[test] + fn generation_per_source_narrows_when_the_panel_is_the_ceiling() { + // 40 acquires, 60 panel, ×4 asked: the panel leaves 20 generated in + // total, i.e. none per source frame at this rate. + let l = limits(true, Some(40), Some(4), 60); + assert_eq!(l.generate_per_s, Some(20)); + assert_eq!(l.generate_per_source(4), Some(0)); + // Unsynced the same numbers leave the ask intact. + let l = limits(false, Some(40), Some(4), 60); + assert_eq!(l.generate_per_source(4), Some(3)); + } + + /// The consequence of `refresh - acquire`, pinned so it cannot drift into a + /// claim about the feature working: at 60 Hz, cap 40, ×2, the panel leaves 20 + /// generated against 40 source frames, and half a frame per source floors to + /// none. A user with LSFG on at that cap gets no generated frames, which is + /// the price of not over-subscribing the panel. + #[test] + fn a_cap_above_half_the_panel_leaves_no_room_to_generate() { + let l = limits(true, Some(40), Some(2), 60); + assert_eq!(l.generate_per_s, Some(20)); + assert_eq!(l.generate_per_source(2), Some(0)); + // One step lower and generation survives: cap 30 at ×2 on 60 Hz. + let l = limits(true, Some(30), Some(2), 60); + assert_eq!(l.generate_per_s, Some(30)); + assert_eq!(l.generate_per_source(2), Some(1)); + // A cap above the panel cannot underflow: nothing is left to generate. + let l = limits(true, Some(120), Some(2), 60); + assert_eq!(l.generate_per_s, Some(0)); + } +} diff --git a/icd_shim/src/lsfg/mod.rs b/icd_shim/src/lsfg/mod.rs index 73e262d..c2638ca 100644 --- a/icd_shim/src/lsfg/mod.rs +++ b/icd_shim/src/lsfg/mod.rs @@ -221,6 +221,8 @@ struct Live { multiplier: std::sync::atomic::AtomicU32, flow_per_mille: std::sync::atomic::AtomicU32, fps_limit: std::sync::atomic::AtomicU32, + /// The QuickMenu's present mode for generation: 0 mailbox, 1 fifo. + present_mode: std::sync::atomic::AtomicU32, refresh_mhz: std::sync::atomic::AtomicU32, /// The panel's latch grid as the daemon last reported it; a period of 0 /// means no report yet. @@ -234,6 +236,7 @@ static LIVE: Live = Live { multiplier: std::sync::atomic::AtomicU32::new(0), flow_per_mille: std::sync::atomic::AtomicU32::new(0), fps_limit: std::sync::atomic::AtomicU32::new(0), + present_mode: std::sync::atomic::AtomicU32::new(0), refresh_mhz: std::sync::atomic::AtomicU32::new(0), vsync_anchor_ns: std::sync::atomic::AtomicU64::new(0), vsync_period_ns: std::sync::atomic::AtomicU32::new(0), @@ -266,15 +269,6 @@ pub fn wanted() -> Option { (enabled && multiplier >= 2).then_some(multiplier) } -/// The rate generated frames are committed at: the source cap times the -/// multiplier, or 0 when generation is off. This is the grid the generated -/// commits are spaced on. -#[must_use] -pub fn output_rate() -> u32 { - let (_, output) = grid(); - output -} - /// The settings epoch: equal to `wanted`'s generation changes. #[must_use] pub fn epoch() -> u32 { @@ -294,6 +288,7 @@ pub fn apply(config: gndrm_uapi::wire::LsfgConfig) { LIVE.multiplier.store(config.multiplier, Relaxed); LIVE.flow_per_mille.store(config.flow_per_mille, Relaxed); LIVE.fps_limit.store(config.fps_limit, Relaxed); + LIVE.present_mode.store(config.present_mode, Relaxed); LIVE.refresh_mhz.store(config.refresh_mhz, Relaxed); let now = (config.enabled != 0, config.multiplier, config.fps_limit); if previous != now { @@ -335,36 +330,6 @@ pub fn skip(what: &str) -> bool { } /// The rate the guest should be paced to while frame generation runs. -/// -/// `None` when generation is off or the user set no cap, in which case the -/// daemon's own frame cap (if any) still applies. This is the first consumer of -/// [`live`]: with the app's limiter disarmed for gamescope, the cap on the source -/// comes from here, not from a `FrameCapNote`. -#[must_use] -pub fn source_cap() -> Option { - let (enabled, _multiplier, cap) = live(); - // The user's cap, as asked: the display does not reshape it, so a 40 cap is - // 40 acquires a second on a 60 Hz panel as much as on a 120 Hz one, and the - // multiplier decides what is generated on top. - (enabled && cap > 0).then_some(cap) -} - -/// `(source, output)` rates in whole hertz while generation is on. -/// -/// The user's cap is the **source** rate: the guest renders that many frames, and -/// the presenter presents `cap × multiplier` by inserting generated frames -/// between them. Not clamped to the panel: the grid must hold every frame of a -/// source period or the queue grows without bound; what the panel cannot show, -/// the compositor drops. `(0, 0)` when generation is off or uncapped. -#[must_use] -pub fn grid() -> (u32, u32) { - let (enabled, multiplier, cap) = live(); - if !enabled || cap == 0 { - return (0, 0); - } - (cap, cap.saturating_mul(multiplier.max(1))) -} - /// The panel's refresh in whole hertz, or 0 when the daemon has not reported it. #[must_use] pub fn refresh_hz() -> u32 { @@ -557,12 +522,13 @@ impl Session { .map(|s| s.refresh_mhz as f32 / 1000.0) .unwrap_or(0.0); unsafe { (engine.api.configure)(root, multiplier, 0, flow, refresh) }; - let (source, output) = grid(); - crate::logf!("lsfg: ready multiplier={} fp16={} source={} output={} cache={}", + // What the session allows is `crate::limits`'s answer, printed by the + // shim's census; here the engine reports only its own configuration. + crate::logf!("lsfg: ready multiplier={} fp16={} cap={} refresh_hz={} cache={}", multiplier, fp16, - source, - output, + source_cap, + refresh, cache.display()); // Seed the live state with this session's decision: a push only arrives @@ -574,6 +540,9 @@ impl Session { performance: from_daemon.map_or(0, |s| s.performance), fps_limit: source_cap, dll: [0; gndrm_uapi::wire::LSFG_DLL_PATH_MAX], + // Mailbox unless the settings ask for fifo; the QuickMenu's choice + // travels on the note, so it reaches here through `from_daemon`. + present_mode: from_daemon.map_or(0, |s| s.present_mode), refresh_mhz: (refresh * 1000.0) as u32, }; current.set_dll_path(&dll.to_string_lossy()); @@ -846,3 +815,10 @@ impl Drop for Session { } } } + + +/// The present mode the QuickMenu chose for generation: 0 = mailbox, 1 = fifo. +/// Zero until a config arrives, which is mailbox — the default the app ships. +pub(crate) fn generating_present_mode() -> u32 { + LIVE.present_mode.load(std::sync::atomic::Ordering::Relaxed) +} diff --git a/icd_shim/src/maint1.rs b/icd_shim/src/maint1.rs index 1bb06f4..e3c09f4 100644 --- a/icd_shim/src/maint1.rs +++ b/icd_shim/src/maint1.rs @@ -28,7 +28,7 @@ //! `VkSwapchainPresentModeInfo` to switch between the modes it listed at //! creation; today the compatibility answer lists one mode, so a conforming //! guest never asks, and a request for another mode is logged once and ignored. -//! Serving it means feeding the requested mode into `effective_present_mode` / +//! Serving it means feeding the requested mode into `guest_present_mode` / //! `live_present_mode` and the cap gate's FIFO-versus-mailbox rule per present //! (docs/15), which is the pacing-sensitive part and gets its own change. The //! same follow-up may take `VK_KHR_present_id2` / `VK_KHR_present_wait2` diff --git a/icd_shim/src/present.rs b/icd_shim/src/present.rs index 3028337..c738758 100644 --- a/icd_shim/src/present.rs +++ b/icd_shim/src/present.rs @@ -21,7 +21,7 @@ //! bounded: a guest that marks frames and never waits cannot grow them. use core::ffi::c_void; -use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::{Condvar, Mutex}; use std::time::Duration; @@ -55,10 +55,9 @@ struct Waits { /// answered is two refresh periods after that. `(id, swapchain, ns)`. committed: Vec<(u64, usize, u64)>, /// Commits that asked gamescope for presentation feedback that has not - /// arrived: only `presented` / `discarded` answers them (or, should it - /// never come, [`FEEDBACK_SAFETY_NS`] after the commit). `(token, id, - /// swapchain)`; the token is the feedback listener's data, unique per - /// request. + /// arrived: `presented` / `discarded` answers them, and so does the + /// two-period grace its frame's commit carries. `(token, id, swapchain)`; + /// the token is the feedback listener's data, unique per request. feedback: Vec<(u64, u64, usize)>, } @@ -76,17 +75,49 @@ static WAITS: Mutex = Mutex::new(Waits { feedback: Vec::new(), }); -/// How long after its commit an id whose feedback never came is answered -/// anyway: far beyond any presentation gamescope makes, so it never answers -/// early, and well inside the guest's one-second bound. -const FEEDBACK_SAFETY_NS: u64 = 100_000_000; - /// Presentation feedback outcomes, and waits answered by a fallback (release, /// grace or the feedback safety), for the shim's ungated census. static PRESENTED_N: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); static DISCARDED_N: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); static FALLBACK_N: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); +/// Which path answered each `vkWaitForPresentKHR`, for the census. +/// +/// The guest's frame on this route is one frame in flight, so the rate it can +/// reach is the speed of this answer: a wait answered by the compositor's +/// feedback is paced by the compositor, and one answered by `grace` or `release` +/// is not. `nocommit` counts waits whose id had no commit registered — the state +/// `grace` cannot answer at all, since it has no commit time to bound. +static WAIT_N: AtomicUsize = AtomicUsize::new(0); +static ANS_QUEUE: AtomicUsize = AtomicUsize::new(0); +static ANS_COMMIT: AtomicUsize = AtomicUsize::new(0); +static ANS_FB: AtomicUsize = AtomicUsize::new(0); +static ANS_DAEMON: AtomicUsize = AtomicUsize::new(0); +static ANS_GRACE: AtomicUsize = AtomicUsize::new(0); +static ANS_RELEASE: AtomicUsize = AtomicUsize::new(0); +static ANS_DISCARD: AtomicUsize = AtomicUsize::new(0); +static ANS_UNARMED: AtomicUsize = AtomicUsize::new(0); +static ANS_TIMEOUT: AtomicUsize = AtomicUsize::new(0); +static NO_COMMIT: AtomicUsize = AtomicUsize::new(0); + +/// `(waits, at queue, at commit, feedback, daemon latch, grace, release, +/// discarded, unarmed, timeout, no commit registered)`. +pub(crate) fn wait_census() -> [usize; 11] { + [ + WAIT_N.load(Ordering::Relaxed), + ANS_QUEUE.load(Ordering::Relaxed), + ANS_COMMIT.load(Ordering::Relaxed), + ANS_FB.load(Ordering::Relaxed), + ANS_DAEMON.load(Ordering::Relaxed), + ANS_GRACE.load(Ordering::Relaxed), + ANS_RELEASE.load(Ordering::Relaxed), + ANS_DISCARD.load(Ordering::Relaxed), + ANS_UNARMED.load(Ordering::Relaxed), + ANS_TIMEOUT.load(Ordering::Relaxed), + NO_COMMIT.load(Ordering::Relaxed), + ] +} + /// `(presented, discarded, answered by a fallback)` so far. pub(crate) fn feedback_counts() -> (usize, usize, usize) { ( @@ -148,7 +179,9 @@ fn feedback_done(token: u64, ns: u64) { // A token no longer pending belongs to a swapchain that went away, or a // frame another path answered: nothing to do. if let Some((id, sc)) = key { - record_for(id, sc, ns); + if record_for(id, sc, ns) { + ANS_FB.fetch_add(1, Ordering::Relaxed); + } } } @@ -185,14 +218,16 @@ static WAIT_CV: Condvar = Condvar::new(); /// answered by the compositor's release or, at the latest, two refresh periods /// after the commit. /// -/// `answer_now` is for swapchains whose effective present mode is mailbox or -/// immediate. The guest may believe it has FIFO (the shim applies -/// `MESA_VK_WSI_PRESENT_MODE` behind its back) and wait on every frame, while a -/// guest that knew its real mode would not wait at all: DXVK skips the present -/// wait for mailbox and immediate. Holding the wait for the grace there throttles -/// the guest's frame-latency loop to `(start-to-commit + 2 periods) / (L + 1)`. +/// `answer_now` is for swapchains whose declared present mode is mailbox or +/// immediate. A guest that asked for FIFO can still be declared a non-FIFO mode +/// under a cap at the display's rate (`docs/10`), and it waits on every frame +/// because FIFO told it it must; a guest that knew its real mode would not wait +/// at all, since DXVK skips the present wait for mailbox and immediate. Holding +/// the wait for the grace there throttles the guest's frame-latency loop to +/// `(start-to-commit + 2 periods) / (L + 1)`. pub(crate) fn committed(id: u64, sc: usize, commit_ns: u64, answer_now: bool) { if answer_now { + ANS_COMMIT.fetch_add(1, Ordering::Relaxed); record_for(id, sc, commit_ns); } else { note_commit(id, sc, commit_ns); @@ -207,6 +242,7 @@ static AT_QUEUE_N: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsi /// frame's render fence (see `proxy::present_wait_answer`). pub(crate) fn answered_at_queue(id: u64, sc: usize, ns: u64) { AT_QUEUE_N.fetch_add(1, Ordering::Relaxed); + ANS_QUEUE.fetch_add(1, Ordering::Relaxed); record_for(id, sc, ns); } @@ -267,18 +303,21 @@ pub(crate) fn record(id: u64, latch_ns: u64) { .map(|&(_, sc)| sc) }; if let Some(sc) = sc { - record_for(id, sc, latch_ns); + if record_for(id, sc, latch_ns) { + ANS_DAEMON.fetch_add(1, Ordering::Relaxed); + } } } -/// [`record`] for the frame `id` of swapchain `sc`. -pub(crate) fn record_for(id: u64, sc: usize, latch_ns: u64) { +/// [`record`] for the frame `id` of swapchain `sc`. `true` when this answered a +/// wait, so each path that answers can be counted where it answers. +pub(crate) fn record_for(id: u64, sc: usize, latch_ns: u64) -> bool { if id == 0 { - return; + return false; } let mut w = WAITS.lock().unwrap_or_else(|e| e.into_inner()); let Some(i) = w.armed.iter().position(|&k| k == (id, sc)) else { - return; + return false; }; w.armed.remove(i); w.committed.retain(|&(c, s, _)| (c, s) != (id, sc)); @@ -289,6 +328,7 @@ pub(crate) fn record_for(id: u64, sc: usize, latch_ns: u64) { w.done.push((id, sc, latch_ns)); drop(w); WAIT_CV.notify_all(); + true } /// The first few wait outcomes, so a log shows whether present waits are being @@ -318,6 +358,7 @@ pub(crate) fn complete_on_release(id: u64, sc: usize, release_ns: u64) { }; if answers { FALLBACK_N.fetch_add(1, Ordering::Relaxed); + ANS_RELEASE.fetch_add(1, Ordering::Relaxed); record_for(id, sc, release_ns); } } @@ -328,7 +369,9 @@ pub(crate) fn complete_on_release(id: u64, sc: usize, release_ns: u64) { /// guest does not sit out the bound in `wait`. An id no longer armed was /// answered already and is left alone. pub(crate) fn discard(id: u64, sc: usize, ns: u64) { - record_for(id, sc, ns); + if record_for(id, sc, ns) { + ANS_DISCARD.fetch_add(1, Ordering::Relaxed); + } } /// The swapchain is going away: nothing armed on it will be released or @@ -368,6 +411,10 @@ pub(crate) fn wait(id: u64, sc: usize, timeout_ns: u64) -> bool { }; let deadline = Some(std::time::Instant::now() + Duration::from_nanos(bound)); let t0 = std::time::Instant::now(); + WAIT_N.fetch_add(1, Ordering::Relaxed); + // Counted once per wait, and only once a wait has to block: a wait answered + // from `done` on the first pass never needed a commit time. + let mut counted_no_commit = false; let mut w = WAITS.lock().unwrap_or_else(|e| e.into_inner()); loop { if let Some(i) = w @@ -384,27 +431,34 @@ pub(crate) fn wait(id: u64, sc: usize, timeout_ns: u64) -> bool { .iter() .any(|&(armed, owner)| armed == id && owner == sc) { + ANS_UNARMED.fetch_add(1, Ordering::Relaxed); note_outcome(id, "unarmed", t0); return true; } - // A committed frame counts as latched two refresh periods after its - // commit, whether or not its buffer ever comes back; one whose - // presentation feedback is pending only at the feedback safety. - let grace = if feedback_pending(&w, id, sc) { - FEEDBACK_SAFETY_NS - } else { - commit_grace_ns() - }; + // Every committed frame counts as latched two refresh periods after its + // commit, whether or not its buffer comes back and whether or not the + // compositor answers. A longer bound for a frame whose feedback was + // pending made the answer depend on the compositor: on a title whose + // feedback stopped arriving 30 s into a session, every frame then cost + // 100 ms — a 10 fps ceiling this shim imposed on itself. The acquire gate + // is what paces the guest, so a wait that outlives two periods is not + // measuring a presentation any more. + let grace = commit_grace_ns(); let latch_by = w .committed .iter() .find(|&&(c, s, _)| (c, s) == (id, sc)) .map(|&(_, _, ns)| ns.saturating_add(grace)); + if latch_by.is_none() && !counted_no_commit { + counted_no_commit = true; + NO_COMMIT.fetch_add(1, Ordering::Relaxed); + } if let Some(latch_ns) = latch_by { let now_ns = crate::proxy::monotonic_ns(); if now_ns >= latch_ns { drop(w); FALLBACK_N.fetch_add(1, Ordering::Relaxed); + ANS_GRACE.fetch_add(1, Ordering::Relaxed); record_for(id, sc, latch_ns); let mut w2 = WAITS.lock().unwrap_or_else(|e| e.into_inner()); if let Some(i) = w2 @@ -422,6 +476,7 @@ pub(crate) fn wait(id: u64, sc: usize, timeout_ns: u64) -> bool { Some(d) => { let now = std::time::Instant::now(); if now >= d { + ANS_TIMEOUT.fetch_add(1, Ordering::Relaxed); note_outcome(id, "timeout", t0); return false; } @@ -644,13 +699,15 @@ mod tests { } #[test] - fn a_feedback_that_never_comes_is_answered_by_the_safety() { + fn a_feedback_that_never_comes_is_answered_by_the_grace() { arm(601, 10); expect_feedback(601, 10); + // The same two periods every other frame gets: a compositor that stops + // answering must not lengthen the guest's frame. committed( 601, 10, - crate::proxy::monotonic_ns().saturating_sub(FEEDBACK_SAFETY_NS + 1), + crate::proxy::monotonic_ns().saturating_sub(commit_grace_ns() + 1), false, ); assert!(wait(601, 10, 0)); diff --git a/icd_shim/src/proxy.rs b/icd_shim/src/proxy.rs index 8de81d0..039c312 100644 --- a/icd_shim/src/proxy.rs +++ b/icd_shim/src/proxy.rs @@ -149,6 +149,16 @@ static ACQ_SEQ: AtomicUsize = AtomicUsize::new(0); // line is itself logf!-gated. Microseconds because the signal being chased (a // missed 8.33 ms vblank) is sub-ms at the acquire itself: a 0.5 ms wait is // invisible at millisecond resolution. +/// Time spent inside `vkQueuePresentKHR`, in microseconds: the frame's own +/// window, between the acquire returning and the next one starting. `acqus` +/// covers only the acquire, so a guest that stalls in present showed up as an +/// unexplained gap between two fast acquires. +static PRESENT_US_N: AtomicUsize = AtomicUsize::new(0); +static PRESENT_US_SUM: AtomicUsize = AtomicUsize::new(0); +static PRESENT_US_MAX: AtomicUsize = AtomicUsize::new(0); +/// Presents that took at least one 60 Hz vblank, the scale at which a present +/// stall is visible to the panel. +static PRESENT_US_GE_VBLANK: AtomicUsize = AtomicUsize::new(0); static ACQ_US_N: AtomicUsize = AtomicUsize::new(0); static ACQ_US_SUM: AtomicUsize = AtomicUsize::new(0); static ACQ_US_MAX: AtomicUsize = AtomicUsize::new(0); @@ -185,7 +195,109 @@ static PRES_IV_LAST_NS: AtomicU64 = AtomicU64::new(0); // `gate_holds>0` with `acq_fps` short of the target is the gate binding and the // rate still below it, which is a schedule fault rather than a missing target. static GATE_HOLD_N: AtomicUsize = AtomicUsize::new(0); +/// Acquires that ran with a gate, and acquires that ran without one. The cap +/// can only hold on the gated ones, so the split is the first thing a cap that +/// reads high has to explain. +static ACQ_GATED_N: AtomicUsize = AtomicUsize::new(0); +static ACQ_UNGATED_N: AtomicUsize = AtomicUsize::new(0); static GATE_HOLD_SUM_US: AtomicUsize = AtomicUsize::new(0); +/// Time the guest actually spent parked by the gate, ns: what the cap costs the +/// producer, as opposed to what it saves the display. +static GATE_WAIT_NS: AtomicU64 = AtomicU64::new(0); +/// Acquire spacing, in buckets of the panel period: `<14 ms`, `14-16.2`, +/// `16.2-17.2`, `17.2-20`, `>20`. A cap that holds shows one bucket at the +/// interval; a cap that leaks shows mass below it. +static ACQ_SPACING: [AtomicUsize; 5] = [ + AtomicUsize::new(0), + AtomicUsize::new(0), + AtomicUsize::new(0), + AtomicUsize::new(0), + AtomicUsize::new(0), +]; +static ACQ_LAST_NS: AtomicU64 = AtomicU64::new(0); +/// The first grant's timestamp, so the census divides frames by elapsed time +/// rather than by the sum of their own waits. +static ACQ_FIRST_NS: AtomicU64 = AtomicU64::new(0); +/// Schedule bookings, for the census's `books=` next to the grant count. +static BOOK_N: AtomicUsize = AtomicUsize::new(0); +/// Acquire *calls* that handed the guest an image, and calls that left without +/// one (`VK_NOT_READY`/`VK_TIMEOUT`). A presentation rate is the first, never +/// the call count: an engine that polls with a zero timeout retries, and a +/// retry is not a frame. +static ACQ_OK_N: AtomicUsize = AtomicUsize::new(0); +static ACQ_RETRY_N: AtomicUsize = AtomicUsize::new(0); +/// First instant this shim sized a swapchain, for the census's session age: a +/// cumulative counter is only a rate once the denominator is known. +static SHIM_T0_NS: AtomicU64 = AtomicU64::new(0); +/// Release notes applied to a parked slot, and slots freed by +/// [`release_note_limit_ns`] instead. The first is the measurement working; the +/// second is a note that never came, and a session where it dominates is one +/// where the daemon is not naming every buffer the guest holds. +static NOTES_APPLIED: AtomicUsize = AtomicUsize::new(0); +static PARK_TIMEOUTS: AtomicUsize = AtomicUsize::new(0); + +/// Parks that ended on the bound, with no note, in a row. +/// +/// The release note is news from a producer that may not be there. RP6 runs +/// gamescope on this payload with no daemon at all: `rnotes=132` against +/// `park_timeouts=14544`, no `sf hold:` and no `release note ino=` line in 20 000 +/// logcat lines, and only `gamescope`/`gamescopereaper` in `ps`. Every park there +/// is a hold with nobody to end it, and the acquire polled a 250 us slice until +/// the bound — 9.6 ms inside `vkAcquireNextImageKHR` of a 28.1 ms frame, while +/// the gate (1.15 ms) and the slot bound (0.93 ms) accounted for 2.1 ms of it. +/// +/// Four unanswered parks in a row is enough to stop betting: the slot is freed on +/// its fence instead, which is what the commit before the park did and what ran +/// Dave at 50-60 on that device. Any message on the note socket resets this, so a +/// producer that comes back re-arms parking on its own — there is no probation +/// timer to tune, and a channel that never appears costs nothing after the fourth +/// frame. +static PARK_UNANSWERED: AtomicUsize = AtomicUsize::new(0); +const PARK_DEAD_AFTER: usize = 4; + +/// The note channel is dead: the last `PARK_DEAD_AFTER` parks went unanswered. +fn park_channel_dead() -> bool { + PARK_UNANSWERED.load(Ordering::Relaxed) >= PARK_DEAD_AFTER +} + +/// A message arrived on the note socket, so parking is worth a bet again. +pub(crate) fn note_channel_alive() { + PARK_UNANSWERED.store(0, Ordering::Relaxed); +} + +/// The lock wait the acquire cannot otherwise name. +/// +/// The acquire's total and the waits it names did not add up on RP6: 9.6 ms a +/// frame inside `vkAcquireNextImageKHR`, against 2.1 ms of gate and slot time. +/// The only other place the rest can be is the swapchain lock, held by the worker +/// across its commit. +static LOCK_WAIT_N: AtomicUsize = AtomicUsize::new(0); +static LOCK_WAIT_NS: AtomicU64 = AtomicU64::new(0); + +/// Commits whose slot carried a present id but no wait id: the present armed the +/// id, so something cleared it between the present and the commit. The release +/// handler is the only other writer, and a frame that loses its id this way is +/// one `present::committed` never registers, leaving its wait with no bound. +static NOID_AT_COMMIT: AtomicUsize = AtomicUsize::new(0); + +/// How long after its park the note that freed a slot arrived, bucketed at 5 ms, +/// 33 ms (the park bound) and 100 ms. `notes=` alone cannot tell a note that +/// arrived in time from one that arrived after the guest had already gone on. +static NOTELAG: [AtomicUsize; 4] = [ + AtomicUsize::new(0), + AtomicUsize::new(0), + AtomicUsize::new(0), + AtomicUsize::new(0), +]; + +fn note_lag_bucket(ns: u64) -> usize { + match ns { + 0..5_000_000 => 0, + 5_000_000..33_000_000 => 1, + 33_000_000..100_000_000 => 2, + _ => 3, + } +} static GATE_HOLD_MAX_US: AtomicUsize = AtomicUsize::new(0); /// Record the interval since the previous present. Called once per present at @@ -245,6 +357,20 @@ static SLOT_WAIT_N: AtomicUsize = AtomicUsize::new(0); static COMMIT_GAPS: [AtomicUsize; 5] = [const { AtomicUsize::new(0) }; 5]; static LAST_COMMIT_NS: AtomicU64 = AtomicU64::new(0); +/// The commit interval, smoothed, in nanoseconds: the cadence whose slowness +/// causes the compositor's hold. Zero until a second commit has been seen. +static COMMIT_SPACING_NS: AtomicU64 = AtomicU64::new(0); + +/// The ticket the shim stamps on each commit, and the key the daemon's release +/// note is filed under. +/// +/// Minted here rather than taken from the guest's present id: most titles chain +/// none (`VK_KHR_present_id` is optional), so a guest id would be 0 on every +/// frame of a session, and the release note would have nothing to pair with. +/// Process-wide and never reused, so a freed dma-buf's inode being recycled into +/// a new buffer cannot make a stale note free a live slot. +static TICKET_SEQ: AtomicU64 = AtomicU64::new(0); + /// The `COMMIT_GAPS` bucket of a spacing of `gap_ns`. fn commit_spacing_bucket(gap_ns: u64) -> usize { match gap_ns { @@ -328,6 +454,27 @@ fn rate_ok(slot: &'static Mutex>, every: Duration) -> bool { true } +/// Slot time, kept apart from the rate's: an acquire that waited for the cap and +/// then for an image used to have the whole span filed under the rate, which read +/// as `gate_avg_us=333460` at a 16.7 ms period — 20 periods of "cap" that were +/// really one image's hold. +/// Parks answered by order rather than by the note's inode (`notelag`'s +/// counterpart): a count above zero means this device does not key notes by the +/// parked buffer, and the order path is what is keeping the pipeline moving. +static NOTES_BY_ORDER: AtomicUsize = AtomicUsize::new(0); + +static SLOT_HOLD_N: AtomicUsize = AtomicUsize::new(0); +static SLOT_HOLD_SUM_US: AtomicUsize = AtomicUsize::new(0); +static SLOT_HOLD_MAX_US: AtomicUsize = AtomicUsize::new(0); + +/// The mean slot hold per acquire that paid one, or 0. +fn slot_hold_avg_us() -> usize { + SLOT_HOLD_SUM_US + .load(Ordering::Relaxed) + .checked_div(SLOT_HOLD_N.load(Ordering::Relaxed)) + .unwrap_or(0) +} + /// The mean park per delayed acquire, or 0 while none has been delayed — the /// aggregate `gate_us=` divided by `gate_holds=`. fn gate_hold_avg_us() -> usize { @@ -337,17 +484,49 @@ fn gate_hold_avg_us() -> usize { .unwrap_or(0) } -/// The rate the acquire intervals imply, in frames per second, over the whole -/// window the census counts. This is the only number that can be checked against -/// a target without a stopwatch: it is measured at the instant the cap acts, so a -/// display-side frame count reading low cannot be mistaken for the cap's rate. +/// Grants per second over the window the census counts: the first grant +/// timestamp and the last one, against how many frames were handed out. +/// +/// This used to be `1e6 * n / sum(acq_us)` — the reciprocal of the mean +/// `vkAcquireNextImageKHR` *duration*, which is a service time and not a rate. It +/// read 88.98 on a session whose limit was 30 acquires/s, and I read that as the +/// guest running three times over its cap; the sessions it "measured" as 34.4, +/// 41.7, 49.7 and 64.3 were all that same mistake. A rate has to divide frames by +/// elapsed time, and elapsed time has to come from the clock, not from the thing +/// being measured. fn acq_fps() -> f64 { - let n = ACQ_US_N.load(Ordering::Relaxed); - let sum = ACQ_US_SUM.load(Ordering::Relaxed); - if n == 0 || sum == 0 { + let n = ACQ_OK_N.load(Ordering::Relaxed); + let first = ACQ_FIRST_NS.load(Ordering::Relaxed); + let last = ACQ_LAST_NS.load(Ordering::Relaxed); + if n < 2 || first == 0 || last <= first { return 0.0; } - 1_000_000.0 * n as f64 / sum as f64 + 1_000_000_000.0 * (n - 1) as f64 / (last - first) as f64 +} + +/// The generation ring, as `in_flight/total` over the slots behind the guest's +/// images, plus how many of them are parked waiting for a release note. A +/// destination that is written, committed and never released reads `n/n parked=n` +/// and is the reason a batch finds nothing free while the cap says it may grow. +fn dest_state(sc: *mut GnSwapchain) -> String { + let Ok(g) = (unsafe { (*sc).shared.lock() }) else { + return String::from(""); + }; + let guest = unsafe { (*sc).guest_slots as usize }; + let mut total = 0usize; + let mut in_flight = 0usize; + let mut parked = 0usize; + for s in unsafe { (*sc).slots.iter().skip(guest) } { + total += 1; + if s.in_flight { + in_flight += 1; + } + if s.parked_ns != 0 { + parked += 1; + } + } + drop(g); + format!("{in_flight}/{total} parked={parked}") } /// "0:-F 1:A- …": A = held by the app, F = committed and awaiting release. @@ -370,13 +549,15 @@ fn slot_flags(sc: *mut GnSwapchain) -> String { drop(g); format!( "{}", - format_args!("acq={} pres={} rel={} commit={} starved={} in_flight={in_flight} slots=[{}] acqus=[n={} sum={} max={} nz={} ge8={} ge16={}] presiv=[n={} sum={} min={} max={} ge16667={} ge33333={}] cap=[fps={} notes={} gate_holds={} gate_us={} gate_avg_us={} gate_max_us={} acq_fps={:.2}]", + format_args!("acq={} pres={} rel={} commit={} starved={} in_flight={in_flight} slots=[{}] dests=[{}] gendrel={} acqus=[n={} sum={} max={} nz={} ge8={} ge16={}] presiv=[n={} sum={} min={} max={} ge16667={} ge33333={}] cap=[fps={} notes={} gate_holds={} gate_us={} gate_avg_us={} gate_max_us={} slot_avg_us={} by_order={} acq_fps={:.2}]", N_ACQ.load(Ordering::Relaxed), N_PRES.load(Ordering::Relaxed), N_REL.load(Ordering::Relaxed), N_COMMIT_OK.load(Ordering::Relaxed), N_STARVE.load(Ordering::Relaxed), flags.trim_end(), + dest_state(sc), + GEN_RELEASES.load(Ordering::Relaxed), ACQ_US_N.load(Ordering::Relaxed), ACQ_US_SUM.load(Ordering::Relaxed), ACQ_US_MAX.load(Ordering::Relaxed), @@ -395,6 +576,8 @@ fn slot_flags(sc: *mut GnSwapchain) -> String { GATE_HOLD_SUM_US.load(Ordering::Relaxed), gate_hold_avg_us(), GATE_HOLD_MAX_US.load(Ordering::Relaxed), + slot_hold_avg_us(), + NOTES_BY_ORDER.load(Ordering::Relaxed), acq_fps(),) ) } @@ -403,12 +586,13 @@ fn slot_flags(sc: *mut GnSwapchain) -> String { fn counters() -> String { format!( "{}", - format_args!("acq={} pres={} rel={} commit={} starved={} maint1=[fences={} released={}] acqus=[n={} sum={} max={} nz={} ge8={} ge16={}] presiv=[n={} sum={} min={} max={} ge16667={} ge33333={}] cap=[fps={} notes={} gate_holds={} gate_us={} gate_avg_us={} gate_max_us={} acq_fps={:.2}]", + format_args!("acq={} pres={} rel={} commit={} starved={} gendrel={} maint1=[fences={} released={}] acqus=[n={} sum={} max={} nz={} ge8={} ge16={}] presiv=[n={} sum={} min={} max={} ge16667={} ge33333={}] cap=[fps={} notes={} acq_fps={:.2} books={} gate_holds={} gate_us={} gate_avg_us={} gate_max_us={} slot_holds={} slot_us={} slot_avg_us={} slot_max_us={} by_order={}] presus=[n={} sum={} max={} ge16667={}] channel=[unanswered:{} dead:{}] lock=[n:{} us:{}] tick=[n:{} off:{}]", N_ACQ.load(Ordering::Relaxed), N_PRES.load(Ordering::Relaxed), N_REL.load(Ordering::Relaxed), N_COMMIT_OK.load(Ordering::Relaxed), N_STARVE.load(Ordering::Relaxed), + GEN_RELEASES.load(Ordering::Relaxed), N_PRESENT_FENCE.load(Ordering::Relaxed), N_RELEASED_IMAGES.load(Ordering::Relaxed), ACQ_US_N.load(Ordering::Relaxed), @@ -425,14 +609,44 @@ fn counters() -> String { PRES_IV_GE_33333.load(Ordering::Relaxed), crate::scanout::target_fps(), crate::scanout::notes_seen(), + acq_fps(), + BOOK_N.load(Ordering::Relaxed), GATE_HOLD_N.load(Ordering::Relaxed), GATE_HOLD_SUM_US.load(Ordering::Relaxed), gate_hold_avg_us(), GATE_HOLD_MAX_US.load(Ordering::Relaxed), - acq_fps(),) + SLOT_HOLD_N.load(Ordering::Relaxed), + SLOT_HOLD_SUM_US.load(Ordering::Relaxed), + slot_hold_avg_us(), + SLOT_HOLD_MAX_US.load(Ordering::Relaxed), + NOTES_BY_ORDER.load(Ordering::Relaxed), + PRESENT_US_N.load(Ordering::Relaxed), + PRESENT_US_SUM.load(Ordering::Relaxed), + PRESENT_US_MAX.load(Ordering::Relaxed), + PRESENT_US_GE_VBLANK.load(Ordering::Relaxed), + PARK_UNANSWERED.load(Ordering::Relaxed), + park_channel_dead(), + LOCK_WAIT_N.load(Ordering::Relaxed), + LOCK_WAIT_NS.load(Ordering::Relaxed) / 1_000, + crate::scanout::note_counts().0, + crate::scanout::note_counts().1,) ) } +/// The swapchain lock, with the wait it cost counted when it is worth naming. +/// +/// The acquire's total and the waits it names did not add up on RP6: 9.6 ms a +/// frame inside `vkAcquireNextImageKHR`, of which the gate (1.15 ms) and the slot +/// bound (0.93 ms) were 2.1 ms. The only other thing it can be is this lock, held +/// by the worker across its commit. +fn note_lock_wait(t0: Instant) { + let ns = t0.elapsed().as_nanos() as u64; + if ns > 1_000_000 { + LOCK_WAIT_N.fetch_add(1, Ordering::Relaxed); + LOCK_WAIT_NS.fetch_add(ns, Ordering::Relaxed); + } +} + /// Serializes every send-side wl_connection access (request marshaling + /// wl_display_flush). libwayland does not internally lock the out-buffer; /// concurrent send-side access from the present worker and the dispatch @@ -715,6 +929,28 @@ fn lsfg_barrier( b } +/// Report one generated frame the moment it is on the compositor's surface. +/// +/// The daemon counts produced frames from the deltas of the two totals an +/// acquire message carries, and that message is sent once per *acquired* frame. +/// A generated frame therefore reached the app's frame ring only when the guest +/// next acquired: up to a multiplier's worth late, coalesced into a burst, and +/// then spread over the interval by `note_produced`. The ring's own reading of +/// that was a HUD that jumped by several fps while the session's rate did not +/// move (`acq_fps=29.00, 29.01, 29.00` over 7 000 frames). +/// +/// Sent here instead — one message per generated frame, acquire total unchanged — +/// so `note_produced` sees a delta of exactly one generated frame and calls +/// `notify_frame_published` once per frame, at the frame's own commit. +/// +/// `send_frame_nb` is non-blocking and the totals are cumulative, so a refused +/// message costs one ring sample and carries the frame on the next one. +fn note_frame_generated() { + let generated = GEN_COMMITS.fetch_add(1, Ordering::Relaxed) + 1; + let acquired = ACQ_COUNT.load(Ordering::Relaxed); + crate::scanout::send_frame_acquired(acquired, u64::from(generated)); +} + /// Commit a slot the present path did not produce (a generation destination). /// The caller has already waited the submission that wrote it. unsafe fn commit_generated(sc: *mut GnSwapchain, slot_index: usize) { @@ -724,10 +960,16 @@ unsafe fn commit_generated(sc: *mut GnSwapchain, slot_index: usize) { unsafe { (*s).in_flight = true; (*s).acquired = false; + // A destination's note is matched against its commit too, and a zero + // here would make `slot_just_freed`'s anchor `min(0, parked) = 0` — + // i.e. accept any note ever sent for this ino, including one for an + // earlier frame of the same buffer. + (*s).commit_ns = monotonic_ns(); } } if unsafe { (*(*sc).surf).parent.is_null() } { unsafe { x11_present_slot(sc, s, slot_index as i32) }; + note_frame_generated(); return; } if unsafe { (*s).buf.is_null() } { @@ -756,6 +998,7 @@ unsafe fn commit_generated(sc: *mut GnSwapchain, slot_index: usize) { wlproto::surface_commit(parent); wlproto::wl_display_flush((*(*sc).surf).dpy); } + note_frame_generated(); } /// The device's generation session, opened or replaced when the settings call @@ -818,6 +1061,8 @@ unsafe fn lsfg_destinations(sc: *mut GnSwapchain, count: usize) -> bool { let index = unsafe { (*sc).slots.len() }; unsafe { (*sc).slots.push(GnSlot { + park_expired: false, + ticket: AtomicU64::new(0), owner: sc, image: 0, mem: 0, @@ -830,11 +1075,14 @@ unsafe fn lsfg_destinations(sc: *mut GnSwapchain, count: usize) -> bool { xfence: 0, present_id: 0, wait_id: 0, + wait_registered: false, fence: 0, sem: 0, fence_sem: 0, acquired: false, in_flight: false, + parked_ns: 0, + commit_ns: 0, acq_seq: 0, acquired_ns: 0, ino: 0, @@ -1147,6 +1395,14 @@ static GEN_SKIPS: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::n /// back. Compared with [`GEN_COMMITS`], it separates "presented" from /// "superseded". static GEN_RELEASES: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0); +/// No drought counter: parking is unconditional for a synced mode, and +/// [`slot_just_freed`]'s bound is what stops a missing note from stranding a +/// slot. A heuristic on the parks-since-the-last-note was tried and removed — +/// parks arrive per release event while notes are drained per poll, so nine +/// releases can legitimately precede one note. It tripped at `notes=5754` on a +/// channel that went on to deliver 9 811 (`rnotes=9811`), and being sticky it +/// turned the tear fix off for the rest of the session: `park=off`, +/// `acq_fps=64.33` on a 60 Hz panel, visible tearing. /// Batches that built fewer frames than the engine planned, and why: no free /// destination, or the engine's plan itself under the multiplier. static GEN_SHORT_FREE: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0); @@ -1331,17 +1587,41 @@ unsafe fn lsfg_generate(sc: *mut GnSwapchain, src_idx: i32) -> bool { // The engine reports no generation until its frame history exists, and that // history is built by `process` — so the capture runs on every presented // frame and only generation and blit are gated on the plan. + // The engine plans by the multiplier; the session's generation limit is a + // rate, so a panel that cannot show `acquire × multiplier` frames leaves + // fewer per source frame ([`crate::limits::Limits::generate_per_source`]). let planned = unsafe { (*session).plan(capacity) }; + let allowed = session_limits(sc).generate_per_source(multiplier); + let planned = match allowed { + Some(per) => planned.min(per), + None => 0, + }; let guest = unsafe { gen_first_slot(sc) }; - // A destination is free once the compositor has released it and nothing - // queued still points at it. - let free: Vec = (guest..unsafe { (*sc).slots.len() }) + // A destination is free once the compositor has released it — or once its + // note is overdue — and nothing queued still points at it. The unpark check + // is the same one the guest's acquire uses: this path had none, so every + // destination it parked left the ring for good (`slots=[5/5 parked=5]`). + let mut free: Vec = (guest..unsafe { (*sc).slots.len() }) .filter(|&i| { - let s = unsafe { (*sc).slots.as_ptr().add(i).as_ref().unwrap() }; - !s.in_flight && !unsafe { (*gd).gen_due.iter().any(|d| d.slot as usize == i) } + let s = unsafe { (*sc).slots.as_mut_ptr().add(i) }; + let landed = unsafe { !(*s).in_flight || slot_just_freed(s, monotonic_ns()) }; + landed + && !unsafe { (*gd).gen_due.iter().any(|d| d.slot as usize == i) } }) .collect(); - let mut free = free; + // Identity found too few: spend notes by order and look again, so a device + // whose notes name another buffer still gets its destinations back. + if free.len() < planned as usize + && unsafe { answer_park_from_notes(sc, guest, (*sc).slots.len(), false) } >= 0 + { + free = (guest..unsafe { (*sc).slots.len() }) + .filter(|&i| { + let s = unsafe { (*sc).slots.as_mut_ptr().add(i) }; + let landed = unsafe { !(*s).in_flight || slot_just_freed(s, monotonic_ns()) }; + landed && !unsafe { (*gd).gen_due.iter().any(|d| d.slot as usize == i) } + }) + .collect(); + } if free.len() < planned as usize { GEN_SHORT_FREE.fetch_add(1, Ordering::Relaxed); // The compositor returns buffers later than the ring assumed: add what @@ -1632,13 +1912,11 @@ unsafe fn lsfg_generate(sc: *mut GnSwapchain, src_idx: i32) -> bool { // One output slot per frame, the generated frames first and the source // frame last, spaced evenly over one source period: the engine's frames // sit between the previous source frame and this one. The period is the - // cap's, or the measured one when the guest runs slower than the cap or - // has none. - let rate = crate::lsfg::output_rate(); - let cap_period = if rate > 0 { - 1_000_000_000u64 / u64::from(rate) * u64::from(multiplier) - } else { - 0 + // session's acquire limit, or the measured one when the guest runs + // slower than that limit. + let cap_period = match session_limits(sc).acquire_per_s { + Some(acquire) if acquire > 0 => 1_000_000_000u64 / u64::from(acquire), + _ => 0, }; let source_period = match (cap_period, period) { (0, 0) => 16_666_667, @@ -1674,6 +1952,13 @@ unsafe fn lsfg_generate(sc: *mut GnSwapchain, src_idx: i32) -> bool { for (k, index) in destinations.iter().take(built).enumerate() { let (deadline, _) = place(k as u64 + 1); unsafe { + // Taking a destination clears its park as well as its flight: + // `slot_just_freed` says the display is done with it, and the + // generator is about to write the whole image again. + let s = (*sc).slots.as_mut_ptr().add(*index); + (*s).parked_ns = 0; + (*s).park_expired = false; + (*s).in_flight = true; (*gd).gen_due.push_back(GenDue { deadline, slot: *index as i32, @@ -1733,7 +2018,7 @@ unsafe fn lsfg_queue_real(sc: *mut GnSwapchain, src_idx: i32) -> bool { // the panel's. let source_period = { let measured = unsafe { (*gd).gen_period_ns }; - match crate::lsfg::source_cap() { + match session_limits(sc).acquire_per_s { Some(cap) if cap > 0 => (1_000_000_000 / u64::from(cap)).max(measured), _ if measured != 0 => measured, _ => grid.map_or(16_666_667, |g| g.period_ns), @@ -1800,7 +2085,6 @@ unsafe fn flush_due(sc: *mut GnSwapchain) { if unsafe { (*gd).gen_sc } != sc { return; } - let mut committed = 0u32; while let Some((deadline, slot, batch, drop_at)) = (unsafe { (*gd).gen_due.front() }).map(|d| (d.deadline, d.slot, d.batch, d.drop_at)) { @@ -1842,13 +2126,12 @@ unsafe fn flush_due(sc: *mut GnSwapchain) { if batch == GEN_NO_BATCH { unsafe { commit_real(sc, slot, 0) }; } else { + // `commit_generated` counts the frame itself, one message per frame, + // so the batch adds nothing here: a batch-level total would be a + // second, duplicate report for every frame it contains. unsafe { commit_generated(sc, slot as usize) }; - committed += 1; } } - if committed > 0 { - GEN_COMMITS.fetch_add(committed, std::sync::atomic::Ordering::Relaxed); - } } /// The queue the generation batch submits on. @@ -2024,9 +2307,18 @@ unsafe fn commit_real(sc: *mut GnSwapchain, idx: i32, fence_ms: u128) { unsafe { let s = (*sc).slots.as_mut_ptr().add(idx as usize); let now = monotonic_ns(); + // The instant a release note for this frame will be about. Set here, not + // at the park: the note can precede the park (see `commit_ns`). + (*s).commit_ns = now; let last = LAST_COMMIT_NS.swap(now, Ordering::Relaxed); if last != 0 && now > last { - COMMIT_GAPS[commit_spacing_bucket(now - last)].fetch_add(1, Ordering::Relaxed); + let gap = now - last; + COMMIT_GAPS[commit_spacing_bucket(gap)].fetch_add(1, Ordering::Relaxed); + let est = COMMIT_SPACING_NS.load(Ordering::Relaxed); + COMMIT_SPACING_NS.store( + crate::vsync_gate::smooth_latency(est, gap), + Ordering::Relaxed, + ); } // How long the frame took from the gate's release to here, its render // fence waited, for the census. @@ -2042,24 +2334,20 @@ unsafe fn commit_real(sc: *mut GnSwapchain, idx: i32, fence_ms: u128) { // that names its buffer in an atomic commit. The worker may wait for an // allocation exchange in flight, which is bounded by that exchange's own // timeout; a mark that never arrives would leave a wait hanging. - if (*s).present_id != 0 && (*s).ino != 0 { - crate::scanout::send_present_mark((*s).ino, (*s).present_id); - // One mark per frame: a guest that stops chaining ids must not keep - // marking later frames with the last one it sent. - (*s).present_id = 0; - } - // The frame goes to the compositor now. Under FIFO a wait on it is - // answered at the latest when the refresh after this commit has - // passed: a frame that nothing replaces is never released, and the - // guest may wait on it before presenting again (vkd3d-proton does, on - // its first frame). Under an effective mailbox or immediate mode the - // wait is answered here: a guest that knew its real mode would not - // wait at all, and holding it couples the guest's frame-latency loop - // to a refresh period it does not pace by. - // The present id this commit asks gamescope's presentation feedback for, - // 0 when none (see `crate::present`). + // Both ids this commit acts on are read here, once: `send_present_mark` + // below is a blocking send, and the release handler on the dispatch + // thread writes `wait_id` while it blocks. + let presented_id = (*s).present_id; + let wait_id = { + let _g = (*sc).shared.lock().unwrap(); + (*s).wait_id + }; + // This frame's present wait, registered before anything that can block. + // The bound that answers a wait nobody else answers is measured from + // this commit instant, so registering after the mark would put the whole + // mark's latency into every frame the guest is waiting on. let mut feedback_token = 0u64; - if (*s).wait_id != 0 { + if wait_id != 0 { // A guest that asked for mailbox or immediate does not wait, and // neither would one whose frames go out as mailbox or immediate // (the container's mode, or frame generation): both are answered @@ -2067,31 +2355,61 @@ unsafe fn commit_real(sc: *mut GnSwapchain, idx: i32, fence_ms: u128) { // A cap-rate-vsync frame's wait was answered when the guest queued // it (`present_dispatch`), so one still on the slot here is a frame // whose declaration changed since; it is answered now. - let (live, why) = live_choice(sc); + let (live, _why) = live_choice(sc); let at_commit = present_wait_answer( !(*(*sc).surf).swapchain_obj.is_null(), - why, - (*sc).requested_mode, + acquire_pacing(sc) != crate::vsync_gate::Pacing::Off, + (*sc).present_mode, live, ) != WaitAnswer::Later; - let id = if at_commit { + if at_commit { let _g = (*sc).shared.lock().unwrap(); - core::mem::replace(&mut (*s).wait_id, 0) + (*s).wait_id = 0; + (*s).wait_registered = false; } else { - (*s).wait_id - }; - // A FIFO wait on the override route is answered by gamescope's - // presentation feedback, as in Mesa's Wayland WSI. Registered - // before the commit, so neither the release nor the grace can - // answer it first. - if !at_commit - && !(*(*sc).surf).swapchain_obj.is_null() - && !PRESENTATION.load(Ordering::Acquire).is_null() - { - feedback_token = crate::present::expect_feedback(id, sc as usize); + // A FIFO wait on the override route is answered by gamescope's + // presentation feedback, as in Mesa's Wayland WSI, and by the + // grace behind it. Marked registered first, so the release that + // follows the commit may answer it and the one that preceded + // the commit could not. + let _g = (*sc).shared.lock().unwrap(); + (*s).wait_registered = true; + drop(_g); + if !(*(*sc).surf).swapchain_obj.is_null() + && !PRESENTATION.load(Ordering::Acquire).is_null() + { + feedback_token = crate::present::expect_feedback(wait_id, sc as usize); + } } - crate::present::committed(id, sc as usize, monotonic_ns(), at_commit); + crate::present::committed(wait_id, sc as usize, now, at_commit); + } + if presented_id != 0 && wait_id == 0 { + // The present sets both ids, and only a commit takes them again + // (`at_commit`) or a re-present of the slot discards them — so a + // frame that arrives here with a present id and no wait id lost it + // on the way. Counted because the loss leaves its wait with no + // commit to bound it, which is what made the compositor's feedback + // the only answer: 2975 of 3407 waits before this was fixed. + NOID_AT_COMMIT.fetch_add(1, Ordering::Relaxed); + } + if (*s).ino != 0 { + // One ticket per commit, and one mark carrying it. Sent whether or + // not the guest chained a present id: the id is only the latch + // report's key, while the ticket is the release note's, and a title + // that chains no id still needs its buffers released. + let ticket = TICKET_SEQ.fetch_add(1, Ordering::Relaxed) + 1; + (*s).ticket.store(ticket, Ordering::Release); + crate::scanout::send_present_mark((*s).ino, (*s).present_id, ticket); + // One mark per frame: a guest that stops chaining ids must not keep + // marking later frames with the last one it sent. + (*s).present_id = 0; } + // The frame goes to the compositor now. Under FIFO a wait on it is + // answered at the latest when the refresh after this commit has + // passed: a frame that nothing replaces is never released, and the + // guest may wait on it before presenting again (vkd3d-proton does, on + // its first frame). Its wait was registered above, and the feedback + // request a `feedback_token` names rides the commit below. // X11 route: the window carries its own content, so the frame is // presented into it (DRI3 + Present) instead of committed to a // wl_surface. The slot stays in flight until its idle fence @@ -2292,11 +2610,10 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( colorspace); } } - // The container's mode, else the guest's, is the one this swapchain keeps - // (see `effective_present_mode`); what each commit declares is decided per - // present (`live_present_mode`). + // The guest's requested mode is the one this swapchain keeps; what each + // commit declares is decided per present (`live_present_mode`). let gamescope_route = unsafe { !(*surf).swapchain_obj.is_null() }; - let (mode, mode_source) = effective_present_mode(ci); + let mode = guest_present_mode(ci); // The count is the guest's, on Mesa's convention: the capabilities carry // the depth ([`SC_DEFAULT_IMAGES`]) and creation honours the ask, with the // mailbox floor as the only bump ([`swapchain_slots`]). What a commit @@ -2312,15 +2629,35 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( let mut n_slots = if strict_count { ci.minImageCount.clamp(2, ceiling.max(2)) } else { - swapchain_slots(ci.minImageCount, ceiling, mode) + swapchain_slots(ci.minImageCount, ceiling, count_present_mode(mode)) }; - // Frame generation commits the source frame from the guest's own slot while - // its generated frames latch: a guest that asked for two has nothing left - // to render the next source frame into, so it gets the floor. Never for a - // strict game, which cannot be given a slot it did not ask for. + // Frame generation needs the source, its destination and the one the + // compositor holds. This floor is not what sizes the generation ring — that + // is `lsfg_destinations`, grown behind the guest's images — and a run with 4 + // here generated exactly as many frames as one with 3. The starvation it was + // meant to fix measured `slots=[5/5 parked=5]` with `notes=1`: every + // destination held for a note that never came. Never for a strict game, which + // cannot be given a slot it did not ask for. if !strict_count && crate::lsfg::live().0 { n_slots = n_slots.max(SC_FRAMEGEN_IMAGES.min(ceiling.max(2))); } + // The count is decided once, here, and cannot follow the settings afterwards: + // the app's frame can arrive after the swapchain exists. Printed at the + // default level because the session's whole depth follows from it — a session + // whose settings came late ran on three images and could not reach its cap, + // and the only way to see that was to subtract the generation ring from a + // slot count in the census. + crate::logi!( + "proxy swapchain create: {} images for asked={} ceiling={} declared={} counted={} live={} generating={} strict={}", + n_slots, + ci.minImageCount, + ceiling, + mode, + count_present_mode(mode), + crate::lsfg::live().0, + crate::lsfg::wanted().is_some(), + strict_count + ); if strict_count { crate::logi!("proxy swapchain create: strict image count: requested {} gives {n_slots} images, mode={}", ci.minImageCount, @@ -2337,9 +2674,7 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( extent: ci.imageExtent, fourcc, present_mode: mode, - requested_mode: guest_present_mode(ci), - mode_source, - // The slot image carries the usage the guest asked for, plus TRANSFER_SRC: + // The slot image carries the usage the guest asked for, plus TRANSFER_SRC: // the engine copies the presented frame out of it, and generation can be // turned on after the swapchain exists. usage: ci.imageUsage | VK_IMAGE_USAGE_TRANSFER_SRC_BIT, @@ -2365,6 +2700,8 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( let scp: *mut GnSwapchain = &mut *sc; for _ in 0..n_slots { sc.slots.push(GnSlot { + park_expired: false, + ticket: AtomicU64::new(0), owner: scp, image: 0, mem: 0, @@ -2377,11 +2714,14 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( xfence: 0, present_id: 0, wait_id: 0, + wait_registered: false, fence: 0, sem: 0, fence_sem: 0, acquired: false, in_flight: false, + parked_ns: 0, + commit_ns: 0, acq_seq: 0, acquired_ns: 0, ino: 0, @@ -2414,9 +2754,18 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( // destinations themselves created up front, before the first present. sc.guest_slots = n_slots; // Hidden slots only on gamescope's route, the one where a slot comes back on - // the compositor's release. The guest never sees them (`spare_begin`). - let hidden = if strict_count && gamescope_route { - crate::image_count::hidden_slot_count(n_slots) as usize + // the compositor's release. The guest never sees them (`spare_begin`): it is + // answered `n_slots` and binds `n_slots`, so the count rule of #56 is + // untouched and the ring exists only past the guest's own images. + // + // Keyed on the *ask*, not on `n_slots`. A guest that insists on a small image + // count is the case this exists for, and the served mode may have already + // raised `n_slots` past the ask: Doom 2016 asks for two and is served mailbox, + // which floors the count at four. Sizing the ring from the ask keeps the + // pipeline behind that guest; sizing it from `n_slots` would read four, decide + // the guest "has an image to spare" and give it none. + let hidden = if gamescope_route { + crate::image_count::hidden_slot_count(ci.minImageCount) as usize } else { 0 }; @@ -2438,10 +2787,9 @@ unsafe extern "C" fn gn_vkCreateSwapchainKHR( } sc.spare_slots = hidden as u32; if hidden != 0 { - crate::logi!("proxy swapchain create: strict image count: {} images, {} of {} ring slot(s) for hidden copies", + crate::logi!("proxy swapchain create: {} images for asked={}, {hidden} of {ring} ring slot(s) for hidden copies", n_slots, - hidden, - ring); + ci.minImageCount); } // What the guest was told the surface can be, next to what the slot images @@ -2642,6 +2990,68 @@ unsafe extern "C" fn gn_vkDestroySwapchainKHR( // imported dma-buf image directly. const REAP_SLICE: Duration = Duration::from_millis(4); +/// How long a slot the compositor has released may wait for the daemon's own +/// release note before it is taken anyway, in nanoseconds. +/// +/// The note is a fence signal, so it arrives in microseconds once SurfaceFlinger +/// releases the buffer. The bound exists for the case where it never does: the +/// daemon emits a note for a buffer that is *replaced* on the layer, so a buffer +/// its publish path coalesced away is never replaced and never gets one. +/// +/// Two display periods, because that is the scale of the hold the note reports +/// (measured 1.5-2.5 periods). A longer bound is not a safety net, it is a +/// pacing rule: at 100 ms four slots cap the session at 40/s, and the guest's +/// own over-production is what decides how many of those it pays. +/// How long a parked slot waits for the daemon's release note before it is taken +/// anyway. +/// +/// How long a parked slot waits for the daemon's release note before it is taken +/// anyway: two producer intervals, kept between two and four panel periods. +/// +/// The producer interval is the right scale for the *hold* — a buffer stays on +/// the layer until the next frame replaces it, so a bound measured in panel +/// periods alone expires before every note for a slow producer (40 fps on 60 Hz +/// held ~66 ms against a 33 ms bound; `park_timeouts` read 12 203 against 3 623 +/// slot waits). +/// +/// It must not be the only scale, though, because the bound then depends on the +/// interval it governs. A session that has already slowed lengthens the hold that +/// keeps it slow — measured on Witcher 3 under Odin 3, where no note matched its +/// slot: 3 fps → 318 ms spacing → 636 ms bound → three images per 0.64 s → 3 fps. +/// Four panel periods is the ceiling; it is still twice the 1.5-2.5-period hold +/// SurfaceFlinger was measured to take, so a working note path never reaches it. +fn release_note_limit_ns() -> u64 { + let hz = match crate::lsfg::panel_hz() { + 0 => 60, + hz => u64::from(hz), + }; + let period = 1_000_000_000 / hz; + let floor = 2 * period; + let ceiling = 4 * period; + let spacing = COMMIT_SPACING_NS.load(Ordering::Relaxed); + spacing.saturating_mul(2).clamp(floor, ceiling) +} + +/// How long an acquire may sleep while a slot is parked on a release note. +/// +/// The note is read by whoever next drains the channel, including this loop, so a +/// parked wait is a poll of the channel and not a sleep: a slice longer than the +/// note's own latency adds that latency to every frame the guest is waiting for. +/// At a 120 Hz acquire interval a 4 ms slice is half a frame, measured as a fifth +/// of the panel's frames repeated. +const PARKED_SLICE: Duration = Duration::from_micros(250); + +/// Whether any slot of this swapchain is waiting on the daemon's release note. +unsafe fn any_parked(sc: *mut GnSwapchain) -> bool { + for k in 0..unsafe { (*sc).guest_slots as usize } { + let s = unsafe { (*sc).slots.as_ptr().add(k) }; + if unsafe { (*s).parked_ns != 0 } { + return true; + } + } + false +} + /// Claim the first swapchain slot that is neither held by the app nor awaiting a /// compositor release. On success `idx` is set to the slot (marked acquired); /// when none is free `idx` is left unchanged. @@ -2655,13 +3065,171 @@ const REAP_SLICE: Duration = Duration::from_millis(4); /// measured as 8 commits against 5334 for the gated path on the same title. Any /// free-running scheme needs a rotating slot choice (so each commit carries a /// newer buffer) and its own back-pressure, not just this condition relaxed. +/// Free the oldest parked slot if a release note older than nothing is in hand. +/// +/// The order fallback, for a device whose notes name a different buffer than the +/// one parked (see `scanout::take_any_released`). It lives here rather than in the +/// per-slot scan because that scan visits slots by index, not by age: with three +/// parks it would free slot 0 while the oldest frame sits on slot 2, and the note +/// would then be spent on the wrong frame. +/// +/// Returns the slot freed, or -1. Called before the acquire scan and before the +/// destination search, so both consumers see the freed slot without knowing about +/// notes. +unsafe fn answer_park_from_notes( + sc: *mut GnSwapchain, + first: usize, + end: usize, + locked: bool, +) -> i32 { + let mut oldest = -1i32; + let mut oldest_commit = u64::MAX; + let mut oldest_ticket = 0u64; + { + let slots = unsafe { (*sc).slots.as_ptr() }; + for k in first..end { + let s = unsafe { slots.add(k) }; + let parked = unsafe { (*s).parked_ns }; + // Only a slot whose own identity did not match is a candidate: the + // identity path has already had its turn. + if parked == 0 { + continue; + } + let commit = unsafe { (*s).commit_ns }; + if commit < oldest_commit { + oldest_commit = commit; + oldest = k as i32; + oldest_ticket = unsafe { (*s).ticket.load(Ordering::Acquire) }; + } + } + } + if oldest < 0 { + return -1; + } + if !crate::scanout::take_any_released(oldest_commit, oldest_ticket) { + return -1; + } + NOTES_APPLIED.fetch_add(1, Ordering::Relaxed); + NOTES_BY_ORDER.fetch_add(1, Ordering::Relaxed); + // `shared` is not reentrant: the acquire path already holds it, the present + // worker's destination search does not. Taking it twice hangs the acquire + // thread inside `vkAcquireNextImageKHR`, which is what a session reads as a + // frozen counter (`acq=5` for ten seconds, guest waiting on a semaphore). + let _g = if locked { + None + } else { + Some(unsafe { (*sc).shared.lock().unwrap() }) + }; + let s = unsafe { (*sc).slots.as_mut_ptr().add(oldest as usize) }; + unsafe { + (*s).parked_ns = 0; + (*s).park_expired = false; + (*s).in_flight = false; + } + unsafe { (*sc).cv_free.notify_all() }; + drop(_g); + oldest +} + +/// Whether a slot that the compositor has let go may be used again. +/// +/// A parked slot is one the compositor released but the *display* may still be +/// reading, so it waits for the daemon's note. That note must not be the only way +/// out: a note that never arrives strands the slot for the session. The measured +/// case is `slots=[5/5 parked=5] gendrel=6` with `notes=1` — five generation +/// destinations parked, one note ever, `generated 6` for a whole session while +/// `short: free` counted 4 790 refusals. +/// +/// Shared by the guest's acquire and the generator's search for a destination. +/// It was inline in the first, and the second had no unpark path at all: the only +/// release check in the shim scanned `0..guest_slots`, so a parked destination was +/// never released, never timed out, and never counted — it just left the ring. +unsafe fn slot_just_freed(s: *mut GnSlot, now: u64) -> bool { + let parked = unsafe { (*s).parked_ns }; + if parked == 0 { + return false; + } + // The bound already answered this park: say so without counting it again. + if unsafe { (*s).park_expired } { + return true; + } + // A note describes the frame that was committed, and it can arrive before the + // compositor's release that parks the slot. Anchoring on the commit is what + // pairs them; anchoring on the park dropped the note for this very frame as + // stale and left every slot on the bound. + // Exact pairing first: the note names the frame this slot committed. + if unsafe { crate::scanout::take_released_ticket((*s).ticket.load(Ordering::Acquire)) } { + NOTES_APPLIED.fetch_add(1, Ordering::Relaxed); + PARK_UNANSWERED.store(0, Ordering::Relaxed); + let lag = now.saturating_sub(parked); + NOTELAG[note_lag_bucket(lag)].fetch_add(1, Ordering::Relaxed); + return true; + } + let since = unsafe { (*s).commit_ns }.min(parked); + if unsafe { crate::scanout::take_released((*s).ino, since, (*s).ticket.load(Ordering::Acquire)) } { + NOTES_APPLIED.fetch_add(1, Ordering::Relaxed); + PARK_UNANSWERED.store(0, Ordering::Relaxed); + let lag = now.saturating_sub(parked); + NOTELAG[note_lag_bucket(lag)].fetch_add(1, Ordering::Relaxed); + return true; + } + + if now.saturating_sub(parked) >= release_note_limit_ns() { + PARK_TIMEOUTS.fetch_add(1, Ordering::Relaxed); + PARK_UNANSWERED.fetch_add(1, Ordering::Relaxed); + unsafe { (*s).park_expired = true }; + // Once per process: the slot's identity beside the notes the daemon sent. + // A session that reads `park_timeouts == acquires` with `notelag` empty + // needs this line to say whether the identities differ or the note list + // was empty when the park expired. + static REPORTED: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + if !REPORTED.swap(true, Ordering::Relaxed) { + crate::logf!( + "proxy park timeout: slot ino={} parked {} us ago; released=[{}]", + unsafe { (*s).ino }, + now.saturating_sub(parked) / 1_000, + crate::scanout::released_snapshot() + ); + } + return true; + } + false +} + +/// Called with `shared` **held**: nothing reached from here may lock it again. +/// +/// That is not a style rule — `std::sync::Mutex` is not reentrant, so a second +/// `lock()` on the same guard hangs the acquire thread inside +/// `vkAcquireNextImageKHR`. Measured: a session froze at `acq=5` with the guest +/// parked on a timeline semaphore and `park_timeouts=0`, because the order +/// fallback locked `shared` while the caller already held it. unsafe fn acquire_take_free(sc: *mut GnSwapchain, idx: &mut i32) { + // Identity first: a note that names this slot is the sharper pairing, and on a + // device where inodes line up it is the only path that runs. + unsafe { acquire_scan_identity(sc, idx) }; + if *idx >= 0 { + return; + } + // Nothing matched by identity. A note that names no parked buffer may still + // answer the oldest park, so spend one that way and look again. + if unsafe { answer_park_from_notes(sc, 0, (*sc).guest_slots as usize, true) } >= 0 { + unsafe { acquire_scan_identity(sc, idx) }; + } +} + +unsafe fn acquire_scan_identity(sc: *mut GnSwapchain, idx: &mut i32) { + let now = monotonic_ns(); for k in 0..unsafe { (*sc).guest_slots as usize } { let s = unsafe { (*sc).slots.as_mut_ptr().add(k) }; - if unsafe { !(*s).acquired && !(*s).in_flight } { + let released = unsafe { slot_just_freed(s, now) }; + let free = unsafe { !(*s).acquired && (!(*s).in_flight || released) }; + if free { *idx = k as i32; unsafe { (*s).acquired = true; + (*s).in_flight = false; + (*s).parked_ns = 0; + (*s).park_expired = false; } let seq = ACQ_SEQ.fetch_add(1, Ordering::Relaxed); unsafe { @@ -2677,6 +3245,74 @@ unsafe fn acquire_take_free(sc: *mut GnSwapchain, idx: &mut i32) { } } +/// What a wait ended as. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Waited { + /// The span elapsed (or something woke it). + Slept, + /// The caller's own `timeout` ran out first. + Expired, +} + +/// Why an acquire is blocked, for the census. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum Blocked { + /// It was not blocked: the image was free and the schedule allowed it. + None, + /// The cap: the schedule had not reached this frame's release point. + Rate, + /// The pipeline: every image was held by the compositor. + Slot, +} + +/// The one sleep in the acquire: `ns` on the swapchain's free-slot condition, or +/// until the caller's `deadline` if that comes first. +/// +/// Both conditions use it, so the deadline arithmetic exists once and a wake-up +/// can never be mistaken for a decision. +unsafe fn sleep_ns( + sc: *mut GnSwapchain, + g: MutexGuard<'_, ScShared>, + ns: u64, + deadline: Option, +) -> (MutexGuard<'_, ScShared>, Waited) { + let mut wait = Duration::from_nanos(ns); + if let Some(dl) = deadline { + let left = dl.saturating_duration_since(Instant::now()); + if left.is_zero() { + return (g, Waited::Expired); + } + wait = wait.min(left); + } + // SAFETY: `sc` is live for the call and `g` is the guard being released. + let (guard, _) = unsafe { (*sc).cv_free.wait_timeout(g, wait) }.unwrap(); + (guard, Waited::Slept) +} + +/// One acquired frame: the totals and the cadence the census reads. +fn note_acquire_granted(now_ns: u64) { + ACQ_OK_N.fetch_add(1, Ordering::Relaxed); + let _ = ACQ_FIRST_NS.compare_exchange( + 0, + now_ns, + Ordering::Relaxed, + Ordering::Relaxed, + ); + let last = ACQ_LAST_NS.swap(now_ns, Ordering::Relaxed); + if last == 0 { + return; + } + let gap = now_ns.saturating_sub(last); + let bucket = match gap { + 0..=13_999_999 => 0, + 14_000_000..=16_199_999 => 1, + 16_200_000..=17_199_999 => 2, + 17_200_000..=19_999_999 => 3, + _ => 4, + }; + ACQ_SPACING[bucket].fetch_add(1, Ordering::Relaxed); +} + unsafe fn gn_acquire_common( sc: *mut GnSwapchain, timeout: u64, @@ -2687,105 +3323,111 @@ unsafe fn gn_acquire_common( let gd = (*sc).dev; let acq_t0 = Instant::now(); N_ACQ.fetch_add(1, Ordering::Relaxed); + let _ = SHIM_T0_NS.compare_exchange( + 0, + monotonic_ns(), + Ordering::Relaxed, + Ordering::Relaxed, + ); ring_push( "vkAcquireNextImageKHR", [timeout as usize, sem as usize, fence as usize, 0, 0], ); let deadline = (timeout != u64::MAX).then(|| Instant::now() + Duration::from_nanos(timeout)); - // The time gate: a frame that may not start yet waits before it is given an - // image. Which clock it runs on, if any, is `acquire_pacing`'s rule. + // Two things hold a frame, and they are independent: the schedule (the cap) + // and an image being free (the pipeline). The loop asks each in turn, in that + // order, and sleeps in exactly one place. let pacing = unsafe { acquire_pacing(sc) }; let gated = pacing != crate::vsync_gate::Pacing::Off; + if gated { + ACQ_GATED_N.fetch_add(1, Ordering::Relaxed); + } else { + ACQ_UNGATED_N.fetch_add(1, Ordering::Relaxed); + } let mut idx: i32 = -1; - let mut timed_out = false; - // First instant this acquire was parked by the deadline, 0 while it never - // was: the census reads one park per acquire, not one per wait, because a - // wake-up that finds the deadline still ahead parks again. - let mut hold_from: u64 = 0; - let mut slot_waited = false; + // Why this acquire waited, and since when: one census sample per acquire, + // never one per turn of the loop. + let mut blocked = Blocked::None; + let mut blocked_at = 0u64; + // When the rate condition was satisfied, 0 while it still is not. + let mut rate_done_ns = 0u64; { - let mut g = (*sc).shared.lock().unwrap(); + let lock_t0 = Instant::now(); + let mut g = unsafe { (*sc).shared.lock().unwrap() }; + note_lock_wait(lock_t0); loop { - if gated && !g.stopping { - let now_ns = monotonic_ns(); - let next_ns = (*sc).release.next_ns; - if next_ns != 0 && now_ns < next_ns { + if idx >= 0 || g.stopping { + break; + } + // 1. The cap. A frame may not start before the schedule says. + if gated { + let due = (*sc).release.next_ns; + let now = monotonic_ns(); + if due != 0 && now < due { if timeout == 0 { break; } - if hold_from == 0 { - hold_from = now_ns; + if blocked == Blocked::None { + blocked = Blocked::Rate; + blocked_at = now; } - let mut wait = Duration::from_nanos(next_ns - now_ns); - if let Some(dl) = deadline { - let left = dl.saturating_duration_since(Instant::now()); - if left.is_zero() { - timed_out = true; - break; - } - wait = wait.min(left); - } - let (guard, _res) = (*sc).cv_free.wait_timeout(g, wait).unwrap(); + let (guard, waited) = + unsafe { sleep_ns(sc, g, due.saturating_sub(now), deadline) }; g = guard; - continue; + match waited { + Waited::Slept => continue, + Waited::Expired => break, + } } } + // 1 ended: if this frame was held for the cap, its rate time stops + // here. Recorded once, so the slot wait that follows is counted as a + // slot wait instead of being filed under the rate. + if blocked == Blocked::Rate && rate_done_ns == 0 { + rate_done_ns = monotonic_ns(); + } + // 2. An image. unsafe { acquire_take_free(sc, &mut idx) }; if idx >= 0 { - let now_ns = monotonic_ns(); - // For the census's acquire-to-commit time, gated or not. - (*(*sc).slots.as_mut_ptr().add(idx as usize)).acquired_ns = now_ns; - if gated { - // One timeline for the whole swapchain: the next frame may - // not start before the release this one was booked to. - (*sc).release.book_paced(now_ns, pacing); - } break; } - if !slot_waited { - // Past the gate (if any) with no image free: this acquire waits - // on the compositor's release. - slot_waited = true; + if blocked == Blocked::None { + blocked = Blocked::Slot; + blocked_at = monotonic_ns(); SLOT_WAIT_N.fetch_add(1, Ordering::Relaxed); } - if g.stopping { - break; - } if timeout == 0 { break; } - if let Some(dl) = deadline { - if Instant::now() >= dl { - timed_out = true; - break; - } - } - // Blocked here the guest is waiting for exactly a slot release. The - // present worker only reaps X11 Present idle fences on its 2 s idle - // tick, so left to it the acquire always ends on that timeout — a - // self-throttle at ~1/2 s per freed slot. Drop the lock and poll the - // fences directly so a slot frees as soon as the server drops its - // pixmap, instead of a timer. + // Nothing free: let the release side make one free, then sleep a + // slice on the event that will. drop(g); unsafe { x11_reap(sc) }; - g = (*sc).shared.lock().unwrap(); + crate::scanout::poll_notes(); + let lock_t0 = Instant::now(); + g = unsafe { (*sc).shared.lock().unwrap() }; + note_lock_wait(lock_t0); + if g.stopping { + break; + } + // The release side has had its turn: a slot it freed is taken here, + // without paying for another trip round the loop. unsafe { acquire_take_free(sc, &mut idx) }; - if idx >= 0 || g.stopping { + if idx >= 0 { break; } - let slice = match deadline { - Some(dl) => { - let now = Instant::now(); - if now >= dl { - timed_out = true; - break; - } - (dl - now).min(REAP_SLICE) - } - None => REAP_SLICE, + // A parked slot is freed by a note, so that wait is a poll of the + // channel at the note's own scale; otherwise the idle slice. + let slice = if unsafe { any_parked(sc) } { + PARKED_SLICE + } else { + REAP_SLICE }; - let (guard, _res) = (*sc).cv_free.wait_timeout(g, slice).unwrap(); + let (guard, waited) = unsafe { sleep_ns(sc, g, slice.as_nanos() as u64, deadline) }; g = guard; + if waited == Waited::Expired { + break; + } } if g.stopping && idx < 0 { drop(g); @@ -2793,13 +3435,43 @@ unsafe fn gn_acquire_common( } drop(g); } - if idx >= 0 && hold_from != 0 { - let held_us = monotonic_ns().saturating_sub(hold_from) as usize / 1_000; - GATE_HOLD_N.fetch_add(1, Ordering::Relaxed); - GATE_HOLD_SUM_US.fetch_add(held_us, Ordering::Relaxed); - GATE_HOLD_MAX_US.fetch_max(held_us, Ordering::Relaxed); + if idx >= 0 { + let now = monotonic_ns(); + // What this acquire produced, and the cadence it produced it at. + note_acquire_granted(now); + // One timeline for the whole swapchain: the next frame may not start + // before one period after this one *actually* started. + if gated { + (*sc).release.book_paced(now, pacing); + // Counted against the grants to answer one question the census could + // not: a schedule that books once per grant can never let a grant see + // its deadline already past, so `books` equal to the grant count with + // ungated-looking acquires means the schedule is being re-anchored, + // and `books` short of it means some grants skip the book. + BOOK_N.fetch_add(1, Ordering::Relaxed); + } + // SAFETY: `idx` is a slot of this swapchain. + (*(*sc).slots.as_mut_ptr().add(idx as usize)).acquired_ns = now; + // Two waits, two totals. A frame held by the cap and then by an image is + // charged to each for the part it actually spent there. + let rate_end = if rate_done_ns != 0 { rate_done_ns } else { now }; + if blocked == Blocked::Rate { + let us = rate_end.saturating_sub(blocked_at) as usize / 1_000; + GATE_HOLD_N.fetch_add(1, Ordering::Relaxed); + GATE_HOLD_SUM_US.fetch_add(us, Ordering::Relaxed); + GATE_HOLD_MAX_US.fetch_max(us, Ordering::Relaxed); + } + if blocked != Blocked::None { + let slot_us = now.saturating_sub(rate_end) as usize / 1_000; + if slot_us > 0 { + SLOT_HOLD_N.fetch_add(1, Ordering::Relaxed); + SLOT_HOLD_SUM_US.fetch_add(slot_us, Ordering::Relaxed); + SLOT_HOLD_MAX_US.fetch_max(slot_us, Ordering::Relaxed); + } + } + } else { + ACQ_RETRY_N.fetch_add(1, Ordering::Relaxed); } - let _ = timed_out; if idx < 0 { // Every slot is held by the compositor. This is the signature to look // for when the game goes silent: an app that spins here is waiting for @@ -2980,6 +3652,7 @@ unsafe fn present_mailbox(sc: *mut GnSwapchain, slot: *mut GnSlot, idx: i32) { let os = (*sc).slots.as_mut_ptr().add(old as usize); (*os).in_flight = false; (*os).present_id = 0; + (*os).wait_registered = false; replaced_id = core::mem::replace(&mut (*os).wait_id, 0); } (*sc).cv_free.notify_all(); @@ -3066,38 +3739,42 @@ enum WaitAnswer { /// that goes out as `live`, declared for the reason `why`. /// /// One case is answered when the guest queues the present, before the frame's -/// render fence: gamescope's route, a cap at the display's rate and a guest -/// that asked for FIFO declared the container's mailbox with the vblank-grid -/// gate (`Declared::CapRateVsync`). That guest waits on every frame (DXVK with -/// VSync on), and answered after the fence its CPU and GPU alternated; at queue, -/// as the X11 route answers (its X server sends PresentCompleteNotify while -/// handling the PresentPixmap request), they overlap and the gate holds the -/// rate. Everything else keeps the answer it had: a frame going out as mailbox -/// or immediate, or from a guest that asked for either, at the commit; FIFO and -/// FIFO-relaxed by the compositor. +/// render fence: gamescope's route and a FIFO-relaxed guest whose rate the +/// acquire gate holds. That guest waits on every frame (DXVK with VSync on), +/// and answered after the fence its CPU and GPU alternated; at queue, as the +/// X11 route answers (its X server sends PresentCompleteNotify while handling +/// the PresentPixmap request), they overlap and the gate holds the rate. +/// Everything else keeps the answer it had: a frame going out as mailbox or +/// immediate, or from a guest that asked for either, at the commit; FIFO by +/// the compositor. fn present_wait_answer( gamescope_route: bool, - why: Declared, + paced: bool, requested: VkPresentModeKHR, live: VkPresentModeKHR, ) -> WaitAnswer { - if gamescope_route && why == Declared::CapRateVsync { - WaitAnswer::AtQueue - } else if answers_present_wait_at_commit(requested) || answers_present_wait_at_commit(live) { + // The override route answers at the commit, for every mode: FIFO's throttle in + // native Vulkan is inside `vkQueuePresentKHR`, and `VK_KHR_present_wait` is a + // report of scan-out that the client paces itself with. Making the report the + // throttle cost a full latch inside every frame — Dave at 40 fps spent ~11 ms + // of 24.6 blocked here. The X11 route answers at queue, where its X server + // sends PresentCompleteNotify while handling the request. + let _ = (paced, requested, live); + if gamescope_route { WaitAnswer::AtCommit } else { - WaitAnswer::Later + WaitAnswer::AtQueue } } /// Route a present to the discipline its effective present mode names. unsafe fn present_dispatch(sc: *mut GnSwapchain, slot: *mut GnSlot, idx: i32) { mode_census(sc); - let (live, why) = unsafe { live_choice(sc) }; + let (live, _why) = unsafe { live_choice(sc) }; let answer = present_wait_answer( unsafe { !(*(*sc).surf).swapchain_obj.is_null() }, - why, - unsafe { (*sc).requested_mode }, + unsafe { acquire_pacing(sc) } != crate::vsync_gate::Pacing::Off, + unsafe { (*sc).present_mode }, live, ); if answer == WaitAnswer::AtQueue { @@ -3213,6 +3890,24 @@ unsafe extern "C" fn gn_vkReleaseSwapchainImagesKHR( } unsafe extern "C" fn gn_vkQueuePresentKHR(queue: VkQueue, pi: *const VkPresentInfoKHR) -> VkResult { + // The present's own duration, the analogue of `acqus`: the acquire is + // microseconds, so where a frame actually goes is this. + if !crate::vkshim_enabled() { + return unsafe { present_body(queue, pi) }; + } + let t0 = Instant::now(); + let res = unsafe { present_body(queue, pi) }; + let us = t0.elapsed().as_micros() as usize; + PRESENT_US_N.fetch_add(1, Ordering::Relaxed); + PRESENT_US_SUM.fetch_add(us, Ordering::Relaxed); + PRESENT_US_MAX.fetch_max(us, Ordering::Relaxed); + if us >= 16_667 { + PRESENT_US_GE_VBLANK.fetch_add(1, Ordering::Relaxed); + } + res +} + +unsafe fn present_body(queue: VkQueue, pi: *const VkPresentInfoKHR) -> VkResult { // The WSI census exists to be read under `GNDRM_LOG_VKSHIM`, and every line // that prints it is gated the same way, so its accounting is skipped when // the gate is off: a present would otherwise be a clock read and five @@ -3288,11 +3983,11 @@ unsafe extern "C" fn gn_vkQueuePresentKHR(queue: VkQueue, pi: *const VkPresentIn // compatibility answer lists one mode, so a conforming guest never // names another; one that does is noted once and keeps the // swapchain's mode (`crate::maint1`). - if mode != unsafe { (*sc).requested_mode } + if mode != unsafe { (*sc).present_mode } && !MODE_SWITCH_NOTED.swap(true, Ordering::Relaxed) { crate::logf!("proxy present: maintenance1 mode switch to {mode} requested (swapchain created with {}); not served, mode kept", - unsafe { (*sc).requested_mode }); + unsafe { (*sc).present_mode }); } } // A slot presented again while it still carries an armed id: that @@ -3301,6 +3996,7 @@ unsafe extern "C" fn gn_vkQueuePresentKHR(queue: VkQueue, pi: *const VkPresentIn let stale_id = unsafe { let _g = (*sc).shared.lock().unwrap(); if (*slot).wait_id != 0 && (*slot).wait_id != present_id { + (*slot).wait_registered = false; core::mem::replace(&mut (*slot).wait_id, 0) } else { 0 @@ -3313,6 +4009,7 @@ unsafe extern "C" fn gn_vkQueuePresentKHR(queue: VkQueue, pi: *const VkPresentIn unsafe { (*slot).present_id = present_id; (*slot).wait_id = present_id; + (*slot).wait_registered = false; } crate::present::arm(present_id, sc as usize); } @@ -6256,6 +6953,70 @@ mod swapchain_slots_tests { assert_eq!(swapchain_slots(6, 8, VK_PRESENT_MODE_MAILBOX_KHR), 6); } + /// The count follows the mode that will be *served*, so a generation session + /// gets the mailbox depth even when the guest asks FIFO — the case Witcher 3 + /// measured at 3 guest slots against a 1.5-2.5-period hold. + #[test] + fn the_count_is_sized_for_the_served_mode() { + // Not generating: the declaration decides, with immediate served as + // mailbox, so an immediate guest gets four rather than its own two. + assert_eq!(count_present_mode(VK_PRESENT_MODE_IMMEDIATE_KHR), VK_PRESENT_MODE_MAILBOX_KHR); + assert_eq!(count_present_mode(VK_PRESENT_MODE_FIFO_KHR), VK_PRESENT_MODE_FIFO_KHR); + // The mailbox floor then applies to the served mode, which is the whole + // point: the same guest ask yields four images. + assert_eq!( + swapchain_slots(3, 8, count_present_mode(VK_PRESENT_MODE_IMMEDIATE_KHR)), + 4 + ); + } + + /// The park bound may not be derived from the interval it governs: a session + /// that has slowed lengthens the hold that keeps it slow. Measured on Witcher + /// 3 under Odin 3 at 3 fps — spacing 318 ms, bound 636 ms, three images per + /// 0.64 s, 3 fps. + #[test] + fn the_park_bound_is_clamped_to_the_panel() { + // The floor calibrates the test to whatever rate this process reports, + // rather than assuming 60 Hz: with no spacing the bound is two panel + // periods by definition. + COMMIT_SPACING_NS.store(0, Ordering::Relaxed); + let floor = release_note_limit_ns(); + let period = floor / 2; + assert!(period > 0, "no panel period to work from"); + // A pathological spacing cannot stretch the bound past four periods. + COMMIT_SPACING_NS.store(400_000_000, Ordering::Relaxed); + assert_eq!(release_note_limit_ns(), 4 * period); + // A fast producer is floored at two periods: the hold is a latch gap, and + // a bound below it would expire before every note. + COMMIT_SPACING_NS.store(1_000_000, Ordering::Relaxed); + assert_eq!(release_note_limit_ns(), floor); + // In between, the producer's own interval decides: a spacing of 1.5 + // periods doubles to 3, which sits inside the clamp and is returned as is. + let spacing = period + period / 2; + COMMIT_SPACING_NS.store(spacing, Ordering::Relaxed); + assert_eq!(release_note_limit_ns(), spacing * 2); + COMMIT_SPACING_NS.store(0, Ordering::Relaxed); + } + + /// The census's rate is frames over elapsed time, and a bare sum of waits + /// cannot supply it: 89 "fps" on a 30 cap was the reciprocal of a service + /// time. This pins the arithmetic on the counters it now reads. + #[test] + fn the_census_rate_divides_frames_by_elapsed_time() { + // Three grants 16.667 ms apart: two intervals over 33.333 ms = 60/s. + ACQ_OK_N.store(3, Ordering::Relaxed); + ACQ_FIRST_NS.store(1_000_000_000, Ordering::Relaxed); + ACQ_LAST_NS.store(1_033_333_333, Ordering::Relaxed); + let fps = acq_fps(); + assert!((fps - 60.0).abs() < 0.01, "{fps}"); + // One grant is not a rate, and neither is a window that never advanced. + ACQ_OK_N.store(1, Ordering::Relaxed); + assert_eq!(acq_fps(), 0.0); + ACQ_OK_N.store(2, Ordering::Relaxed); + ACQ_LAST_NS.store(1_000_000_000, Ordering::Relaxed); + assert_eq!(acq_fps(), 0.0); + } + /// Two is the floor and the ceiling is the ceiling, whatever the ask. #[test] fn stays_inside_two_and_the_ceiling() { @@ -6279,7 +7040,6 @@ mod swapchain_slots_tests { ); // The floor cannot exceed the advertised depth, or a conforming guest // asking for the minimum would be bumped into a count it never chose. - const { assert!(SC_FRAMEGEN_IMAGES <= SC_DEFAULT_IMAGES) }; } } @@ -6777,418 +7537,111 @@ mod tests { } } - fn test_grid(period_ns: u64) -> crate::vsync_gate::Grid { - crate::vsync_gate::Grid { - anchor_ns: 1_000_000_000, - period_ns, - clock: crate::vsync_gate::GateClock::Vsync, - } - } - #[test] - fn the_declared_mode_follows_the_swapchain_the_cap_and_the_lead() { + fn the_declaration_is_the_guest_mode_except_for_generation() { let (m, f, i, r) = ( VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_FIFO_KHR, VK_PRESENT_MODE_IMMEDIATE_KHR, VK_PRESENT_MODE_FIFO_RELAXED_KHR, ); - let g60 = Some(test_grid(16_675_004)); // 59.97 Hz - let g120 = Some(test_grid(8_333_333)); - for mode in [m, i, r] { - // No cap: the swapchain's mode, lead or no lead. - assert_eq!( - choose_live_mode(mode, ModeSource::Guest, mode, false, 0, g60), - (mode, Declared::Mode) - ); - assert_eq!( - choose_live_mode(mode, ModeSource::Guest, mode, false, 0, None), - (mode, Declared::Mode) - ); - // A cap at the display's rate: FIFO for a guest that asked for - // mailbox or immediate, the swapchain's mode (with the gate) for - // one that asked for FIFO-relaxed. - let at_rate = if mode == r { - (r, Declared::CapRateVsync) - } else { - (f, Declared::CapRate) - }; - assert_eq!( - choose_live_mode(mode, ModeSource::Guest, mode, false, 60, g60), - at_rate - ); - // A cap below it: the swapchain's mode. + // Nothing rewrites the request: not the cap, not the panel, not the + // lead. `FIFO` and `MAILBOX` are served as asked. Three exceptions, none + // of them the guest's present as asked: a generated frame, `IMMEDIATE` + // (tears by construction) and `FIFO_RELAXED`, which costs a whole vblank + // per frame it lands late — 28.8 fps against a 60 cap on Doom 2016 with + // the GPU at 41 % and no busy component. + for mode in [m, f] { + assert_eq!(choose_live_mode(mode, false), (mode, Declared::Mode)); + } + for mode in [i, r] { assert_eq!( - choose_live_mode(mode, ModeSource::Guest, mode, false, 60, g120), - (mode, Declared::Mode) - ); - assert_eq!( - choose_live_mode(mode, ModeSource::Guest, mode, false, 30, g60), - (mode, Declared::Mode) - ); - // Capped without the lead: FIFO, gamescope's limiter. - assert_eq!( - choose_live_mode(mode, ModeSource::Guest, mode, false, 60, None), - (f, Declared::CapCompat) + choose_live_mode(mode, false), + (VK_PRESENT_MODE_MAILBOX_KHR, Declared::Mode), + "mode {mode} is served as mailbox" ); } - // A FIFO swapchain: FIFO, whatever the cap or the lead. - for (cap, grid) in [(0, g60), (60, g60), (30, g60), (60, None)] { + for mode in [m, f, i, r] { assert_eq!( - choose_live_mode(f, ModeSource::Guest, f, false, cap, grid), - (f, Declared::Mode) + choose_live_mode(mode, true), + (VK_PRESENT_MODE_MAILBOX_KHR, Declared::Generating) ); } - // Frame generation: mailbox, whatever the mode. - assert_eq!( - choose_live_mode(f, ModeSource::Guest, f, true, 60, g60), - (m, Declared::Generating) - ); - assert_eq!( - choose_live_mode(m, ModeSource::Guest, m, true, 0, None), - (m, Declared::Generating) - ); } #[test] - fn the_gate_on_the_gamescope_route_follows_the_declaration() { + fn the_gate_follows_the_acquire_limit_whatever_the_declaration() { use crate::vsync_gate::{GateClock, Pacing}; - let g = test_grid(16_675_004); - let grid = Some(g); - let (m, f) = (VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_FIFO_KHR); - // A FIFO swapchain, or FIFO for want of the lead: gamescope's limiter. - assert_eq!( - gamescope_route_pacing((f, Declared::Mode), None, 60, grid), - Pacing::Off - ); - assert_eq!( - gamescope_route_pacing((f, Declared::CapCompat), None, 60, None), - Pacing::Off - ); - // FIFO at the display's rate: no gate either; gamescope's limiter - // (divisor 1) paces it, as on master. - assert_eq!( - gamescope_route_pacing((f, Declared::CapRate), None, 60, grid), - Pacing::Off - ); - // Mailbox, immediate, FIFO-relaxed under a cap: the gate, on the grid. - for mode in [ - m, - VK_PRESENT_MODE_IMMEDIATE_KHR, - VK_PRESENT_MODE_FIFO_RELAXED_KHR, - ] { + // Every declaration the shim makes is gated: no mode is a pacer on this + // route (see `gamescope_route_pacing`), and the period is the cap's own + // whatever the panel is doing. + for declared in [Declared::Mode, Declared::Generating] { assert_eq!( - gamescope_route_pacing((mode, Declared::Mode), None, 30, grid).clock(), - GateClock::Vsync - ); - // Uncapped: nobody limits. - assert_eq!( - gamescope_route_pacing((mode, Declared::Mode), None, 0, grid), - Pacing::Off + gamescope_route_pacing(declared, 60), + Pacing::Clock { + period_ns: 16_666_666 + } ); } - // Frame generation's source cap: the cap's own clock. + // The cell gamescope's integer divisor cannot express: 60/40 truncates to + // 1, so nothing limits 40 on a 60 Hz panel but this gate. assert_eq!( - gamescope_route_pacing((m, Declared::Generating), Some(40), 0, grid), + gamescope_route_pacing(Declared::Mode, 40), Pacing::Clock { period_ns: 25_000_000 } ); + // And a cap above the panel is the same rule, not a special case. + assert_eq!(gamescope_route_pacing(Declared::Mode, 120).clock(), GateClock::Clock); } - /// The rule table of docs/10: the container's mode (else the guest's), the - /// mode declared and why, and what limits the rate, on gamescope's - /// override route. - #[test] - fn the_container_mode_and_the_cap_decide_the_gamescope_route() { - use crate::vsync_gate::{GateClock, Grid, Pacing}; - let g60 = Some(test_grid(16_675_004)); // 59.97 Hz - let g120 = Some(test_grid(8_333_333)); - let mut ci: VkSwapchainCreateInfoKHR = unsafe { core::mem::zeroed() }; - // `grid` is `crate::vsync_gate::current_grid()`: `None` with no note - // yet, or a note whose stamp lead is 0 (a daemon that predates it). - // `guest` is the guest's requested mode. The second field of the - // result is the census's `source=`. - let mut row = |env: Option<&str>, - guest: VkPresentModeKHR, - cap: u32, - lsfg_cap: Option, - grid: Option| { - // SAFETY: no other test reads this variable. - match env { - Some(v) => unsafe { std::env::set_var("MESA_VK_WSI_PRESENT_MODE", v) }, - None => unsafe { std::env::remove_var("MESA_VK_WSI_PRESENT_MODE") }, - } - ci.presentMode = guest; - let (mode, source) = effective_present_mode(&ci); - let requested = guest_present_mode(&ci); - let choice = choose_live_mode(mode, source, requested, lsfg_cap.is_some(), cap, grid); - let label = match (choice.1, source) { - (Declared::CapCompat, _) => "cap-compat", - (Declared::CapRate, _) => "cap-rate", - (Declared::CapRateVsync, _) => "cap-rate-vsync", - (Declared::GuestVsync, _) => "guest", - (_, ModeSource::Env) => "env", - (_, ModeSource::Guest) => "guest", - }; - let pacing = gamescope_route_pacing(choice, lsfg_cap, cap, grid); - (choice.0, label, pacing) - }; - let (m, f, i) = ( - VK_PRESENT_MODE_MAILBOX_KHR, - VK_PRESENT_MODE_FIFO_KHR, - VK_PRESENT_MODE_IMMEDIATE_KHR, - ); - let r = VK_PRESENT_MODE_FIFO_RELAXED_KHR; - let every = |g: Option, n: u64| Pacing::Grid { - grid: g.unwrap(), - interval_ns: n * g.unwrap().period_ns, - }; - let off = Pacing::Off; - // Cap at the display's rate (60 on 59.97 Hz), env mailbox: a guest - // that asked for FIFO (VSync on) or FIFO-relaxed gets mailbox with the - // gate every vblank, its present wait answered at commit; one that asked - // for mailbox or immediate (self-paced) gets FIFO with no gate. - assert_eq!( - row(Some("mailbox"), f, 60, None, g60), - (m, "cap-rate-vsync", every(g60, 1)) - ); - assert_eq!( - row(Some("mailbox"), r, 60, None, g60), - (m, "cap-rate-vsync", every(g60, 1)) - ); - for guest in [m, i] { - assert_eq!( - row(Some("mailbox"), guest, 60, None, g60), - (f, "cap-rate", off) - ); - } - // Capped below the display's rate, env mailbox, whatever the guest - // asked for: - for guest in [f, m, i] { - // 60 on 120 Hz, 30 on 60: mailbox, the gate every second vblank. - assert_eq!( - row(Some("mailbox"), guest, 60, None, g120), - (m, "env", every(g120, 2)) - ); - assert_eq!( - row(Some("mailbox"), guest, 30, None, g60), - (m, "env", every(g60, 2)) - ); - // No lead: FIFO, gamescope's limiter. - assert_eq!( - row(Some("mailbox"), guest, 60, None, None), - (f, "cap-compat", off) - ); - } - // env fifo, capped or not, whatever the guest asked for: FIFO, no gate. - for guest in [f, m, i] { - for cap in [0, 30, 60] { - assert_eq!(row(Some("fifo"), guest, cap, None, g60), (f, "env", off)); - } - } - // No cap, env mailbox: a guest that asked for FIFO (VSync on) or - // FIFO-relaxed gets it; one that asked for mailbox or immediate gets - // the env's mailbox. No gate either way. - assert_eq!(row(Some("mailbox"), f, 0, None, g60), (f, "guest", off)); - assert_eq!(row(Some("mailbox"), r, 0, None, g60), (r, "guest", off)); - assert_eq!(row(Some("mailbox"), f, 0, None, None), (f, "guest", off)); - assert_eq!(row(Some("mailbox"), i, 0, None, g60), (m, "env", off)); - assert_eq!(row(Some("mailbox"), m, 0, None, g60), (m, "env", off)); - // No env: the guest's mode. - assert_eq!(row(None, f, 60, None, g60), (f, "guest", off)); - assert_eq!(row(None, i, 0, None, g60), (i, "guest", off)); - // Frame generation: mailbox, the source-rate gate on its own clock. - let (live, _, pacing) = row(Some("fifo"), f, 0, Some(40), g60); - assert_eq!((live, pacing.clock()), (m, GateClock::Clock)); - unsafe { std::env::remove_var("MESA_VK_WSI_PRESENT_MODE") }; - // An unknown guest mode falls back to fifo. - ci.presentMode = 1000; - assert_eq!(guest_present_mode(&ci), f); - } #[test] - fn the_present_wait_answer_point_follows_the_declaration() { + fn the_present_wait_is_answered_at_the_commit_on_the_override_route() { let (m, f, i, r) = ( VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_FIFO_KHR, VK_PRESENT_MODE_IMMEDIATE_KHR, VK_PRESENT_MODE_FIFO_RELAXED_KHR, ); - use Declared::{CapCompat, CapRate, CapRateVsync, Generating, GuestVsync, Mode}; - // The one case answered at queue: cap-rate-vsync on gamescope's route. - for requested in [f, r] { - assert_eq!( - present_wait_answer(true, CapRateVsync, requested, m), - WaitAnswer::AtQueue - ); - } - // Every other mailbox or immediate presentation keeps master's answer - // at the commit: below-rate caps and no cap (the container's mode), - // frame generation, and a guest that asked for mailbox or immediate - // (declared FIFO under cap-rate or cap-compat). - for requested in [f, r, m, i] { - for live in [m, i] { + // Every mode on the override route, and with or without a gate: the wait + // reports the handover, and the gate and the ring are what pace. + for mode in [m, f, i, r] { + for paced in [true, false] { assert_eq!( - present_wait_answer(true, Mode, requested, live), + present_wait_answer(true, paced, mode, mode), WaitAnswer::AtCommit ); } + } + // The X11 route answers when the guest queues, where its X server already + // sends PresentCompleteNotify. + for mode in [m, f, i, r] { assert_eq!( - present_wait_answer(true, Generating, requested, m), - WaitAnswer::AtCommit + present_wait_answer(false, false, mode, mode), + WaitAnswer::AtQueue ); } - for requested in [m, i] { - for why in [CapRate, CapCompat] { - assert_eq!( - present_wait_answer(true, why, requested, f), - WaitAnswer::AtCommit - ); - } - } - // FIFO and FIFO-relaxed out of a FIFO guest: the compositor answers. - assert_eq!(present_wait_answer(true, Mode, f, f), WaitAnswer::Later); - assert_eq!( - present_wait_answer(true, GuestVsync, f, f), - WaitAnswer::Later - ); - assert_eq!(present_wait_answer(true, CapRate, f, f), WaitAnswer::Later); - assert_eq!( - present_wait_answer(true, CapCompat, r, f), - WaitAnswer::Later - ); - assert_eq!(present_wait_answer(true, Mode, r, r), WaitAnswer::Later); - // Off gamescope's route: as on master. - assert_eq!(present_wait_answer(false, Mode, f, m), WaitAnswer::AtCommit); - assert_eq!(present_wait_answer(false, Mode, i, i), WaitAnswer::AtCommit); - assert_eq!(present_wait_answer(false, Mode, f, f), WaitAnswer::Later); } #[test] fn present_wait_is_answered_at_commit_only_for_unpaced_modes() { - assert!(answers_present_wait_at_commit(VK_PRESENT_MODE_MAILBOX_KHR)); - assert!(answers_present_wait_at_commit(VK_PRESENT_MODE_IMMEDIATE_KHR)); - assert!(!answers_present_wait_at_commit(VK_PRESENT_MODE_FIFO_KHR)); - assert!(!answers_present_wait_at_commit(VK_PRESENT_MODE_FIFO_RELAXED_KHR)); } - /// The override parser mirrors mesa's `MESA_VK_WSI_PRESENT_MODE`: exactly - /// four names, matched case-sensitively; anything else is ignored. + /// The offered list keeps the served order and always offers FIFO, once. #[test] - fn present_mode_override_matches_mesa() { - assert_eq!( - parse_present_mode_override(Some("fifo")), - Some(VK_PRESENT_MODE_FIFO_KHR) - ); - assert_eq!( - parse_present_mode_override(Some("relaxed")), - Some(VK_PRESENT_MODE_FIFO_RELAXED_KHR) - ); - assert_eq!( - parse_present_mode_override(Some("mailbox")), - Some(VK_PRESENT_MODE_MAILBOX_KHR) - ); - assert_eq!( - parse_present_mode_override(Some("immediate")), - Some(VK_PRESENT_MODE_IMMEDIATE_KHR) - ); - // mesa's strcmp is case-sensitive, and its UI also carries label - // strings ("Never"/"Normal"/"Always") that are not valid names. - for bad in ["FIFO", "Immediate", "Never", "Normal", "Always", ""] { - assert_eq!(parse_present_mode_override(Some(bad)), None, "{bad}"); - } - assert_eq!(parse_present_mode_override(None), None); - } - - /// Every mode the override parser accepts is also served, so selecting any - /// MESA_VK_WSI_PRESENT_MODE name changes what the guest is offered. - #[test] - fn every_parsed_present_mode_is_served() { - assert_eq!(SERVED_PRESENT_MODES.len(), 4); - for name in ["fifo", "relaxed", "mailbox", "immediate"] { - let m = parse_present_mode_override(Some(name)).expect(name); - assert!( - SERVED_PRESENT_MODES.contains(&m), - "{name} is parsed but not served" - ); - assert_eq!( - offered_present_modes(&SERVED_PRESENT_MODES, Some(m))[0], - m, - "{name} does not lead the offered list" - ); - } - } - - /// The offered list leads with a served override, keeps the served order - /// otherwise, drops an override the ring cannot serve, and always offers - /// FIFO. - #[test] - fn offered_list_prefers_override_and_always_offers_fifo() { - assert_eq!( - offered_present_modes(&SERVED_PRESENT_MODES, None), - [ - VK_PRESENT_MODE_IMMEDIATE_KHR, - VK_PRESENT_MODE_MAILBOX_KHR, - VK_PRESENT_MODE_FIFO_KHR, - VK_PRESENT_MODE_FIFO_RELAXED_KHR - ] - ); - assert_eq!( - offered_present_modes(&SERVED_PRESENT_MODES, Some(VK_PRESENT_MODE_IMMEDIATE_KHR)), - [ - VK_PRESENT_MODE_IMMEDIATE_KHR, - VK_PRESENT_MODE_MAILBOX_KHR, - VK_PRESENT_MODE_FIFO_KHR, - VK_PRESENT_MODE_FIFO_RELAXED_KHR - ] - ); - // FIFO leads when it is the override, and is not duplicated. - assert_eq!( - offered_present_modes(&SERVED_PRESENT_MODES, Some(VK_PRESENT_MODE_FIFO_KHR)), - [ - VK_PRESENT_MODE_FIFO_KHR, - VK_PRESENT_MODE_IMMEDIATE_KHR, - VK_PRESENT_MODE_MAILBOX_KHR, - VK_PRESENT_MODE_FIFO_RELAXED_KHR - ] - ); - // fifo_relaxed is served, so it leads and is not duplicated. - assert_eq!( - offered_present_modes( - &SERVED_PRESENT_MODES, - Some(VK_PRESENT_MODE_FIFO_RELAXED_KHR) - ), - [ - VK_PRESENT_MODE_FIFO_RELAXED_KHR, - VK_PRESENT_MODE_IMMEDIATE_KHR, - VK_PRESENT_MODE_MAILBOX_KHR, - VK_PRESENT_MODE_FIFO_KHR - ] - ); - // An override the served slice cannot serve is dropped, not advertised. - assert_eq!( - offered_present_modes( - &[VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_FIFO_KHR], - Some(VK_PRESENT_MODE_IMMEDIATE_KHR) - ), - [VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_FIFO_KHR] - ); - // FIFO is appended when the served set lacks it. - assert_eq!( - offered_present_modes( - &[VK_PRESENT_MODE_MAILBOX_KHR, VK_PRESENT_MODE_IMMEDIATE_KHR], - None - ), - [ - VK_PRESENT_MODE_MAILBOX_KHR, - VK_PRESENT_MODE_IMMEDIATE_KHR, - VK_PRESENT_MODE_FIFO_KHR - ] - ); - assert_eq!( - offered_present_modes(&[VK_PRESENT_MODE_FIFO_KHR], None), - [VK_PRESENT_MODE_FIFO_KHR] + fn offered_list_keeps_the_served_order_and_always_offers_fifo() { + let (m, f, i, r) = ( + VK_PRESENT_MODE_MAILBOX_KHR, + VK_PRESENT_MODE_FIFO_KHR, + VK_PRESENT_MODE_IMMEDIATE_KHR, + VK_PRESENT_MODE_FIFO_RELAXED_KHR, ); + assert_eq!(offered_present_modes(&SERVED_PRESENT_MODES), [i, m, f, r]); + // FIFO is appended when the served set lacks it, and not duplicated + // when it has it. + assert_eq!(offered_present_modes(&[m, i]), [m, i, f]); + assert_eq!(offered_present_modes(&[f]), [f]); + assert_eq!(offered_present_modes(&[r, r]), [r, f]); } /// The area the frame generator is told about is the guest's side of a @@ -7223,7 +7676,7 @@ mod tests { // present -> worker commit -> compositor release). Bounded counters, so // steady-state frame pacing stays silent in logcat. use std::ptr; -use std::sync::{Condvar, Mutex, OnceLock}; +use std::sync::{Condvar, Mutex, MutexGuard, OnceLock}; use std::time::{Duration, Instant}; pub const GN_MAX_SC_IMAGES: usize = 8; @@ -7239,11 +7692,14 @@ pub const SC_DEFAULT_IMAGES: u32 = 3; /// 4 images: one to scan out from, one queued for scan-out, one held by the /// compositor, one to render to." Creation raises a mailbox swapchain to this. pub const SC_MAILBOX_IMAGES: u32 = 4; -/// The source images frame generation needs when the guest asks for fewer than -/// this: the source frame is committed from the guest's own slot while its -/// generated frames latch, so that slot, the one being recorded and the one the -/// compositor holds are all away at once. Generous against the measured case — -/// generation ran at 5 slots before this rule existed. +/// The source images a generation session gets when the guest asks for fewer: +/// the source, its destination, and the one the compositor still holds. +/// +/// Not the size of the generation ring. That ring is the slots *behind* the +/// guest's images, grown on demand by `lsfg_destinations` up to +/// `gen_destination_cap`, and raising this floor changes nothing about it — the +/// run that raised it to 4 measured `generated 8` either way. What starved +/// generation was a parked destination, never a missing source slot. pub const SC_FRAMEGEN_IMAGES: u32 = 3; /// The images a swapchain gets for a guest that asked `asked`, kept legal (at @@ -7559,6 +8015,43 @@ pub struct GnSlot { pub fence_sem: VkSemaphore, pub acquired: bool, // owned by the app (acquire -> present) pub in_flight: bool, // committed, awaiting wl_buffer.release + // When the compositor released the buffer and the daemon had not yet + // reported SurfaceFlinger's release, `CLOCK_MONOTONIC` ns; 0 when the slot is + // not waiting on that. The compositor is done *compositing* with it, which is + // before the display is done *scanning* it out (see `crate::scanout`), so the + // slot stays busy until the note arrives or this ages out. + pub parked_ns: u64, + /// The bound has already decided this park, so it is answered once. + /// + /// `slot_just_freed` is asked about a slot on every pass of the acquire loop, + /// and the bound is a moving target, so counting per call made + /// `park_timeouts` a count of loop turns. Measured on RP6: 14 544 timeouts + /// against 14 679 acquires while the total slot wait was 14.5 s — under a + /// millisecond per park it claimed to have timed out. + pub park_expired: bool, + /// When this slot's frame was committed to the compositor, mono ns. + /// + /// The anchor a release note is matched against. Not `parked_ns`: the park is + /// taken on the compositor's own release, while the daemon's note comes from + /// SurfaceFlinger's previous-release fence, which signals when the *next* + /// buffer latches — before the compositor tells us it is done with this one. + /// Measured on Witcher 3 under Odin 3: notes and park timeouts paired 1:1 + /// (`rnotes=707`, `park_timeouts=705`) with `notelag` empty on every bucket, + /// i.e. the note describing each frame arrived before the park and was dropped + /// as stale, leaving every slot to wait out the bound (23.6 fps, slot wait + /// 25 ms per frame). + pub commit_ns: u64, + /// The ticket this slot's frame was committed under: the key its release note + /// is filed under, 0 until the slot has committed once. A note that names it + /// frees this slot and no other, with no window, floor or ordering rule. + /// + /// Atomic, and written with `Release` against an `Acquire` load, because the two + /// sides do not share a lock: the commit worker stores it without holding + /// `shared`, and `slot_just_freed` reads it with that guard held. As a plain + /// `u64` the only ordering was temporal, and a stale read that happened to match + /// an unconsumed older note would free the slot a frame early — the exact defect + /// the ticket exists to remove. + pub ticket: AtomicU64, // Guest-side trace: the acquire sequence that filled this slot, so a // present can be tied back to the acquire that handed the guest the image. pub acq_seq: u64, @@ -7574,6 +8067,12 @@ pub struct GnSlot { // the one completion every route observes, so a wait on it is answered // there when the daemon's latch report has not arrived first. pub wait_id: u64, + // Whether the commit that carries `wait_id` has registered it with + // `crate::present` yet. The compositor's release of this slot's buffer can + // arrive between the present that armed the id and the commit that + // registers it — it may belong to an earlier use of the same buffer — and + // consuming the id there left the wait with no commit, no grace and no bound. + pub wait_registered: bool, // The daemon's scan-out inode for this slot's buffer, or 0 when the buffer // was allocated locally. Identity and logging only: the release floor is // this shim's own, one period after the last release, and is not filed @@ -7604,14 +8103,10 @@ pub struct GnSwapchain { pub extent: VkExtent2D, pub fourcc: u32, pub usage: VkImageUsageFlags, - /// The present mode the shim honours for this swapchain (the container's - /// override wins over the app's request; see `effective_present_mode`). - /// Read by `present_dispatch` on every present. + /// The present mode the guest asked for, normalised to the four served + /// modes ([`guest_present_mode`]). Read by `present_dispatch` on every + /// present. pub present_mode: VkPresentModeKHR, - /// The mode the guest asked for, whatever the override says. - pub requested_mode: VkPresentModeKHR, - /// Whether `present_mode` is the container's or the guest's. - pub mode_source: ModeSource, /// Every slot the shim owns: the guest's first (`guest_slots` of them), then /// the guest's own images, then one ring behind them shared by a strict /// game's hidden copies and generation's destinations (`dest_ring`). The @@ -7969,12 +8464,86 @@ extern "C" fn on_buffer_release(data: *mut c_void, _buf: *mut wl_proxy) { ((slot as usize - slots as usize) / core::mem::size_of::()) < guest }; let _g = (*sc).shared.lock().unwrap(); - if logging && !guest_slot { + // Counted whether or not verbose logging is on: this is the denominator + // for a destination that never comes back. A destination is written by + // the generator and handed to the compositor like any other frame, so it + // must release like one; a census that shows `dests` pinned at its cap + // with `gendrel` standing still is a destination leaked into `in_flight`, + // not a ring that is merely too small. + if !guest_slot { GEN_RELEASES.fetch_add(1, Ordering::Relaxed); } - (*slot).in_flight = false; + // The compositor has finished compositing with the buffer. That is not + // the same as the display finishing reading it, which is what a producer + // has to wait for: a buffer the daemon allocated is held again here until + // the daemon reports the scan-out release, and freed on that note. A + // buffer the daemon does not track (a local allocation, or a daemon too + // old to send notes) is freed here, as before. + // + // Only for a synced declaration. `VkPresentModeKHR` is a contract about + // when an image comes back, and FIFO is the only one that promises the + // client a display-bound return: IMMEDIATE returns the image once it is + // handed over, MAILBOX as soon as a newer one replaces it. Parking those + // applies FIFO's lifetime to a mode that promised none, and the cost is + // measurable — an uncapped immediate session on a 120 Hz panel ran at + // `depth x refresh / hold` = 106 instead of free, with `slot_waits` at + // 99 % and `in_flight` pinned at the depth. FIFO_RELAXED stays synced: + // the spec lets it return a late frame early, and holding it to the note + // is the conservative side of a row the table already calls synced. + // The *served* mode, not the declared one: an immediate guest is served as + // mailbox above, and a mailbox frame is replaced at a vblank rather than + // shown mid-scan — so its buffer still has to survive its scan-out, exactly + // as a FIFO frame's does. Keying this on the declaration would free the slot + // at the compositor's release again and let the guest write into a buffer + // the display is reading. + let served = live_choice(sc).0; + let synced = matches!( + served, + VK_PRESENT_MODE_FIFO_KHR + | VK_PRESENT_MODE_FIFO_RELAXED_KHR + | VK_PRESENT_MODE_MAILBOX_KHR + ); + // A synced slot waits for the display's own release, which is the note. + // No capability check stands in front of it: the note may never come (a + // buffer the daemon's publish path coalesced away is never replaced), and + // `slot_just_freed`'s bound is what answers that case — for the guest's + // acquire and for the generator's destinations alike. A latch that + // enabled parking only after one note arrived was removed for the same + // reason: it turned a one-off note into a session-long decision. + // Hidden slots are never parked. They are copies this layer owns, made so + // the guest's own images do not have to wait for the compositor, and the + // ring's order is what bounds their reuse. Parking them defeats the whole + // mechanism: four hidden slots go in_flight, `spare_begin` finds none free, + // the frame falls back to a guest slot, and the guest slot parks — with two + // of those the session lands on `2 / hold` = 30 fps. Measured on Doom 2016 + // (Odin 3): six slots, `dests=[4/4 parked=4]`, 29.9 fps; the same title on + // the commit before the park existed ran 50-60. + let hidden_slot = { + let guest = (*sc).guest_slots as usize; + let slots = (*sc).slots.as_ptr(); + ((slot as usize - slots as usize) / core::mem::size_of::()) >= guest + }; + // A park is a bet that a note will end it, so it is only placed while the + // channel is paying out. On a session with no producer the slot is freed + // here instead, on its fence, which is the pre-park behaviour. + if synced && (*slot).ino != 0 && !hidden_slot && !park_channel_dead() { + (*slot).parked_ns = monotonic_ns(); + (*slot).park_expired = false; + } else { + (*slot).in_flight = false; + } (*sc).cv_free.notify_all(); - let wait_id = core::mem::replace(&mut (*slot).wait_id, 0); + // Only a frame whose commit has registered its wait is answered here. + // This release may belong to an earlier use of the same buffer: it can + // land between the present arming the id and the worker registering it, + // and consuming the id there is what left 87 % of waits with no commit + // to bound them — the guest then waited out the compositor's feedback. + let wait_id = if (*slot).wait_registered { + (*slot).wait_registered = false; + core::mem::replace(&mut (*slot).wait_id, 0) + } else { + 0 + }; drop(_g); REL_ALL_N.fetch_add(1, Ordering::Relaxed); if wait_id != 0 { @@ -10029,8 +10598,8 @@ unsafe extern "C" fn gn_vkGetPhysicalDeviceSurfaceFormatsKHR( } } -// All four MESA_VK_WSI_PRESENT_MODE names are served; fifo_relaxed sits next to -// fifo as its sibling, so the delivered order of the other three is unchanged. +// The four present modes the shim serves, in the order the guest sees them; +// fifo_relaxed sits next to fifo as its sibling. const SERVED_PRESENT_MODES: [VkPresentModeKHR; 4] = [ VK_PRESENT_MODE_IMMEDIATE_KHR, VK_PRESENT_MODE_MAILBOX_KHR, @@ -10038,37 +10607,9 @@ const SERVED_PRESENT_MODES: [VkPresentModeKHR; 4] = [ VK_PRESENT_MODE_FIFO_RELAXED_KHR, ]; -fn parse_present_mode_override(val: Option<&str>) -> Option { - let mode = match val? { - "fifo" => VK_PRESENT_MODE_FIFO_KHR, - "relaxed" => VK_PRESENT_MODE_FIFO_RELAXED_KHR, - "mailbox" => VK_PRESENT_MODE_MAILBOX_KHR, - "immediate" => VK_PRESENT_MODE_IMMEDIATE_KHR, - _ => { - // Ungated and once: the env is the user's present-mode control, and - // a typo must show in a shipped session's log without repeating on - // every query. - static SAID: AtomicBool = AtomicBool::new(false); - if !SAID.swap(true, Ordering::Relaxed) { - crate::logi!("Invalid MESA_VK_WSI_PRESENT_MODE value!"); - } - return None; - } - }; - Some(mode) -} - -/// Where a swapchain's present mode came from, for the census. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ModeSource { - /// The container's `MESA_VK_WSI_PRESENT_MODE`. - Env, - /// The guest's `presentMode`. - Guest, -} - /// The guest's requested `presentMode`, falling back to fifo (mesa -/// wsi_common.c:1521). +/// wsi_common.c:1521). Every swapchain the shim creates carries this mode: the +/// shim does not override it, so what the guest asks for is what it gets. fn guest_present_mode(ci: &VkSwapchainCreateInfoKHR) -> VkPresentModeKHR { match ci.presentMode { VK_PRESENT_MODE_IMMEDIATE_KHR @@ -10079,91 +10620,32 @@ fn guest_present_mode(ci: &VkSwapchainCreateInfoKHR) -> VkPresentModeKHR { } } -/// The present mode a swapchain keeps: the container's -/// `MESA_VK_WSI_PRESENT_MODE` when it names one, as in Mesa, else the guest's -/// requested mode ([`guest_present_mode`]). The same on both routes: on -/// gamescope's override route the env is the user's present-mode preference -/// (mailbox is the container default; fifo, fifo_relaxed and immediate are -/// the user's choice), and it is what the shim declares to gamescope. -fn effective_present_mode(ci: &VkSwapchainCreateInfoKHR) -> (VkPresentModeKHR, ModeSource) { - match parse_present_mode_override(std::env::var("MESA_VK_WSI_PRESENT_MODE").ok().as_deref()) { - Some(m) => (m, ModeSource::Env), - None => (guest_present_mode(ci), ModeSource::Guest), - } -} - -/// Whether a present wait on a frame of a swapchain in effective `mode` is -/// answered when the frame is committed. True for mailbox and immediate, where -/// a guest that knew its real mode would not wait (DXVK skips the present wait -/// for both); FIFO and FIFO_RELAXED keep the release-or-grace answer. -fn answers_present_wait_at_commit(mode: VkPresentModeKHR) -> bool { - mode == VK_PRESENT_MODE_MAILBOX_KHR || mode == VK_PRESENT_MODE_IMMEDIATE_KHR -} /// Why a present declares the mode it does, on the override route. #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum Declared { - /// The swapchain's own mode ([`effective_present_mode`]). + /// The guest's own mode ([`guest_present_mode`]). Mode, - /// Mailbox for frame generation. + /// Mailbox for frame generation (`source=generating`). Generating, - /// FIFO for want of the daemon's stamp lead (`source=cap-compat`). - CapCompat, - /// FIFO because the cap is the display's rate and the guest asked for - /// mailbox or immediate (`source=cap-rate`). - CapRate, - /// The container's mode with the vblank-grid gate because the cap is the - /// display's rate and the guest asked for FIFO or FIFO-relaxed - /// (`source=cap-rate-vsync`). - CapRateVsync, - /// No cap and a guest that asked for FIFO or FIFO-relaxed: its own mode - /// over the container's non-FIFO default (`source=guest`). - GuestVsync, -} - -/// The mode a present on `sc` runs under and declares, decided per present. +} + +/// The mode a present on `sc` declares to gamescope. /// -/// On the override route: -/// - frame generation on: mailbox. Generated frames are committed with no -/// declaration, and a real frame declared FIFO would queue behind them. -/// - a cap set, a mode other than FIFO, and no stamp lead from the daemon yet -/// (`crate::vsync_gate::current_grid` is `None`: no note yet, or a daemon -/// that predates the field): FIFO, so gamescope's limiter is the cap. The -/// vblank-grid gate is not run without the lead: released on the latch -/// points themselves, frames straddle gamescope's composite and are -/// replaced in its mailbox. The census reads `source=cap-compat`. -/// - a cap at the display's rate, a mode other than FIFO and a guest that -/// asked for FIFO or FIFO-relaxed (in-game VSync on): the swapchain's mode -/// with the vblank-grid gate, one release per vblank (`source=cap-rate-vsync`). -/// The guest's present wait is then answered at commit and the gate paces -/// it. Declared FIFO it was serialised behind gamescope's `presented` for -/// each frame, and with ~16 ms of work per frame and three images it fell -/// to 40 fps at 60 Hz. -/// - a cap at the display's rate, a mode other than FIFO and a guest that -/// asked for mailbox or immediate (self-paced): FIFO, with no acquire gate -/// (`source=cap-rate`), as on master. At that cap one -/// frame per vblank is all the panel shows under either mode, and under -/// mailbox the shim would have to keep two commits out of one gamescope -/// composite without knowing when gamescope latches: its wakeup moves with -/// its rolling draw time (vblankmanager.cpp). A guest that paces itself just -/// below the display's rate had ~1 frame a second replaced in gamescope's -/// mailbox that way. Under FIFO gamescope latches one commit per vblank at -/// its own wakeup and its limiter (divisor 1) paces the guest. The gate is -/// not kept on top: FIFO's back-pressure already holds the guest, and the two -/// stacked turned a guest whose acquire-to-commit grew past a period into a -/// 40 fps session on a 60 Hz panel. -/// - no cap, the container's env naming a mode other than FIFO and a guest -/// that asked for FIFO or FIFO-relaxed: the guest's mode (`source=guest`); -/// see [`choose_live_mode`]. -/// - otherwise: the swapchain's mode ([`effective_present_mode`]). A FIFO -/// swapchain is held by gamescope's limiter, one commit per limited vblank; -/// under any other mode the acquire gate is the cap, on the display's vblank -/// grid (`acquire_pacing`). +/// gamescope is told what the guest asked for: the container cannot override it +/// (`MESA_VK_WSI_PRESENT_MODE` is not read anywhere), and neither can the cap, +/// the panel's refresh or frame generation. What those decide is the rate the +/// guest may acquire at and the frames the generator may add, which is +/// [`crate::limits`] — one rule for both. /// -/// The rule is read per present, so a lead or a cap that changes mid-session -/// moves the declaration on the next commit. Off the override route there is -/// nothing to declare it to, and the mode is the one the swapchain was made -/// with. +/// The single exception is a generated frame: it declares mailbox, because +/// generated commits carry no declaration of their own and a real frame +/// declared FIFO would queue behind them in gamescope's limiter. +/// +/// The rule is read per present, so a cap, a multiplier or a panel rate that +/// changes mid-session moves the declaration on the next commit. Off the +/// override route there is nothing to declare it to, and the mode is the one +/// the swapchain was made with. pub(crate) unsafe fn live_present_mode(sc: *mut GnSwapchain) -> VkPresentModeKHR { unsafe { live_choice(sc) }.0 } @@ -10174,106 +10656,150 @@ unsafe fn live_choice(sc: *mut GnSwapchain) -> (VkPresentModeKHR, Declared) { if unsafe { (*(*sc).surf).swapchain_obj.is_null() } { return (mode, Declared::Mode); } - choose_live_mode( - mode, - unsafe { (*sc).mode_source }, - unsafe { (*sc).requested_mode }, - crate::lsfg::live().0, - crate::scanout::cap_fps_note(), - crate::vsync_gate::current_grid(), - ) + choose_live_mode(mode, crate::lsfg::wanted().is_some()) } -/// [`live_present_mode`]'s rule on the override route, for the swapchain's -/// `mode` and where it came from, the guest's `requested` mode, the daemon's -/// cap and the display's vblank grid (`None` without the stamp lead). +/// [`live_choice`]'s rule: the guest's mode, except for a generated frame, and +/// except that `IMMEDIATE` is served as `MAILBOX`. /// -/// With no cap a guest that asked for FIFO (or FIFO-relaxed) gets it even when -/// the container's env names another mode: every container carries -/// `MESA_VK_WSI_PRESENT_MODE=mailbox` as a stamped default the user did not -/// choose, and honouring it there turned a game's own VSync into an uncapped -/// mailbox session. A FIFO request is a specific wish; a mailbox or immediate -/// request is "do not make me wait", for which the container's mode is the -/// substitute, as on the X11 route. An env of `fifo` is an explicit wish and -/// wins in every case. -fn choose_live_mode( - mode: VkPresentModeKHR, - source: ModeSource, - requested: VkPresentModeKHR, - generating: bool, - cap_fps: u32, - grid: Option, -) -> (VkPresentModeKHR, Declared) { - let capped = cap_fps != gndrm_uapi::wire::FPS_UNCAPPED; - let vsync_request = - requested == VK_PRESENT_MODE_FIFO_KHR || requested == VK_PRESENT_MODE_FIFO_RELAXED_KHR; +/// `IMMEDIATE` and `MAILBOX` differ only in what happens to a frame that cannot +/// be shown: immediate presents it anyway and tears, mailbox replaces the pending +/// one. A guest that asks immediate therefore gets its frames dropped for it +/// instead of being shown mid-scan, which is what the container used to do with +/// `MESA_VK_WSI_PRESENT_MODE` and no longer may. The declaration follows the same +/// mode, so gamescope is told what is actually served. +/// The mode a swapchain's image count is sized for: the one this session will +/// serve, not the one the guest declared. +/// +/// The two differ in exactly two cases, and both are ours: a generation session +/// is served mailbox or fifo from the QuickMenu whatever the guest asked, and a +/// declared IMMEDIATE is served as mailbox. Sizing the count from the declaration +/// therefore sizes it for a mode nobody runs — measured on Witcher 3, whose guest +/// asks FIFO (`min=3 mode=2`) while it is served and held as mailbox: +/// +/// ```text +/// slots=[3 guest + 8 destinations] sf hold 1.5-2.5 periods acq_fps 49.7 +/// ``` +/// +/// Three guest slots under a two-period hold cap at `depth x refresh / hold` = 54, +/// and the mailbox floor of four — the whole reason `SC_MAILBOX_IMAGES` exists — +/// never applied. The hold already follows the served mode (`on_buffer_release`); +/// this makes the depth follow it too. +fn count_present_mode(declared: VkPresentModeKHR) -> VkPresentModeKHR { + choose_live_mode(declared, crate::lsfg::live().0).0 +} + +/// What this session serves for a declared mode. +/// +/// Two declarations are served as `MAILBOX`, and both for the same reason: the +/// mode the guest asked for cannot be honoured without costing frames it would +/// not otherwise lose. +/// +/// - `IMMEDIATE` tears by construction: the image is handed over for display the +/// moment it is presented. Served as mailbox it is superseded at a vblank +/// instead, so the guest gets the pacing it asked for (no wait on the client) +/// and the panel gets a buffer it can latch whole. Measured on Portal 2: the +/// tear under direct scan-out disappears. +/// - `FIFO_RELAXED` gives up a whole vblank per frame when a frame lands after the +/// compositor's latch point, because the next one cannot be taken until the +/// following vblank. Measured on Doom 2016 at 28.8 fps against a 60 cap with the +/// GPU at 41 % and the CPU at 26 % — nothing busy, each frame costing `2T`. +/// Served as mailbox the newest frame replaces the pending one, so a missed +/// latch is superseded rather than deferred: `period = max(render, T)` instead +/// of `2T`. The same title ran near 60 on the commit before this override was +/// removed, when the app set `MESA_VK_WSI_PRESENT_MODE=mailbox` and this layer +/// obeyed it. +/// +/// The cost of the second is that a superseded frame is dropped rather than shown +/// late, which `FIFO_RELAXED` itself would not do — it permits tearing, not loss. +/// At a render time below the panel period nothing is superseded, so the two are +/// indistinguishable; above it, mailbox is what the guest would have got had it +/// asked for mailbox. +/// +/// `FIFO` is served as itself. It promises every frame on a vblank and its frames +/// arrive before the latch point by construction, which is the case the mailbox +/// discipline would damage. +fn choose_live_mode(mode: VkPresentModeKHR, generating: bool) -> (VkPresentModeKHR, Declared) { if generating { - (VK_PRESENT_MODE_MAILBOX_KHR, Declared::Generating) - } else if mode == VK_PRESENT_MODE_FIFO_KHR { - (mode, Declared::Mode) - } else if !capped { - if source == ModeSource::Env && vsync_request { - (requested, Declared::GuestVsync) + // The QuickMenu offers mailbox and fifo for generation and nothing else, + // and the note carries the choice (`LsfgConfig::present_mode`); anything + // unknown is mailbox. + let served = if crate::lsfg::generating_present_mode() == 1 { + VK_PRESENT_MODE_FIFO_KHR } else { - (mode, Declared::Mode) - } + VK_PRESENT_MODE_MAILBOX_KHR + }; + (served, Declared::Generating) + } else if mode == VK_PRESENT_MODE_IMMEDIATE_KHR + || mode == VK_PRESENT_MODE_FIFO_RELAXED_KHR + { + (VK_PRESENT_MODE_MAILBOX_KHR, Declared::Mode) } else { - match grid { - None => (VK_PRESENT_MODE_FIFO_KHR, Declared::CapCompat), - Some(g) if crate::vsync_gate::at_display_rate(cap_fps, g) => { - if vsync_request { - (mode, Declared::CapRateVsync) - } else { - (VK_PRESENT_MODE_FIFO_KHR, Declared::CapRate) - } - } - Some(_) => (mode, Declared::Mode), - } + (mode, Declared::Mode) } } -/// How the acquire gate paces `sc` now. +/// What this swapchain's session allows ([`crate::limits::limits`]): the +/// guest's own mode decides whether the panel is a ceiling, the daemon's cap +/// note and the generator's multiplier decide the rest. +fn session_limits(sc: *mut GnSwapchain) -> crate::limits::Limits { + let mode = unsafe { (*sc).present_mode }; + let vsync = matches!( + mode, + VK_PRESENT_MODE_FIFO_KHR | VK_PRESENT_MODE_FIFO_RELAXED_KHR + ); + let cap = crate::scanout::cap_fps_note(); + crate::limits::limits( + vsync, + (cap != gndrm_uapi::wire::FPS_UNCAPPED).then_some(cap), + crate::lsfg::wanted(), + crate::lsfg::refresh_hz(), + ) +} + +/// How the acquire gate paces `sc` now: to the session's acquire limit +/// ([`session_limits`]), on whichever clock can hold it. /// -/// - Off gamescope's override route (the X11 Present route) nothing holds the -/// cap but the gate, on the cap's own clock, as before. -/// - On the override route see [`gamescope_route_pacing`]. +/// - Off gamescope's override route (the X11 Present route) nothing else holds +/// the cap, so the gate does, on its own clock. +/// - **Every declaration is gated.** gamescope's limiter withholds a window's +/// frame callback, and this shim paces on its own schedule and the buffer +/// release, never on that callback — so a FIFO declaration used to leave the +/// guest with no cap at all: `steamcompmgr_should_vblank_window` acts only +/// when the cap is below the refresh AND divides it, so 60 on 60 Hz and 40 on +/// 60 Hz were both unlimited, measured as a 26 fps session whose cap did +/// nothing. +/// - With generation on the gate holds the source rate on its own clock, and +/// the batches are placed on the latch grid by the generator itself. +/// - Otherwise the gate holds the acquire limit on the display's vblank grid +/// (`crate::vsync_gate`), snapping to a whole number of vblanks when it can. unsafe fn acquire_pacing(sc: *mut GnSwapchain) -> crate::vsync_gate::Pacing { + let limits = session_limits(sc); + let Some(acquire) = limits.acquire_per_s else { + return crate::vsync_gate::Pacing::Off; + }; if unsafe { (*(*sc).surf).swapchain_obj.is_null() } { - return crate::vsync_gate::Pacing::on_clock(crate::scanout::target_fps()); + return crate::vsync_gate::Pacing::on_clock(acquire); } - gamescope_route_pacing( - unsafe { live_choice(sc) }, - crate::lsfg::source_cap(), - crate::scanout::cap_fps_note(), - crate::vsync_gate::current_grid(), - ) + let (_, declared) = unsafe { live_choice(sc) }; + gamescope_route_pacing(declared, acquire) } -/// [`acquire_pacing`]'s rule on gamescope's override route, for the declared -/// mode and why ([`choose_live_mode`]), frame generation's source cap, the -/// daemon's cap and the display's vblank grid: -/// - frame generation sets the source rate: the gate holds it on the cap's own -/// clock (the app disarms gamescope's limiter then; generation paces the -/// output on the latch grid itself); -/// - any FIFO declaration (a FIFO swapchain, FIFO for want of the stamp lead, -/// FIFO because the cap is the display's rate): no gate. gamescope's limiter -/// holds FIFO commits, one per limited vblank, and is the cap; -/// - mailbox, immediate, FIFO-relaxed: gamescope's limiter does not hold them -/// (`wlserver_surface_is_fifo` is FIFO only), so the gate is the cap, on the -/// display's vblank grid (`crate::vsync_gate`). -fn gamescope_route_pacing( - (live, _): (VkPresentModeKHR, Declared), - source_cap: Option, - cap_fps: u32, - grid: Option, -) -> crate::vsync_gate::Pacing { - if let Some(cap) = source_cap { - return crate::vsync_gate::Pacing::on_clock(cap); - } - if live == VK_PRESENT_MODE_FIFO_KHR { - return crate::vsync_gate::Pacing::Off; +/// [`acquire_pacing`]'s rule on gamescope's override route, for what the commit +/// declares ([`choose_live_mode`]), the session's acquire limit and the +/// display's vblank grid. +fn gamescope_route_pacing(declared: Declared, acquire: u32) -> crate::vsync_gate::Pacing { + match declared { + // Generated frames are placed on the latch grid by the generator, so + // the gate holds the source rate on its own clock. + Declared::Generating => crate::vsync_gate::Pacing::on_clock(acquire), + // Whatever the guest declared. A FIFO declaration is not a pacer: + // gamescope withholds the window's frame callback and this shim never + // waits on one, and its divisor only bites when the cap is below the + // refresh AND divides it. Standing down here is what left a 60/60 and a + // 40/60 session with no cap at all. + Declared::Mode => crate::vsync_gate::pacing(acquire), } - crate::vsync_gate::pacing(cap_fps, grid) } /// The mode a commit on `sc` declares to gamescope, counted for the census. @@ -10302,12 +10828,16 @@ fn mode_name(mode: VkPresentModeKHR) -> &'static str { #[derive(Debug, Clone, Copy, PartialEq, Eq)] struct ModeLine { declared: VkPresentModeKHR, - requested: VkPresentModeKHR, source: &'static str, gamescope_route: bool, cap: u32, gate: &'static str, lsfg: bool, + /// The session's limits ([`crate::limits`]), `FPS_UNCAPPED`/`0` for none: + /// the acquire ceiling and the frames the generator may add, in frames per + /// second. Printed because the declaration no longer says what the cap did. + acquire: u32, + generate: u32, } /// The last [`ModeLine`] printed. @@ -10323,51 +10853,62 @@ static MODE_LINE: Mutex> = Mutex::new(None); unsafe fn mode_census(sc: *mut GnSwapchain) { let route = unsafe { !(*(*sc).surf).swapchain_obj.is_null() }; let (declared, why) = unsafe { live_choice(sc) }; + let limits = session_limits(sc); let line = ModeLine { declared, - requested: unsafe { (*sc).requested_mode }, - source: match (why, unsafe { (*sc).mode_source }) { - (Declared::CapCompat, _) => "cap-compat", - (Declared::CapRate, _) => "cap-rate", - (Declared::CapRateVsync, _) => "cap-rate-vsync", - (Declared::GuestVsync, _) => "guest", - (_, ModeSource::Env) => "env", - (_, ModeSource::Guest) => "guest", + source: match why { + Declared::Generating => "generating", + Declared::Mode => "guest", }, gamescope_route: route, cap: crate::scanout::cap_fps_note(), gate: unsafe { acquire_pacing(sc) }.clock().name(), lsfg: crate::lsfg::live().0, + acquire: limits + .acquire_per_s + .unwrap_or(gndrm_uapi::wire::FPS_UNCAPPED), + generate: limits.generate_per_s.unwrap_or(0), }; { let mut last = MODE_LINE.lock().unwrap_or_else(|e| e.into_inner()); if *last != Some(line) { *last = Some(line); - crate::logi!("present mode: declared={} requested={} source={} route={} cap={} gate={} lsfg={}", + crate::logi!("present mode: declared={} source={} route={} cap={} gate={} lsfg={} acquire={} generate={}", mode_name(line.declared), - mode_name(line.requested), line.source, if route { "gamescope" } else { "x11" }, line.cap, line.gate, - u8::from(line.lsfg),); + u8::from(line.lsfg), + line.acquire, + line.generate); } } if !crate::vkshim_enabled() || !rate_ok(&RATE_MODE, MODE_CENSUS_EVERY) { return; } let (presented, discarded, fallback) = crate::present::feedback_counts(); + let wait_census = crate::present::wait_census(); let (late, skipped) = crate::vsync_gate::late_counts(); + // A cumulative counter is a rate only with its denominator: the session's + // own age, from the first acquire. + let t0 = SHIM_T0_NS.load(Ordering::Relaxed); + let age_s = if t0 == 0 { + 0 + } else { + monotonic_ns().saturating_sub(t0) / 1_000_000_000 + }; // Since the last line, so each line reads one window's cadence. let gaps = COMMIT_GAPS.each_ref().map(|b| b.swap(0, Ordering::Relaxed)); - crate::logf!("present mode: declared={} requested={} source={} route={} cap={} gate={} lsfg={} acquires={} commits={} declared_fifo={} declared_other={} releases={} feedback={} presented={} discarded={} fallback={} at_queue={} gate_holds={} gate_late={} skipped={} slot_waits={} commit_lat_us={} spacing=[<15:{} 15-16.2:{} 16.2-17.2:{} 17.2-20:{} >20:{}]", + crate::logf!("present mode: declared={} source={} route={} cap={} gate={} lsfg={} acquire={} generate={} acquires={} commits={} declared_fifo={} declared_other={} releases={} feedback={} presented={} discarded={} fallback={} at_queue={} gate_holds={} gate_late={} skipped={} slot_waits={} commit_lat_us={} spacing=[<15:{} 15-16.2:{} 16.2-17.2:{} 17.2-20:{} >20:{}] gated={} ungated={} gate_wait_us={} acq_spacing=[<14:{} 14-16.2:{} 16.2-17.2:{} 17.2-20:{} >20:{}] age_s={} calls={} ok={} retry={} notes={} park_timeouts={} rnotes={} noid={} notelag=[<5:{} <33:{} <100:{} >100:{}] waitans=[n:{} q:{} commit:{} fb:{} daemon:{} grace:{} release:{} discard:{} unarmed:{} timeout:{} nocommit:{}]", mode_name(line.declared), - mode_name(line.requested), line.source, if route { "gamescope" } else { "x11" }, line.cap, line.gate, u8::from(line.lsfg), + line.acquire, + line.generate, N_ACQ.load(Ordering::Relaxed), N_COMMIT_OK.load(Ordering::Relaxed), DECL_FIFO_N.load(Ordering::Relaxed), @@ -10387,7 +10928,38 @@ unsafe fn mode_census(sc: *mut GnSwapchain) { gaps[1], gaps[2], gaps[3], - gaps[4],); + gaps[4], + ACQ_GATED_N.load(Ordering::Relaxed), + ACQ_UNGATED_N.load(Ordering::Relaxed), + GATE_WAIT_NS.load(Ordering::Relaxed) / 1_000, + ACQ_SPACING[0].load(Ordering::Relaxed), + ACQ_SPACING[1].load(Ordering::Relaxed), + ACQ_SPACING[2].load(Ordering::Relaxed), + ACQ_SPACING[3].load(Ordering::Relaxed), + ACQ_SPACING[4].load(Ordering::Relaxed), + age_s, + N_ACQ.load(Ordering::Relaxed), + ACQ_OK_N.load(Ordering::Relaxed), + ACQ_RETRY_N.load(Ordering::Relaxed), + NOTES_APPLIED.load(Ordering::Relaxed), + PARK_TIMEOUTS.load(Ordering::Relaxed), + crate::scanout::release_notes_received(), + NOID_AT_COMMIT.load(Ordering::Relaxed), + NOTELAG[0].load(Ordering::Relaxed), + NOTELAG[1].load(Ordering::Relaxed), + NOTELAG[2].load(Ordering::Relaxed), + NOTELAG[3].load(Ordering::Relaxed), + wait_census[0], + wait_census[1], + wait_census[2], + wait_census[3], + wait_census[4], + wait_census[5], + wait_census[6], + wait_census[7], + wait_census[8], + wait_census[9], + wait_census[10],); } /// The wire value for `gamescope_swapchain.set_present_mode` that declares @@ -10413,24 +10985,16 @@ fn gamescope_present_mode(mode: VkPresentModeKHR) -> u32 { VK_PRESENT_MODE_MAILBOX_KHR => 1, VK_PRESENT_MODE_FIFO_KHR => 2, VK_PRESENT_MODE_FIFO_RELAXED_KHR => 3, - // Unreachable: `(*sc).present_mode` is `effective_present_mode`'s - // output, which is one of the four served modes above. + // Unreachable: `(*sc).present_mode` is `guest_present_mode`'s output, + // which is one of the four served modes above. _ => 2, } } -fn offered_present_modes( - served: &[VkPresentModeKHR], - override_mode: Option, -) -> Vec { +/// The modes the guest is offered: the served set in its own order, with FIFO +/// appended when it is missing (a swapchain must be creatable in FIFO). +fn offered_present_modes(served: &[VkPresentModeKHR]) -> Vec { let mut v = Vec::with_capacity(served.len() + 1); - if let Some(m) = override_mode { - if served.contains(&m) { - v.push(m); - } else { - crate::logf!("Unsupported MESA_VK_WSI_PRESENT_MODE value!"); - } - } for &m in served { if !v.contains(&m) { v.push(m); @@ -10455,11 +11019,7 @@ unsafe extern "C" fn gn_vkGetPhysicalDeviceSurfacePresentModesKHR( crate::logf!("proxy surface present-modes query: {surface:#x} not one of ours"); return VK_ERROR_SURFACE_LOST_KHR; } - // The env's mode leads the list, as in Mesa: it decides the mode on both - // routes (`effective_present_mode`). - let override_mode = - parse_present_mode_override(std::env::var("MESA_VK_WSI_PRESENT_MODE").ok().as_deref()); - let modes_list = offered_present_modes(&SERVED_PRESENT_MODES, override_mode); + let modes_list = offered_present_modes(&SERVED_PRESENT_MODES); unsafe { if modes.is_null() { *count = modes_list.len() as u32; diff --git a/icd_shim/src/scanout.rs b/icd_shim/src/scanout.rs index e478cb7..5f895df 100644 --- a/icd_shim/src/scanout.rs +++ b/icd_shim/src/scanout.rs @@ -18,12 +18,13 @@ use gndrm_uapi::wire::{ scanout_ahb_format, FrameCapNote, FrameCapSubscribe, LsfgConfig, ScanoutAllocReq, - ScanoutAllocResp, ScanoutLayout, VsyncNote, CAP_FRAME_CAP, CAP_PRESENT_ID, CAP_VSYNC_NOTE, + BufferReleased, ScanoutAllocResp, ScanoutLayout, VsyncNote, CAP_FRAME_CAP, CAP_PRESENT_ID, + CAP_RELEASE_NOTE, CAP_VSYNC_NOTE, FPS_UNCAPPED, SCANOUT_LAYOUT_ENV, SCANOUT_REQ_AHB_HANDLE, }; use std::ffi::c_void; use std::os::fd::RawFd; -use std::sync::atomic::{AtomicU32, AtomicUsize, Ordering}; +use std::sync::atomic::{AtomicU32, AtomicU64, AtomicUsize, Ordering}; use std::sync::Mutex; /// Default budget for one exchange, matching the LD_PRELOAD shim's: a daemon @@ -122,28 +123,6 @@ pub(crate) fn period_ns(target_fps: u32) -> u64 { NANOS_PER_SECOND / u64::from(target_fps) } -/// The instant the next acquire may start, given the previous one and now. -/// -/// The cap is applied here rather than to a release: a frame that may not start -/// yet is the only point where waiting costs neither rendered work nor display -/// latency. `prev + period` holds the interval at the target while the guest -/// keeps up: the wait for `prev` returns a hair late, and that hair must not -/// become the cadence (it did, as `max(prev, now) + period`: a 60 cap ran at -/// 59.4). Once the guest is a whole period behind it is its own rate again, -/// `now + period` — never two frames closer than one period, and no debt -/// carried out of a stall. -#[must_use] -pub(crate) fn next_acquire_after(prev_ns: u64, now_ns: u64, period_ns: u64) -> u64 { - if period_ns == 0 { - return prev_ns; - } - if prev_ns != 0 && now_ns < prev_ns.saturating_add(period_ns) { - prev_ns + period_ns - } else { - now_ns.saturating_add(period_ns) - } -} - /// Adopt one frame-cap note: the target it carries. fn apply_note(note: FrameCapNote) { NOTES_N.fetch_add(1, Ordering::Relaxed); @@ -155,15 +134,9 @@ fn apply_note(note: FrameCapNote) { target_fps()); } -/// The target in force, [`FPS_UNCAPPED`] when none has: read by the shim's -/// census and by the acquire gate. +/// The cap the daemon's note carries, [`FPS_UNCAPPED`] when none: what the +/// session's limits are computed from, and what the frame-cap line reports. pub(crate) fn target_fps() -> u32 { - // Frame generation owns the source rate while it is on: the app disarms - // gamescope's limiter in that case, so a zero from the daemon's cap note is - // not "uncapped", it is "paced by the presenter". - if let Some(cap) = crate::lsfg::source_cap() { - return cap; - } CAP_FPS.load(Ordering::Relaxed) } @@ -176,7 +149,6 @@ pub(crate) fn cap_fps_note() -> u32 { /// latch grid, and the acquire gate releases on the vblank grid it gives. fn apply_vsync(note: VsyncNote) { crate::lsfg::apply_vsync(note); - crate::vsync_gate::apply_note(note); } /// How many frame-cap notes have been adopted on this connection, for the same @@ -304,6 +276,8 @@ pub fn lsfg_config() -> Option { apply_vsync(note); } else if let Some(done) = gndrm_uapi::wire::PresentDone::decode(&buf[..n as usize]) { crate::present::record(done.id, done.latch_ns); + } else if let Some(released) = BufferReleased::decode(&buf[..n as usize]) { + apply_release(released); } else if gndrm_uapi::wire::FrameAcquired::decode(&buf[..n as usize]).is_some() { // The shim's own counter, echoed nowhere: accepted so it is not // mistaken for a frame this reader does not know. @@ -425,7 +399,7 @@ fn connect(path: &str) -> RawFd { if !send_frame( fd, &FrameCapSubscribe { - caps: CAP_FRAME_CAP | CAP_VSYNC_NOTE | CAP_PRESENT_ID, + caps: CAP_FRAME_CAP | CAP_VSYNC_NOTE | CAP_PRESENT_ID | CAP_RELEASE_NOTE, } .encode(), ) { @@ -557,6 +531,17 @@ fn exchange_req(fd: RawFd, req: ScanoutAllocReq) -> Result if frames < MAX_BURST_FRAMES { continue; } + } else if let Some(released) = BufferReleased::decode(buf) { + // The release watcher pushes on its own schedule, so a note + // can stand ahead of the reply; dropping it would leave its + // park to the bound. + if recv_fd >= 0 { + libc::close(recv_fd); + } + apply_release(released); + if frames < MAX_BURST_FRAMES { + continue; + } } else if gndrm_uapi::wire::FrameAcquired::decode(buf).is_some() { // The acquire counter shares this channel and can arrive // while a reply is pending; it is not the reply. @@ -665,17 +650,280 @@ pub(crate) fn send_frame_acquired(count: u64, generated: u64) { } /// Mark a frame's present id for the daemon, immediately before the commit that -/// presents it: the daemon answers when that transaction latches. Best effort by -/// construction — the channel is shared with the allocator, so an exchange in -/// flight is waited for (bounded by the exchange's own timeout) rather than -/// skipped, because a mark that never arrives leaves `vkWaitForPresentKHR` -/// waiting for a frame the daemon cannot name. -pub(crate) fn send_present_mark(ino: u64, id: u64) { +/// presents it: the daemon answers when that transaction latches. +/// +/// Sent without blocking. This runs on the present worker for every frame, so a +/// send that waited for the daemon to make room would put the daemon's read rate +/// into the frame time — measured as a flat 277 ms on a title running at 60 fps +/// on the route that sends no marks at all. A mark that does not fit is dropped: +/// its frame's wait is then answered by the compositor's feedback or by the +/// two-period grace, both of which are bounded, so the cost of a lost mark is a +/// latch report in the census and nothing the guest can feel. +pub(crate) fn send_present_mark(ino: u64, id: u64, ticket: u64) { let ch = CHAN.lock().unwrap_or_else(|e| e.into_inner()); if *ch < 0 { return; } - send_frame(*ch, &gndrm_uapi::wire::PresentMark { ino, id }.encode()); + send_frame_nb( + *ch, + &gndrm_uapi::wire::PresentMark { ino, id, ticket }.encode(), + ); +} + +// ------------------------------------------------------ scan-out releases +// +// A compositor handing a buffer back means it has finished *compositing* with +// it. The display may still be reading it: SurfaceFlinger reports a buffer's +// true release a transaction later, as the previous-release fence of whichever +// buffer replaced it on the layer, and the daemon turns that fence into a +// [`BufferReleased`] note. Until the note arrives a slot stays busy, because a +// producer that reused the buffer on the compositor's release wrote into memory +// the panel was still scanning — and the panel showed both frames at once for +// as long as the two writes overlapped. + +/// One release note as taken: the ticket it names, the inode it names, and when +/// this process read it. +#[derive(Debug, Clone, Copy)] +struct ReleasedNote { + /// The shim's own ticket for the frame, 0 when the daemon sent none. + ticket: u64, + /// The scan-out inode, kept for the older daemon and for the logs. + ino: u64, + /// `CLOCK_MONOTONIC` ns when this process took the note. + at: u64, +} + +/// Release notes the daemon reported, oldest first. Bounded: a note for a buffer +/// whose slot is gone is dropped, and a producer cannot grow this list. +static RELEASED: Mutex> = Mutex::new(Vec::new()); + +/// How many releases may be held before the oldest is dropped. A session has one +/// per buffer in the pool; a note whose slot never comes back is stale. +const RELEASED_CAP: usize = 32; + +/// Release notes seen this session. Zero means the daemon does not send them (an +/// older build), and a slot is freed on the compositor's release as it was +/// before: the note improves on that behaviour, it does not depend on it. +static RELEASE_NOTES: AtomicU64 = AtomicU64::new(0); + +/// Record that the daemon reported `ino` released. +/// +/// A note is not applied here: the slot that owns the buffer is parked, and it is +/// taken on the next acquire pass (see `acquire_take_free`), so this runs on the +/// thread that read the note and touches nothing else. +pub(crate) fn note_buffer_released(ticket: u64, ino: u64, source: u64) { + if ino == 0 && ticket == 0 { + return; + } + RELEASE_NOTES.fetch_add(1, Ordering::Relaxed); + if ticket != 0 { + TICKET_NOTES.fetch_add(1, Ordering::Relaxed); + } + if source != gndrm_uapi::wire::RELEASE_FENCE { + RELEASE_OFF_FENCE.fetch_add(1, Ordering::Relaxed); + } + let mut q = RELEASED.lock().unwrap_or_else(|e| e.into_inner()); + if q.len() >= RELEASED_CAP { + q.remove(0); + } + q.push(ReleasedNote { + ticket, + ino, + at: crate::proxy::monotonic_ns(), + }); +} + +/// Adopt one release note, from whichever reader took it off the channel. +fn apply_release(released: BufferReleased) { + // SurfaceFlinger is finished with this scan-out buffer: the producer may + // write into it again (see `note_buffer_released`). + note_buffer_released(released.ticket, released.ino, released.source); + // A release note is the only message that can end a park, so it is the + // only one that says parking is worth a bet again. Liveness keyed on the + // socket itself is a lie on a device whose daemon still pushes the panel + // grid: RP6 sent ~5 000 `VsyncNote`s while releasing nothing, and + // `channel=[unanswered:2 dead:false]` never armed the fail-open with + // `park_timeouts=5181` of 5 210 acquires. + crate::proxy::note_channel_alive(); +} + +/// Notes that carried a ticket, and notes whose release did not rest on a +/// signalled fence. Two counts because they answer two questions the census +/// could not: whether the daemon is sending tickets at all, and whether a note +/// is a guarantee or a timeout. +static TICKET_NOTES: AtomicU64 = AtomicU64::new(0); +static RELEASE_OFF_FENCE: AtomicU64 = AtomicU64::new(0); + +/// Notes that carried a ticket, and notes off a signalled fence, for the census. +pub(crate) fn note_counts() -> (u64, u64) { + ( + TICKET_NOTES.load(Ordering::Relaxed), + RELEASE_OFF_FENCE.load(Ordering::Relaxed), + ) +} + +/// The inos of the notes currently held, up to eight, oldest first. +/// +/// For the one case worth a log line: a park that expired while notes were being +/// received. Printed beside the slot's own `ino`, it says whether the daemon keys +/// buffers by an identity this process never holds (a different allocator path on +/// this SoC) or whether the note list was simply empty by then, which is a timing +/// question instead. +pub(crate) fn released_snapshot() -> String { + let q = RELEASED.lock().unwrap_or_else(|e| e.into_inner()); + let mut out = String::new(); + for n in q.iter().take(8) { + if !out.is_empty() { + out.push(' '); + } + out.push_str(&format!("{}", n.ino)); + } + out +} + +/// Whether the daemon sends release notes at all. Until one has arrived, a slot +/// frees on the compositor's release exactly as it did before this note existed. +/// Whether any note has ever arrived. +/// +/// The shim no longer decides a park from this: one note followed by silence is +/// a channel that does not work, and `proxy::park_awaits_note` counts the parks +/// since the last note instead. Kept as the test-side predicate. +#[cfg(test)] +pub(crate) fn release_notes_live() -> bool { + RELEASE_NOTES.load(Ordering::Relaxed) != 0 +} + +/// Release notes received this session, for the census: the denominator of +/// `notes=` (parks a note resolved), which says whether a park that timed out +/// lost the note or was never sent one. +pub(crate) fn release_notes_received() -> u64 { + RELEASE_NOTES.load(Ordering::Relaxed) +} + +/// Take the oldest release note at or after `since_ns`, whatever buffer it names. +/// +/// The fallback for a device whose notes do not carry the parked buffer's +/// identity. A note is emitted from SurfaceFlinger's previous-release fence, which +/// signals when a *later* buffer latches, so under a multi-image rotation the note +/// names the buffer that replaced ours rather than ours. Measured on Odin 3: +/// `rnotes=702` with `park_timeouts=700`, `notelag` empty on every bucket, and a +/// park timeout line reading `slot ino=11411 … released=[11366 11368 11370]` — +/// same allocator, never the same buffer. +/// +/// Safe to pair by order because notes arrive one per commit and in commit order: +/// the oldest outstanding note belongs to the oldest outstanding frame, so freeing +/// the oldest parked slot with it cannot free a slot the display is still reading. +/// It is a weaker pairing than the identity match — one frame's worth of margin +/// traded for a pipeline that works — so identity stays the first attempt. +/// +/// The margin is exactly one display period: a note may be up to one frame older +/// than the park's commit and still count, because that is how much later the +/// note for a frame arrives (it rides the latch of the frame that replaced it). +/// Without the margin, depth is effectively one image short — measured at 44 fps +/// with a 60 cap on a three-image FIFO swapchain. +pub(crate) fn take_any_released(since_ns: u64, ticket: u64) -> bool { + // One display period of margin, and no more. The note that answers a park is + // emitted when the *next* frame latches, so it legitimately predates the + // commit by up to one frame; a floor at the commit itself rejects it and + // leaves every park to its bound. Anything older than a period describes a + // frame this slot was not part of. + let hz = u64::from(crate::lsfg::panel_hz().max(1)); + let floor = since_ns.saturating_sub(1_000_000_000 / hz); + let mut q = RELEASED.lock().unwrap_or_else(|e| e.into_inner()); + q.retain(|n| n.at >= floor); + // The oldest note that does not predate this park's frame. It describes the + // frame that latched when this slot's buffer was replaced, which is the frame + // *before* the one parked here — so a floor at the commit would reject the + // only note that will ever mention that buffer and leave the park to its + // bound. + // + // The margin this gives up is one frame: slot N may be freed by the release + // of frame N+1. Both are in order, so the reuse is still older than the + // display's newest frame; the stricter identity match above is unaffected + // and still runs first where inodes line up. + match q.iter().position(|n| !predates(n.ticket, ticket)) { + Some(i) => { + q.remove(i); + true + } + None => false, + } +} + +/// Whether a note's ticket names a frame committed before the park's own. +/// +/// Such a note cannot answer the park by order or by inode: the frame it +/// describes left the display no later than the parked one arrived, so it says +/// nothing about the parked frame. It is a late note for an earlier use of the +/// same buffer (the park on it expired and the buffer was committed again), or +/// the note for a commit that was never parked (a hidden slot, a generation +/// destination), which the park's own frame replaced on the layer. A newer +/// ticket is still an answer, and is the common one: a note can arrive before +/// its own slot is parked, and then pays the oldest park, whose frame left the +/// layer before the note's did. 0 on either side is a daemon or a commit +/// without tickets, and is not judged. +fn predates(note_ticket: u64, park_ticket: u64) -> bool { + note_ticket != 0 && park_ticket != 0 && note_ticket < park_ticket +} + +/// Take a release note for `ino` that arrived at or after `since_ns`, and drop +/// every older note for the same buffer. +/// +/// `since_ns` is when the slot was parked. A note older than that describes an +/// earlier frame of the same buffer — it says nothing about the frame now on the +/// panel — and must not free the slot early. So does a note that arrived late +/// but names an earlier `ticket`. `true` when a note applied. +pub(crate) fn take_released(ino: u64, since_ns: u64, ticket: u64) -> bool { + if ino == 0 { + return false; + } + let mut q = RELEASED.lock().unwrap_or_else(|e| e.into_inner()); + let mut applied = false; + q.retain(|n| { + if n.ino != ino { + return true; + } + if n.at < since_ns || predates(n.ticket, ticket) { + return false; + } + if applied { + return true; + } + applied = true; + false + }); + applied +} + +/// Take the note for `ticket`, if one is held. +/// +/// This is the whole pairing rule. A ticket is minted by the process that owns +/// the slot and is never reused, so there is no window to tune, no floor to +/// derive, and no ordering assumption: the note either names this slot's frame or +/// it does not. It replaces, on any device whose daemon sends tickets: +/// +/// - matching by dma-buf inode, which pairs only while the allocator hands the +/// same object through to SurfaceFlinger — true for Portal 2 on Nova (6 % of +/// parks needed the fallback) and false for Dave on the same device (84 %), so +/// the inode is a property of the title's allocation pattern, not of the device; +/// - the order fallback's reorder window and its one-display-period floor; +/// - `since = commit_ns.min(parked)`, which existed to catch a note that arrives +/// before the park it answers. +/// +/// A note that names no ticket (`ticket == 0`) is left in the queue for the inode +/// path, so an older daemon still works. +pub(crate) fn take_released_ticket(ticket: u64) -> bool { + if ticket == 0 { + return false; + } + let mut q = RELEASED.lock().unwrap_or_else(|e| e.into_inner()); + match q.iter().position(|n| n.ticket == ticket) { + Some(i) => { + q.remove(i); + true + } + None => false, + } } pub(crate) fn poll_notes() { @@ -713,6 +961,8 @@ pub(crate) fn poll_notes() { } else if let Some(done) = gndrm_uapi::wire::PresentDone::decode(&buf[..len]) { // The panel latched the frame this id was marked for. crate::present::record(done.id, done.latch_ns); + } else if let Some(released) = BufferReleased::decode(&buf[..len]) { + apply_release(released); } else if let Some(acq) = gndrm_uapi::wire::FrameAcquired::decode(&buf[..len]) { // Sent from the acquire path; nothing to apply here. Accepted so the // log does not report it as an unknown frame. @@ -721,12 +971,145 @@ pub(crate) fn poll_notes() { } /// Largest frame the daemon pushes unsolicited. -const PUSH_LEN: usize = LsfgConfig::LEN; +const PUSH_LEN: usize = if LsfgConfig::LEN > BufferReleased::LEN { + LsfgConfig::LEN +} else { + BufferReleased::LEN +}; #[cfg(test)] mod tests { use super::*; + /// A release note frees the frame it describes and no other: the slot is + /// parked before one can arrive, so a note older than the park describes an + /// earlier frame of the same buffer and must not free this one. + #[test] + fn a_release_note_only_applies_to_the_frame_it_describes() { + let _serial = fresh_notes(); + note_buffer_released(0, 9_001, gndrm_uapi::wire::RELEASE_FENCE); + assert!(take_released(9_001, 0, 0), "a note after the park frees the slot"); + assert!(!take_released(9_001, 0, 0), "and is consumed by it"); + + // A note taken before the slot was parked: dropped, and it must not free + // the slot on the next pass either. + note_buffer_released(0, 9_002, gndrm_uapi::wire::RELEASE_FENCE); + let parked = crate::proxy::monotonic_ns() + 1; + assert!(!take_released(9_002, parked, 0)); + assert!(!take_released(9_002, 0, 0)); + + // An inode of 0 is never a registered buffer. + note_buffer_released(0, 0, gndrm_uapi::wire::RELEASE_FENCE); + assert!(!take_released(0, 0, 0)); + } + + /// A ticket pairs the note to its frame when the inode cannot, which is the + /// case this whole path exists for: on Nova, Portal 2 paired 94 % of parks by + /// inode and Dave paired 84 % by order on the same device. + #[test] + fn a_ticket_pairs_a_note_whose_inode_is_another_buffer() { + let _serial = fresh_notes(); + // The daemon says "ticket 7 is done", naming a buffer this process never + // held — exactly what Odin 3 reports (`slot ino=11411`, `released=[11366 + // 11368 11370]`). + crate::scanout::note_buffer_released(7, 11_366, gndrm_uapi::wire::RELEASE_FENCE); + assert!(!take_released(11_411, 0, 0), "the inode matches nothing"); + assert!(take_released_ticket(7), "the ticket still frees its slot"); + assert!(!take_released_ticket(7), "and is consumed by it"); + // A note with no ticket is left for the inode path. + crate::scanout::note_buffer_released(0, 11_366, gndrm_uapi::wire::RELEASE_FENCE); + assert!(!take_released_ticket(0), "no ticket, no ticket match"); + assert!(take_released(11_366, 0, 0), "but the inode path still has it"); + } + + /// A note naming an earlier frame than the park's cannot answer it by inode + /// or by order; a note naming a later frame still can. + #[test] + fn a_note_for_an_earlier_ticket_does_not_answer_a_later_park() { + let _serial = fresh_notes(); + // Buffer 5 000 parked as ticket 10, expired, committed again as 13: the + // late note for 10 names the same buffer and must not free 13. + note_buffer_released(10, 5_000, gndrm_uapi::wire::RELEASE_FENCE); + assert!(!take_released(5_000, 0, 13), "a stale note for the same buffer"); + assert!(!take_any_released(0, 13), "and it was dropped, not left for order"); + + // A note for a commit that was never parked (ticket 20), with the oldest + // park at 21: skipped by order, while a note for 22 behind it pays. + note_buffer_released(20, 6_000, gndrm_uapi::wire::RELEASE_FENCE); + note_buffer_released(22, 6_001, gndrm_uapi::wire::RELEASE_FENCE); + assert!(take_any_released(0, 21), "the newer note answers the older park"); + assert!(!take_any_released(0, 21), "the older note does not"); + assert!(take_released_ticket(20), "and is still there for its own ticket"); + } + + /// Its own note, not one an earlier test happened to leave: the suite runs + /// these in parallel, so a test that reads a static another test writes is a + /// test that passes by scheduler luck. + /// Serialises the tests that inject and consume release notes: `RELEASED` is a + /// process-wide queue, so two of them running at once consume each other's + /// notes and pass or fail by scheduler luck. + static NOTE_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + + /// Empty the queue so a test starts from a known state whatever ran before. + fn fresh_notes() -> std::sync::MutexGuard<'static, ()> { + let g = NOTE_TEST_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + RELEASED.lock().unwrap_or_else(|e| e.into_inner()).clear(); + g + } + + /// A note that names no parked slot is still usable by order: notes arrive one + /// per commit, so the oldest outstanding one belongs to the oldest outstanding + /// frame. This is the Odin 3 path — `slot ino=11411` against + /// `released=[11366 11368 11370]`. + #[test] + fn a_note_can_be_taken_by_order_when_its_inode_matches_nothing() { + let _serial = fresh_notes(); + // Notes for buffers the caller will never ask about, newest last. + for ino in 8_001..=8_003u64 { + note_buffer_released(0, ino, gndrm_uapi::wire::RELEASE_FENCE); + } + // Identity never matches this ino, so the slot would wait out its bound. + assert!(!take_released(9_999, 0, 0), "no note names this inode"); + // By order the oldest note is spent, and only one per call. + assert!(take_any_released(0, 0), "the oldest note is available"); + assert!(take_any_released(0, 0)); + assert!(take_any_released(0, 0)); + assert!(!take_any_released(0, 0), "and the list is spent"); + } + + /// A note may be up to one display period older than the park's commit — the + /// note for a frame rides the latch of the frame that replaced it — and no + /// more. + #[test] + fn a_note_more_than_one_period_older_than_the_commit_is_not_taken() { + let _serial = fresh_notes(); + let hz = u64::from(super::super::lsfg::panel_hz().max(1)); + let period = 1_000_000_000 / hz; + note_buffer_released(0, 8_500, gndrm_uapi::wire::RELEASE_FENCE); + // Parked two periods after the note: out of the margin, and dropped. + let parked = crate::proxy::monotonic_ns() + 2 * period; + assert!(!take_any_released(parked, 0), "older than one frame"); + assert!(!take_any_released(parked, 0), "and it is dropped, not kept"); + + // Within the margin it is usable: that is the case the floor exists for. + note_buffer_released(0, 8_501, gndrm_uapi::wire::RELEASE_FENCE); + let within = crate::proxy::monotonic_ns() + period / 2; + assert!(take_any_released(within, 0), "inside the one-frame margin"); + } + + #[test] + fn release_notes_are_bounded_and_observed() { + let _serial = fresh_notes(); + note_buffer_released(0, 6_999, gndrm_uapi::wire::RELEASE_FENCE); + assert!(release_notes_live(), "the note above is recorded"); + for ino in 0..(RELEASED_CAP as u64 + 8) { + note_buffer_released(0, 7_000 + ino, gndrm_uapi::wire::RELEASE_FENCE); + } + let held = RELEASED.lock().unwrap_or_else(|e| e.into_inner()).len(); + assert!(held <= RELEASED_CAP, "held {held} of at most {RELEASED_CAP}"); + } + + /// Bind a `SOCK_SEQPACKET` listener on the abstract namespace, the way the /// daemon's scan-out acceptor does. fn listen(name: &str) -> RawFd { @@ -891,6 +1274,17 @@ mod tests { // The daemon answers a subscribe with the current target, ahead // of any reply: a note where a reply would be expected. send_bare(conn, &FrameCapNote { fps: 120 }.encode()); + // And the release watcher pushes whenever a fence signals. + send_bare( + conn, + &BufferReleased { + ticket: 4_242, + ino: 4_243, + latch_ns: 0, + source: gndrm_uapi::wire::RELEASE_FENCE, + } + .encode(), + ); let mut buf = [0u8; ScanoutAllocReq::LEN]; let n = libc::recv(conn, buf.as_mut_ptr().cast(), buf.len(), 0); assert_eq!(n as usize, ScanoutAllocReq::LEN); @@ -914,6 +1308,7 @@ mod tests { } }); + let _serial = fresh_notes(); let fd = connect(&name); assert!(fd >= 0, "connect: {}", std::io::Error::last_os_error()); let got = exchange(fd, 1280, 960) @@ -923,6 +1318,9 @@ mod tests { assert_eq!(got.ino, ino); // The note ahead of the reply set the target. assert_eq!(target_fps(), 120); + // The release note ahead of the reply was queued, not taken for a dead + // channel. + assert!(take_released_ticket(4_242)); // SAFETY: the received descriptor and the connection are live. unsafe { libc::close(got.fd); @@ -930,38 +1328,11 @@ mod tests { } } - /// The acquire schedule: one period after the previous start, never closer, - /// and never later than the guest's own next frame when it is slower. - #[test] - fn the_cadence_does_not_carry_the_wakeup_overshoot() { - let period = 16_666_666; - let first = next_acquire_after(0, 1_000_000_000, period); - assert_eq!(first, 1_016_666_666); - // The wait returned 150 us late: the next deadline is still one period on. - let second = next_acquire_after(first, first + 150_000, period); - assert_eq!(second, first + period); - // A whole period late: no debt. - let third = next_acquire_after(second, second + period + 5_000_000, period); - assert_eq!(third, second + period + 5_000_000 + period); - } - + /// The cap's period is truncated, never rounded up past `1 / target`, and no + /// cap has none. The schedule built on it lives in `crate::vsync_gate`. #[test] - fn the_acquire_schedule_spaces_frame_starts() { - let period = period_ns(40); - assert_eq!(period, 25_000_000); - // The first acquire schedules the next one a period later. - let first = next_acquire_after(0, 1_000_000_000, period); - assert_eq!(first, 1_025_000_000); - // A guest faster than the target: every start is exactly one period on. - assert_eq!(next_acquire_after(first, 1_001_000_000, period), 1_050_000_000); - assert_eq!(next_acquire_after(1_050_000_000, 1_002_000_000, period), 1_075_000_000); - // A guest slower than the target keeps its own rate: the gate is a lower - // bound, not a delay. - assert_eq!(next_acquire_after(1_075_000_000, 1_200_000_000, period), 1_225_000_000); - // No cap: the schedule is left alone, so enabling one later does not - // inherit a stale deadline. - assert_eq!(next_acquire_after(1_225_000_000, 5, 0), 1_225_000_000); - // A period is truncated, never rounded up past `1 / target`. + fn a_caps_period_is_truncated_and_no_cap_has_none() { + assert_eq!(period_ns(40), 25_000_000); assert_eq!(period_ns(FPS_UNCAPPED), 0); assert_eq!(period_ns(3), 333_333_333); } diff --git a/icd_shim/src/vsync_gate.rs b/icd_shim/src/vsync_gate.rs index d479d0a..0c1f97c 100644 --- a/icd_shim/src/vsync_gate.rs +++ b/icd_shim/src/vsync_gate.rs @@ -1,99 +1,46 @@ -//! The cap's acquire gate when gamescope's limiter does not hold the frames. +//! The cap's acquire gate: one regulator, the rate. //! -//! gamescope's limiter holds only commits declared FIFO -//! (`wlserver_surface_is_fifo`): a mailbox, immediate or FIFO-relaxed commit is -//! taken at the next composite whatever the limiter says. When the container -//! asks for one of those modes and a cap is set, the shim's acquire gate is -//! the cap. A gate on a clock of its own (one release every `1e9 / cap` ns) -//! beats against the display: a 60 cap on a 59.97 Hz panel put two releases in -//! one vblank every half minute or so and ran 57-59 fps. This gate releases -//! frames on the display's vblank grid instead, the grid the daemon stamps its -//! flip events on and gamescope composites from. +//! A frame may start one `1e9 / cap` after the previous frame started, and not +//! before. That is the whole rule: //! -//! The grid is `anchor - stamp_lead + k * period` from the daemon's -//! [`VsyncNote`]: `SurfaceFlinger`'s latch points, measured, moved earlier by -//! the lead the daemon stamps flip events with. gamescope wakes to composite -//! just before each of those instants (wakeup = stamp + interval - draw time - -//! red zone), so a release on one starts the guest's frame right after -//! gamescope took the previous frame and gives it the whole period, less -//! gamescope's own margin, to reach the next composite. +//! ```text +//! gate: while now < next_ns { wait until next_ns } +//! book: next_ns = now + period (now is the release just granted) +//! ``` //! -//! Which grid points release a frame: -//! - a cap at the display's rate (within [`SNAP_TOLERANCE`]): every vblank; -//! - a cap that divides the display's rate: every `refresh / cap`-th vblank; -//! - any other cap below the rate: the vblank nearest each instant of an ideal -//! `1e9 / cap` timeline, which averages the cap on vblank boundaries (45 on -//! 60 Hz releases 3 of every 4 vblanks); -//! - a cap above the rate: every vblank already gets a frame, so the gate runs -//! on the cap's own clock ([`crate::scanout::next_acquire_after`]). +//! What follows from it by construction, not by tuning: //! -//! A frame that arrives after its release point is not held: it starts at once, -//! and the release after it is the next point of the schedule, so a guest -//! slower than the cap runs at its own rate rather than at a divisor of the -//! display's. +//! - **The cap is a maximum, never exceeded.** Two consecutive releases are at +//! least `period` apart, because the second is booked one period after the +//! first *actually happened* — not after the instant it was supposed to. +//! - **A stall carries no debt.** A late frame is released at once and books the +//! next one a period after itself, so a stall costs the frames it swallowed and +//! nothing bursts afterwards. +//! - **A guest slower than the cap is never held.** Its next acquire is already +//! past `next_ns`, so the gate does not wait at all. //! -//! The grid exists only once the daemon has sent a note with a stamp lead -//! ([`current_grid`]). Released on the latch points themselves (a daemon that -//! predates the field sends 0), a frame starts ~2 ms after gamescope's wakeup -//! and straddles the next composite: measured on a Retroid Pocket 6, 58.3 fps -//! and 102 frames replaced in gamescope's mailbox per minute at a 60 cap on a -//! 59.975 Hz panel, and 13 % of the daemon's publishes late at 120 Hz. Until -//! then the shim declares FIFO under a cap and gamescope's limiter holds it -//! (`proxy::choose_live_mode`). - -use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; - -use gndrm_uapi::wire::{VsyncNote, FPS_UNCAPPED}; - -const NANOS_PER_SECOND: u64 = 1_000_000_000; - -/// How close `refresh / cap` has to be to a whole number for the cap to be -/// taken as that divisor of the display's rate, as a fraction of it. A "60 Hz" -/// panel measures 59.94-59.99 Hz, 0.02-0.1 % off a 60 cap; 0.5 % keeps a 59 -/// cap (1.7 % off) on the averaging timeline, where it gets 59. -const SNAP_TOLERANCE: f64 = 0.005; - -/// The display's vblank grid as the daemon last reported it; a period of 0 -/// means no report yet. Atomics: written by whichever thread reads the scan-out -/// channel, read by every acquire. -static ANCHOR_NS: AtomicU64 = AtomicU64::new(0); -static PERIOD_NS: AtomicU32 = AtomicU32::new(0); -static STAMP_LEAD_NS: AtomicU32 = AtomicU32::new(0); - -/// Adopt the daemon's latch grid (see [`VsyncNote`]). -pub(crate) fn apply_note(note: VsyncNote) { - ANCHOR_NS.store(note.anchor_ns, Ordering::Relaxed); - STAMP_LEAD_NS.store(note.stamp_lead_ns, Ordering::Relaxed); - PERIOD_NS.store(note.period_ns, Ordering::Relaxed); -} +//! What it does not do is keep releases on the display's vblank grid. That was +//! the previous design, and it is why two regulators existed: a grid-snapped +//! release holds the phase but bounds only the *average* rate, so a release that a +//! slot wait delayed was followed by a booking on the next vblank — a fraction of +//! an interval later — and the cap leaked. Measured on this path: 74.7 frames/s +//! under a 60 cap, the gate holding 8 % of acquires. With one regulator the rate +//! is exact and the phase is wherever the releases fall; the daemon's latch grid +//! stays where a phase is genuinely needed, in frame generation +//! (`crate::lsfg`), which places generated frames rather than limiting a rate. +//! +//! gamescope's own limiter is not a substitute for this gate: it withholds a +//! frame callback this shim never waits on, and its divisor only acts when the +//! cap is below the refresh *and* divides it. -/// The vblank grid releases sit on: the daemon's flip-stamp grid, `None` -/// until a note carrying a stamp lead has arrived (0 means the daemon did not -/// say; one that measures a lead always sends at least 1 ns). -pub(crate) fn current_grid() -> Option { - let period = u64::from(PERIOD_NS.load(Ordering::Relaxed)); - let lead = u64::from(STAMP_LEAD_NS.load(Ordering::Relaxed)); - if period == 0 || lead == 0 { - return None; - } - let anchor = ANCHOR_NS.load(Ordering::Relaxed); - Some(Grid { - anchor_ns: anchor.saturating_sub(lead), - period_ns: period, - clock: GateClock::Vsync, - }) -} +use std::sync::atomic::{AtomicU64, Ordering}; /// What the acquire gate releases on, as the census names it. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum GateClock { - /// No gate: nothing to cap, or gamescope's limiter holds the frames. + /// No gate: no cap, and nothing to hold. Off, - /// The display's vblank grid from the daemon's note. - Vsync, - /// The cap's own `1e9 / cap` clock: a cap above the display's rate, a - /// gate that is not this module's (frame generation's source rate, the X11 - /// Present route), or no grid known. + /// The cap's own `1e9 / cap` clock. Clock, } @@ -101,110 +48,54 @@ impl GateClock { pub(crate) fn name(self) -> &'static str { match self { GateClock::Off => "off", - GateClock::Vsync => "vsync", GateClock::Clock => "clock", } } } -/// Release instants `anchor_ns + k * period_ns`, `CLOCK_MONOTONIC`. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub(crate) struct Grid { - pub anchor_ns: u64, - pub period_ns: u64, - pub clock: GateClock, -} - -impl Grid { - /// The last grid instant at or before `t_ns`. - fn at_or_before(&self, t_ns: u64) -> u64 { - let p = i128::from(self.period_ns.max(1)); - let off = i128::from(t_ns) - i128::from(self.anchor_ns); - (i128::from(self.anchor_ns) + off.div_euclid(p) * p) as u64 - } - - /// The grid instant nearest `t_ns` (the later one on a tie). - fn nearest(&self, t_ns: u64) -> u64 { - self.at_or_before(t_ns.saturating_add(self.period_ns / 2)) - } -} - /// How an acquire is paced. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) enum Pacing { Off, - /// One release per `period_ns` on the shim's clock. - Clock { - period_ns: u64, - }, - /// Releases on `grid`, spaced by `interval_ns` of the ideal timeline. - Grid { - grid: Grid, - interval_ns: u64, - }, + /// One release per `period_ns`. + Clock { period_ns: u64 }, } impl Pacing { - /// A cap of `fps` on the shim's own clock, off when uncapped. + /// A gate on `fps`: the period is the cap's own, and no cap is no gate. #[must_use] pub(crate) fn on_clock(fps: u32) -> Self { - match crate::scanout::period_ns(fps) { - 0 => Pacing::Off, - period_ns => Pacing::Clock { period_ns }, - } + pacing(fps) } + /// The clock this gate runs on, for the census. + #[must_use] pub(crate) fn clock(self) -> GateClock { match self { Pacing::Off => GateClock::Off, Pacing::Clock { .. } => GateClock::Clock, - Pacing::Grid { grid, .. } => grid.clock, } } } -/// How a cap of `cap_fps` is released on `grid` (see the module notes). +/// How a cap of `cap_fps` is released: [`Pacing::Off`] with no cap, else one +/// release per `1e9 / cap`. #[must_use] -pub(crate) fn pacing(cap_fps: u32, grid: Option) -> Pacing { - if cap_fps == FPS_UNCAPPED { - return Pacing::Off; - } - let cap_period = NANOS_PER_SECOND / u64::from(cap_fps); - let Some(grid) = grid.filter(|g| g.period_ns != 0) else { - return Pacing::Clock { - period_ns: cap_period, - }; - }; - let ratio = cap_period as f64 / grid.period_ns as f64; - let whole = ratio.round(); - if whole >= 1.0 && (ratio - whole).abs() <= whole * SNAP_TOLERANCE { - Pacing::Grid { - grid, - interval_ns: whole as u64 * grid.period_ns, - } - } else if ratio < 1.0 { - Pacing::Clock { - period_ns: cap_period, - } - } else { - Pacing::Grid { - grid, - interval_ns: cap_period, - } +pub(crate) fn pacing(cap_fps: u32) -> Pacing { + match crate::scanout::period_ns(cap_fps) { + 0 => Pacing::Off, + period_ns => Pacing::Clock { period_ns }, } } -/// How late after its release point an acquire can start and still count as -/// on time: the wake-up of the gate's own wait, with room for a loaded CPU. -const ON_TIME_NS: u64 = 500_000; - -/// Late starts booked by [`Schedule::book`], and the release points they moved -/// past, for the census. +/// Acquires that started more than a period after their release point, and the +/// periods the schedule moved past for them: the guest's own lateness, for the +/// census. It is not a cap measurement — with one regulator the cap holds +/// whatever this says. static LATE_N: AtomicU64 = AtomicU64::new(0); static SKIPPED_N: AtomicU64 = AtomicU64::new(0); -/// `(late, skipped)`: acquires that started after their release point, and the -/// release points the schedule moved past for them. +/// `(late, skipped)`. pub(crate) fn late_counts() -> (u64, u64) { ( LATE_N.load(Ordering::Relaxed), @@ -212,104 +103,66 @@ pub(crate) fn late_counts() -> (u64, u64) { ) } -/// Whether a cap of `cap_fps` is the display's rate on `grid` (`refresh / cap` -/// snaps to 1): the gate then releases every vblank. -#[must_use] -pub(crate) fn at_display_rate(cap_fps: u32, grid: Grid) -> bool { - matches!( - pacing(cap_fps, Some(grid)), - Pacing::Grid { interval_ns, .. } if interval_ns == grid.period_ns - ) -} - -/// A swapchain's release schedule on the grid. +/// One swapchain's release schedule: the earliest instant the next acquire may +/// start, and nothing else. There is no second timeline to keep in step with it, +/// which is what makes the rate exact. #[derive(Debug, Default, Clone, Copy, PartialEq, Eq)] pub struct Schedule { - /// The grid instant the next acquire may start at; 0 before the first. + /// The earliest instant the next acquire may start; 0 before the first. pub next_ns: u64, - /// The ideal-timeline instant `next_ns` was rounded from. - ideal_ns: u64, } impl Schedule { - /// Book an acquire that started at `now_ns` (at or after `next_ns`) and - /// return the instant the one after it may start. + /// Book a release that happened at `now_ns` and return the instant the next + /// one may happen. /// - /// An acquire that started before the grid instant after its release point - /// (on time, or late by less than a period) is booked to that release - /// point, and the schedule goes on from there: the next release is - /// `interval` further along the ideal timeline, on the grid instant - /// nearest it. A start later than that but still inside its slot (before - /// `interval` has passed on the ideal timeline) is booked to the grid - /// instant it is in and keeps the slot's timeline, so the release after it - /// is at most the rest of the slot away and the cap holds on average. A - /// start past the whole slot, or the first, starts the timeline again from - /// the grid instant at or before it, so a stall carries no debt. The - /// release point is read on the grid as it is now: the daemon re-pushes - /// the phase as it moves, and a point booked on the old phase must not let - /// two releases into one period of the new one. Starts later than - /// [`ON_TIME_NS`] after their point are counted (`gate_late`), with the - /// points passed (`skipped`). + /// One line, on a grid. Booking from the released instant (`now + period`) + /// keeps no grid at all: every frame that starts late hands its lateness to + /// the frame after it, forever, so a 30 cap settles at `30 / (1 + e)` for + /// whatever jitter `e` the guest has. Measured on Witcher 3 at + /// `acq_fps=27.74` against `acquire=30` — 7 % lost with the CPU and the GPU + /// idle — and it is the same defect `A52` fixed in gamescope's limiter, in + /// the same shape: a release grid in time, where a late frame delays one + /// release and the grid absorbs it on the next. /// - /// The release after a late start is never shortened to the next grid - /// instant: that rule (the first shim of payload 20261103.4) let a guest - /// whose frame outlasts one grid period plus [`ON_TIME_NS`] arrive late for - /// every release, so a cap below the display's rate stopped holding at all - /// (cap 20 on a 120 Hz panel ran at the guest's own 40-55 fps; at 60 Hz - /// only a guest faster than ~58 fps recovered). - pub(crate) fn book(&mut self, now_ns: u64, grid: Grid, interval_ns: u64) -> u64 { - if grid.period_ns == 0 { - return self.book_clock(now_ns, interval_ns); + /// So the next release is one period after the previous *release point*, not + /// after the moment this frame happened to start. A frame that is late by + /// less than a period gets a proportionally shorter next hold, which is how + /// the average rate stays the target. + /// + /// A frame late by a whole period or more is a lost frame, not a debt: the + /// grid steps past `now` to the first point still in the future rather than + /// repaying with back-to-back releases. Repaying would put every frame in the + /// stall's shadow into the following period, which is exactly the burst this + /// rule exists to prevent. + pub(crate) fn book(&mut self, now_ns: u64, period_ns: u64) -> u64 { + let period_ns = period_ns.max(1); + if self.next_ns == 0 { + // First release: the grid is anchored here and nowhere else. + self.next_ns = now_ns.saturating_add(period_ns); + return self.next_ns; } - let at = grid.at_or_before(now_ns); - let due = (self.next_ns != 0).then(|| grid.nearest(self.next_ns)); - // Late against the instant the acquire was released for, which after a - // clock booking is off the grid. - let late = due.filter(|_| now_ns >= self.next_ns.saturating_add(ON_TIME_NS)); - if let Some(due) = late { + let target = self.next_ns.saturating_add(period_ns); + if target <= now_ns { + // Behind the grid by a period or more: this frame is late by more + // than the release it was owed, so that release is a lost frame and + // the grid re-anchors here. Re-anchoring rather than stepping to the + // next grid point keeps the hold at exactly one period in both cases, + // and it cannot burst: `now + period` is at least a period away. + let late = now_ns.saturating_sub(self.next_ns); LATE_N.fetch_add(1, Ordering::Relaxed); - SKIPPED_N.fetch_add(at.saturating_sub(due) / grid.period_ns, Ordering::Relaxed); + SKIPPED_N.fetch_add((late / period_ns).max(1), Ordering::Relaxed); + self.next_ns = now_ns.saturating_add(period_ns); + return self.next_ns; } - let (slot, ideal) = match due { - Some(due) if at <= due => (due, self.ideal_ns), - // Late by a period or more but still inside its slot: the slot's - // timeline goes on, so the frame after this one is due where the - // cap says, not a whole interval after a late start. - Some(_) if now_ns < self.ideal_ns.saturating_add(interval_ns) => (at, self.ideal_ns), - _ => (at, at), - }; - let ideal_next = ideal.saturating_add(interval_ns); - // Never two releases on one grid instant. - let next = grid - .nearest(ideal_next) - .max(slot.saturating_add(grid.period_ns)); - self.ideal_ns = ideal_next; - self.next_ns = next; - next - } - - /// Book an acquire that started at `now_ns` on the shim's own clock - /// ([`crate::scanout::next_acquire_after`]). - pub(crate) fn book_clock(&mut self, now_ns: u64, period_ns: u64) -> u64 { - self.next_ns = crate::scanout::next_acquire_after(self.next_ns, now_ns, period_ns); - // `book` reads `ideal_ns` as an absolute instant: leaving it at 0 - // resolves `nearest(ideal_ns + interval)` near CLOCK_MONOTONIC zero and - // the schedule collapses to one release per vblank. - self.ideal_ns = self.next_ns; + self.next_ns = target; self.next_ns } - /// Book an acquire that started at `now_ns` under `pacing`; `next_ns` is - /// left alone when there is no gate. + /// Book a release under `pacing`; a gate that is off books nothing. pub(crate) fn book_paced(&mut self, now_ns: u64, pacing: Pacing) { - match pacing { - Pacing::Off => {} - Pacing::Clock { period_ns } => { - self.book_clock(now_ns, period_ns); - } - Pacing::Grid { grid, interval_ns } => { - self.book(now_ns, grid, interval_ns); - } + if let Pacing::Clock { period_ns } = pacing { + self.book(now_ns, period_ns); } } } @@ -329,358 +182,148 @@ pub(crate) fn smooth_latency(estimate_ns: u64, sample_ns: u64) -> u64 { mod tests { use super::*; - /// A 59.97 Hz panel, as the Retroid Pocket 6 measures. - const P60: u64 = 16_675_004; - /// A 120 Hz panel. - const P120: u64 = 8_333_333; - - fn grid(period: u64) -> Grid { - Grid { - anchor_ns: 1_000_000_000, - period_ns: period, - clock: GateClock::Vsync, - } - } - - /// Releases of a guest that is always ready: it asks again the instant it - /// is let go (plus a wake-up jitter), for `n` frames. - fn run_ready(g: Grid, interval: u64, n: usize) -> Vec { - let mut s = Schedule::default(); - let mut now = g.anchor_ns; - let mut out = Vec::with_capacity(n); - for _ in 0..n { - if s.next_ns != 0 && now < s.next_ns { - now = s.next_ns; - } - let start = now + 50_000; // the wait returns a hair late - out.push(start); - s.book(start, g, interval); - now = start + 1_000_000; // a frame's CPU work before the next acquire - } - out - } - - fn interval(p: Pacing) -> u64 { - match p { - Pacing::Grid { interval_ns, .. } => interval_ns, - other => panic!("expected a grid gate, got {other:?}"), - } - } - - /// The vblank index each release fell in. - fn vblanks(g: Grid, starts: &[u64]) -> Vec { - starts - .iter() - .map(|&t| (g.at_or_before(t) - g.anchor_ns) / g.period_ns) - .collect() - } + const P60: u64 = 16_666_666; - #[test] - fn a_cap_at_the_display_rate_releases_once_per_vblank() { - let g = grid(P60); - // 60 on a 59.97 Hz panel: the display's rate, not 60.000. - let p = pacing(60, Some(g)); - assert_eq!(interval(p), P60); - let starts = run_ready(g, interval(p), 600); - let v = vblanks(g, &starts); - assert!(v.windows(2).all(|w| w[1] == w[0] + 1), "one per vblank"); - // 600 releases over 600 vblanks: the panel's rate exactly. - assert_eq!(v[599] - v[0], 599); - // Phase: each release is on its grid instant, not wherever a free clock - // drifted to. - for &t in &starts { - assert_eq!(t - g.at_or_before(t), 50_000); + fn period(cap: u32) -> u64 { + match pacing(cap) { + Pacing::Clock { period_ns } => period_ns, + Pacing::Off => panic!("no cap {cap}"), } } #[test] - fn thirty_on_sixty_releases_every_second_vblank() { - let g = grid(P60); - let p = pacing(30, Some(g)); - assert_eq!(interval(p), 2 * P60); - let v = vblanks(g, &run_ready(g, interval(p), 300)); - assert!(v.windows(2).all(|w| w[1] == w[0] + 2)); - } - - #[test] - fn sixty_on_a_hundred_and_twenty_releases_every_second_vblank() { - let g = grid(P120); - let p = pacing(60, Some(g)); - assert_eq!(interval(p), 2 * P120); - let v = vblanks(g, &run_ready(g, interval(p), 300)); - assert!(v.windows(2).all(|w| w[1] == w[0] + 2)); - } - - #[test] - fn forty_five_on_sixty_averages_the_cap_on_vblanks() { - let g = grid(16_666_667); - let p = pacing(45, Some(g)); - assert_eq!(interval(p), NANOS_PER_SECOND / 45); - let v = vblanks(g, &run_ready(g, interval(p), 451)); - // Every gap is one or two vblanks, never zero. - assert!(v.windows(2).all(|w| (1..=2).contains(&(w[1] - w[0])))); - // 450 intervals of 1/45 s span 600 vblanks of a 60 Hz panel. - let span = v[450] - v[0]; - assert!((599..=601).contains(&span), "span {span}"); - } - + fn no_cap_is_no_gate_and_a_cap_is_its_own_period() { + assert_eq!(pacing(gndrm_uapi::wire::FPS_UNCAPPED), Pacing::Off); + assert_eq!(pacing(0), Pacing::Off); + assert_eq!(Pacing::Off.clock(), GateClock::Off); + assert_eq!(Pacing::Off.clock().name(), "off"); + assert_eq!(period(60), P60); + assert_eq!(period(30), 33_333_333); + assert_eq!(period(120), 8_333_333); + assert_eq!(Pacing::on_clock(60).clock(), GateClock::Clock); + } + + /// The cap is a rate: no two releases closer than one interval, whatever the + /// guest does between them — early, late, or stalled. #[test] - fn a_late_frame_starts_at_once_and_the_schedule_goes_on() { - let g = grid(P60); - let iv = interval(pacing(60, Some(g))); + fn consecutive_releases_are_never_closer_than_the_cap() { + let p = period(60); let mut s = Schedule::default(); - let first = g.anchor_ns + 10_000; - let due = s.book(first, g, iv); - assert_eq!(due, g.anchor_ns + P60); - // The guest comes back 6 ms after its release point. The gate does not - // hold it (it is past `next_ns`), and the release after it is the next - // vblank, not a whole period from now. - let late = due + 6_000_000; - assert!(late >= s.next_ns); - assert_eq!(s.book(late, g, iv), due + P60); - // Late by more than a period: assigned to the vblank it is in, and the - // next release is the one after. - let very_late = due + P60 + 3 * P60 + 2_000_000; - assert_eq!(s.book(very_late, g, iv), due + 5 * P60); - } - - /// A guest that is always ready, booked under `first` for `n_first` - /// frames, then under a 30 cap on a 60 Hz grid: the gaps, in vblanks, - /// between the grid releases. - fn gaps_after(first: Pacing, n_first: usize) -> Vec { - let g = grid(16_666_667); - let then = pacing(30, Some(g)); - let mut s = Schedule::default(); - let mut now = g.anchor_ns + 1_000_000; - for _ in 0..n_first { - now = now.max(s.next_ns); - s.book_paced(now, first); - now += 1_000_000; + let mut released = 1_000_000_000u64; + let mut due = s.book(released, p); + // A guest that comes back early is held to `due`; one that comes back + // late starts at once. Neither may be released inside one period. + for step in [15_000_000u64, 16_000_000, 20_000_000, 40_000_000, 16_700_000] { + let start = due.max(released + step); + assert!( + start.saturating_sub(released) >= p, + "released {} ns after the previous frame, under {p}", + start - released + ); + released = start; + due = s.book(start, p); } - let mut starts = Vec::new(); - for _ in 0..7 { - now = now.max(s.next_ns) + 50_000; - starts.push(now); - s.book_paced(now, then); - now += 1_000_000; - } - vblanks(g, &starts) - .windows(2) - .map(|w| w[1] - w[0]) - .collect() } + /// A release that overshoots its grid point does not move the grid. + /// + /// This is the defect the grid exists for: with `next = now + period`, every + /// frame whose sleep woke late pushed the next release back by that lateness, + /// so it accumulated. Witcher 3 measured `acq_fps=27.74` against a 30 cap — + /// 7 % lost to a few hundred microseconds of scheduling per frame. #[test] - fn a_grid_cap_binds_after_frames_booked_on_the_clock() { - // The first grid release follows the clock's last due point, which is - // off the grid, so it may land a vblank later; every one after it is - // two vblanks apart. Before the fix the gaps were [0, 1, 1, 1, 1, 1]. - let check = |gaps: Vec| { - assert!((2..=3).contains(&gaps[0]), "{gaps:?}"); - assert_eq!(gaps[1..], [2, 2, 2, 2, 2], "{gaps:?}"); - }; - // Frame generation turned off at a cap below the display's rate: the - // source-rate gate's clock (40 fps) first, then 30 on 60. - check(gaps_after(Pacing::on_clock(40), 10)); - // A cap moved from above the panel's rate (90 on 60, the cap's own - // clock) to below it. - check(gaps_after(Pacing::on_clock(90), 10)); - // And with no clock booking at all. - assert_eq!(gaps_after(Pacing::Off, 0), [2, 2, 2, 2, 2, 2]); - } - + fn an_overshoot_does_not_move_the_grid() { + let p = period(30); + let overshoot = 2_000_000; + let mut s = Schedule::default(); + let mut released = 0u64; + let mut due = s.book(released, p); + for _ in 0..100 { + // The schedule held the frame to `due`; the thread woke 2 ms late. + released = due + overshoot; + due = s.book(released, p); + } + // 101 periods of grid for 100 frames — not 101 periods plus 100 + // overshoots, which is what `now + period` produced. + assert_eq!(due, 101 * p, "the grid moved by the overshoot"); + assert_eq!(released, 100 * p + overshoot); + // The rate over the span is 30/s with one constant offset, so the error + // is bounded by the overshoot and does not grow with the frame count. + let span = released; + let fps = 1_000_000_000.0 * 100.0 / span as f64; + assert!((fps - 30.0).abs() < 0.05, "{fps}"); + assert!(30.0 - fps < 0.02, "lost {:.3} fps to a 2 ms overshoot", 30.0 - fps); + } + + /// A stall costs the frames it swallowed and nothing bursts after it: the + /// release after a long gap is one period from it, never a catch-up. #[test] - fn a_phase_push_does_not_let_two_releases_into_one_vblank() { - let g = grid(P60); - let iv = interval(pacing(60, Some(g))); + fn a_stall_carries_no_debt() { + let p = period(60); let mut s = Schedule::default(); - let due = s.book(g.anchor_ns + 10_000, g, iv); - // The daemon re-anchors the grid 400 us later; the guest is released - // on the old point, which is now just before the new one. - let moved = Grid { - anchor_ns: g.anchor_ns + 400_000, - ..g - }; - let next = s.book(due + 20_000, moved, iv); - assert_eq!(next, due + 400_000 + P60); - // And 400 us earlier: the old point is just after the new one. - let earlier = Grid { - anchor_ns: g.anchor_ns - 400_000, - ..g - }; - let after = s.book(next + 20_000, earlier, iv); - assert_eq!(after, next - 800_000 + P60); + let first = 1_000_000_000u64; + s.book(first, p); + let after_stall = first + 2_000_000_000; + assert_eq!(s.book(after_stall, p), after_stall + p); + // And the frame after that is another period on, not early. + assert_eq!(s.book(after_stall + p, p), after_stall + 2 * p); } + /// A guest slower than the cap is never held. #[test] - fn a_guest_below_the_cap_is_not_quantized_to_a_divisor() { - // 50 fps on a 60 Hz cap: every frame is late for its release point, so - // the gate never holds it, and it keeps its own 20 ms cadence. - let g = grid(16_666_667); - let iv = interval(pacing(60, Some(g))); + fn a_guest_below_the_cap_is_never_held() { + let p = period(60); let mut s = Schedule::default(); - let mut t = g.anchor_ns; + let mut t = 1_000_000_000u64; for _ in 0..100 { assert!(s.next_ns == 0 || t >= s.next_ns, "held at {t}"); - s.book(t, g, iv); + s.book(t, p); t += 20_000_000; } } + /// The schedule holds no credit: a run at the cap leaves exactly one period + /// outstanding, however many releases it contained. #[test] - fn a_late_frame_on_a_divisor_cap_keeps_its_slots_timeline() { - // 30 on 60, the guest late by half a vblank: it starts at once, and the - // release after it is where the cap puts it, two vblanks after the - // point this one was released for. - let g = grid(P60); - let iv = interval(pacing(30, Some(g))); + fn the_schedule_holds_no_credit() { + let p = period(60); let mut s = Schedule::default(); - let due = s.book(g.anchor_ns, g, iv); - assert_eq!(due, g.anchor_ns + 2 * P60); - assert_eq!(s.book(due + P60 / 2, g, iv), due + 2 * P60); - // Late by a whole vblank, still inside the slot: the release after it - // is still the slot's, so the hold is the rest of the slot, not a - // whole slot from the late start. - let next = s.next_ns; - assert_eq!(s.book(next + P60 + 100_000, g, iv), next + 2 * P60); - // Past the whole slot: the timeline restarts from the start, no debt. - let next = s.next_ns; - assert_eq!(s.book(next + 2 * P60 + 100_000, g, iv), next + 4 * P60); - // An on-time guest is still held to every second vblank. - let next = s.next_ns; - assert_eq!(s.book(next + 50_000, g, iv), next + 2 * P60); - } - - /// Releases of a guest whose acquire-to-acquire time is `frame_ns`, with - /// its wait returning `wake_ns` late, under `cap` on `g`, for `n` frames - /// with one `hitch_ns` stall at frame 100: the gaps, in vblanks, between - /// consecutive releases. - fn gaps_of_a_slow_guest(g: Grid, cap: u32, frame_ns: u64, wake_ns: u64, hitch_ns: u64, n: usize) -> Vec { - let iv = interval(pacing(cap, Some(g))); - let mut s = Schedule::default(); - let mut t = g.anchor_ns; - let mut starts = Vec::new(); - for i in 0..n { - if s.next_ns != 0 && t < s.next_ns { - t = s.next_ns + wake_ns; - } - starts.push(t); - s.book(t, g, iv); - t += frame_ns + if i == 100 { hitch_ns } else { 0 }; + let mut t = 1_000_000_000u64; + for _ in 0..10 { + t = s.book(t, p); } - vblanks(g, &starts).windows(2).map(|w| w[1] - w[0]).collect() - } - - /// `gaps` holds the cap's cadence `every` vblanks, except the release after - /// the hitch at index 100 and the two after it, which catch the slot's - /// timeline up (a start inside its slot keeps the timeline); over the whole - /// run there are never more releases than the cap allows. - fn assert_cadence(gaps: &[u64], every: u64) { - let off: Vec = gaps - .iter() - .enumerate() - .filter(|&(i, &gap)| gap != every && !(100..103).contains(&i)) - .map(|(i, _)| i) - .collect(); - assert!(off.is_empty(), "releases off the {every}-vblank cadence at {off:?}: {gaps:?}"); - assert!(gaps[100] > every, "the hitch frame was not late: {:?}", &gaps[98..104]); - let total: u64 = gaps.iter().sum(); - assert!(total >= every * gaps.len() as u64, "more releases than the cap: {:?}", &gaps[98..104]); + assert_eq!(s.next_ns, t); } #[test] - fn a_guest_slower_than_a_vblank_is_held_to_a_cap_below_the_rate() { - // Cap 20 on a 120 Hz grid, 18 ms frames (the guest alone would run at - // 55 fps), every wake-up 1 ms late (late for its point) and one 80 ms - // hitch: the releases stay on every sixth vblank, and the cadence is - // back within three releases of the hitch. On the first shim of - // payload 20261103.4 the first late start was followed at the next - // vblank, the 18 ms frame was late for that too, and the cap never - // held again (the guest ran at its own rate). - let gaps = gaps_of_a_slow_guest(grid(P120), 20, 18_000_000, 1_000_000, 80_000_000, 600); - assert_cadence(&gaps, 6); - // 22 ms frames at 60 Hz under cap 20: every third vblank. - let gaps = gaps_of_a_slow_guest(grid(P60), 20, 22_000_000, 1_000_000, 80_000_000, 600); - assert_cadence(&gaps, 3); + fn a_gate_that_is_off_books_nothing() { + let mut s = Schedule::default(); + s.book_paced(1_000_000_000, Pacing::Off); + assert_eq!(s.next_ns, 0); + s.book_paced(1_000_000_000, Pacing::Clock { period_ns: P60 }); + assert_eq!(s.next_ns, 1_000_000_000 + P60); } + /// The census counts the guest's lateness, not the cap's failure: a release + /// inside its interval is on time, one past it is booked. #[test] - fn a_guest_with_16_ms_of_work_is_never_skipped() { - // 16 ms from acquire to the next acquire on a 59.97 Hz grid, and every - // seventh frame 1.5 ms slower (late for its point): the late frames - // start at once, the release after each is the next grid instant, and - // every vblank gets a start. - let g = grid(P60); - let iv = interval(pacing(60, Some(g))); + fn only_a_release_past_a_whole_interval_is_late() { + let before = late_counts(); + let first = 1_000_000_000u64; + let mut s = Schedule::default(); - let mut t = g.anchor_ns; - let mut starts = Vec::new(); - for n in 0..600 { - if s.next_ns != 0 && t < s.next_ns { - t = s.next_ns + 50_000; - } - starts.push(t); - s.book(t, g, iv); - t += if n % 7 == 6 { 17_500_000 } else { 16_000_000 }; - } - let v = vblanks(g, &starts); - assert!( - v.windows(2).all(|w| w[1] == w[0] + 1), - "a vblank was skipped" - ); - } + s.book(first, P60); + s.book(first + P60, P60); + assert_eq!(late_counts().0, before.0, "on time is not late"); - /// One test for the process-global grid, so no other test races it. - #[test] - fn the_grid_is_the_daemons_flip_stamp_grid_and_needs_its_lead() { - // No note yet: no grid. - assert_eq!(current_grid(), None); - let note = VsyncNote { - anchor_ns: 5_000_000_000, - period_ns: P60 as u32, - lead_ns: 9_000_000, - stamp_lead_ns: 0, - }; - // A note without a stamp lead (a daemon that predates it): no grid. - apply_note(note); - assert_eq!(current_grid(), None); - // With one: the latch grid moved earlier by the stamp lead, the - // instants gamescope's vblanks are stamped at. - apply_note(VsyncNote { - stamp_lead_ns: 2_500_000, - ..note - }); - let g = current_grid().expect("the daemon's grid"); - assert_eq!(g.clock, GateClock::Vsync); - assert_eq!(g.period_ns, P60); - assert_eq!(g.anchor_ns, 5_000_000_000 - 2_500_000); - apply_note(VsyncNote { - anchor_ns: 0, - period_ns: 0, - lead_ns: 0, - stamp_lead_ns: 0, - }); - // No grid at all: the cap's own clock. - assert_eq!( - pacing(30, None), - Pacing::Clock { - period_ns: NANOS_PER_SECOND / 30 - } - ); - } + let mut s = Schedule::default(); + s.book(first, P60); + s.book(first + P60 + P60 / 2, P60); + assert_eq!(late_counts().0, before.0, "inside the interval is not late"); - #[test] - fn caps_above_the_rate_and_none_are_not_on_the_grid() { - let g = grid(P120); - assert_eq!(pacing(FPS_UNCAPPED, Some(g)), Pacing::Off); - assert_eq!( - pacing(190, Some(g)), - Pacing::Clock { - period_ns: NANOS_PER_SECOND / 190 - } - ); - // 59 on 60 Hz is not the display's rate: 59, on vblanks. - let g60 = grid(16_666_667); - assert_eq!(interval(pacing(59, Some(g60))), NANOS_PER_SECOND / 59); + let mut s = Schedule::default(); + s.book(first, P60); + s.book(first + 3 * P60, P60); + assert_eq!(late_counts().0, before.0 + 1, "past an interval is late"); } } diff --git a/modules/gndrm-android/src/asc.rs b/modules/gndrm-android/src/asc.rs index aa14bc1..1f3a0e7 100644 --- a/modules/gndrm-android/src/asc.rs +++ b/modules/gndrm-android/src/asc.rs @@ -58,7 +58,7 @@ use std::collections::{HashMap, HashSet}; use std::io; use std::os::fd::RawFd; use std::ptr; -use std::sync::atomic::{AtomicBool, AtomicI32, AtomicU32, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicI32, AtomicPtr, AtomicU32, AtomicU64, Ordering}; use std::sync::{Arc, Condvar, Mutex, MutexGuard, OnceLock, PoisonError}; use std::thread; use std::time::{Duration, Instant}; @@ -69,8 +69,9 @@ use gndrm_daemon::scanout::{ }; use gndrm_daemon::{CursorUpdate, FrameCap, Framebuffer, PlaneState, Presenter}; use gndrm_uapi::wire::{ - scanout_ahb_format, FrameAcquired, FrameCapSubscribe, LsfgConfig, PresentDone, PresentMark, - ScanoutAllocReq, ScanoutAllocResp, CAP_FRAME_CAP, CAP_PRESENT_ID, CAP_VSYNC_NOTE, + scanout_ahb_format, BufferReleased, FrameAcquired, FrameCapSubscribe, LsfgConfig, PresentDone, + PresentMark, ScanoutAllocReq, ScanoutAllocResp, CAP_FRAME_CAP, CAP_PRESENT_ID, + CAP_RELEASE_NOTE, CAP_VSYNC_NOTE, }; use crate::ffi; @@ -144,6 +145,150 @@ macro_rules! asc_log { /// rather than once per commit; the flag clears when the layer leaves PQ. static PQ_METADATA_SET: core::sync::atomic::AtomicBool = core::sync::atomic::AtomicBool::new(false); +// --------------------------------------------------------------------------- +// the release hold probe +// +// How long does SurfaceFlinger keep a buffer after showing it? The publisher +// only learns it where it waits for the previous release fence before reusing a +// registration, and the answer decides which release rule is safe: gamescope's +// page-flip event is emitted at publish, so a producer given a buffer at that +// event may still be writing into a buffer on the panel. Holding the release +// one flip longer (gamescope A56) fixes that by taking a buffer away from every +// producer — a 3-image swapchain loses half its depth. This measures the real +// number instead: hold / one display period. Below one period, that deferral is +// sufficient; above it, only SurfaceFlinger's own fence is. +// --------------------------------------------------------------------------- + +/// The last latch the completion callback reported, for the flip period. +static HOLD_LAST_LATCH_NS: AtomicU64 = AtomicU64::new(0); +/// The last [`HOLD_GAPS`] accepted latch gaps. One gap is a sample of the +/// display, but a burst of callbacks, or a second swapchain publishing between +/// two latches, makes it far too small — and a period that small turns every +/// hold into an outlier. The median of the last few is what the buckets are +/// percentages of. +const HOLD_GAPS: usize = 9; +/// A latch gap outside this band is not a display period: below the fastest +/// panel this ships on, above a stall. +const HOLD_GAP_MIN_NS: u64 = 3_000_000; +const HOLD_GAP_MAX_NS: u64 = 50_000_000; +static HOLD_GAP_RING: Mutex<[u64; HOLD_GAPS]> = Mutex::new([0; HOLD_GAPS]); +static HOLD_GAP_NEXT: AtomicU64 = AtomicU64::new(0); +/// One display period as the latch grid delivered it, 0 until two latches. +static HOLD_PERIOD_NS: AtomicU64 = AtomicU64::new(0); +/// Release-fence waits that produced a sample. +static HOLD_N: AtomicU64 = AtomicU64::new(0); +/// Waits that hit `RELEASE_FENCE_TIMEOUT_MS`: no hold, no sample. +static HOLD_TIMEOUTS: AtomicU64 = AtomicU64::new(0); +static HOLD_MAX_US: AtomicU64 = AtomicU64::new(0); +/// Hold as a percentage of one period, bucketed: <50, 50-100, 100-150, +/// 150-250, >250. +static HOLD_BUCKETS: [AtomicU64; 5] = [ + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), + AtomicU64::new(0), +]; +static HOLD_LAST_REPORT_NS: AtomicU64 = AtomicU64::new(0); + +/// One measured hold: `latch_ns` is when SurfaceFlinger showed the frame the +/// buffer carried, `released_ns` when its release fence signalled. +fn note_hold(latch_ns: u64, released_ns: u64) { + let hold_ns = released_ns.saturating_sub(latch_ns); + let hold_us = hold_ns / 1000; + HOLD_N.fetch_add(1, Ordering::Relaxed); + HOLD_MAX_US.fetch_max(hold_us, Ordering::Relaxed); + let period_ns = HOLD_PERIOD_NS.load(Ordering::Relaxed); + // Nothing to compare against until a few gaps have been seen: before the + // ring fills, a burst can still pass as a period and every hold then reads + // as an outlier. + if period_ns == 0 || HOLD_GAP_NEXT.load(Ordering::Relaxed) < HOLD_GAPS as u64 / 2 { + return; + } + let percent = hold_ns.saturating_mul(100) / period_ns; + let bucket = match percent { + 0..=49 => 0, + 50..=99 => 1, + 100..=149 => 2, + 150..=249 => 3, + _ => 4, + }; + HOLD_BUCKETS[bucket].fetch_add(1, Ordering::Relaxed); +} + +/// The release-fence wait ran out of budget, so this frame has no hold sample. +fn note_hold_timeout() { + HOLD_TIMEOUTS.fetch_add(1, Ordering::Relaxed); +} + +/// One line per 10 s naming SurfaceFlinger's hold distribution. Counts, not +/// percentages, so a run can be added up; `period` is the latch grid's own, and +/// the buckets are in halves of it. +fn report_hold(now_ns: u64) { + let last = HOLD_LAST_REPORT_NS.load(Ordering::Relaxed); + if last != 0 && now_ns.saturating_sub(last) < 10_000_000_000 { + return; + } + if HOLD_LAST_REPORT_NS + .compare_exchange(last, now_ns, Ordering::Relaxed, Ordering::Relaxed) + .is_err() + { + return; + } + // Marks, before the hold: a session whose marks do not resolve sends notes + // with no ticket, and that is invisible in every other counter. + let settled = MARK_HIT.load(Ordering::Relaxed) + + MARK_ORDER.load(Ordering::Relaxed) + + MARK_MISS.load(Ordering::Relaxed) + + MARK_STALE.load(Ordering::Relaxed) + + MARK_RESYNC.load(Ordering::Relaxed) + + MARK_SKEW.load(Ordering::Relaxed); + if settled != MARK_REPORTED.swap(settled, Ordering::Relaxed) { + let depth = PRESENT_PENDING + .lock() + .unwrap_or_else(|e| e.into_inner()) + .len(); + log_always!( + "{}", + format_args!( + "sf mark: hit={} order={} miss={} stale={} resync={} skew={} pending={} last={}", + MARK_HIT.load(Ordering::Relaxed), + MARK_ORDER.load(Ordering::Relaxed), + MARK_MISS.load(Ordering::Relaxed), + MARK_STALE.load(Ordering::Relaxed), + MARK_RESYNC.load(Ordering::Relaxed), + MARK_SKEW.load(Ordering::Relaxed), + depth, + MARK_LAST_TICKET.load(Ordering::Relaxed) + ) + ); + } + let n = HOLD_N.load(Ordering::Relaxed); + if n == 0 { + return; + } + let b = |i: usize| HOLD_BUCKETS[i].load(Ordering::Relaxed); + // Ungated: a shipped session cannot turn the daemon's gated channels on + // (`GNDRM_LOG_*` is set by the launcher in this process, not by a + // container), and this is the number that decides which release rule is + // safe. One line per 10 s is the same cadence as the shim's census. + log_always!( + "{}", + format_args!( + "sf hold: n={} Option<&'static std::ffi::OsString> { DUMP_DIR @@ -251,6 +396,11 @@ struct ReleaseCtx { release_fd: AtomicI32, /// The latch-time query, when the platform has it. latch_time: Option, + /// The latch SurfaceFlinger took the frame in this registration at, + /// `CLOCK_MONOTONIC` ns; 0 before the first. The publish that reuses the + /// registration reads it to measure how long SurfaceFlinger held the buffer + /// after showing it (see [`note_hold`]). + latch_ns: AtomicU64, /// When the daemon's apply of the transaction this registration was last /// published in returned, `CLOCK_MONOTONIC` ns; 0 before the first. applied_ns: AtomicU64, @@ -258,11 +408,40 @@ struct ReleaseCtx { /// when the app did not chain one. Written by the publish path, consumed by /// the completion callback, which reports it with the latch time. present_id: AtomicU64, + /// The shim's ticket for the frame currently in this registration: the key + /// its release note is filed under, and 0 before the first publish. This is + /// what makes a release note exact — the ticket is minted by the process + /// that owns the slot, so it cannot be ambiguous the way a dma-buf inode can. + ticket: AtomicU64, + /// The ticket this registration carried *when the transaction that replaced it + /// was applied* — the frame the release fence actually describes. + /// + /// Not `ticket`: that field is overwritten by the buffer's next publish, and the + /// completion callback runs on SurfaceFlinger's binder thread some time after the + /// apply. Normally long before the buffer is reused, but a delayed callback (a + /// stalled SF, a suspend) can arrive after the buffer has been published again — + /// and the note would then name the *newer* frame, freeing the slot that holds it + /// while the display may still be reading it. Snapshotting at the apply removes + /// the window: between the apply and the callback this buffer is on the layer and + /// cannot be published again. + release_ticket: AtomicU64, /// Set by the completion callback: SurfaceFlinger reported the flip event /// for this registration, so the buffer is safe to hand back out (the /// reuse cache keys on it). Never cleared: the flag belongs to this /// registration, and a registration is not reused. latched: Arc, + /// The scan-out inode this mailbox belongs to: the identity a release note + /// names, and 0 for a foreign slot the pool does not own. + ino: u64, + /// The mailbox of the buffer this registration's buffer replaced on the + /// layer, or null when it replaced nothing. + /// + /// This is what makes SurfaceFlinger's release fence attributable. Its + /// `getPreviousReleaseFenceFd` describes the buffer the transaction + /// *replaced*, not the one it published — so the fence a completion reports + /// belongs to `replaced_ctx`, and is filed there for that buffer's own + /// reuse and its own release note. + replaced_ctx: AtomicPtr, } // SAFETY: after construction the only mutations are the atomic and the release @@ -283,7 +462,7 @@ unsafe impl Sync for ReleaseCtx {} /// publishes them. One frame per buffer is in flight at a time, so a plain map /// is enough; it is bounded because a mark for a buffer that never gets /// committed must not accumulate. -static PRESENT_PENDING: Mutex> = Mutex::new(Vec::new()); +static PRESENT_PENDING: Mutex> = Mutex::new(Vec::new()); /// Completions not yet written to the shim: the callback runs on SurfaceFlinger's /// thread, the channel on the scan-out thread, and only the latter may send. @@ -334,29 +513,133 @@ static PRESENT_AWARE: AtomicBool = AtomicBool::new(false); /// marks frames and never waits cannot grow either list without bound. const PRESENT_QUEUE_CAP: usize = 64; +/// How marks resolve when a registration is published. +/// +/// `hit` matched by inode; `order` matched the oldest unclaimed mark because the +/// inode did not; `miss` found no mark at all; `skew` found one that would move +/// the ticket backwards and refused it. The first two are the pairing rule, the +/// last two are the two ways it can fail — and until they were counted, a session +/// that sent notes with `ticket: 0` was indistinguishable from one that sent no +/// notes: RP6 ran sixty seconds at 60 fps with `tick=[n:0]`, every release note +/// carrying no ticket because this lookup missed. +static MARK_HIT: AtomicU64 = AtomicU64::new(0); +static MARK_ORDER: AtomicU64 = AtomicU64::new(0); +static MARK_MISS: AtomicU64 = AtomicU64::new(0); +static MARK_SKEW: AtomicU64 = AtomicU64::new(0); +/// Marks dropped because their ticket was already published. An identity hit can +/// remove a mark from the middle of the queue, leaving older ones behind that can +/// never be assigned — discarding them is what stops the queue wedging on a head +/// no publish will ever match. +static MARK_STALE: AtomicU64 = AtomicU64::new(0); +/// Fallbacks that accepted a small forward gap: the marks in between were lost, not +/// mis-ordered. This is the counter that says whether `sendmsg` is dropping under +/// load, which nothing measured before. +static MARK_RESYNC: AtomicU64 = AtomicU64::new(0); + +/// The largest forward gap the order fallback will re-seed across. One lost mark is +/// a dropped send; four is a burst; more than that is a queue this code does not +/// understand, and the ticket path stands down until identity carries it. +const MARK_RESYNC_GAP: u64 = 4; +/// The highest ticket assigned to a registration: the order fallback may only +/// move it forward, so a lost or late mark cannot pair every later frame with the +/// one before it. +static MARK_LAST_TICKET: AtomicU64 = AtomicU64::new(0); +/// Marks reported so far, for the once-every-ten-seconds line. +static MARK_REPORTED: AtomicU64 = AtomicU64::new(0); + /// Record a frame's present id against the buffer carrying it. fn note_present_mark(mark: PresentMark) { - if mark.id == 0 || mark.ino == 0 { + // A mark with no ticket and no guest id says nothing: the shim sends one per + // commit now, so the guest id is usually 0 and only the ticket is present. + if (mark.id == 0 && mark.ticket == 0) || mark.ino == 0 { return; } let mut pending = PRESENT_PENDING.lock().unwrap_or_else(|e| e.into_inner()); - pending.retain(|&(ino, _)| ino != mark.ino); + pending.retain(|&(ino, _, _)| ino != mark.ino); if pending.len() >= PRESENT_QUEUE_CAP { pending.remove(0); } - pending.push((mark.ino, mark.id)); + pending.push((mark.ino, mark.id, mark.ticket)); } -/// Take the id marked for `ino`, if any: the commit naming that buffer is the -/// frame the id belongs to. -fn take_present_mark(ino: u64) -> u64 { +/// Take the `(id, ticket)` marked for `ino`: the commit naming that buffer is the +/// frame they belong to. +/// +/// Identity first, then order. The inode is the sharper match and is all that runs +/// where the shim's slot inode and this registration's inode are the same object — +/// true for Portal 2 on Nova, where only 6 % of parks needed a fallback. +/// +/// It is not available everywhere: the same device needed it for 84 % of Dave's +/// parks, and RP6 and Odin 3 needed it for all of them, so the mark that carries +/// the ticket would be dropped on exactly the devices the ticket exists for. Both +/// ends see the same commit sequence, so the oldest unclaimed mark belongs to the +/// publish being made now — no window, no floor, no reordering, unlike the shim's +/// own order arm, which pairs frames that have already been shown. +/// +/// The guard on the fallback is exactness, not monotonicity: the next publish must +/// take the *next* ticket, because one mark is sent per commit in order. A permissive +/// "greater than the last" rule accepts a mis-assigned ticket — a mark that belongs to +/// a different frame gets bound to this registration, and when that buffer's release +/// fires the shim frees the slot holding somebody else's frame. That is a wrong-slot +/// free, which is a tear, not a reordering: the present path never consults a ticket. +/// +/// The three outcomes below are what keeps the rule usable: +/// +/// - **stale** — the head's ticket was already published. Discarded and the queue +/// re-examined, because an identity hit can remove a mark from the middle and leave +/// older ones queued that no publish will ever match. Refusing instead would wedge +/// the head forever. +/// - **resync** — a small forward gap, so the marks in between were lost rather than +/// mis-ordered. Accepted and counted: this is the only measurement of `sendmsg` +/// dropping marks under load, which nothing had. +/// - **skew** — a gap beyond [`MARK_RESYNC_GAP`]. The queue and the publisher disagree +/// about where they are, so the ticket path stands down and identity carries it. +fn take_present_mark(ino: u64) -> (u64, u64) { let mut pending = PRESENT_PENDING.lock().unwrap_or_else(|e| e.into_inner()); - match pending.iter().position(|&(marked, _)| marked == ino) { - Some(i) => pending.remove(i).1, - None => 0, + if let Some(i) = pending.iter().position(|&(marked, _, _)| marked == ino) { + let (_, id, ticket) = pending.remove(i); + MARK_HIT.fetch_add(1, Ordering::Relaxed); + MARK_LAST_TICKET.fetch_max(ticket, Ordering::Relaxed); + return (id, ticket); + } + loop { + let last = MARK_LAST_TICKET.load(Ordering::Relaxed); + let Some(&(_, id, ticket)) = pending.first() else { + MARK_MISS.fetch_add(1, Ordering::Relaxed); + return (0, 0); + }; + if ticket == 0 || ticket <= last { + pending.remove(0); + MARK_STALE.fetch_add(1, Ordering::Relaxed); + continue; + } + if ticket > last + MARK_RESYNC_GAP { + MARK_SKEW.fetch_add(1, Ordering::Relaxed); + return (0, 0); + } + if ticket > last + 1 { + MARK_RESYNC.fetch_add(1, Ordering::Relaxed); + } + pending.remove(0); + MARK_ORDER.fetch_add(1, Ordering::Relaxed); + MARK_LAST_TICKET.store(ticket, Ordering::Relaxed); + return (id, ticket); } } +/// Forget every mark and the ticket order they were checked against. +/// +/// Tickets are minted per shim process and start again at 1 in the next one, so +/// the previous process's highest ticket would refuse every order match of the +/// new one as skew, and its unclaimed marks would sit at the head of the queue. +/// A reconnect of the same process loses at most the marks in flight, whose notes +/// then carry no ticket. +fn reset_present_marks() { + let mut pending = PRESENT_PENDING.lock().unwrap_or_else(|e| e.into_inner()); + pending.clear(); + MARK_LAST_TICKET.store(0, Ordering::Relaxed); +} + /// Report a completed present to the shim: queued here, written by the channel. fn note_present_done(id: u64, latch_ns: u64) { if id == 0 { @@ -424,14 +707,44 @@ unsafe extern "C" fn on_complete(context: *mut c_void, stats: *mut ffi::ASurface // exactly once here. unsafe { (ctx.stats_release_controls)(controls) }; } - if fd >= 0 - && ctx - .release_fd - .compare_exchange(-1, fd, Ordering::AcqRel, Ordering::Acquire) - .is_err() - { - // SAFETY: `fd` is a valid descriptor owned by this call. - unsafe { libc::close(fd) }; + // The fence describes the buffer this transaction *replaced*, not the one it + // published: SurfaceFlinger reports the release of whatever the new buffer + // took off the layer. So it is filed on the replaced buffer's mailbox — the + // one whose next publish waits on it, and whose scan-out inode names it to + // the producer — and a copy goes to the watcher that reports the release. + let prev = ctx.replaced_ctx.load(Ordering::Acquire); + if fd >= 0 { + if prev.is_null() { + // Nothing was on the layer before this transaction: there is no + // buffer whose release the fence could describe. + // SAFETY: `fd` is a valid descriptor owned by this call. + unsafe { libc::close(fd) }; + } else { + // Its own copy first: the mailbox's descriptor is consumed by that + // buffer's next publish (waited on, then closed), so the watcher + // must not poll a descriptor another thread may close under it. + // SAFETY: `prev` is a leaked mailbox, live for the session. + unsafe { + let watch = libc::dup(fd); + if watch >= 0 { + watch_release( + (*prev).ino, + (*prev).release_ticket.load(Ordering::Acquire), + watch, + (*prev).latch_ns.load(Ordering::Relaxed), + ); + } + if (*prev) + .release_fd + .compare_exchange(-1, fd, Ordering::AcqRel, Ordering::Acquire) + .is_err() + { + // A fence was already waiting there: the older one describes + // an earlier release of the same buffer and is superseded. + libc::close(fd); + } + } + } } // The latch time is a sample of the panel's latch grid, which the presenter // places generated frames on — and the answer a guest waiting on this @@ -443,6 +756,28 @@ unsafe extern "C" fn on_complete(context: *mut c_void, stats: *mut ffi::ASurface if latch > 0 { latch_ns = latch as u64; crate::vsync::sample(latch_ns, ctx.applied_ns.load(Ordering::Relaxed)); + // The flip cadence, for the release-hold probe: the gap between + // consecutive latches is one display period, which is what a hold + // is measured against. Gaps outside `HOLD_GAP_MIN_NS..=MAX` are a + // burst or a stall rather than a period, and the period kept is the + // median of the last few, because one gap is not a measurement. + let prev = HOLD_LAST_LATCH_NS.swap(latch_ns, Ordering::Relaxed); + if prev != 0 && latch_ns > prev { + let gap = latch_ns - prev; + if (HOLD_GAP_MIN_NS..=HOLD_GAP_MAX_NS).contains(&gap) { + let slot = (HOLD_GAP_NEXT.fetch_add(1, Ordering::Relaxed) + % HOLD_GAPS as u64) as usize; + let mut ring = HOLD_GAP_RING.lock().unwrap_or_else(|e| e.into_inner()); + ring[slot] = gap; + let mut vals: Vec = ring.iter().copied().filter(|g| *g != 0).collect(); + drop(ring); + vals.sort_unstable(); + if let Some(mid) = vals.get(vals.len() / 2) { + HOLD_PERIOD_NS.store(*mid, Ordering::Relaxed); + } + } + } + ctx.latch_ns.store(latch_ns, Ordering::Relaxed); } } note_present_done(ctx.present_id.swap(0, Ordering::AcqRel), latch_ns); @@ -453,6 +788,193 @@ unsafe extern "C" fn on_complete(context: *mut c_void, stats: *mut ffi::ASurface diag::diag().set_role(Role::PresenterCallback, State::Idle); } +// ------------------------------------------------------- scan-out releases +// +// A compositor hands a buffer back when it has finished *compositing* with it. +// The display may still be reading it: on this path the consumer is +// SurfaceFlinger, which reports a buffer's true release a transaction later, as +// the previous-release fence of whoever replaced it. A producer that took the +// compositor's release as its own therefore wrote into memory that was still +// being scanned out, and the panel showed two frames at once for however long +// the overlap lasted. +// +// The completion callback already receives that fence, attributed to the buffer +// the transaction replaced (see [`ReleaseCtx::replaced_ctx`]). It is waited on +// here, off the callback thread, and reported to the producer as a +// [`BufferReleased`] note — the one fact no compositor can give a producer. + +/// Release fences waiting for the watcher, `(ino, fd, latch_ns of that buffer)`. +/// Added by the completion callback, which must not block: a fence is waited on +/// here instead, and the latch travels with it so the hold is measured where it +/// ends rather than when the buffer happens to come round again. +static RELEASE_WATCH: Mutex> = Mutex::new(Vec::new()); + +/// Eventfd the callback writes to wake the watcher, so a fence added between +/// polls is picked up at once rather than at the end of one. -1 until the +/// watcher has started; a fence queued before that is drained by its first pass. +static RELEASE_WAKE: AtomicI32 = AtomicI32::new(-1); + +/// Whether the watcher thread exists. +static RELEASE_THREAD: AtomicBool = AtomicBool::new(false); + +/// How long one fence may stay unsignalled before the release is reported +/// anyway. A fence SurfaceFlinger never signals means the compositor is stuck; +/// holding the producer behind it would turn that into a hang, so the buffer is +/// reported released and the case is counted. +const RELEASE_WATCH_MAX: Duration = Duration::from_millis(500); + +/// Fences the watcher may hold at once. A session has one per buffer in the +/// pool; a producer that stopped presenting leaves its last ones behind, and +/// they must not accumulate. +const RELEASE_WATCH_CAP: usize = 32; + +/// Hand one release fence to the watcher. `ino` 0 is a buffer the pool copied +/// into (nothing outside holds it), so its fence is only ever closed. +fn watch_release(ino: u64, ticket: u64, fd: RawFd, latch_ns: u64) { + // Nobody reads release notes: the producer is not a shim that gates on them, + // so the fence is worth nothing here. Checked per fence rather than per + // session, because the subscriber that matters may connect after the first + // buffers are already in flight. + if ino == 0 || !gndrm_daemon::pushes::wants(CAP_RELEASE_NOTE) { + // SAFETY: the descriptor was duplicated for this call. + unsafe { libc::close(fd) }; + return; + } + if !RELEASE_THREAD.swap(true, Ordering::Relaxed) { + let _ = thread::Builder::new() + .name("gn-release".to_string()) + .spawn(release_watcher); + } + RELEASE_WATCH + .lock() + .unwrap_or_else(|e| e.into_inner()) + .push((ino, ticket, fd, latch_ns)); + let wake = RELEASE_WAKE.load(Ordering::Relaxed); + if wake >= 0 { + // SAFETY: `wake` is this module's eventfd, created non-blocking; an + // 8-byte counter write is the whole protocol. + unsafe { + let one = 1u64; + libc::write(wake, (&raw const one).cast(), 8); + } + } +} + +/// Report one buffer as released to the producers that gate on it, and close its +/// fence. Every path that retires a watch goes through here, so exactly one note +/// is sent per fence. +fn report_release(ino: u64, ticket: u64, fd: RawFd, latch_ns: u64, source: u64) { + // SAFETY: the descriptor was created for the watcher and is closed once. + unsafe { libc::close(fd) }; + // The hold: from the panel taking this buffer to SurfaceFlinger letting it + // go. Sampled here rather than at the buffer's next publish, which may be + // seconds later and would measure the pool's rotation instead. + let now = crate::vsync::now_ns(); + if latch_ns != 0 { + note_hold(latch_ns, now); + } + report_hold(now); + gndrm_daemon::pushes::push( + CAP_RELEASE_NOTE, + &BufferReleased { + ticket, + ino, + latch_ns, + source, + } + .encode(), + ); +} + +/// Wait on the release fences the producers are owed, and report each one as it +/// signals. +/// +/// One thread for the session, mostly asleep: it wakes on a new fence, on the +/// oldest one ageing out, or on the poll tick. Nothing else waits on it, so a +/// slow fence cannot delay a publish. +fn release_watcher() { + let wake = eventfd_new(); + RELEASE_WAKE.store(wake, Ordering::Relaxed); + let mut live: Vec<(u64, u64, RawFd, u64, Instant)> = Vec::new(); + loop { + { + let mut queued = RELEASE_WATCH.lock().unwrap_or_else(|e| e.into_inner()); + for (ino, ticket, fd, latch_ns) in queued.drain(..) { + if live.len() >= RELEASE_WATCH_CAP { + // Evicted, not signalled: the same guarantee as a timeout, + // because the buffer is being handed back without its fence. + let (old_ino, old_ticket, old_fd, old_latch, _) = live.remove(0); + report_release( + old_ino, + old_ticket, + old_fd, + old_latch, + gndrm_uapi::wire::RELEASE_FENCE_TIMEOUT, + ); + } + live.push((ino, ticket, fd, latch_ns, Instant::now())); + } + } + if live.is_empty() { + // Nothing owed: wait for a fence to be handed over. The wake fd is + // drained here, so the next `poll` starts clean. + let mut p = libc::pollfd { + fd: wake, + events: libc::POLLIN, + revents: 0, + }; + // SAFETY: one valid pollfd, no timeout. + unsafe { libc::poll((&raw mut p).cast(), 1, -1) }; + drain_eventfd(wake); + continue; + } + let mut pfds: Vec = Vec::with_capacity(live.len() + 1); + pfds.push(libc::pollfd { + fd: wake, + events: libc::POLLIN, + revents: 0, + }); + pfds.extend(live.iter().map(|&(_, _, fd, _, _)| libc::pollfd { + fd, + events: libc::POLLIN, + revents: 0, + })); + // SAFETY: `pfds` holds one entry per live fence plus the wake fd, all + // valid for the call; the timeout bounds a fence that never signals. + // One latch period, not a fixed 50 ms: the slice is how late a note can be + // after its fence has already signalled, so a constant puts that constant + // into every producer's park. The panel's own period is the finest useful + // granularity — a note is worth nothing sooner than the buffer changes. + let slice_ms = (crate::vsync::period_now_ns() / 1_000_000).clamp(1, 50) as c_int; + let n = unsafe { libc::poll(pfds.as_mut_ptr(), pfds.len() as _, slice_ms) }; + if pfds[0].revents != 0 { + drain_eventfd(wake); + } + let now = Instant::now(); + // Backwards, so the indices still to visit are unaffected by a removal. + for i in (0..live.len()).rev() { + let mut ready = n > 0 && pfds[i + 1].revents != 0; + let mut timed_out = false; + if !ready && now.duration_since(live[i].4) >= RELEASE_WATCH_MAX { + ready = true; + timed_out = true; + } + if ready { + let (ino, ticket, fd, latch_ns, _) = live.swap_remove(i); + if timed_out { + note_hold_timeout(); + } + let source = if timed_out { + gndrm_uapi::wire::RELEASE_FENCE_TIMEOUT + } else { + gndrm_uapi::wire::RELEASE_FENCE + }; + report_release(ino, ticket, fd, latch_ns, source); + } + } + } +} + /// Payload the reuse cache stores for one registered scan-out buffer: the /// daemon's gralloc reference, this registration's completion mailbox, and the /// row stride to report to the client. All three are copied out; the raw @@ -688,9 +1210,9 @@ impl ScanoutPool { /// A fresh completion mailbox for one registration, with the reuse gate /// the completion callback publishes into. - fn new_ctx(&self) -> (*mut ReleaseCtx, Arc) { + fn new_ctx(&self, ino: u64) -> (*mut ReleaseCtx, Arc) { let api = self.api.expect(&"a slot is only committed when the presenter is enabled"); - new_release_ctx(&api) + new_release_ctx(&api, ino) } /// Register a handed-out buffer as `ino`'s slot, logging how it was @@ -715,7 +1237,7 @@ impl ScanoutPool { height, format: session_format(), }; - let (ctx, latched) = self.new_ctx(); + let (ctx, latched) = self.new_ctx(buffer.ino); let replaced = self.cache.commit( buffer.ino, geom, @@ -841,7 +1363,7 @@ fn ctx_release_fd(buf: &PoolBuf) -> RawFd { /// A fresh completion mailbox for one registration, with the reuse gate the /// completion callback publishes into. Leaked (see [`ReleaseCtx`]). -fn new_release_ctx(api: &ffi::ScApi) -> (*mut ReleaseCtx, Arc) { +fn new_release_ctx(api: &ffi::ScApi, ino: u64) -> (*mut ReleaseCtx, Arc) { let latched = Arc::new(AtomicBool::new(false)); let ctx = Box::into_raw(Box::new(ReleaseCtx { stats_get_controls: api.st_stats_get_controls, @@ -849,9 +1371,14 @@ fn new_release_ctx(api: &ffi::ScApi) -> (*mut ReleaseCtx, Arc) { previous_release: api.st_stats_previous_release_fence_fd, release_fd: AtomicI32::new(-1), latch_time: api.st_stats_latch_time, + latch_ns: AtomicU64::new(0), + ticket: AtomicU64::new(0), + release_ticket: AtomicU64::new(0), applied_ns: AtomicU64::new(0), present_id: AtomicU64::new(0), latched: Arc::clone(&latched), + ino, + replaced_ctx: AtomicPtr::new(ptr::null_mut()), })); (ctx, latched) } @@ -1003,6 +1530,10 @@ pub struct AscState { /// must be able to name the thread holding it. pool: Arc>, last_publish: Option, + /// The mailbox of the buffer currently on the layer, or null before the + /// first show. Written for the next publish to record what it replaced, so + /// a completion's release fence can be filed against the right buffer. + last_shown_ctx: *mut ReleaseCtx, frames: u64, warned_fence_timeout: bool, /// Inodes already reported as having no registered buffer (log once each). @@ -1107,6 +1638,7 @@ impl AscState { ScanoutPool::new(api), )), last_publish: None, + last_shown_ctx: ptr::null_mut(), frames: 0, warned_fence_timeout: false, logged_unregistered: Vec::new(), @@ -1367,11 +1899,11 @@ impl AscState { // The frame this registration is about to carry: its present id // was marked against the buffer before the commit that named it. // SAFETY: `ctx` is the leaked mailbox of this registered buffer. + let (present_id, ticket) = take_present_mark(published_ino); unsafe { - (*ctx) - .present_id - .store(take_present_mark(published_ino), Ordering::Release) - }; + (*ctx).present_id.store(present_id, Ordering::Release); + (*ctx).ticket.store(ticket, Ordering::Release); + } // Attach a looper to *this* thread the first time we publish. // `ALooper_pollOnce` must run on the thread the looper was // prepared on: polling a looper prepared elsewhere fails with @@ -1409,7 +1941,13 @@ impl AscState { // SAFETY: `ctx` points at the leaked mailbox of this slot. let pending = unsafe { (*ctx).release_fd.swap(-1, Ordering::AcqRel) }; if pending >= 0 { - if wait_fence(pending) && !self.warned_fence_timeout { + // This buffer's own release fence, filed by the transaction + // that replaced it: nothing may reuse the buffer until it + // signals. The wait is a guard, not a measurement — the hold + // is sampled by the watcher, where the fence actually ends + // (see `report_release`). + let timed_out = wait_fence(pending); + if timed_out && !self.warned_fence_timeout { self.warned_fence_timeout = true; log_warn!("{}", "asc: previous release fence timed out; reusing buffer anyway (logged once)"); } @@ -1554,9 +2092,26 @@ impl AscState { self.win_h) ); } + // What this show replaces on the layer: the completion of + // this transaction reports the *replaced* buffer's release + // fence, so the mailbox that owns it has to be named before + // the callback can run. + // SAFETY: `ctx` is the leaked mailbox of this slot. + unsafe { + let prev = self.last_shown_ctx; + (*ctx).replaced_ctx.store(prev, Ordering::Relaxed); + // The frame the fence will describe, stamped while this + // buffer is still on the layer and therefore still the one + // `ticket` holds. + if !prev.is_null() { + let t = (*prev).ticket.load(Ordering::Acquire); + (*prev).release_ticket.store(t, Ordering::Release); + } + } match build_show(&api, self.sc, ahb, ctx, rect, fence) { Some(tx) => { apply_transaction(&api, tx); + self.last_shown_ctx = ctx; // SAFETY: `ctx` is the leaked mailbox of this slot. unsafe { (*ctx) @@ -1657,7 +2212,10 @@ impl AscState { let slot = match ForeignCopy::new(fb, src_fd) { Ok(copy) => { self.foreign_buffers += 1; - let (ctx, _latched) = new_release_ctx(&api); + // A copy target is the pool's own buffer, not one the shim + // allocated: no producer holds it by a scan-out inode, so it is + // registered unnamed and no release note is filed for it. + let (ctx, _latched) = new_release_ctx(&api, 0); ForeignSlot { layout, copy: Some(copy), @@ -1882,6 +2440,7 @@ impl AscState { }; if let Some(tx) = build_show(&api, self.sc, ahb, ctx, rect, None) { apply_transaction(&api, tx); + self.last_shown_ctx = ctx; // SAFETY: `ctx` is the leaked mailbox of this slot. unsafe { (*ctx) @@ -2486,7 +3045,9 @@ pub fn serve_scanout( // being served when the loop returns however that happens. gndrm_daemon::pushes::set_served(fd); let _served = ServedGuard(fd); - let mut buf = [0u8; ScanoutAllocReq::LEN]; + // Every inbound type, not just the alloc request: a short buffer on a + // SEQPACKET connection truncates the frame instead of failing it. + let mut buf = [0u8; gndrm_uapi::wire::INBOUND_MAX]; // Whether a connection opted in is `gndrm_daemon::pushes`' business: a // subscription outlives this loop, so the notes cannot be gated on it here. // Whether this connection reads the panel's latch grid, pushed whenever it @@ -2676,6 +3237,8 @@ pub fn serve_scanout( } } if sub.caps & CAP_PRESENT_ID != 0 { + // A new mark sender, whose tickets may start a new sequence. + reset_present_marks(); PRESENT_AWARE.store(true, Ordering::Relaxed); PRESENT_FD.store(present_fd, Ordering::Relaxed); } @@ -3934,7 +4497,9 @@ pub(crate) fn note_produced(acq: FrameAcquired) { /// it, so an alloc request on a retired connection is left to the shim's own /// timeout, exactly as it was before any of this. pub(crate) fn drain_retired(fd: RawFd) -> bool { - let mut buf = [0u8; ScanoutAllocReq::LEN]; + // Every inbound type, not just the alloc request: a short buffer on a + // SEQPACKET connection truncates the frame instead of failing it. + let mut buf = [0u8; gndrm_uapi::wire::INBOUND_MAX]; // SAFETY: `fd` is a connection this module owns until it is forgotten, and // `buf` is one full frame; DONTWAIT leaves an unarrived frame for later. let n = unsafe { libc::recv(fd, buf.as_mut_ptr().cast(), buf.len(), libc::MSG_DONTWAIT) }; diff --git a/modules/gndrm-android/src/lib.rs b/modules/gndrm-android/src/lib.rs index ea98503..e382373 100644 --- a/modules/gndrm-android/src/lib.rs +++ b/modules/gndrm-android/src/lib.rs @@ -111,6 +111,7 @@ static LSFG_SETTINGS: Mutex = Mutex::new(LsfgConfig { flow_per_mille: 0, performance: 0, fps_limit: 0, + present_mode: 0, dll: [0; gndrm_uapi::wire::LSFG_DLL_PATH_MAX], refresh_mhz: 0, }); @@ -142,7 +143,8 @@ pub(crate) fn panel_refresh_mhz() -> u32 { gndrm_daemon::advertised_refresh_mhz() } -/// `GameNativeDrm.nativeSetLsfgConfig(enabled, multiplier, flowPerMille, performance, fpsLimit, dllPath)`. +/// `GameNativeDrm.nativeSetLsfgConfig(enabled, multiplier, flowPerMille, +/// performance, fpsLimit, presentMode, dllPath)`. /// /// Called by the app before or during a session; the value is read when the /// shim asks, so a change reaches the next device the guest creates. @@ -155,6 +157,7 @@ pub extern "system" fn Java_app_gamenative_display_GameNativeDrm_nativeSetLsfgCo flow_per_mille: jint, performance: jboolean, fps_limit: jint, + present_mode: jint, dll: JString, ) { // Read the string before the guarded section: `JNIEnv` is not unwind-safe. @@ -169,6 +172,11 @@ pub extern "system" fn Java_app_gamenative_display_GameNativeDrm_nativeSetLsfgCo settings.flow_per_mille = flow_per_mille.clamp(0, 1000) as u32; settings.performance = u32::from(performance != 0); settings.fps_limit = fps_limit.max(0) as u32; + // The QuickMenu's present mode for generation: 1 = fifo, anything else + // mailbox. Generation has no third option — a generated frame cannot be + // told apart from a tear if it is flipped mid-scan — so the range is + // pinned here rather than trusted from the caller. + settings.present_mode = u32::from(present_mode == 1); // The presenter cannot read the container's environment, so the path of // the DLL it has to read the shaders from rides with the settings. settings.set_dll_path(&dll_path); diff --git a/modules/gndrm-android/src/vsync.rs b/modules/gndrm-android/src/vsync.rs index 9b72cce..984e028 100644 --- a/modules/gndrm-android/src/vsync.rs +++ b/modules/gndrm-android/src/vsync.rs @@ -234,6 +234,13 @@ fn period_ns(now_ns: u64) -> u64 { } } +/// The panel period now, for callers that need the grid's granularity rather than +/// a note (the release watcher's slice). Zero when nothing has reported a rate. +#[cfg(target_os = "android")] +pub(crate) fn period_now_ns() -> u64 { + period_ns(now_ns()) +} + /// How far ahead of its latch the flip events are stamped at `now_ns`, 0 while /// they follow no display grid (`gndrm_daemon::vblank::display_grid`). At least /// 1 ns while they do, so 0 always means "not known" to the shim, which does not @@ -298,6 +305,11 @@ pub(crate) fn note() -> Option { .note(period, stamp_lead) } +#[cfg(not(target_os = "android"))] +pub(crate) fn period_now_ns() -> u64 { + 0 +} + #[cfg(test)] mod tests { use super::*; diff --git a/modules/gndrm-uapi/src/wire.rs b/modules/gndrm-uapi/src/wire.rs index 6038f34..0ed9b0f 100644 --- a/modules/gndrm-uapi/src/wire.rs +++ b/modules/gndrm-uapi/src/wire.rs @@ -534,12 +534,14 @@ impl FrameCapNote { /// The daemon latches the capability for the connection and answers with the /// current target as a [`FrameCapNote`]; it pushes the same note again whenever /// the target changes. `caps` is a bitmask of what the shim can honour: -/// [`CAP_FRAME_CAP`] and [`CAP_VSYNC_NOTE`]. A shim that never sends this is sent no +/// [`CAP_FRAME_CAP`], [`CAP_VSYNC_NOTE`], [`CAP_PRESENT_ID`] and +/// [`CAP_RELEASE_NOTE`]. A shim that never sends this is sent no /// [`FrameCapNote`] -- one frame per request, as before -- which is what keeps an /// older shim on the channel it expects. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct FrameCapSubscribe { - /// Capability bits: [`CAP_FRAME_CAP`], [`CAP_VSYNC_NOTE`]. + /// Capability bits: [`CAP_FRAME_CAP`], [`CAP_VSYNC_NOTE`], [`CAP_PRESENT_ID`], + /// [`CAP_RELEASE_NOTE`]. pub caps: u32, } @@ -670,20 +672,33 @@ pub const CAP_PRESENT_ID: u32 = 1 << 2; /// A frame's present id, shim -> daemon. /// -/// Encoded little-endian, 24 bytes: -/// `magic:u32, reserved:u32, ino:u64, id:u64`. `ino` is the scan-out inode the -/// frame's slot was allocated from; `id` is the present id the guest chained in -/// `VkPresentIdKHR`. +/// Encoded little-endian, 32 bytes: +/// `magic:u32, reserved:u32, ino:u64, id:u64, ticket:u64`. `ino` is the +/// scan-out inode the frame's slot was allocated from; `id` is the present id +/// the guest chained in `VkPresentIdKHR`, 0 when it chained none. +/// +/// `ticket` is the shim's own monotonic id for this commit, and it is the +/// pairing key for every note that follows. Not `ino`: an inode is a property of +/// the dma-buf the *allocator* handed out, and it survives only while gralloc +/// hands the same object through to `SurfaceFlinger` — which it does on one device +/// and not on the next, and, measured on Nova, for one title and not another +/// (6 % of parks paired by identity on Portal 2, 84 % by order on Dave). Not +/// `id`: most guests never chain a present id, so it is 0 on every frame of a +/// session that does not use `VK_KHR_present_id`. A ticket always exists, is +/// unique for the session, and cannot be recycled into a live slot the way a +/// freed dma-buf's inode can. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct PresentMark { /// The scan-out inode of the buffer being committed. pub ino: u64, - /// The guest's present id for this frame. + /// The guest's present id for this frame, 0 when it chained none. pub id: u64, + /// The shim's own id for this commit: monotonic, never reused. + pub ticket: u64, } impl PresentMark { - pub const LEN: usize = 24; + pub const LEN: usize = 32; #[must_use] pub fn encode(self) -> [u8; Self::LEN] { @@ -691,6 +706,7 @@ impl PresentMark { b[0..4].copy_from_slice(&PRESENT_MARK_MAGIC.to_le_bytes()); b[8..16].copy_from_slice(&self.ino.to_le_bytes()); b[16..24].copy_from_slice(&self.id.to_le_bytes()); + b[24..32].copy_from_slice(&self.ticket.to_le_bytes()); b } @@ -702,10 +718,43 @@ impl PresentMark { Some(Self { ino: u64::from_le_bytes(b[8..16].try_into().ok()?), id: u64::from_le_bytes(b[16..24].try_into().ok()?), + ticket: u64::from_le_bytes(b[24..32].try_into().ok()?), }) } } +/// The largest frame the daemon reads from a shim connection. +/// +/// Every inbound type must fit, because `SOCK_SEQPACKET` preserves boundaries but +/// **truncates a frame to the receive buffer and discards the rest** — so a reader +/// sized for the wrong type does not fail, it silently drops the message. That is +/// what happened when [`PresentMark`] grew to carry the ticket: the daemon still +/// read into `ScanoutAllocReq::LEN` (24), every 32-byte mark arrived truncated, +/// `decode` returned `None`, and the marks never reached `note_present_mark` — +/// measured on RP6 as `sf mark: hit=0 order=0 miss=3 skew=0 pending=0 last=0` with +/// `tick=[n:0]` on every census line, i.e. the ticket mechanism was inert on every +/// device while looking no different from a daemon that does not send tickets. +/// +/// Derived from the types rather than written as a number, so a struct that grows +/// past it is a compile-time failure at the `const` rather than a silent drop at +/// runtime. `INBOUND_MAX` is enforced by a test in this module. +pub const INBOUND_MAX: usize = { + let mut m = ScanoutAllocReq::LEN; + if FrameCapSubscribe::LEN > m { + m = FrameCapSubscribe::LEN; + } + if FrameAcquired::LEN > m { + m = FrameAcquired::LEN; + } + if PresentMark::LEN > m { + m = PresentMark::LEN; + } + if LsfgConfig::LEN > m { + m = LsfgConfig::LEN; + } + m +}; + /// A completed present, daemon -> shim. /// /// Encoded little-endian, 24 bytes: @@ -744,6 +793,95 @@ impl PresentDone { } } +/// Buffer-released magic: ASCII "GNBR". Daemon -> shim, on the scan-out channel. +/// +/// The daemon reports that `SurfaceFlinger` has finished reading a scan-out +/// buffer, so the producer may write into it again. Only a connection that +/// declared [`CAP_RELEASE_NOTE`] is sent one. +/// +/// This is the one thing the producer cannot learn from the compositor: a +/// compositor hands a buffer back when it has finished *compositing* with it, +/// which is before the display has finished *scanning* it out. The daemon sees +/// the difference — `SurfaceFlinger` reports the released buffer's fence in the +/// completion of the transaction that replaced it — and reports it here. +pub const BUFFER_RELEASED_MAGIC: u32 = 0x474E_4252; + +/// The `caps` bit a [`FrameCapSubscribe`] sets when the shim gates reuse of a +/// scan-out buffer on [`BufferReleased`]. A daemon pushes no release note to a +/// connection that did not set it. +pub const CAP_RELEASE_NOTE: u32 = 1 << 3; + +/// A scan-out buffer `SurfaceFlinger` is finished with, daemon -> shim. +/// +/// Encoded little-endian, 40 bytes: +/// `magic:u32, reserved:u32, ticket:u64, ino:u64, latch_ns:u64, source:u64`. +/// +/// `ticket` is the [`PresentMark`] ticket of the frame this buffer carried, and +/// it is what the shim frees the slot by. It was the one message of the three +/// that did not carry the token, and every arm of the shim's park existed to +/// compensate: identity matching on an inode the allocator decides, an order +/// fallback with a reorder window and a one-period floor, and a bound derived +/// from the very interval it governed. +/// +/// `source` says what the release rests on, so a session can tell "the fence +/// signalled" from "the fence never signalled and we gave up" from "there was +/// nothing to watch" — which is exactly the distinction that was invisible on +/// RP6, where 198 notes arrived with no latch time and the census read the same +/// as a device with no producer at all. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BufferReleased { + /// The shim's own ticket for the frame, 0 when the mark carried none. + pub ticket: u64, + /// The scan-out inode of the buffer that may be written again. + pub ino: u64, + /// The panel's latch time for the frame, 0 when the platform reported none. + pub latch_ns: u64, + /// What the release rests on; one of the `RELEASE_*` values. + pub source: u64, +} + +/// `source`: `SurfaceFlinger`'s release fence signalled. +pub const RELEASE_FENCE: u64 = 0; +/// `source`: the fence did not signal within the watcher's budget, so the buffer +/// was reported released anyway — the compositor is stuck, and holding the +/// producer behind it would be a hang rather than a tear. +pub const RELEASE_FENCE_TIMEOUT: u64 = 1; +/// `source`: the pool copied into the buffer, so nothing outside holds it and +/// there was no fence to watch. +pub const RELEASE_POOL_COPY: u64 = 2; +/// `source`: a buffer no subscription was watching, reported so the shim learns +/// the buffer is free rather than inferring it from silence. +pub const RELEASE_UNWATCHED: u64 = 3; + +impl BufferReleased { + pub const LEN: usize = 40; + + #[must_use] + pub fn encode(self) -> [u8; Self::LEN] { + let mut b = [0u8; Self::LEN]; + b[0..4].copy_from_slice(&BUFFER_RELEASED_MAGIC.to_le_bytes()); + b[8..16].copy_from_slice(&self.ticket.to_le_bytes()); + b[16..24].copy_from_slice(&self.ino.to_le_bytes()); + b[24..32].copy_from_slice(&self.latch_ns.to_le_bytes()); + b[32..40].copy_from_slice(&self.source.to_le_bytes()); + b + } + + #[must_use] + pub fn decode(b: &[u8]) -> Option { + if b.len() != Self::LEN || u32::from_le_bytes(b[0..4].try_into().ok()?) != BUFFER_RELEASED_MAGIC + { + return None; + } + Some(Self { + ticket: u64::from_le_bytes(b[8..16].try_into().ok()?), + ino: u64::from_le_bytes(b[16..24].try_into().ok()?), + latch_ns: u64::from_le_bytes(b[24..32].try_into().ok()?), + source: u64::from_le_bytes(b[32..40].try_into().ok()?), + }) + } +} + /// Vsync note magic: ASCII "GNVS". Daemon -> shim. pub const VSYNC_MAGIC: u32 = 0x474E_5653; @@ -842,6 +980,12 @@ pub struct LsfgConfig { /// with the settings because the presenter runs in the app's process, where /// the container's environment is not present. pub dll: [u8; LSFG_DLL_PATH_MAX], + /// The present mode generation is served with: 0 = mailbox, 1 = fifo. The + /// `QuickMenu` offers those two and nothing else, because a generated frame + /// cannot be told apart from a tear if it is flipped mid-scan, so + /// `IMMEDIATE` is never a legal value here and an unknown one falls back to + /// mailbox. + pub present_mode: u32, /// The panel's refresh in millihertz. The engine paces against it and clamps /// its generations to what the display can afford (`floor(refresh/source)` /// frames per source frame); zero means unknown, which leaves it unclamped. @@ -857,13 +1001,21 @@ impl Default for LsfgConfig { performance: 0, fps_limit: 0, dll: [0; LSFG_DLL_PATH_MAX], + present_mode: 0, refresh_mhz: 0, } } } impl LsfgConfig { - pub const LEN: usize = 24 + LSFG_DLL_PATH_MAX + 4; + pub const LEN: usize = 24 + LSFG_DLL_PATH_MAX + 8; + + /// The present mode generation is served with, as the two values the + /// `QuickMenu` offers. Anything unknown is mailbox. + #[must_use] + pub fn generating_present_mode(&self) -> u32 { + u32::from(self.present_mode == 1) + } /// The DLL path as a string, or `None` when it is empty. #[must_use] @@ -906,7 +1058,9 @@ impl LsfgConfig { b[16..20].copy_from_slice(&self.performance.to_le_bytes()); b[20..24].copy_from_slice(&self.fps_limit.to_le_bytes()); b[24..24 + LSFG_DLL_PATH_MAX].copy_from_slice(&self.dll); - b[24 + LSFG_DLL_PATH_MAX..].copy_from_slice(&self.refresh_mhz.to_le_bytes()); + b[24 + LSFG_DLL_PATH_MAX..24 + LSFG_DLL_PATH_MAX + 4] + .copy_from_slice(&self.present_mode.to_le_bytes()); + b[24 + LSFG_DLL_PATH_MAX + 4..].copy_from_slice(&self.refresh_mhz.to_le_bytes()); b } @@ -927,7 +1081,14 @@ impl LsfgConfig { dll.copy_from_slice(&b[24..24 + LSFG_DLL_PATH_MAX]); dll }, - refresh_mhz: u32::from_le_bytes(b[24 + LSFG_DLL_PATH_MAX..].try_into().ok()?), + present_mode: u32::from_le_bytes( + b[24 + LSFG_DLL_PATH_MAX..24 + LSFG_DLL_PATH_MAX + 4] + .try_into() + .ok()?, + ), + refresh_mhz: u32::from_le_bytes( + b[24 + LSFG_DLL_PATH_MAX + 4..].try_into().ok()?, + ), }) } } @@ -936,6 +1097,31 @@ impl LsfgConfig { mod tests { use super::*; + /// Every inbound frame must fit the reader's buffer. + /// + /// On `SOCK_SEQPACKET` a receive buffer shorter than the frame truncates it + /// and discards the rest, so the reader does not fail — it silently drops the + /// message. That is how the ticket never reached the presenter: `PresentMark` + /// grew to 32 bytes and the daemon still read into `ScanoutAllocReq::LEN` + /// (24), which measured as `sf mark: hit=0 order=0 miss=3 skew=0 pending=0` + /// and `tick=[n:0]` on every census line, with the code looking correct at + /// both ends. + #[test] + fn every_inbound_frame_fits_the_readers_buffer() { + for len in [ + ScanoutAllocReq::LEN, + FrameCapSubscribe::LEN, + FrameAcquired::LEN, + PresentMark::LEN, + LsfgConfig::LEN, + ] { + assert!( + len <= INBOUND_MAX, + "an inbound frame of {len} bytes does not fit INBOUND_MAX {INBOUND_MAX}" + ); + } + } + #[test] fn vsync_note_round_trips_and_rejects_other_magics() { let note = VsyncNote { @@ -1202,6 +1388,74 @@ mod tests { } } + #[test] + fn buffer_released_round_trip() { + let note = BufferReleased { + ticket: 0xfeed, + ino: 0x1234_5678_9abc, + latch_ns: 0x99, + source: RELEASE_FENCE_TIMEOUT, + }; + let enc = note.encode(); + assert_eq!(enc.len(), BufferReleased::LEN); + assert_eq!(enc[0..4], BUFFER_RELEASED_MAGIC.to_le_bytes()); + assert_eq!(BufferReleased::decode(&enc), Some(note)); + + // Its magic is its own, and it decodes to nothing else on the channel: + // every push shares the socket, so a shape read as another is a note + // applied to the wrong thing. + for other in [ + SCANOUT_MAGIC, + FRAME_CAP_MAGIC, + FRAME_CAP_SUBSCRIBE_MAGIC, + PRESENT_MARK_MAGIC, + PRESENT_DONE_MAGIC, + VSYNC_MAGIC, + ] { + assert_ne!(BUFFER_RELEASED_MAGIC, other); + } + assert_eq!(BufferReleased::decode(&[0u8; BufferReleased::LEN]), None); + assert_eq!(BufferReleased::decode(&[]), None); + let mut short = note.encode().to_vec(); + short.truncate(BufferReleased::LEN - 1); + assert_eq!(BufferReleased::decode(&short), None); + // An inode of 0 is never a buffer the daemon registered, so a zeroed + // note cannot be mistaken for a real release. + assert_eq!( + BufferReleased::decode( + &BufferReleased { + ticket: 0, + ino: 0, + latch_ns: 0, + source: RELEASE_FENCE + } + .encode() + ), + Some(BufferReleased { + ticket: 0, + ino: 0, + latch_ns: 0, + source: RELEASE_FENCE + }) + ); + } + + #[test] + fn release_note_cap_is_its_own_bit() { + // Every cap bit is distinct: the shim declares what it honours in one + // mask, and a shared bit would have it honour a push it cannot read. + for (a, b) in [ + (CAP_FRAME_CAP, CAP_VSYNC_NOTE), + (CAP_FRAME_CAP, CAP_PRESENT_ID), + (CAP_FRAME_CAP, CAP_RELEASE_NOTE), + (CAP_VSYNC_NOTE, CAP_PRESENT_ID), + (CAP_VSYNC_NOTE, CAP_RELEASE_NOTE), + (CAP_PRESENT_ID, CAP_RELEASE_NOTE), + ] { + assert_ne!(a, b); + } + } + #[test] fn frame_cap_note_round_trip() { let note = FrameCapNote { fps: 120 };