Repository navigation
Budgeted wakes issued after the region unlock; host lock flag instead of SIGUSR1 masking - #1
Draft
utkarshdalal wants to merge 3 commits into
Draft
utkarshdalal wants to merge 3 commits into
utkarshdalal wants to merge 3 commits into
Conversation
A semaphore release, auto-reset event set or mutex unlock woke every waiter on the object. Each wake cost the signaling thread one futex syscall while holding the region lock, and every extra woken waiter then took the region lock twice, failed to acquire and parked again. Unity's job system releases a semaphore to its parked workers many times per frame: on The Forest (Snapdragon 8 Elite) the main thread made ~41k syscalls/s and spent about a quarter of its time in NtReleaseSemaphore -> sem_release. Now: - A single-object, non-alertable waiter registers its node with registered = 2 (REG_SINGLE). Readers only ever test registered against 0, so the shared layout and older builds are unaffected in both directions. - wake_object takes a budget: the release count for a semaphore, 1 for an auto-reset event, a freed mutex or an abandoned mutex. It wakes at most that many REG_SINGLE waiters. A woken single-object waiter always retries the acquire before any timeout or exit, so each unit lands on a thread that takes the object or finds it already taken. - The budget is charged only when futex_wake reports a thread woken, so a waiter that is spinning, about to park or dead costs nothing and the next one is woken too. - Multi-object, wait-all and alertable waiters are always woken, as before. Close, pulse, manual-reset events and dead-process reaping still wake everyone. - Waiters are woken oldest first, so a contended object cannot starve an early waiter. - NTSYNC_WAKE_ALL=1 restores wake-all. - New stats: threads_woken, wakes_deferred. Tests: release count, oldest-first order, auto vs manual events, mutex handoff, and multi-object and alertable waiters. Two stress tests release exactly as many units as consumers need, with thieves and mixed waiter kinds, so a lost wakeup fails them. The full suite and the process-death soaks pass on device (8 Elite, 3 runs).
Wine waits alertably unless the caller asks otherwise, so the workers of a Unity job system registered an alert node next to the semaphore node and were never counted as single-object waiters: every release still woke all of them (1 deferred wake in 18.8 M signal ops on The Forest). A wait-any waiter with several nodes now registers its object nodes as REG_ANY, which wake_object budgets like REG_SINGLE. Such a waiter can return without the object it was woken for (its alert fired, an earlier object in its list was signaled, or a handle went invalid). It then wakes the next waiter in its place, so the unit is never left in the object with waiters asleep. Wait-all waiters and alert nodes are always woken, as before. Tests: two deterministic ones park a decoy ahead of a consumer and make the decoy return on its alert or on another object; both fail with the forwarding removed. The stress test now fires alerts and a second object at its consumers. The Unity simulation's workers wait alertably and it reports the cost of sem_release.
…of SIGUSR1 masking The cost of a semaphore release on Unity's job semaphore was the region lock around each wake, not the number of threads woken. - Wakes after the unlock: wake walks detach nodes and bump their seq under the lock and collect the futex words (up to 16, inline beyond that); the FUTEX_WAKEs are issued after the unlock, so a woken waiter no longer runs into the lock its waker still holds. A budgeted wake is charged at detach; if its futex_wake finds nobody parked, a second locked walk wakes one more waiter per such unit (what the inline walk did within the same walk). Applies to signal ops and to wakes a waiter passes on; close, pulse and reaping stay inline. - ntsync_set_lock_flag_hook: a host whose SIGUSR1 handler suspends threads (Wine) can give each thread a flag the library sets while it holds or waits for the region lock. The handler marks the signal pending and returns while the flag is set; the library swaps the flag to 0 after the unlock and re-raises SIGUSR1 if it was pending. This replaces two pthread_sigmask syscalls per acquisition. Without the hook, or for threads it returns NULL for, masking stays as before. - Region lock: trylock first, so lock_contended counts contention without NTSYNC_DEBUG and the debug timedlock runs only when the lock is busy; the debug lock-owner tid comes from bionic's cached gettid(). - Stats: wakes_unparked, wake_rewalks, sigusr1_deferred appended to the stats line. - sim_unity_release_cost: 7 alertable workers on one semaphore, 300 single-unit releases per 60 fps frame in bursts, per-job completion events; reports main-thread CPU per release, wake walks and lock acquisitions, and per-thread CPU and switches. - Tests: a budgeted wake that lands on a dead waiter goes to the next one; SIGUSR1 delivered to a thread inside the region lock is deferred, delivered once, and does not wedge the lock, with a negative control that wedges when the handler ignores the flag.
utkarshdalal
pushed a commit
to GameNative/proton-wine
that referenced
this pull request
Nov 16, 2026
…c-android to wakes after unlock - ci: pin ntsync-android 6342614 (GameNative/ntsync-android#1): release wakes are issued after the region unlock, and the library takes a per-thread lock flag from the host instead of masking SIGUSR1. - common/dlls_ntdll_unix_server_c_ntsync_lock_flag.patch: ntdll_thread_data gets ntsync_lock_flag; server_init_process registers it with ntsync_set_lock_flag_hook on aarch64 and x86_64. - arm64ec/dlls_ntdll_unix_signal_arm64_c_ntsync_lock_flag.patch, x86_64/dlls_ntdll_unix_signal_x86_64_c_ntsync_lock_flag.patch: usr1_handler marks the signal pending and returns while the flag is set; the library re-raises it right after the unlock. - android/ntsync_android/ntsync_user.h synced with the library header.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
FUTEX_WAKEs after the unlock. A woken waiter therefore no longer runs into the lock its waker still holds. Close, pulse and dead-process reaping stay inline.ntsync_set_lock_flag_hook. A host whose SIGUSR1 handler suspends threads (Wine) can give each thread a flag that the library sets while the thread holds or waits for the region lock. The handler marks the signal pending and returns. After the unlock, the library atomically swaps the flag to 0 and re-raises SIGUSR1 if a signal was pending. This replaces the twopthread_sigmasksyscalls per acquisition. Without the hook, or for a thread the hook returns NULL for, masking is unchanged. Wine side: ntsync: budgeted wakes after unlock, SIGUSR1 deferral through a lock flag (test build) proton-wine#138.trylockfirst, solock_contendedcounts contention withoutNTSYNC_DEBUG. The debug owner tid comes from bionic's cachedgettid()instead of a syscall.Safety
futex_wakefinds nobody parked (the waiter is spinning, between its seq sample and the futex call, or dead), a second locked walk wakes one more waiter per such unit. This is what the inline walk did. Over-waking is safe; stranding a unit is not. In game this happened 113 times/s, with 28 second walks/s.FUTEX_WAKEon a node that was already freed and reused is a spurious wake. Its new owner re-checks under the lock and parks again.owedmask). Forwarded wakes are also issued after the unlock.wakes_unparked,wake_rewalks,sigusr1_deferred.How the policy was chosen:
sim_unity_release_costThe simulation models the game's job system: 7 alertable workers on one semaphore, 300 single-unit releases per 60 fps frame in bursts, per-job completion events, and 60 µs jobs. Main and workers are pinned.
Validation. On the code before this commit,
NTSYNC_WAKE_ALL=1versus default reproduces the in-game direction:The simulation does not reproduce the contended unlocks seen on the main thread in game.
Candidates (Odin 3,
NTSYNC_DEBUGoff, 3 alternating repetitions):Wake-all becomes much worse once wakes leave the lock. Its batching came from the waker holding the lock through several roughly 10 µs futex wakes, which kept the woken extras from re-registering. So the budget stays the default, and
NTSYNC_WAKE_ALL=1stays for comparison. With idle-proof cores (SIM_HOT=1), budget with both changes is also the cheapest at 3.4 µs, against 9.5 µs for the old wake-all.Tests
dead_waiter_does_not_strand_a_unit: the oldest waiter is a killed process; the unit must reach the live waiter. It fails with the second walk disabled.sigusr1_in_region_lock_is_deferred_not_lost: 300 suspends of a thread that is busy taking the lock. None wedged, every signal suspended the thread exactly once, and 234 landed inside the lock.sigusr1_in_region_lock_wedges_without_deferral: the negative control, where the handler ignores the flag. 100 of 300 wedged.In game: The Forest, Odin 3
Built through proton-wine#138. New Game, the same pad loop, measured from about 3 minutes into gameplay. The scene was daytime forest, as in the f465609 run, but signal operations per frame were higher (about 786 vs 635), so fps is not compared.
pthread_mutex_unlockshareNtReleaseSemaphoreshareWhat did not help. The release share did not fall. Each walk now costs about one syscall, down from about 5. However, the game took 2.3 times more walks per second than in the f465609 run, and woke 0.91 threads per signal operation (previously 0.66). The roughly 7 µs kernel cost of each
FUTEX_WAKEis now nearly all of the release cost. Cutting it further means fewer wakes from the main thread. That is not possible with a per-unit budget, and wake-all is worse once the wakes leave the lock.No stall, stuck wait or forwarded wake was seen. 4 SIGUSR1 suspends were deferred out of the lock in game.
Status: draft.