Skip to content

Apply cross-process work inside a syscall callback - #11

Closed
utkarshdalal wants to merge 1 commit into
gamenativefrom
invalidation-reentry-under-cross-process-work
Closed

utkarshdalal wants to merge 1 commit into
gamenativefrom
invalidation-reentry-under-cross-process-work

Conversation

@utkarshdalal

@utkarshdalal utkarshdalal commented Nov 15, 2026 •

Copy link
Copy Markdown

Adds patch 0013 to the series. Pairs with GameNative/proton-wine#133. Either change on its own stops the hang, and each fixes a real bug on its side.

Problem

FEX drains the cross-process work list from HandleSyscall, PreCompile and SyncThreadContext. A memory-write entry ends in InvalidateIntervalInternal, which holds CodeInvalidationMutex while the allocator may free memory. ntdll only skips its memory notifications for calls made inside a syscall callback. The drain does not run inside one, so the nested NtFreeVirtualMemory called back into NotifyMemoryFree on the same thread. That locked CodeInvalidationMutex again and hung.

Stack captured on the device (one thread, newest frame first):

SleepConditionVariableSRW
__shared_mutex_base::lock                        CodeInvalidationMutex, second time
InvalidationTracker::InvalidateIntervalInternal
InvalidationTracker::InvalidateAlignedInterval
recursive_mutex::lock                            NotifyMemoryFree (ThreadCreationMutex, recursive)
VirtualFree
ContextImpl::InvalidateThreadCachedCodeRange
InvalidationTracker::InvalidateIntervalInternal  CodeInvalidationMutex held
BTCpu64NotifyMemoryDirty
ProcessPendingCrossProcessEmulatorWork
ECSyscallHandler::HandleSyscall

The lock is never taken from a system call or from the unix fault handler. ntdll only queues the entry there, without locks; FEX applies it later on the PE side.

Change

The three drain sites go through one helper. It holds the existing ScopedCallbackDisable across ProcessPendingCrossProcessEmulatorWork, so the thread counts as inside a syscall callback, the same as in ResetToConsistentState. The class moves above the exception namespace so the helper can use it. Upstream main has the same unguarded drain; the patch applies there unchanged.

Not changed: on a page FEX has trapped, NtQueryVirtualMemory reports what patch 0002 tracks for the guest. Using Wine's write-exception interface (ProcessManageWritesToExecutableMemory / VmPageDirtyStateInformation) instead of the protection trap was tried and dropped:

  • Turning it on arms every exec+write page in the process, including FEX's own RWX code buffer. The next JIT write raised STATUS_IN_PAGE_ERROR and the process hung.
  • Wine has no kernel-write path. KernelWriteToExecutableSignal is declared and never read, so a system call writing into an armed page fails.

Results

AYN Odin 3, CI run 43691753463 (2611.1-gn6-2117faf). The test is an x86-64 program: code runs in an RWX page, then system calls write into the page through the ntdll export and through a raw syscall, including 10,000 writes while two other threads execute from the page.

Wine FEX 2611-gn-6 This PR
Earlier FEX-Emu#131 build, queues the write entry without the work-list guard hangs in the concurrent case all cases pass, no hang
GameNative/proton-wine#133 all cases pass all cases pass
Stock 11.0-2 system call writes fail with c0000005 same; FEX cannot make those writes succeed on its own

With either fix in place, code a system call overwrote runs as the new code on the next call, 1,000 of 1,000 on both paths. Self-modifying code: 10,000 same-thread and 10,000 cross-thread iterations, 0 wrong in every run.

Games:

Note: another series branch (gn-7, stale NoExec retry) is in progress. The two touch different patches; whichever merges second renumbers.

@utkarshdalal

Copy link
Copy Markdown
Author

Closing: the Wine-side patch (GameNative/proton-wine#133) is sufficient for this. It guards the cross-process work drain in ntdll, so the re-entry cannot happen with any FEX build.

utkarshdalal added a commit to GameNative/proton-wine that referenced this pull request Nov 15, 2026
…133)

Builds on #131 and includes its commits. Merge #131 first; this PR then
adds one commit.

## What changes

Two patch files in `android/patches/arm64ec/`, both registered in
`build-scripts/build-step-arm64ec.sh` after #131's patch:

- `dlls_ntdll_unix_virtual_c_report_emulated_code_write.patch`: when
#131 makes a write-protected RWX page writable for a system call, ntdll
also queues a `CrossProcessMemoryWrite` entry for that page on the
emulator's cross-process work list. The emulator applies it the next
time it drains the list and drops its translations of the page, so code
that a system call overwrote is translated again. If no entry is free,
the work list is flagged for a full flush, so the write is never
dropped.
- `dlls_ntdll_signal_arm64ec_c_cross_process_work_callback.patch`:
`ProcessPendingCrossProcessEmulatorWork` now applies the work inside a
syscall callback (`enter_syscall_callback` / `leave_syscall_callback`,
cleared only when this call set it), as every other emulator
notification already does.

## Why the second patch is needed

Without it, the first entry hangs the thread. Stack captured on the
device (FEX 2512, one thread, newest frame first):

```
SleepConditionVariableSRW
std::__shared_mutex_base::lock                  CodeInvalidationMutex, second time
InvalidationTracker::InvalidateIntervalInternal
InvalidationTracker::InvalidateAlignedInterval
recursive_mutex::lock                           NotifyMemoryFree
VirtualFree                                     allocator, inside the invalidation
ContextImpl::InvalidateThreadCachedCodeRange
InvalidationTracker::InvalidateIntervalInternal CodeInvalidationMutex held
BTCpu64NotifyMemoryDirty
ProcessPendingCrossProcessEmulatorWork
ECSyscallHandler::HandleSyscall
```

FEX holds its code invalidation lock while its allocator frees memory.
ntdll only skips memory notifications inside a syscall callback, so the
nested `NtFreeVirtualMemory` was reported back to FEX, which took the
same non-recursive lock again on the same thread. The lock is not taken
from the system call or the unix fault handler: the entry is queued
there without locks and applied later on the PE side.

Upstream Wine has the same drain without the guard. Nothing in Wine
queues `CrossProcessMemoryWrite` today, so there it only shows up with
work queued by another process.

## Results

AYN Odin 3, stock Proton 11.0-2 tree with this PR's `ntdll.so` /
`ntdll.dll` from CI run 43691683416. The test is an x86-64 program that
runs code in an RWX page, then lets system calls write into the page,
both through the ntdll export and through a raw `syscall` instruction.

| Case | Stock 11.0-2 | #131 only | #131 + this PR |
|---|---|---|---|
| `NtQuerySystemInformation` into a page whose code already ran |
`c0000005` | 0 | 0 |
| `NtReadVirtualMemory` overwrites code that already ran, then call it
(1,000×, ntdll export) | fails, 1,000 stale | succeeds, **1,000 stale**
| 0 stale |
| Same through a raw `syscall` | fails, 1,000 stale | succeeds, **1,000
stale** | 0 stale |
| 10,000 syscall writes into the page while 2 threads execute from it |
8,550 errors | 0 errors | 0 errors, no hang |
| Self-modifying code, 10,000 same-thread + 10,000 cross-thread | 0
wrong | 0 wrong | 0 wrong |
| 20,000 × (data write + call) on one RWX page | 1654 ms | 1653 ms |
1587 ms (FEX with the GameNative/FEX#11 fix) / 1493 ms (gn-6) |

The work-list guard was checked against the build without it, the first
version of #131, which queues the entry but has no guard. With FEX
2611-gn-6 it hung in the concurrent case; with GameNative/FEX#11, or
with this PR's guard, it does not. Either side's change stops the hang
on its own.

Hogwarts Legacy, real Steam client headless, overlay off, gamescope,
with this build and the FEX#11 build: the game reaches its first screen,
the WB EULA with Accept, at 31 fps on the HUD.

Known, not changed here:
- `NtQueryVirtualMemory` reports what FEX's guest-visible protection
says. With the GameNative FEX series that is 0x40 (patch 0002); a stock
FEX reports 0x20 for a page it trapped.
- A `WriteProcessMemory` from another process into code the target
already ran still runs the old code in the target. The bytes change, but
the target never drops its translation, even after a later system call.
The same happens on stock 11.0-2; it is a separate path
(`CrossProcessFlushCache` from kernelbase).

---------

Co-authored-by: Utkarsh Dalal <utkarsh.dalal@toptal.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant