Skip to content

ntdll: report system call writes into emulated code to the emulator - #133

Merged
utkarshdalal merged 3 commits into
proton_11.0-2from
arm64ec-report-syscall-writes-to-emulator
Nov 15, 2026
Merged

utkarshdalal merged 3 commits into
proton_11.0-2from
arm64ec-report-syscall-writes-to-emulator

Conversation

@utkarshdalal

@utkarshdalal utkarshdalal commented Nov 15, 2026 •

Copy link
Copy Markdown

Builds on #131 and includes its commits. Merge #131 first; this PR then adds one commit.

What changes

Two patch files in android/patches/arm64ec/, both registered in build-scripts/build-step-arm64ec.sh after #131's patch:

  • dlls_ntdll_unix_virtual_c_report_emulated_code_write.patch: when ntdll: let system calls write into RWX pages the emulator write-protected #131 makes a write-protected RWX page writable for a system call, ntdll also queues a CrossProcessMemoryWrite entry for that page on the emulator's cross-process work list. The emulator applies it the next time it drains the list and drops its translations of the page, so code that a system call overwrote is translated again. If no entry is free, the work list is flagged for a full flush, so the write is never dropped.
  • dlls_ntdll_signal_arm64ec_c_cross_process_work_callback.patch: ProcessPendingCrossProcessEmulatorWork now applies the work inside a syscall callback (enter_syscall_callback / leave_syscall_callback, cleared only when this call set it), as every other emulator notification already does.

Why the second patch is needed

Without it, the first entry hangs the thread. Stack captured on the device (FEX 2512, one thread, newest frame first):

SleepConditionVariableSRW
std::__shared_mutex_base::lock                  CodeInvalidationMutex, second time
InvalidationTracker::InvalidateIntervalInternal
InvalidationTracker::InvalidateAlignedInterval
recursive_mutex::lock                           NotifyMemoryFree
VirtualFree                                     allocator, inside the invalidation
ContextImpl::InvalidateThreadCachedCodeRange
InvalidationTracker::InvalidateIntervalInternal CodeInvalidationMutex held
BTCpu64NotifyMemoryDirty
ProcessPendingCrossProcessEmulatorWork
ECSyscallHandler::HandleSyscall

FEX holds its code invalidation lock while its allocator frees memory. ntdll only skips memory notifications inside a syscall callback, so the nested NtFreeVirtualMemory was reported back to FEX, which took the same non-recursive lock again on the same thread. The lock is not taken from the system call or the unix fault handler: the entry is queued there without locks and applied later on the PE side.

Upstream Wine has the same drain without the guard. Nothing in Wine queues CrossProcessMemoryWrite today, so there it only shows up with work queued by another process.

Results

AYN Odin 3, stock Proton 11.0-2 tree with this PR's ntdll.so / ntdll.dll from CI run 43691683416. The test is an x86-64 program that runs code in an RWX page, then lets system calls write into the page, both through the ntdll export and through a raw syscall instruction.

Case Stock 11.0-2 #131 only #131 + this PR
NtQuerySystemInformation into a page whose code already ran c0000005 0 0
NtReadVirtualMemory overwrites code that already ran, then call it (1,000×, ntdll export) fails, 1,000 stale succeeds, 1,000 stale 0 stale
Same through a raw syscall fails, 1,000 stale succeeds, 1,000 stale 0 stale
10,000 syscall writes into the page while 2 threads execute from it 8,550 errors 0 errors 0 errors, no hang
Self-modifying code, 10,000 same-thread + 10,000 cross-thread 0 wrong 0 wrong 0 wrong
20,000 × (data write + call) on one RWX page 1654 ms 1653 ms 1587 ms (FEX with the GameNative/FEX#11 fix) / 1493 ms (gn-6)

The work-list guard was checked against the build without it, the first version of #131, which queues the entry but has no guard. With FEX 2611-gn-6 it hung in the concurrent case; with GameNative/FEX#11, or with this PR's guard, it does not. Either side's change stops the hang on its own.

Hogwarts Legacy, real Steam client headless, overlay off, gamescope, with this build and the FEX#11 build: the game reaches its first screen, the WB EULA with Accept, at 31 fps on the HUD.

Known, not changed here:

  • NtQueryVirtualMemory reports what FEX's guest-visible protection says. With the GameNative FEX series that is 0x40 (patch 0002); a stock FEX reports 0x20 for a page it trapped.
  • A WriteProcessMemory from another process into code the target already ran still runs the old code in the target. The bytes change, but the target never drops its translation, even after a later system call. The same happens on stock 11.0-2; it is a separate path (CrossProcessFlushCache from kernelbase).

Utkarsh Dalal added 3 commits November 15, 2026 10:35
…cted

On ARM64EC the emulator write-protects RWX pages it has translated code
from, to catch self-modifying code. A system call that writes its output
into such a page (for example NtQuerySystemInformation into a buffer that
shares a page with code the program generated) faulted on the unix side and
returned STATUS_ACCESS_VIOLATION, where Windows completes the call. Make the
page writable again for the call and queue a memory-write notification so the
emulator drops its translations of that page.
Queuing CrossProcessMemoryWrite from the signal handler hung a test program
in the emulator's handling of it. Only lift the write protection; the
emulator's translations of a page a system call wrote into are not dropped.
A system call that writes into an RWX page the emulator write-protected now
queues a memory-write entry on the cross-process work list, so the emulator
drops its translations of that page the next time it processes the list. When
no free entry is left the work list is flagged for a full flush instead.

ProcessPendingCrossProcessEmulatorWork now applies the work inside a syscall
callback. The emulator invalidates code under its own lock and may allocate or
free memory while doing so; outside a callback those calls were reported back to
it, and it took the same lock again on the same thread and hung.
@utkarshdalal
utkarshdalal merged commit a23e554 into proton_11.0-2 Nov 15, 2026
3 checks passed
@utkarshdalal
utkarshdalal deleted the arm64ec-report-syscall-writes-to-emulator branch November 15, 2026 10:30
utkarshdalal added a commit to utkarshdalal/GameNative that referenced this pull request Nov 17, 2026
…efaults for new containers (#2454)

* Proton 11.0-2 rebuild 12 in the manifest and as the default for new containers

Rebuild 12 is GameNative/proton-wine proton_11.0-2 at a23e5548d (release
proton-11.0-2-20261115): rebuild 11 plus GameNative/proton-wine#133, where
ntdll tells the emulator when a system call writes into code it has already
translated, so that code is translated again (arm64ec only). Over rebuild 10
it also has GameNative/proton-wine#128. The x86_64 build is the same source
as rebuild 11.

New containers take proton-11.0-2-arm64ec-12 on every device class that took
rebuild 9. Existing containers keep the version they have. The rebuild 12 ids
join BionicSteamAssetsDependency so those installs get the Proton 11
steam.exe and lsteamclient handling.

* FEX 2611-gn-7 in the manifest and as the default

Series 7 is GameNative/FEX at 6e1003396 (tag gn-7): gn-6 plus
GameNative/FEX#10, a fix for a start-up hang in the stale NoExec retry on
protected launchers, on upstream FEX-2611.1 (gn-6 was on FEX-2611). The
pack keeps the 2611-gn line, so overlays with a 2611-gn floor resolve to it.

---------

Co-authored-by: Utkarsh Dalal <utkarsh.dalal@toptal.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant