Skip to content

Fix the offline code cache on WoW64 and ship the offline compilers - #7

Open
utkarshdalal wants to merge 3 commits into
gamenativefrom
code-cache-fixes
Open

utkarshdalal wants to merge 3 commits into
gamenativefrom
code-cache-fixes

Conversation

@utkarshdalal

Copy link
Copy Markdown

Tried FEX's offline AOT code cache (FEX_ENABLECODECACHINGWIP=1) on a 32-bit game (Blazing Chrome) on a Poco F6 with 2610-gn-4. Capture works, but a compiled cache kills the game the moment it loads, and the 64-bit caches crash winhandler. Two bugs in FEX-2610, neither fixed upstream yet:

  • Since upstream Avoid double offset relocations FEX-Emu/FEX#5840 the JIT records relocation offsets per compiled unit, but the cache writer and loader still treat them as offsets into the whole code buffer. Every cache carried stale host pointers and guest addresses for the compiler's load base. 0008 rebases them when a cache is generated or validated and sorts them on save.
  • The WoW64 loader maps the cache file read-only and then patches relocations into it. The first write faults during process start-up, before FEX's handler is up, so the guest dies silently. CodeCache: Mark executable memory as EC code on ARM64EC FEX-Emu/FEX#5658 fixed the same thing for ARM64EC only. 0010 gives WoW64 a writable copy too.

Plus two compiler fixes found on the way: 0007 (reads SilentLog before loading config, crashes at startup) and 0009 (process-all passes child arguments unquoted, breaks on "blazing chrome.exe").

The package now also ships FEXOfflineCompiler32.exe / 64.exe in system32 so caches can be generated inside the prefix on the device. They only matter when the cache flag is on. The cache header hash is the tree hash, so compilers and DLLs from one build go together.

Tested so far: a cache built from a 126 KB code map with the old binaries parsed clean but was corrupt as described. The fixed binaries are being tried on the device now; result to follow here.

All four go upstream after this.

The code map capture worked on a 32-bit game, but every cache the offline
compiler produced was corrupt and the WoW64 loader faulted while patching
the read-only file mapping. Four patches fix the relocation rebasing, the
WoW64 load path, a compiler startup crash and process-all quoting. The
.wcp now also carries FEXOfflineCompiler32/64.exe so caches can be built
inside the prefix on the device.
@utkarshdalal

Copy link
Copy Markdown
Author

Device result with the patched 2610 build (hash 4706b291) on the Poco F6, 32-bit Blazing Chrome: the regenerated cache is now well formed (relocation offsets span the whole code buffer, sorted; no stale host pointers; guest addresses follow the SerializedBase rule), so 0008 does what it says. But the game still exits silently about 2 s after "Cache load: 41844 blocks" with validation on or off, before any window, same as the unpatched build. 0010 is therefore not sufficient for WoW64 on its own. Looking at the WoW64 load path further before this is ready; do not merge yet.

@utkarshdalal

Copy link
Copy Markdown
Author

Device result with the full series (through 0013, hash 8a3f34dd) on the Poco F6, 32-bit Blazing Chrome, Proton 11.0-2-arm64ec-9. Cache regenerated with the matching compiler from a 126 KB code map taken on a cold run (3,394 units, 41,844 entry points, 18.7 MB).

With FEX_ENABLECODECACHINGWIP=1 and validation off, the cache loads and the game runs: load phases log in order and end in "Cache load: enabled", ntdll loads, no exception, gameplay input works, Alt+F4 exits cleanly.

Timing from the final guest environment (both runs after the one-time prefix build):

warm (cache) cold
to splash hidden 4.9 s 25.8 s
to game window 13.7 s 36.9 s
first 60 s after window, worst frame 305 ms 366 ms
first 60 s, p95 max 61 ms 86 ms
steady fps 59 59

Part of the start-up gain is the warm page cache, not only the FEX cache. The warm run still wrote an 86 KB code map (misses), against 127 KB cold, so coverage of one play session is far from complete.

Open items, both left as is:

  • FEX_ENABLECODECACHEVALIDATION=1 overflows the 1 MB start-up stack on WoW64 (validation recompiles every block inside BTCpuProcessInit). 0011 only makes a fault there survivable; validation itself needs to move off the start-up path.
  • The 64-bit caches built from an ARM64EC process's code map (ntdll/kernel32/kernelbase) crashed winhandler with the old relocation bug; not re-tested with 0008.

@utkarshdalal

Copy link
Copy Markdown
Author

Second game, 64-bit this time: Balatro (LÖVE) on the same device with the full series. process-all built caches for all 36 images in its code map (19.9 MB, 9,001 units, 50,280 blocks; love.dll 9.4 MB, steamclient64 4.4 MB, lua51 2.0 MB, SDL2 1.4 MB, 23 Wine DLLs at 8–38 KB each). On the warm launch all 36 loaded through the ARM64EC path, zero failures, zero crashes, and the run's new code map had 245 entries against 9,001 cold, so about 97 % of the blocks hit the cache.

cold warm, all caches
to splash / window (excl. one-time setup) 5.1 / 6.4 s 3.3 / 4.5 s
first 60 s worst frame 469 ms 217 ms
after 60 s worst frame 1032 ms 101 ms
average fps 54.7 55.0

One thing worth knowing for anyone building caches for Wine's own DLLs: the code map's file id is the exact PE (name, timestamp, size) the game ran with. When the offline compiler is pointed at a Wine builtin by its real name, Wine substitutes its own copy of that DLL, so the compiler silently maps a different image and the cache comes out with invalid instructions. Compiling from renamed copies with the builtin marker patched out gave matching ids and clean caches. The compiler should probably refuse an image whose file id does not match the map.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant