Skip to content

Game code generator in C++: no Python in the releases - #861

Merged
peiikai merged 8 commits into
developfrom
feature/gen-cpp
Nov 20, 2026
Merged

peiikai merged 8 commits into
developfrom
feature/gen-cpp

Conversation

@peiikai

@peiikai peiikai commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

The game code generator (tools/gen.py and its modules) is ported to C++ and runs inside the app. No release ships Python any more.

What changed

  • runtime/gen/: the generator in C++. The lifter, the portable FP/SIMD lowering, the patches, the NSO loader with its own LZ4 block decoder, and a small AArch64 evaluator for the jump tables in place of Unicorn. No Capstone (generation never used it). make lasgen builds it as a command-line tool; make gen uses it.
  • runtime/gamebuild.c: setup runs the generator on a thread per module instead of starting the lasgen helper, with the same progress and error reporting. The setup log says how long each module took.
  • Removed from the builds: the PyInstaller lasgen helper (macOS, Windows, Linux), Python with Capstone, Unicorn and lz4 in the APK (gen.zip now holds only zig's lib and the headers), the Windows no-CFG spawn path and its longjmp fix (Game files setup fails: "The game code didn't build, make sure there is 2GB free" #337, Beta 6rc For Windows ARM crashes during compile #779), and the x64 helper Windows ARM64 ran under emulation. Unicorn (GPL 2.0) was the last GPL component in the releases.
  • ci, nightly and release workflows, the README licence table and the build docs updated.
  • Docs: the README's "Building from source" rewritten for every platform (game files in extracted/ + make gen, or the start screen with tools/fetch_zig.sh; an iOS-from-source section; Python only for tests and some dependency scripts), the platform guides and CONTRIBUTING likewise, and code comments now point at runtime/gen/ instead of the Python. runtime/cpu.h and simd.h are untouched, so players keep their compiled objects after updating (only the generation step reruns).
  • make lasgen builds with the host compiler, so make gen PLATFORM=ios and cross ARCH= builds can run it.
  • The updater removes the old helper (lasgen/lasgen.exe and lasgen/_internal/) once the new version has reached the start screen, and the lasgen folder only if nothing else is in it. Only the Windows zip needs this; the macOS app and the AppImage are replaced whole.
  • tools/gen.py stays as the reference until this has shipped. A generator, patch or chunk-cut change goes into both; tools/test_gen_cpp.py checks they agree.

How it was tested

  • Byte-identical C: tools/test_gen_cpp.py compared every file for 1.0.0, 1.0.1 and 1.1.0, main and sdk, --simd asm / c, with and without --dyn-base (24 cases, all identical). The C++ lifter and SIMD lowering were also fuzzed against the Python on several million instruction words (0 differences).
  • Speed: main takes about 5 s instead of 30 s, peak memory about 300 MB per module.
  • A clean first-time setup from each packaged build (draft dry run), then gameplay:
    • macOS DMG (1.0.0): setup 2:35, played.
    • Windows x64 (i9-12900K, RTX 3080 Ti): setup 198 s, played.
    • Windows ARM64 (VM, lavapipe): setup 487 s, played. The generator now runs in the ARM64 exe.
    • Linux aarch64 AppImage (Debian VM, llvmpipe): setup 316 s (game code 266 s; 335 s with the Python generator), played.
    • Android, Retroid Pocket 5: cold setup 529 s, 60 fps.
    • iOS export from the desktop: iPad (1.0.0) and iPhone 15 Pro (1.1.0), both played at 60 fps.
    • Linux x86_64 AppImage: built, no Python inside; not run (same generated C as the others).
  • CI green on every platform.

Not in this PR

  • nightly.yml on main still has the helper steps; it changes in its own PR so both copies stay identical.
  • --prepare still needs a Vulkan driver (it opens a window); unchanged by this branch.

🤖 Generated with Claude Code

peiikai and others added 8 commits November 19, 2026 12:43
A port of tools/gen.py and its modules: the lifter, the portable SIMD
lowering, the patches, the NSO loader with its own LZ4 block decoder,
and a small AArch64 evaluator for the jump tables in place of Unicorn.
The output is byte-identical to gen.py; tools/test_gen_cpp.py runs both
and compares. The Python stays in tools/ as the reference for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
runtime/gamebuild.c runs the C++ generator (runtime/gen/) on a thread per
module instead of starting the lasgen helper, with the same progress and
error reporting. The HLE names and the randomizer's code variants are
compiled in. The generator's evaluator now tracks unknown FP values so
tables with FP work in their window resolve as with Unicorn, and its
scan cache keeps only kind and target (main: 300 MB instead of 1.2 GB).
make lasgen builds the command-line tool; make gen uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The generator runs inside the app, so the releases drop the frozen lasgen
helper (macOS, Windows, Linux), Python with Capstone, Unicorn and lz4 in
the Android APK, the Windows no-CFG spawn path and its longjmp fix
(#337, #779), and the x64 helper Windows ARM64 ran under emulation.
Unicorn (GPL 2.0) was the last GPL component in the releases.

- Makefile: no gen-helper target or gen-venv; make deps builds no Python
- macOS bundle, Windows zip, Linux AppImage: no lasgen/, no gen_helper.sh
- Android: build_gen_deps.sh fetches zig only; gen.zip holds zig's lib
  and the headers; check_apk.sh allows only those
- ci, nightly, release workflows: helper and unicorn steps removed
- README licence table and build docs updated

tools/gen.py stays as the reference (tools/test_gen_cpp.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Testers on Android and Linux missed the old generator line: the log now
says the generator is built in and how long each module took.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	docs/BUILD-windows.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
README: the two ways to get the game code (extracted/ + make gen, or the
start screen with tools/fetch_zig.sh), Python only for tests and some
dependency scripts, an iOS-from-source section, and the source checkout
as a data folder. The platform guides and CONTRIBUTING say the same.
Comments that pointed at tools/patches.py and gen.py now point at
runtime/gen/ (cpu.h and simd.h left alone: they are part of every
compiled object's id, so players keep their compiled objects).

make lasgen now uses the host compiler (HOSTCXX): with PLATFORM=ios or a
cross ARCH, CXX targets the device and make gen could not run lasgen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An in-place update (the Windows zip) keeps every file the new version
doesn't replace, so the lasgen/ helper of older versions (Python and
Unicorn) stayed in the app folder. Once the updated version reaches the
start screen, its own parts go (lasgen[.exe] and _internal/, which marks
it); the folder too if nothing else is in it. Files of the player's own
there stay, and a lasgen folder without _internal is left alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@peiikai
peiikai merged commit 3dcde48 into develop Nov 20, 2026
15 of 16 checks passed
@peiikai
peiikai deleted the feature/gen-cpp branch November 20, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant