m2: reviving Wirth's Lilith Modula-2 compiler

In the mid-1980s Niklaus Wirth's group at ETH Zürich built the Lilith — a workstation whose entire software stack, the operating system included, was written in Modula-2 and compiled by a single-pass Modula-2 compiler emitting M-Code: the bytecode of a stack machine that was itself defined as a Modula-2 program and executed by the machine's microcode. The hardware is long gone, and most of the software went with it. What survived is source text, and an ETH report.

This project exists to bring that compiler back to life. Not to read it, not to archive it — to run it: to take Wirth's mid-1980s Modula-2 compiler as he wrote it, build it on a modern PC, hand it real Modula-2, and execute the M-Code it emits. Compile, link, run, end to end, with no Lilith anywhere in the loop.

That means writing the two pieces the hardware used to supply — an M-Code emulator and a linker for multi-module programs — and the small runtime library of operating-system services (console, files, clock, heap, command line) that Lilith programs used to call. Those tools then compile Wirth's compiler and run it, and the compiler compiles itself to a fixed point: the second and third generations come out byte-identical. That is the strongest evidence available here that the revival is faithful rather than merely plausible.

It is done without a safety net: no real Lilith to compare against, no pre-existing M-Code emulator, no test oracle beyond the original sources themselves — Wirth's own Modula-2 code, and the ETH report The Personal Computer Lilith (docs/Lilith.txt), whose Appendix 1, "The M-code interpreter", is a Modula-2 program that is the definition of the instruction set.

Under a bar like that, one implementation is not evidence enough, so the four tools — m2c (the compiler), lilith (the emulator), m2l (the linker) and m2dis (the disassembler) — exist twice over: as a C89 translation in src/c/, and as an Oberon-07 port in src/o7/. Both build the same four names into the root bin/ — m2c.exe, lilith.exe, m2l.exe, m2dis.exe — and test/ runs against whichever one is there, so the two mirror each other rather than one being merely a reference for the other. IMPL=c or IMPL=o7 on the top-level make picks which. Everything below describes the tools themselves; docs/port.md describes the port.

• docs/arch.md — how the four pieces fit together, and the M-Code object-file format they all speak.
• docs/opcode.md — the M-Code instruction set, one entry per opcode: what it does to the stack, and where the three sources that define it agree.
• docs/m2c.md — the compiler: module-by-module breakdown, what's known-uncertain, how to build and run it.
• docs/lilith.md — the emulator: the calling convention, opcode coverage, how to build and run it, -trace usage.
• docs/m2l.md — the multi-module linker: object-file merging, the relocation scheme, what's in and out of scope.
• docs/port.md — the Oberon-07 port of the same four tools: how it is built, the language limits it was written against, how it is checked — and the third compiler compiling itself, the g1→g2→g3 fixed point.

Few STRONG rules:

• For better portability - need use C89 and stdint.h
• All file names must be in DOS 8.3 format (prefer when create).

Layout

m2/
  Makefile      top level. `make` builds one implementation into bin/;
                IMPL=c picks the C89 sources, IMPL=o7 (default) the
                Oberon-07 port. The two do not coexist in bin/: each
                overwrites the other's images, which is the point.
  bin/          m2c.exe lilith.exe m2l.exe m2dis.exe -- whichever
                implementation was built last, under the same names
                either way, so test/ is indifferent to which it is.
                `make dist` replaces this with what a user is given:
                five files, the four tools under a 32 suffix and the
                self-hosted compiler as m2c.LNK.  See "Delivery"
                below.
  build/        scratch for both builds, kept out of src/
  docs/         you are here
  lib/          runtime modules (Terminal, FileSystem, Clock, Heap, CmdLine)
  src/
    c/          the C89 implementation, and the live oracle the port is
                checked against. This is what docs/ describes and what
                the port is diffed against. The Makefiles redirect
                every output into build/, so a build writes nothing
                here.
      lilith/ m2c/ m2l/ m2dis/   the four C tools, with their testrun/
      m2.h                       the constants shared by all four
    m2/
      m2c/          Wirth's original Modula-2 source; read-only
      selfhost/     a working copy of it, built and run by the tools
                    in bin/; its Makefile keeps every product out of
                    this directory, under build/m2selfhost/
      testrun/      the artifacts m2c produced from those sources --
                    the goldens dissweep and lnksweep read
    o7/         the Oberon-07 port. The same four tools, written against
                a compiler that rejects most of what the C took for
                granted.
      M2RULES.MK  the shared build rules; each directory includes it
      common/     modules shared by more than one tool
      lilith/     the M-Code emulator        (done)
      m2dis/      the M-Code disassembler    (done)
      m2l/        the multi-module linker    (done)
      m2c/        the compiler               (done — M2C, M2S, M3DL, M3TL,
                                               M3GL, M3PL, M3RL)
  test/         one tree, run against whichever implementation is in
                bin/.  Organised like the tests that ship with the
                Oberon-07 toolchain (its samples/itsy/test):
                inputs, expected output and scratch are three separate
                directories, and the Makefile is the whole harness --
                no helper scripts
    Makefile     the entire harness. `make check` runs every check,
                 `make run` prints what the probes say with nothing
                 compared, `make refresh-<set>` re-records the expected
                 output
    scancases/   malformed Modula-2 inputs, written so that the
                 scanner's error paths are reached on purpose rather
                 than by luck
    tlcases/     Modula-2 fragments for the symbol table's policy
                 branches a real source does not reach
    rocases/     module pairs for read-only export -- the one language
                 fact a single module cannot state
    fix/         what correct output looks like: the C's own answers,
                 recorded once, so a check needs neither the oracle nor
                 python -- and one set for both implementations, which
                 is what makes them mirror each other.  Its README says
                 where each set came from
    out/         scratch: a run happens here, a case that passes
                 removes the files it created -- by name, never the
                 whole directory -- and whatever is left is the only
                 evidence a failure has
    tools/       the models and the scripts the port was checked with,
                 and the C-side helpers.  `make check` needs none of
                 them -- their answers are recorded in test/fix/, and
                 regress.sh reruns the sweeps against the live C as the
                 deeper check
      mkoracle.sh  builds the C tools outside the tree (in
                   /tmp/m2oracle by default) as a live oracle; it is
                   now only `make IMPL=c BINDIR=<dir> install` plus the
                   two C-side helpers below
      scandump.c   a main for the C scanner, so the C and the
                   port can be diffed token by token
      scansweep.py runs both and diffs stdout, err.DAT, err.LST and the
                   exit status
      tldump.c     the same for the C symbol table, which also
                   dumps the chains, the search trees and every
                   object's fields
      tlsweep.py   runs both and diffs that, err.DAT, err.LST and the
                   exit status
      m2ccmp.sh    the six compiler sources through both compilers,
                   .OBJ/.SBL byte for byte
      selfhost.sh  the fixed point: the compiler compiles itself twice,
                   under both toolchains
      m2orig.sh    the third compiler -- the Modula-2 one -- built by
                   the tools in bin/, run under lilith, against the same
                   goldens as the other two, and then over the input that
                   does not compile, where the C is the answer
      errcmp.py    that second half: the malformed sources and the cut
                   module sources, fed to both the third compiler and the
                   C, with the one difference that survives it
                   named and counted rather than tolerated
      octsweep.py  every hex literal that could have been transcribed
                   from a C octal one without converting the base
      oblib/       the two files of the Oberon library this port
                   changes, each in both states, with the account of the
                   defects -- see "Requirements"

src/m2/ — Wirth's own source — is the one read-only tree. A build writes nothing into it: make IMPL=c leaves find src/m2 -newer <build start> empty, and that is how it is checked. To get the C tools somewhere else without disturbing bin/, sh test/tools/mkoracle.sh builds them outside the tree.

Delivery

make dist turns this tree into the thing that gets handed over. The difference from a plain make is that everything a build produced is removed afterwards, so what is left is the two directories a user needs and nothing else.

bin/ ends up holding exactly five files:

m2c32.exe  lilith32.exe  m2l32.exe  m2dis32.exe     target win32con
m2c.LNK                                             Wirth's compiler

One target, and that is a decision rather than a gap. The port is written for a 32-bit Oberon word and its sources say so; on win64con DWAK's INTEGER is 64 bits, and the port is then silently wrong in three of the four tools — m2dis prints wrong word and byte counts, m2l a wrong module count, m2c compiles every REAL to zero, and lilith gets every signed comparison, signed branch, DIV/MOD and range check wrong. None of it raises an error; all of it produces a plausible wrong answer. The measurements are in the port's docs/port.md, under "the port is 32-bit". A missing image is better than a wrong one, so the suffixed name says which target the file is rather than leaving the choice to a guess.

m2c.LNK is the third compiler — the Modula-2 one, from src/m2/selfhost/ — compiled and linked by the tools beside it. It is not a host program: it is M-Code, so it runs under either lilith, and it takes the same command line as the two host compilers. Two things about running it are worth knowing, and both are the original's behaviour rather than the port's:

• it reads no M2LIB. It opens NAME.SBL in the current directory, so the five lib/*.SBL have to be where you are — or copied beside your source — for an IMPORT to resolve. The host compilers search, it does not.
• it writes its diagnostics as the original does: err.LST and err.DAT, the latter holding a record per module compiled rather than only per error, which is why a clean run leaves both files non-empty.

lib/ is kept whole — .DEF and .MOD for the five runtime modules, with the .OBJ, .SBL and .RFC the delivered compiler makes from them. NAME.SBL is what an IMPORT resolves through and NAME.OBJ what the linker needs, so both have to travel with the sources. The set is not hand-maintained: compiling the five with bin/m2c32.exe reproduces all fifteen products byte for byte, which is the check that the library and the compiler that ships beside it agree.

Everything else a build leaves is gone — build/, src/o7/*/build/, test/build/, and test/out/, which a passing run leaves empty anyway. The goldens stay: test/fix/, src/c/*/testrun/ and src/m2/testrun/ are data, not products. src/m2/ is untouched throughout.

One thing does not survive, and it is deliberate. test/ and the scripts under test/tools/ name the tools as bin/m2c.exe, bin/lilith.exe and so on, so make check and regress.sh cannot run in a delivered tree — they look for the development names that make dist has just renamed. The checks belong to the build, not to the delivery: run them before, and deliver after.

Requirements

An Oberon-07 compiler on PATH, as w64oc.exe by default — the tree names the compiler and never where it lives, so OCC= is the way to point it at another one. Its library does not have to be named either: the compiler resolves an import beside its own executable before the current directory, so it finds the library that ships with it (OBLIB= adds an explicit -l back, for a compiler installed elsewhere).

Two files of that library have to be patched: its Windows ArchArgs reads its environment out of freed memory, and its portable Args drops an argument that is quoted and empty. The patches live in test/tools/oblib/ and are installed by sh test/tools/fixoblib.sh <libdir>. docs/port.md has the whole account. Without the first, a program that looks up M2LIB in the environment gets an answer from wherever the heap happened to land; without the second, a guest sees a shorter argument vector than the host passed it.

Quick start

From m2/:

make                    # the Oberon-07 port -> bin/m2c.exe, lilith.exe, ...
make IMPL=c             # the C89 sources -> the same four names

Or build just one tool: make install in src/o7/{m2c,lilith,m2l,m2dis}, or in src/c/{m2c,lilith,m2l,m2dis}.

Every tool answers -? (or /?) with its usage text. The text is the same in both implementations of a tool, and its last line names the one that answered — that is how a build identifies itself, so a binary you found somewhere says what it is:

bin/m2c -?      # ... "m2c: C89 port" or "m2c: Oberon-07 port"
bin/lilith -?   # ... "lilith: C89 port" or "lilith: Oberon-07 port"

(Wirth's own compiler in src/m2/m2c/, the third implementation, prints the same text and ends its help with m2c: Modula-2 original.)

Compile and run a single self-contained module — work in test/out/ or any scratch directory, never in a testrun/ directory, which holds the recorded goldens the checks compare against:

cd test/out
../../bin/m2c RUN1.MOD                   # -> RUN1.OBJ, RUN1.RFC
../../bin/lilith -mem 6 RUN1.OBJ

Compile, link and run a multi-module program (see docs/m2l.md):

cd test/out
cp ../../src/c/m2l/testrun/*.MOD .
../../bin/m2c Helper.MOD HelperImp.MOD Second.MOD SecondImp.MOD Main.MOD
../../bin/m2l -dump -o Main.LNK Main.OBJ Helper.OBJ Second.OBJ
../../bin/lilith -mem 4 Main.LNK         # result = 126

Anything after the object file is passed to the program (read with lib/CmdLine's ArgCount/GetArg). The program name itself is not among them:

cd test/out
cp ../../src/c/m2c/testrun/ARGTEST.MOD .
export M2LIB=$PWD/../../lib/
../../bin/m2c ARGTEST.MOD
../../bin/m2l -o ARGTEST.LNK ARGTEST.OBJ $M2LIB/Terminal.OBJ $M2LIB/CmdLine.OBJ
../../bin/lilith ARGTEST.LNK ab -r "x y" # ArgCount = 3, not 4

The self-hosted compiler takes the same command line as m2c. It is a Modula-2 program, so building it means running the toolchain on itself:

make -C src/m2/selfhost run ARGS="M3DL.MOD M3GL.MOD -r"

That compiles the six modules and links them in build/m2selfhost/src, which is where the compiler then runs — nothing is written beside the sources, and the artifacts a check reads live in src/m2/testrun/.

Everything the tree claims about itself, including the live differential checks against a C build:

sh test/tools/regress.sh        # or: make regression

And the recorded answers, which need neither the oracle nor python:

make check                      # replays against test/fix/

What's verified, in one sentence

m2c compiles real Modula-2 source (single- and multi-module) into byte-correct M-Code object files; lilith loads and correctly executes single-module object files and m2l-linked multi-module programs, including nested procedures with working lexical (static-chain) variable access; m2l correctly merges multi-module object files and renumbers cross-module references; and m2orig compiled by m2c self-hosts to a g2 == g3 byte-identical fixed point (see docs/port.md) — each claim in the docs above is backed by a specific hand-verified test, not just "it didn't crash."

The Oberon-07 port of all four tools is held to the same bar and reaches it: the C tools rebuilt here are the live oracle, and every artifact the port produces — the emulator's trace, the disassembler's text, the linker's .LNK, the compiler's .OBJ, .SBL and .RFC — is compared against it byte for byte, with the self-host fixed point reached independently by both toolchains. The third compiler is held to the same bar and not a lower one: test/tools/m2orig.sh builds Wirth's compiler with the tools in bin/, runs it under bin/lilith.exe over the same five programs the other two are checked on, and compares against the same goldens -- the one word it fills and the C leaves zero being asserted rather than masked, and the four documented results read back out of the code it emitted. Those five programs compile, and that is half of what a compiler does: the same script then feeds this one the input that does not compile -- the malformed sources test/scancases/ keeps for that, and the module sources cut short at a stride -- and compares what it wrote against what the C wrote on the same bytes. That is the second half, and a golden can show none of it: test/tools/errcmp.py carries the comparison, and the one difference that survives it is named and counted there rather than tolerated: the original guarding against an overflow with a constant this emulator's REAL cannot represent.

sh test/tools/regress.sh runs all of it. make check replays the answers the C gave once -- recorded in test/fix/ -- so it needs neither the oracle nor python, and regress.sh now ends by replaying that same make check against a C build as well: one fixture set, two builds, both green, which is what makes "run it under both builds" a check rather than an instruction.

Those recorded answers are shared by both implementations, and that is what makes the mirroring a checkable claim rather than a slogan: make IMPL=c and make IMPL=o7 each put the same four names in bin/, and make check has to come out green against the same test/fix/ either way. One set says it outright — make check-bin drives all four names through a single scenario, compiling a program, linking it, running it under the emulator and disassembling it, and compares both the artifacts against the goldens the tree ships and everything the tools say against one recorded transcript.

License

The tree holds other people's work beside its own, so it is worth being exact about which is which. Three sets of terms, and one thing that should be said out loud rather than left to be discovered.

Wirth's compiler — ETH Zürich, via RAMSES

src/m2/m2c/ is Niklaus Wirth's own source for the one-pass Modula-2 compiler for Lilith. It says so on the second line of M3PL.MOD — "One-pass Modula-2 Compiler for Lilith. Copyright N.Wirth" — its modules are dated 1983–1986, and the compiler announces itself as ETHZ M3L NW 2.8.86. src/m2/selfhost/ is a working copy of the same text. It reached this project as part of RAMSES (Research Aids for Modeling and Simulation of Environmental Systems), the ETH Zürich package the compiler was distributed in — Modula-2 Singlepass compiler was part of RAMSES by ETH Zürich.

docs/Lilith.txt is the report this project works from, The Personal Computer Lilith, © Institut für Informatik, ETH Zürich, 1981 — reproduced by permission, and the document docs/Lilith.pdf was published as. Its Appendix 1, "The M-code interpreter", defines M-Code as a Modula-2 program.

Neither is public domain. The RAMSES/MacMETH License Agreement, which governs the compiler sources, is reproduced in full in LICENSE.TXT, or at https://www.sysecol2.ethz.ch/RAMSES/READ_ME.RAMSES.html. In short: RAMSES is freeware but not public domain — all rights reserved, copyright with the authors and ETH Zürich, redistribution only unaltered and never for profit. Those terms also require a credit, so here it is:

This software was developed by means of the RAMSES package. Its compiler — Wirth's one-pass Modula-2 compiler for Lilith — is the subject of this work.

This project's own work — public domain

Everything else here was written for this project: the C89 translation in src/c/, the Oberon-07 port in src/o7/, the runtime modules in lib/, the whole harness and its test cases in test/, the build files, and the documentation in docs/ that does not come from ETH.

That part is public domain. Use it under the Unlicense or as if it were — no attribution required, no conditions, no warranty.

Build-time dependencies

The Oberon-07 port is compiled by an external Oberon-07 toolchain, which is not vendored here — see "Requirements". The only files from it in this tree are the two patched library modules under test/tools/oblib/, each kept in both its original and its patched state, with the account of what was wrong with it. Those two files are someone else's work and keep the toolchain's own terms: BSD 2-Clause. Their copyright headers are part of the files and are left exactly as they ship — retaining that notice is a condition of the licence, and the reason they are byte-for- byte copies rather than the project's own text.

One edit to the compiler sources

Worth stating plainly. The sources in src/m2/m2c/ are not byte-for-byte the RAMSES distribution: the tree makes exactly one change to them — the usage text M3PL.MOD prints for -? and /?, aligned with the other two implementations of the compiler so that a single help-comparison check has one expected answer in all three. It is recorded, with the check that enforces it, in docs/port.md. Everything else is as Wirth wrote it.