Skip to content

closure: keep the lambda compiler's recursion inside the C stack - #140

Open
buzzom wants to merge 1 commit into
ldmud:masterfrom
buzzom:lambda-recursion-depth
Open

closure: keep the lambda compiler's recursion inside the C stack#140
buzzom wants to merge 1 commit into
ldmud:masterfrom
buzzom:lambda-recursion-depth

Conversation

@buzzom

@buzzom buzzom commented Jul 31, 2026

Copy link
Copy Markdown

compile_value() guards its recursion with a counter starting at
MAX_LAMBDA_LEVELS = 0x8000 (32768), but the C stack cannot take that many
levels, so the guard never fires: the driver crashes first. A deeply nested
value passed to lambda() segfaults instead of raising the existing
"Too deep recursion inside lambda()" error:

mixed v = 1;
for (int i = 0; i < 25000; i++) v = ({ #'+, 1, v });
lambda(0, v);   /* segfault */

compile_lvalue() recurses into nested lvalues without going through
compile_value() and never touches the counter at all, so it is a second,
completely unguarded path that a lower limit alone would not fix:

mixed lv = 'a;
for (int i = 0; i < 100000; i++) lv = ({ #'[, lv, 0 });
lambda(({'a}), ({ #'=, lv, 1 }));   /* segfault */

Measurements

All numbers from current master, gcc on x86_64 Linux, 8 MB stack limit
(ulimit -s 8192), bisected with the reproduction above:

Build survives crashes cost per level
-O2 19000 20000 ~430 bytes (compile_value() + compile_efun_call())
-O1 ASan/UBSan 7000 8000 ~1.1 KB

The lvalue chain is cheaper per level (25000 levels survive, 100000 crash at
-O2), but unbounded all the same.

Fix

  • Lower MAX_LAMBDA_LEVELS to 1000. That stays below 1 MB of C stack in a
    normal build, survives even a 2 MB stack limit in a sanitizer build
    (verified with ulimit -s 2048), and matches MAX_JSON_NESTING_DEPTH in
    pkg-json.c (pkg-json: bound the recursion in json_serialize() #136). With the new limit a value nesting depth of 998 still
    compiles and 999 raises the error cleanly.
  • Let compile_lvalue() take part in the depth accounting through a counting
    wrapper around the renamed compile_lvalue_int(), following the
    regmatch()/regmatch_int() pattern in regexp.c. The function has more
    than ten return statements, so a wrapper is less error-prone than
    restoring the counter at every exit.
  • Drop the stray semicolon at the end of the #define; it was harmless only
    because the macro is used solely in statement position.

Alternatives considered

  • Probe the real stack instead of counting levels, e.g. remember the
    stack address at lambda() entry and compare against a budget derived from
    getrlimit(RLIMIT_STACK). It would adapt to the actual frame size, but the
    driver currently calls getrlimit() nowhere, so this adds new portability
    surface for a limit nobody should ever reach legitimately. I am happy to
    follow up with this variant if you prefer it.
  • assert_stack_gap() only guards the gap between heap and C stack; on a
    64-bit host with the stack far above the heap it never fires (same
    reasoning as in pkg-json: bound the recursion in json_serialize() #136), and it would not have caught either crash.
  • A less drastic constant such as 4096. It works with an 8 MB stack in
    both builds (4096 × 1.1 KB ≈ 4.5 MB), but leaves no margin under smaller
    stack limits: with ulimit -s 2048 a sanitizer build dies far below 4096
    levels. Since even 1000 levels of nesting is far beyond any hand-written or
    generated lambda expression I could find, I chose the safer value.
  • A configure option like the one for LD_REGEXP_RECURSION_LIMIT.
    Possible, but the safe value depends on build flags rather than on the
    mudlib, so a compile-time constant with a documented rationale seemed more
    honest.

Tests

Four new cases in t-efuns.c:

  • a lambda 990 levels deep still compiles and evaluates correctly,
  • an lvalue chain 500 levels deep still compiles,
  • both kinds of 2000-deep nesting raise a catchable error.

Without the fix, both error tests fail with "There was no error." (and depths
of 25000 value levels / 100000 lvalue levels segfault the driver). The full
test suite passes in a normal build and in an ASan/UBSan build; the
sanitizers report nothing from closure.c.

The positive value test nests to the left (({ #'+, val, 1 })) on purpose:
compile-time recursion depth is the same in both directions, but a
right-nested chain would also grow the evaluator stack at call time, which is
a separate resource with its own limit.

Related, not part of this PR

  • compile_value() restores the counter on normal exit but not when
    lambda_error() unwinds. That is pre-existing and harmless: the counter is
    reinitialised at every lambda() call.

compile_value() guards its recursion with a counter starting at
MAX_LAMBDA_LEVELS = 0x8000 (32768), but the C stack cannot take that
many levels: one level of lambda nesting costs about 430 bytes of stack
(compile_value() plus compile_efun_call(), gcc -O2 on x86_64), so with
the common 8 MB stack limit the driver dies at a nesting depth of about
19500 - far below the limit that is supposed to protect it. A deeply
nested value passed to lambda() crashes the driver instead of raising
the "Too deep recursion inside lambda()" error:

    mixed v = 1;
    for (int i = 0; i < 25000; i++) v = ({ #'+, 1, v });
    lambda(0, v);   /* segfault */

Measured on master: a nesting depth of 19000 compiles, 20000 crashes.
An AddressSanitizer build needs about 1.1 KB per level and already dies
between 7000 and 8000.

compile_lvalue() has the same recursion but no guard at all - it never
touches the counter, so lowering the limit alone does not help this
path:

    mixed lv = 'a;
    for (int i = 0; i < 100000; i++) lv = ({ #'[, lv, 0 });
    lambda(({'a}), ({ #'=, lv, 1 }));   /* segfault */

Lower MAX_LAMBDA_LEVELS to 1000 and let compile_lvalue() take part in
the depth accounting through a counting wrapper, following the
regmatch()/regmatch_int() pattern in regexp.c. 1000 levels stay below
1 MB of C stack in a normal build, survive a 2 MB stack limit even in
a sanitizer build, and match MAX_JSON_NESTING_DEPTH in pkg-json.c.
With the new limit a value nesting depth of 998 still compiles and 999
raises the error cleanly.

Also drop the stray semicolon at the end of the #define; it was
harmless only because the macro is used solely in statement position.

Adds four cases to t-efuns.c: values and lvalues nested well below the
limit still compile (and the value case evaluates correctly), and both
kinds of 2000-deep nesting raise a catchable error. Without the fix
both error tests fail with "There was no error." The full test suite
and an ASan/UBSan run pass with the change.
@buzzom
buzzom force-pushed the lambda-recursion-depth branch from 6931395 to 5e8ba33 Compare July 31, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant