Skip to content

An import is discovered once: nested imports were generated 2^depth times - #769

Merged
ASDAlexander77 merged 1 commit into
mainfrom
fix-import-discovery-exponential
Nov 5, 2026
Merged

ASDAlexander77 merged 1 commit into
mainfrom
fix-import-discovery-exponential

Conversation

@ASDAlexander77

Copy link
Copy Markdown
Owner

Problem

A file at the end of a deep import chain took exponential time and memory to compile. powder_sandbox's checks.ts sits eight source imports deep (checks → sandbox → ui → simulation → saves → levels → view → state, plus defs/nri in nearly every file). Its build step at --opt --opt_level=3 -mm=rc ran past 10 GB and several minutes, and the Android build of powder_sandbox_nri.ts reached 19 GB. At -O0 it hit a 16 GB cap after 168 s, so this is MLIRGen, not LLVM or the ownership passes.

Compiled on its own at -O0, each file in the chain takes twice the time and memory of the one it imports: state 2.1 s / 251 MB, view 4.0 s, levels 8.3 s, saves 16 s, simulation 33 s / 3.4 GB, ui and sandbox over 6 GB. A minimal chain of tiny modules gives the same curve: 3.8 s at depth 12, 16.5 s at 14, 62.8 s at 16.

Cause

For every source import, mlirGenInclude runs mlirDiscoverAllDependencies and then mlirCodeGenModule. Discovery works in a throwaway module whose DiscoveryModuleScope starts with an empty emittedFiles. That is deliberate (#371): the throwaway module must hold every declaration. But it means both passes generate all of the import's own imports again, so T(d) = 2·T(d−1).

Fix

A new discoveredImports set survives discovery, as filesInProgress does. An import whose discovery has already run once is generated into the current module without being discovered again, because its declarations are already in the maps. The emittedFiles reset is unchanged, so every module still receives every file. Cost goes from 2^depth to about depth² file generations. A key is added only after the import generated successfully, so a failed import in a cycle is still retried.

Results

before after
checks.ts -O3 -mm=rc >10 GB, minutes 4.8 s / 360 MB
powder_sandbox_nri.ts -O3 -mm=rc (Windows / Android aarch64) 10 GB / 19 GB 4.0 s / 4.0 s
checks.ts -O0 >168 s, >16 GB (capped) 4.0 s / 345 MB
import-chain/deep_chain.ts (20 deep) still running at 150 s 0.3 s

simulation.ts's LLVM IR is identical to alpha113's after normalizing generated numeric ids: 126 defines, 1727 declares, 12505 lines.

Tests

  • New tslang/test/tester/import-chain: a chain of 20 imports with a diamond at each link (every module also imports chain_01). Registered as test-compile-import-chain-deep and test-jit-import-chain-deep, each with TIMEOUT 120.
  • No -shared variants: any nested import (main → m2 → m1) under test-runner -shared already fails with "redefinition of symbol", with or without this change. That is being fixed separately.
  • Full ctest -C Release on Windows: every existing test passed (the only failures were the two -shared variants of the new test, since removed). The new compile and JIT tests pass.

🤖 Generated with Claude Code

…imes

mlirGenInclude ran a discovery pass and then codegen for every source import. Discovery
works in a throwaway module with an empty emittedFiles (#371: that module must hold every
declaration), so each import's own imports were generated in both passes again - an import
d levels deep was generated 2^d times. powder_sandbox's checks.ts sits on a chain of eight
imports: at -O3 -mm=rc it ran past 10 GB and several minutes (16 GB / 168 s at -O0).

An import whose discovery has run once (discoveredImports, which survives discovery like
filesInProgress) is now generated into the current module without being discovered again:
its declarations are in the maps already. Every module still gets every file. checks.ts:
4.8 s / 360 MB at -O3 -mm=rc; simulation.ts's IR is unchanged except generated ids.

Test: import-chain/deep_chain.ts, a chain of 20 imports with a diamond at each link (TIMEOUT
120; unfixed it ran past 150 s, fixed it compiles in 0.3 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ASDAlexander77
ASDAlexander77 merged commit de5f2d8 into main Nov 5, 2026
2 checks passed
@ASDAlexander77
ASDAlexander77 deleted the fix-import-discovery-exponential branch November 5, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant