Repository navigation
An import is discovered once: nested imports were generated 2^depth times - #769
Merged
Merged
Conversation
…imes mlirGenInclude ran a discovery pass and then codegen for every source import. Discovery works in a throwaway module with an empty emittedFiles (#371: that module must hold every declaration), so each import's own imports were generated in both passes again - an import d levels deep was generated 2^d times. powder_sandbox's checks.ts sits on a chain of eight imports: at -O3 -mm=rc it ran past 10 GB and several minutes (16 GB / 168 s at -O0). An import whose discovery has run once (discoveredImports, which survives discovery like filesInProgress) is now generated into the current module without being discovered again: its declarations are in the maps already. Every module still gets every file. checks.ts: 4.8 s / 360 MB at -O3 -mm=rc; simulation.ts's IR is unchanged except generated ids. Test: import-chain/deep_chain.ts, a chain of 20 imports with a diamond at each link (TIMEOUT 120; unfixed it ran past 150 s, fixed it compiles in 0.3 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A file at the end of a deep import chain took exponential time and memory to compile. powder_sandbox's
checks.tssits eight source imports deep (checks → sandbox → ui → simulation → saves → levels → view → state, plusdefs/nriin nearly every file). Its build step at--opt --opt_level=3 -mm=rcran past 10 GB and several minutes, and the Android build ofpowder_sandbox_nri.tsreached 19 GB. At-O0it hit a 16 GB cap after 168 s, so this is MLIRGen, not LLVM or the ownership passes.Compiled on its own at
-O0, each file in the chain takes twice the time and memory of the one it imports: state 2.1 s / 251 MB, view 4.0 s, levels 8.3 s, saves 16 s, simulation 33 s / 3.4 GB, ui and sandbox over 6 GB. A minimal chain of tiny modules gives the same curve: 3.8 s at depth 12, 16.5 s at 14, 62.8 s at 16.Cause
For every source import,
mlirGenIncluderunsmlirDiscoverAllDependenciesand thenmlirCodeGenModule. Discovery works in a throwaway module whoseDiscoveryModuleScopestarts with an emptyemittedFiles. That is deliberate (#371): the throwaway module must hold every declaration. But it means both passes generate all of the import's own imports again, so T(d) = 2·T(d−1).Fix
A new
discoveredImportsset survives discovery, asfilesInProgressdoes. An import whose discovery has already run once is generated into the current module without being discovered again, because its declarations are already in the maps. TheemittedFilesreset is unchanged, so every module still receives every file. Cost goes from 2^depth to about depth² file generations. A key is added only after the import generated successfully, so a failed import in a cycle is still retried.Results
checks.ts-O3 -mm=rcpowder_sandbox_nri.ts-O3 -mm=rc(Windows / Android aarch64)checks.ts-O0import-chain/deep_chain.ts(20 deep)simulation.ts's LLVM IR is identical to alpha113's after normalizing generated numeric ids: 126 defines, 1727 declares, 12505 lines.Tests
tslang/test/tester/import-chain: a chain of 20 imports with a diamond at each link (every module also importschain_01). Registered astest-compile-import-chain-deepandtest-jit-import-chain-deep, each with TIMEOUT 120.-sharedvariants: any nested import (main → m2 → m1) undertest-runner -sharedalready fails with "redefinition of symbol", with or without this change. That is being fixed separately.ctest -C Releaseon Windows: every existing test passed (the only failures were the two-sharedvariants of the new test, since removed). The new compile and JIT tests pass.🤖 Generated with Claude Code