Skip to content

A source import is parsed once per compile, not each time a module generates it - #772

Merged
ASDAlexander77 merged 1 commit into
mainfrom
import-parse-cache
Nov 6, 2026
Merged

ASDAlexander77 merged 1 commit into
mainfrom
import-parse-cache

Conversation

@ASDAlexander77

Copy link
Copy Markdown
Owner

Problem

After #769, a file deep in an import chain still spends most of its compile on its imports. Profiling powder_sandbox's checks.ts at its build flags (--di --opt_level=0 -mm=gc): 3.4 s of a 3.96 s compile is imports, and the biggest part is parsing. mlirGenInclude generates an import into every module that needs it (each discovery module, then theModule), and read and parsed the file again each time.

phase calls time if each file ran once
parse (loadIncludeFile) 211 1.97 s 0.24 s
import codegen 211 0.67 s 0.07 s
discovery 27 0.41 s 0.41 s
/// <reference> .d.ts in discovery 25 0.29 s 0.02 s

nri.ts alone was parsed 20 times.

Fix (MLIRGenModule.cpp, MLIRGenImpl.h)

loadImportFile keeps a parsed import and the files it references in parsedImports, keyed by canonical path, and reuses them.

  • A reused tree has every node's processed flag cleared first, as a fresh parse has it.
  • Everything else MLIRGen writes into the tree (accessor methods added to a class, internal flags) already had to hold across two passes of one parse, discovery then codegen.
  • No MLIRGen map is keyed by an AST node.
  • An import still on the generation stack is never loaded again (filesInProgress), so no pass is still using the nodes when they are reused.

Results

main this PR
checks.ts --di -O0 -mm=gc 4.10 s / 357 MB 2.63 s / 202 MB
checks.ts -O3 -mm=rc 4.82 s / 364 MB 3.25 s / 200 MB
powder_sandbox_nri.ts --di -O0 -mm=gc 4.07 s / 376 MB 2.29 s / 206 MB
simulation.ts --di -O0 -mm=gc 2.50 s / 251 MB 1.54 s / 159 MB

The LLVM IR of checks, powder_sandbox_nri and simulation is identical to main's (30715, 8758 and 10611 lines, after normalizing generated numeric ids).

Tests

  • Full ctest -C Release on Windows: 6651/6651 passed, including import-chain-deep (compile, JIT, -shared) and reference-path-import-nested.

Next

Import codegen still runs 211 times (0.67 s). Cloning the ops of an import already generated, instead of generating them again, would remove most of that.

🤖 Generated with Claude Code

…nerates it

mlirGenInclude generates an import into every module that needs it (each
discovery module, then theModule) and read and parsed the file again each
time. powder_sandbox's checks.ts parsed its imports 211 times: 1.97 s of a
3.96 s compile (nri.ts alone 20 times).

loadImportFile keeps the parsed import and the files it references in
parsedImports, by canonical path. A reused tree has every node's
`processed` flag cleared first, as a fresh parse has it; what else MLIRGen
writes into the tree already had to hold for discovery followed by codegen
of one parse. An import on the generation stack is never reloaded
(filesInProgress), so no pass is still using the nodes.

checks.ts --di -O0 -mm=gc: 4.10 s / 357 MB -> 2.63 s / 202 MB;
-O3 -mm=rc: 4.82 s -> 3.25 s. LLVM IR of checks, powder_sandbox_nri and
simulation is identical to main's. ctest -C Release: 6651/6651 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ASDAlexander77
ASDAlexander77 merged commit 6c2f206 into main Nov 6, 2026
2 checks passed
@ASDAlexander77
ASDAlexander77 deleted the import-parse-cache branch November 6, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant