Skip to content

fix: allow restart-analysis replay in assert_cache_hit mode - #75

Open
cennn wants to merge 1 commit into
mainfrom
fix/assert-cache-hit-restart-analysis
Open

fix: allow restart-analysis replay in assert_cache_hit mode#75
cennn wants to merge 1 commit into
mainfrom
fix/assert-cache-hit-restart-analysis

Conversation

@cennn

@cennn cennn commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Problem

When assert_cache_hit=True (bake-verify mode) and a subgraph has restart_analysis_count > 0, CompilerManager.load() returns None due to restart-analysis skip. compile() then unconditionally raises RuntimeError('cache miss'), even though the cache entry exists.

This causes verify_compile_cache.sh to fail for any model whose compilation triggers RestartAnalysis (e.g. via TensorifyScalarRestartAnalysis from the tensorify_python_scalars pass).

Fix

Change the assert condition from:

if self.compile_config.assert_cache_hit:
    raise RuntimeError(...)

to:

if self.compile_config.assert_cache_hit and cache_entry not in self.cache:
    raise RuntimeError(...)

When the cache entry exists but load() returned None (restart skip), fall through to the normal compile path. standalone_compile will trigger the same TensorifyScalarRestartAnalysis as bake, dynamo re-traces with the correct graph shape, and on retry load() succeeds.

Test

test_assert_cache_hit_restart_analysis.py — two-process integration test:

  • Process 1 (bake): warms cache, encounters RestartAnalysis → restart_analysis_count=1
  • Process 2 (verify): runs with MAGI_COMPILE_ASSERT_CACHE_HIT=1 → must succeed

Athena PR: https://github.com/world-sim-dev/athena/pull/1068

@cennn
cennn force-pushed the fix/assert-cache-hit-restart-analysis branch from 888acf1 to e84eb7a Compare September 10, 2026 14:35
@cennn cennn added the ci:run Trigger CI integration tests label Sep 10, 2026
@github-actions github-actions Bot removed the ci:run Trigger CI integration tests label Sep 10, 2026
When assert_cache_hit=True and a cache entry exists but load() returns None
due to restart_analysis_count > 0, fall through to the normal compile path
instead of raising RuntimeError.  standalone_compile will trigger the same
TensorifyScalarRestartAnalysis that occurred during bake, causing dynamo to
re-trace with the correct graph shape.  On retry, load() succeeds.

Without this fix, verify_compile_cache.sh fails with a false cache miss for
any subgraph that encountered RestartAnalysis during bake.

Add regression test: test_assert_cache_hit_restart_analysis.py
@cennn
cennn force-pushed the fix/assert-cache-hit-restart-analysis branch from e84eb7a to 7beaf7c Compare September 10, 2026 14:45
@cennn cennn added the ci:run Trigger CI integration tests label Sep 10, 2026
@github-actions github-actions Bot removed the ci:run Trigger CI integration tests label Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant