Skip to content

test: guard apply_fix -> al_compile freshness and correct the stale-diagnostic note - #24

Open
Arthurvdv wants to merge 3 commits into
mainfrom
fix/almcp-stale-diagnostics-after-apply-fix
Open

Arthurvdv wants to merge 3 commits into
mainfrom
fix/almcp-stale-diagnostics-after-apply-fix

Conversation

@Arthurvdv

Copy link
Copy Markdown
Member

Summary

What changed

Test files:

  • ApplyFixThenCompileTests.cs (new) — ApplyFixAlMcpFixture launches the full server, applies a code fix, then calls al_compile(onlyErrors: false) and asserts the fixed diagnostic is gone. A skipped al_getdiagnostics variant documents why that endpoint cannot verify fixes.
  • AlMcpProxyTests.csCreateProxy() gains an optional fixtureName parameter so the new fixture can reuse the helper.

Docs & tool descriptions:

  • ApplyFixTool.cs / ApplyFixAllTool.cs — tool descriptions now say: "Verify with al_compile (onlyErrors: false)."
  • README.md — new "Verifying a fix" section after the onlyErrors note.
  • AGENTS.md — new bullet in "Tool patterns" explaining al_compile vs al_getdiagnostics after a fix.

Reproduction outcome

Characterization (10 consecutive runs, no settle delay): 10/10 PASS.

This is the "passes every run" branch of the decision tree. The race between our file write and almcp's ProjectWatcher (FileSystemWatcher) is not reproducible on this machine: al_compile awaits WaitForProcessingAsync before compiling, and the OS delivers the change event quickly enough that the old text is never compiled.

No runtime code change was needed. The test stays as a regression guard.

The al_getdiagnostics variant confirmed the watcher-drain analysis: it returns {"succeeded":true,"diagnostics":[],"errorCount":0} — zero diagnostics — because it reads cached compilation results rather than re-analyzing. This test is [Skip]-ped with documentation.

Likely sources of the original report: using al_getdiagnostics (returns cached, never re-compiles), a child that had run al_downloadsymbols (disposes the workspace), or a project outside the child's --projects set.

Test plan

  • dotnet build --configuration Release — 0 warnings, 0 errors
  • dotnet test --configuration Release — 69 passed, 0 failed, 1 skipped (the al_getdiagnostics variant, intentionally skipped)
  • 10-run loop of ApplyFix_ThenAlCompile_NoLongerReportsFixedDiagnostic — 10/10 pass with no settle delay

Corrects PR #20's known-issue note

PR #20 noted that al_compile might return stale diagnostics after apply_fix, suggesting callers add a delay or call al_getdiagnostics. This was backwards: the child almcp does not serve a cached compilation for al_compile. It has a per-project FileSystemWatcher (ProjectWatcher) whose pending events al_compile drains via WaitForProcessingAsync before compiling, so it always picks up on-disk changes. al_getdiagnostics, on the other hand, returns the last cached compilation result without re-analyzing — it is the endpoint that misses changes. Only a full child process restart produces a truly "clean" state, but al_compile is fresh enough for verification after a fix.

🤖 Generated with Claude Code

Arthurvdv and others added 3 commits September 13, 2026 17:07
Add ApplyFixThenCompileTests with an end-to-end test that applies a code
fix via apply_fix, then calls al_compile (onlyErrors: false) and asserts
the fixed diagnostic is gone. Characterization result: 10/10 pass with
no settle delay — almcp's FileSystemWatcher + WaitForProcessingAsync
drains before compile on this machine. A skipped al_getdiagnostics
variant documents that it returns cached (zero) results instead of
re-analyzing. AlMcpProxyTests.CreateProxy() gains an optional
fixtureName parameter so the new fixture can reuse the helper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Update the apply_fix and apply_fix_all tool descriptions to direct
callers to verify with al_compile (onlyErrors: false) instead of
al_getdiagnostics. Add a "Verifying a fix" section to the README and
a bullet in AGENTS.md explaining the difference: al_compile awaits
almcp's FileSystemWatcher drain, while al_getdiagnostics returns
cached results without re-analyzing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…kipped al_getdiagnostics test

- Pass Cts.Token to ApplyFixTool.ApplyFix so the call respects the test's
  90-second cancellation timeout (CONFIRMED review finding).
- Remove ApplyFix_ThenAlGetDiagnostics_ReportsStaleDiagnostics: its only
  assertion (expects LC0020) contradicted its own skip reason (reports zero
  diagnostics). The knowledge is preserved in the class-level summary
  (PLAUSIBLE review finding).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant