Skip to content

review-commit: budget expiry + failed conclusion call = failed run with no finding #58

Description

@jschatz1

A kai review-commit job fails outright — no finding posted, run marked failed — when the review exhausts its time budget and the salvage path also times out.

Observed on marketing-site run #122 (kai ci trace 122), reviewing c83f66bf1e52 ("landing: rewrite the homepage StoryBrand-style"):

review ended without a conclusion (finish=time_budget) — requesting one from the transcript…
conclusion call failed: context deadline exceeded
timing: review=8m38s
Error: review produced no content (no prose, no risks, intent unknown) — failing instead of posting an empty finding
Exit code: 1

The run made ~24 tool calls over 8m38s of healthy exploration, hit the soft time budget at a turn boundary, and then lost everything it had read because the one salvage call didn't return in time.

The salvage path already has a fix — and it wasn't enough (or wasn't there)

rcConcludeFromTranscript (cmd/kai/review_commit.go:443) was hardened in f20db8a (2026-08-26, first released in v0.35.34) for exactly this failure: trim tool results to 2000 chars, and give the call a fresh 3-minute deadline off context.Background() rather than the run's exhausted context.

Whether the failing job had that fix is not determinable from the log — see the version-stamping issue filed alongside this one. Two possibilities, and the fix differs:

  1. Job predates f20db8a → this is already fixed; the real problem is that CI is running old code, and we couldn't tell.
  2. Job has f20db8a → a fresh 3 minutes still wasn't enough, and the salvage path needs to be more robust.

Resolve which before building anything.

If it is case 2, the levers

  • Model choice. The conclusion call reuses the review model (z-ai/glm-5.2 on this run — a reasoning model). A single 2500-token write-up over a trimmed transcript is a fast-model job; using the slow reasoning model for it is what pushes past 3 minutes.
  • Transcript size. Trimming caps each tool result at 2000 chars, but ~24 calls plus assistant reasoning still adds up. Consider capping total transcript bytes, not just per-result.
  • Retry. One shot, no retry, on a call whose whole purpose is salvage.
  • Degrade instead of failing. Right now: budget expiry → salvage fails → entire run fails with nothing. A finding recording "review ran out of budget after N tool calls, here is what it read" is more useful to a user than a red run with no output.

Why the budget was exhausted at all

The reviewed commit touches src/components/landing/markup.ts and src/app/styles/landing.css — both generated by scripts/gen-landing.mjs and already guarded by that repo's gen:check. The trace shows repeated view/kai_grep calls against them plus the generator and the source HTML. Reviewing generated output is spend with no return; an ignore mechanism for generated paths would likely have kept this run inside its budget.

Acceptance

  • Determine whether the failing job included f20db8a (blocked on the version-stamping issue)
  • Budget expiry never produces a failed run with zero output — either a conclusion lands, or a partial finding is posted saying the review was truncated
  • Conclusion call uses a fast model and/or gets a retry
  • Total transcript size bounded for the conclusion call, not just per-tool-result
  • Generated/derived paths excludable from review so the budget isn't spent on them
  • Regression test: a run that hits FinishReasonTimeBudget with a failing conclusion call still exits non-empty

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions