BPMN eval: 4 pattern tasks fail on turn-count/looping, not timeout (follow-up to #3150)

[nikhil-maryala]

Context

Follow-up from #3150. In the 2026-09-08 claude/sonnet-5 nightly (2026-09-08_04-17-10), 8 BPMN pattern/node tasks failed on budget. #3150 raised turn/max-turn budgets and a verification run (34374073086) confirmed 4 pass. The remaining 4 do not respond to a larger budget — the limit shifts rather than resolves:

Task	Result at bumped budget	Read
patterns/ai_decision_review/brownfield	MAX_TURNS_EXHAUSTED (70 turns), score 0.167	No longer times out; now turn-count bound
patterns/approval_chain/near_miss	MAX_TURNS_EXHAUSTED (70 turns), score 0.000	Looks like looping, not slowness
patterns/composition/nested_review	ERROR — turn_timeout hard-killed at 2700s	Too heavy even at 45 min
patterns/queue_distribution	ERROR (early), cause unclear	Possibly transient — needs a recheck

Why a bigger cap won't fix these

Per the coder-eval timeout guidance: raising turn_timeout shifts the binding limit to max_turns, and a task still blowing a 45-minute turn budget is runaway/heavy, not slow. The fix is turn-count reduction, decomposition, or a skill-guidance change — not a larger cap.

Proposed work

1. queue_distribution — re-dispatch alone to rule out a transient error; if it recovers, just re-include it.
2. ai_decision_review/brownfield & approval_chain/near_miss — read the agent transcripts (blob runs/.../default/<task>/00/task.json) to see whether they loop; if a skill-guidance gap drives extra turns, fix that. Only bump max_turns if the transcript shows genuine, non-repeating progress.
3. composition/nested_review — likely needs decomposition or a tighter authoring path; investigate why one streamed turn exceeds 45 min.

Acceptance

All four reach SUCCESS on a claude/sonnet-5 verification run, with the root cause (looping vs genuinely heavy) documented per task.

[nikhil-maryala]

Plateau note (from #3150 review): external_wait + high_volume_batch have now been bumped twice in ~8 days — #2940 (2026-09-01) set 2400/1800 calling it "a generous ceiling" and root-caused it to Bedrock per-turn latency across 52-193 assistant turns; #3150 raised them again to 3600/2700. The underlying cause (per-turn Bedrock latency on heavy single-turn authoring) is unchanged, so a further raise is likely. Before the next bump, treat this as a decision: either accept the class budget at the contract_variant_wrappers ceiling (3600/2700) as terminal, or decompose these heavy authoring tasks. The four tasks tracked in this issue are the ones where even that ceiling is not enough.

[nikhil-maryala]

Current state on main (post-merge of #3150/#3151/#3152)

Ran all 4 deferred tasks on main at their base budgets, claude/sonnet-5 (run 34411222183): 1/4 PASS. This refines the per-task diagnosis:

Task	Result on main	Refined read
approval_chain/near_miss	✅ SUCCESS	Flaky, not consistently broken. Nightly: timeout @1800s. On the #3150 branch @2700s: MAX_TURNS, score 0.000 (looked like looping). On main @1800s: passes. Non-deterministic → treat as flaky/borderline, likely a modest budget+turn bump with variance tolerance, NOT decomposition. Re-run a few times to characterize.
ai_decision_review/brownfield	❌ ERROR (turn timeout @900s)	Inherits the base 900/1200 default (no override on main). Needs the timeout override like the 8 fixed — but @1800s on the #3150 branch it hit MAX_TURNS, so it needs both a budget bump and turn-count reduction.
composition/nested_review	❌ ERROR (timeout @1800s)	Consistently too heavy — turn_timeout fired at 1800s here and at 2700s on the #3150 branch. Needs decomposition or a tighter authoring path, not a bigger cap.
queue_distribution	❌ MAX_TURNS_EXHAUSTED	Consistent — the earlier one-off ERROR on the #3150 branch is not the real behavior; it is turn-bound (nightly hit 71 turns, cap 70). Needs a max_turns bump justified by a transcript showing genuine progress, or turn-count reduction if it repeats.

Updated proposed work (supersedes the initial list):

1. queue_distribution — read the transcript; if turns are genuine progress, bump max_turns; if repeating, reduce turn count.
2. approval_chain/near_miss — re-run 3-5x to confirm flakiness; if borderline, small budget/turn bump + accept variance.
3. ai_decision_review/brownfield — add timeout override AND investigate why it needs >max_turns at 1800s.
4. composition/nested_review — decomposition / tighter authoring path.

[nikhil-maryala]

Two rows here are closed by PR #3369's verification run 35177570730 (agent=claude, CLAUDE_CODE_MODEL=claude-sonnet-5):

• ai_decision_review/brownfield — SUCCESS 1.0 in 1223s with max_turns untouched at 70
• approval_chain/near_miss — SUCCESS 1.0 in 951s with max_turns untouched at 70

Both were wall-clock-bound (the claude session streams as one turn, so turn_timeout was the binding cap), not looping: transcripts show steady non-repeating progress and tool execution accounts for <50s of the wall. The "bigger cap won't fix these" hypothesis is refuted for these two — they needed a bigger turn budget, which #3369 grants. Still open here: queue_distribution and composition/nested_review (the latter remains budget-insensitive — killed at 1800s on 09-14/15/16 alike).