test(uipath-maestro-bpmn): fix false-negative grading on 3 BPMN eval tasks

[nikhil-maryala]

Summary

Fixes false-negative grading on 3 BPMN eval tasks from the claude-code nightly run 2026-09-08_04-17-10. In each case the agent behaved correctly but a criterion failed it.

error_mapping — check had the wrong casing

The engine seeds the failed element's error under the capital Error key (references/expression-authoring.md:30-35, verified against PO.BpmnEngine in #2780); vars.error never resolves. The agent correctly authored =vars.Error.code == …, but the check regex + prompt required lowercase — and the check docstring cited the very reference it contradicted. Corrected check + prompt to capital Error. Scope note: the check grades casing only; the <uipath:output source="=Error"> resolution binding is tracked in #3171 (the skill's own example currently omits it).

wiki_pageviews — judge failed read-only registry discovery

The llm_judge failed the agent for uip maestro bpmn registry pull/search/get — read-only discovery and the documented authoring path. Whitelisted it, and narrowed the pass clause ("a live HTTP fetch of the modeled pageview data") so it no longer contradicts the whitelist.

minimal_fault_triage — judge treated the 4 named sources as a closed whitelist

The agent used only read-only uip maestro bpmn … --output json reads (no mutation) and was failed for "inspecting beyond permitted sources." Reworded to gate on mutations, while preserving the explicit fail for reading mocks//fixtures//response JSON directly. Dropped traces/spans from the whitelist — no mock serves them in this sandbox.

Verification

Ran all three on this branch via run-coder-eval (agent=claude, claude-sonnet-5, run 34374076677) — all 3 SUCCESS: error_mapping 1.000, wiki_pageviews 1.000, minimal_fault_triage 0.992.

Stronger still, minimal_fault_triage is CI-proven harness-independently: this PR's own smoke run (34278022399) scored its llm_judge 0.98 / SUCCESS on Codex (gpt-5.6-luna), while the identical task on #3152's branch (same harness, without this fix) scores 0.45 / FAILURE (34376337458). 0.45 → 0.98 is direct proof of the diagnosis.

Merge order

Land this before #3152 — that PR's smoke gate is red solely on this judge criterion.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[github-actions]

Claude finished @nikhil-maryala's task in 2m 49s —— View job

---

Coder-eval task lint (advisory)

3 task YAMLs changed (all modified); 0 Critical, 0 High, 0 Medium, 0 Low, 3 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body explicitly states "needs a coder-eval re-run on this branch to confirm the three now pass" — no claim that the changed tasks have been run and passed. Please edit the PR description to add a line like:

Ran skill-bpmn-expr-error-mapping, skill-bpmn-e2e-wiki-pageviews, skill-bpmn-operate-diagnose-minimal-fault-triage and they passed.

Per-task lint

tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/error_mapping.yaml — verdict: OK

No issues. Casing fix from vars.error.code → vars.Error.code is consistent across the YAML (description, prompt, criterion description) and the Python checker (regex, error messages, success message). Ground-truth anchor (vars.Error.code) correctly matches the skill reference docs.

tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml — verdict: OK

No issues. The llm_judge prompt now whitelists read-only registry discovery (uip maestro bpmn registry list/search/get/pull) alongside the already-whitelisted validate and --help. The gate still correctly fails on cloud-side lifecycle mutations and live HTTP data fetches.

tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml — verdict: OK

No issues. Reworded llm_judge prompt properly opens the diagnostic command whitelist (any read-only uip maestro bpmn CLI read) while preserving two important failure gates: (a) lifecycle mutations, and (b) reading mocks//fixtures/ files directly instead of using the mocked CLI.

CLI verb reachability

Skipped — python3 scripts/check-cli-verbs.py was not permitted to run. No command_executed criteria reference uip verbs directly in the changed files (the command_pattern regexes are on the minimal_fault_triage task only), so this axis has low risk of surfacing issues.

Within-PR duplicates

No duplicate clusters detected. All 3 files are modifications to existing tasks, not new additions.

Conclusion

✅ All 3 changed tasks pass the rubric — changes are well-scoped casing fixes and llm_judge prompt refinements that address documented false negatives. One process issue: PR body does not yet claim a passing re-run (High, advisory). The grading-logic changes look correct but should be confirmed with a coder-eval run before or shortly after merge.

---

[github-actions]

Claude Code is working…

I'll analyze this and get back to you.

View job run

[rockymadden]

What

Fixes false-negative grading on 3 BPMN eval tasks from nightly 2026-09-08_04-17-10.

Task	Change
expressions/error_mapping	check regex + prompt vars.error.code -> vars.Error.code
multi_node/wiki_pageviews	judge whitelists read-only uip maestro bpmn registry discovery
operate-diagnose/minimal_fault_triage	judge gates on mutations instead of a closed 4-source whitelist

---

What I confirmed

• ✅ Casing claim holds. references/expression-authoring.md:30-35 says the engine seeds the error under the capital-Error key and that vars.error does not resolve. structural-bpmn.md:340,351 and patterns/failure-escalation-guide.md:15,44,50 agree. The old check contradicted the reference its own docstring cited. No lowercase vars.error is left under tests/tasks/uipath-maestro-bpmn/.
• ✅ Registry claim holds. references/cli-conventions.md:16-17 and SKILL.md:155,161 make registry pull/list/get the mandated discovery path. Failing an agent for following the skill was wrong.
• ✅ The 4-source whitelist really was closed. The old prompt named status, incidents, variables, and asset; the mocks also serve incident get, element-executions, and cursors (fixtures/mocks/responses/manifest.json).
• ✅ The cheat path stays closed. Reading mocks//fixtures//response JSON directly is still an explicit fail, matching initial_prompt:45.

---

Overall findings

🟢 The "needs a re-run" caveat is stale — your own CI already verified a third of this PR

This PR's smoke run 34278022399 ran skill-bpmn-operate-diagnose-minimal-fault-triage and scored llm_judge 0.98, status SUCCESS.

The same task on #3152's branch (same harness, without this fix) scores 0.45, status FAILURE (run 34376337458).

0.45 -> 0.98 on an identical task is direct proof of the diagnosis, and it reproduces on Codex / gpt-5.6-luna, not just claude-sonnet-5. That makes the false negative harness-independent, which is a stronger result than the PR claims.

Fix: put that in the PR body. It also clears the lint bot's High.

🟡 minor — the other two are still unverified

error_mapping is integration and wiki_pageviews is e2e, so neither ran in the PR gate (the smoke step reports Pass rate: 100.0% (1/1)). A targeted coder-eval run on each is cheap and would close the loop.

🟢 Merge this before #3152

#3152's smoke gate is red solely on this judge criterion. Landing this unblocks it.

---

Recommendation

Comment. The diagnosis is correct on all three and one is CI-proven. One in-scope defect in the wiki_pageviews rewrite is worth a one-line fix first, because it leaves alive the same judge ambiguity this PR exists to remove.

---

tl;dr

Good diagnosis, correct fixes. wiki_pageviews's judge prompt now contradicts itself: it whitelists registry pull while the sentence above still forbids "any live HTTP call". Fix that line. Your own smoke run already proved the minimal_fault_triage fix (0.98 here vs 0.45 on #3152), so drop the "needs a re-run" caveat and add the claim.

[nikhil-maryala]

@rockymadden addressed (861384b):

• wiki_pageviews — narrowed the pass clause to "a live HTTP fetch of the modeled pageview data" so it no longer contradicts the registry whitelist below it. That relocated ambiguity was a real defect; good catch.
• error_mapping — scoped the docstring to casing-only; opened BPMN error_mapping: does uipath:errorMapping need <uipath:output source="=Error"> to resolve? (follow-up to #3151) #3171 for the <uipath:output source="=Error"> resolution binding and the skill self-inconsistency (structural-bpmn.md:334-342 omits it). Casing fix stays here.
• minimal_fault_triage — dropped traces/spans (no mock serves them here).
• Body updated: dropped the stale "needs a re-run" caveat and added the evidence — run 34374076677 (3/3 SUCCESS), plus the cross-harness 0.45 → 0.98 proof you pointed out on Codex. Clears the lint High.

Agreed on merge order — this should land before #3152.