From 44caf0471e08cd847a5285a3cd3616832fe8ef66 Mon Sep 17 00:00:00 2001 From: Nikhil Maryala Date: Tue, 8 Sep 2026 15:57:22 -0500 Subject: [PATCH 1/2] test(uipath-maestro-bpmn): fix false-negative grading on 3 BPMN eval tasks From the claude-code nightly run 2026-09-08_04-17-10. Three gating criteria failed correct agent behavior. - error_mapping: the skill (expression-authoring.md, structural-bpmn.md, failure-escalation-guide.md) documents the runtime error key as capital `Error` and warns `vars.error` does NOT resolve. The agent correctly wrote `vars.Error.code`, but the check regex + prompt required lowercase `vars.error.code` - and the check docstring even cited the reference it contradicted. Correct the check regex/messages and the prompt/description to capital `Error`. - wiki_pageviews: the llm_judge failed read-only `uip maestro bpmn registry` discovery as a "live network fetch", though it is the documented way to author `uipath:*` payloads. Whitelist read-only registry discovery; gate on actual cloud lifecycle mutations / live data fetches. - minimal_fault_triage: the llm_judge applied "only inspected [4 sources]" as a closed whitelist and failed read-only inspection the prompt encourages (diagnostic priority ladder); no mutation occurred. Reword to gate on mutations only and allow any read-only diagnostic read. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../error_mapping/check_error_mapping.py | 12 +++++++----- .../error_mapping/error_mapping.yaml | 10 +++++----- .../wiki_pageviews/wiki_pageviews.yaml | 13 ++++++++----- .../operate-diagnose/minimal_fault_triage.yaml | 17 ++++++++++++----- 4 files changed, 32 insertions(+), 20 deletions(-) diff --git a/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py b/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py index 5e280819ac..b1c5b86b80 100644 --- a/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py +++ b/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py @@ -2,8 +2,10 @@ """Structural check for the error-mapping expression eval. Grades that the authored BPMN carries a uipath:errorMapping block whose -condition branches on the runtime error object via `vars.error.code` (matched +condition branches on the runtime error object via `vars.Error.code` (matched as an expression, not a baked literal), per references/expression-authoring.md. +The engine seeds the error under the capital-`Error` key; `vars.error` does not +resolve, so the casing is graded exactly. Reuses the shared uipath-maestro-bpmn check helpers (stdlib ElementTree). """ @@ -27,7 +29,7 @@ require_sequence_integrity, ) -VARS_ERROR_RE = re.compile(r"vars\.error\.code") +VARS_ERROR_RE = re.compile(r"vars\.Error\.code") def main() -> None: @@ -48,16 +50,16 @@ def main() -> None: matching = [c for c in conditions if VARS_ERROR_RE.search(c)] if not matching: - fail(f"no errorMapping condition reads vars.error.code; found: {conditions}") + fail(f"no errorMapping condition reads vars.Error.code; found: {conditions}") # Must be a runtime expression (leading '='), not a baked literal. if not any(c.strip().startswith("=") for c in matching): - fail(f"vars.error.code condition must be an expression (leading '='): {matching}") + fail(f"vars.Error.code condition must be an expression (leading '='): {matching}") require_sequence_integrity(root) require_di_for_visible_elements(root) require_no_private_connector_values(root) - print(f"OK: {path} branches on vars.error.code via uipath:errorMapping") + print(f"OK: {path} branches on vars.Error.code via uipath:errorMapping") if __name__ == "__main__": diff --git a/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/error_mapping.yaml b/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/error_mapping.yaml index 40aa7afe02..7300c81322 100644 --- a/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/error_mapping.yaml +++ b/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/error_mapping.yaml @@ -2,7 +2,7 @@ task_id: skill-bpmn-expr-error-mapping description: > Expression-authoring eval: agent uses the uipath-maestro-bpmn skill to author a uipath:errorMapping block whose condition branches on the runtime error - object via `=vars.error.code == "..."` after a failed activity. Grades the + object via `=vars.Error.code == "..."` after a failed activity. Grades the error-mapping expression shape from references/expression-authoring.md. Authoring only — no cloud effects. tags: [uipath-maestro-bpmn, integration, "mode:build", "lifecycle:generate"] @@ -28,15 +28,15 @@ initial_prompt: | activity fails with a service-unavailable error, the process branches on the runtime error object. Model the error code on a `bpmn:error errorCode` and reference it from a `uipath:errorMapping` whose condition reads the runtime - error via `vars.error.code` (for example - `=vars.error.code == "SERVICE_UNAVAILABLE"`). + error via `vars.Error.code` (for example + `=vars.Error.code == "SERVICE_UNAVAILABLE"`). Requirements: - Discover any `uipath:*` activity payload via the registry (`uip maestro bpmn registry`) and author it from the served template. Do not hand-author registry-owned `uipath:*` XML from prose. - The errorMapping condition must be a read-only expression that reads - `vars.error.code` — not a baked literal and no assignment operators. + `vars.Error.code` — not a baked literal and no assignment operators. - Use synthetic placeholder resource names only. Do not include tenant URLs, folder keys, connection IDs, release keys, real process names, or real user names. @@ -63,7 +63,7 @@ success_criteria: pass_threshold: 1.0 - type: run_command - description: "errorMapping condition branches on vars.error.code and the graph is sound" + description: "errorMapping condition branches on vars.Error.code and the graph is sound" command: "python3 $TASK_DIR/check_error_mapping.py" timeout: 30 expected_exit_code: 0 diff --git a/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml b/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml index bf1a84b630..dbfbade9ad 100644 --- a/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml +++ b/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml @@ -92,8 +92,11 @@ success_criteria: prompt: > Pass if the agent created only local project files and did not run or request approval for BPMN debug, process run, upload, publish, deploy, or - any live HTTP call. Exactly these commands are always acceptable and - must NOT fail this check: `uip maestro bpmn validate` (offline - validation of local files) and `--help` invocations. Fail only if tool - calls or the final answer show a cloud-side lifecycle mutation, live - network fetch, or a request to perform one. + any live HTTP call. These commands are always acceptable and must NOT + fail this check: `uip maestro bpmn validate` (offline validation of local + files), read-only registry discovery (`uip maestro bpmn registry` + list/search/get/pull — the documented way to author `uipath:*` payloads), + and `--help` invocations. Fail only if tool calls or the final answer + show a cloud-side lifecycle mutation (upload, publish, deploy, run, + debug), a live HTTP data fetch to the modeled endpoint, or a request to + perform one. diff --git a/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml b/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml index 377e530a80..d0829cdace 100644 --- a/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml +++ b/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml @@ -95,8 +95,15 @@ success_criteria: include_agent_output: true include_tool_calls: true prompt: > - Pass if the agent only inspected mocked status, incidents, variables, and - deployed asset data, then recommended a safe next action without running - retry, cancel, pause, resume, migrate, goto, debug, process run, upload, - publish, or deploy commands. Fail if tool calls or the final answer show - any lifecycle mutation or any request to perform one. + Pass if the agent gathered evidence only through read-only `uip maestro + bpmn ... --output json` diagnostic commands (status, incidents, incident + get, variables, deployed asset, element-executions, cursors, traces, spans + — any read-only CLI diagnostic read is fine; thorough reading is expected + per the diagnostic priority ladder), then recommended a safe next action + without running retry, cancel, pause, resume, migrate, goto, debug, + process run, upload, publish, or deploy commands. Fail if tool calls or + the final answer show (a) a lifecycle mutation or a request to perform + one, or (b) reading the mock/fixture inputs directly — `mocks/`, + `fixtures/`, or response JSON files — instead of the mocked `uip` CLI. Do + NOT fail merely for running additional read-only `uip maestro bpmn` + diagnostic commands beyond the minimum. From 861384bc829d29265df8685481cb4ca08a0ee2f9 Mon Sep 17 00:00:00 2001 From: Nikhil Maryala Date: Wed, 9 Sep 2026 13:14:09 -0500 Subject: [PATCH 2/2] test(uipath-maestro-bpmn): tighten criteria per review on #3151 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - wiki_pageviews: pass sentence said "any live HTTP call" while whitelisting registry pull (which is one) — narrow to "a live HTTP fetch of the modeled pageview data" so the pass and fail clauses agree. - minimal_fault_triage: drop traces/spans from the whitelist — no mock serves them in this sandbox (unmocked returns exit 1); trailing "any read-only CLI diagnostic read" already covers future additions. - error_mapping check: scope the docstring to what it grades (casing only); the resolution binding is not graded and the skill example omits it — tracked in #3171. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../error_mapping/check_error_mapping.py | 12 +++++++++--- .../wiki_pageviews/wiki_pageviews.yaml | 16 ++++++++-------- .../operate-diagnose/minimal_fault_triage.yaml | 4 ++-- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py b/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py index b1c5b86b80..76bd220f4d 100644 --- a/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py +++ b/tests/tasks/uipath-maestro-bpmn/expressions/error_mapping/check_error_mapping.py @@ -3,9 +3,15 @@ Grades that the authored BPMN carries a uipath:errorMapping block whose condition branches on the runtime error object via `vars.Error.code` (matched -as an expression, not a baked literal), per references/expression-authoring.md. -The engine seeds the error under the capital-`Error` key; `vars.error` does not -resolve, so the casing is graded exactly. +as an expression, not a baked literal), per references/expression-authoring.md +— the engine seeds the error under the capital-`Error` key, so the casing is +graded exactly. + +Scope: this check grades the condition string and its casing only. It does NOT +verify the `` binding that the same reference +says `vars.Error` needs to resolve at runtime — the skill's own canonical +example currently omits that binding, so grading it here would be premature. +Tracked in issue #3171. Reuses the shared uipath-maestro-bpmn check helpers (stdlib ElementTree). """ diff --git a/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml b/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml index dbfbade9ad..9e9aec53dd 100644 --- a/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml +++ b/tests/tasks/uipath-maestro-bpmn/multi_node/wiki_pageviews/wiki_pageviews.yaml @@ -92,11 +92,11 @@ success_criteria: prompt: > Pass if the agent created only local project files and did not run or request approval for BPMN debug, process run, upload, publish, deploy, or - any live HTTP call. These commands are always acceptable and must NOT - fail this check: `uip maestro bpmn validate` (offline validation of local - files), read-only registry discovery (`uip maestro bpmn registry` - list/search/get/pull — the documented way to author `uipath:*` payloads), - and `--help` invocations. Fail only if tool calls or the final answer - show a cloud-side lifecycle mutation (upload, publish, deploy, run, - debug), a live HTTP data fetch to the modeled endpoint, or a request to - perform one. + a live HTTP fetch of the modeled pageview data. These commands are always + acceptable and must NOT fail this check: `uip maestro bpmn validate` + (offline validation of local files), read-only registry discovery + (`uip maestro bpmn registry` list/search/get/pull — the documented way to + author `uipath:*` payloads), and `--help` invocations. Fail only if tool + calls or the final answer show a cloud-side lifecycle mutation (upload, + publish, deploy, run, debug), a live HTTP data fetch to the modeled + endpoint, or a request to perform one. diff --git a/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml b/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml index d0829cdace..5ead3817e2 100644 --- a/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml +++ b/tests/tasks/uipath-maestro-bpmn/operate-diagnose/minimal_fault_triage.yaml @@ -97,8 +97,8 @@ success_criteria: prompt: > Pass if the agent gathered evidence only through read-only `uip maestro bpmn ... --output json` diagnostic commands (status, incidents, incident - get, variables, deployed asset, element-executions, cursors, traces, spans - — any read-only CLI diagnostic read is fine; thorough reading is expected + get, variables, deployed asset, element-executions, cursors — any + read-only CLI diagnostic read is fine; thorough reading is expected per the diagnostic priority ladder), then recommended a safe next action without running retry, cancel, pause, resume, migrate, goto, debug, process run, upload, publish, or deploy commands. Fail if tool calls or