Skip to content

fix(agent): classify LLM provider 400s as USER with an actionable detail - #1072

Open
ionut-mihalache-uipath wants to merge 1 commit into
mainfrom
fix/pc-5002-llm-400-user-category
Open

fix(agent): classify LLM provider 400s as USER with an actionable detail#1072
ionut-mihalache-uipath wants to merge 1 commit into
mainfrom
fix/pc-5002-llm-400-user-category

Conversation

@ionut-mihalache-uipath

Copy link
Copy Markdown
Contributor

Problem

An LLM gateway 400 fell into the unclassified branch of _classify, so it surfaced as HTTP_ERROR / UNKNOWN with detail set to the HTTP reason phrase — the two words "Bad Request".

That is the single largest contributor to the fleet's two-word failure messages (PC-5002), and it buries causes the customer can actually act on. The motivating case: a job that failed 192/192 runs on gpt-4.1-mini-e2e-custom because max_tokens=65535 exceeded the model's 32768 completion-token limit — a value Agent Builder itself had written into the model settings. The customer saw "Bad Request", categorized Unknown.

Change

400 now maps to LLM_PROVIDER_BAD_REQUEST / UiPathErrorCategory.USER, with a title of "LLM provider rejected the request".

For detail, precedence is:

  1. The gateway's ProblemDetails detail — first-party UiPath text, and more specific, so it still wins.
  2. Otherwise a canned message pointing at the agent's model settings (output-token limit, temperature, effort) and noting that the provider's own message is on the LLM call span.

The provider body is deliberately not read out into the error. It may carry customer PII, and it is already recorded on the LLM call span, which is tenant-scoped. USER is not in _SHOULD_WRAP_CATEGORIES, so the canned detail has to stand on its own — tests pin that it isn't prefixed with the generic wrapper.

_classify now returns a fourth element, fallback_detail. None means "fall back to the HTTP reason phrase", so only statuses whose cause we cannot name are left with it.

What is deliberately not in scope

404 stays UNKNOWN. Every LLM-gateway 404 observed in prod over 30 days was a missing or unreachable model deployment — BYO relay not connected, Azure DeploymentNotFound, a retired Bedrock model — which is Deployment, not User. It needs its own decision on its own evidence rather than being folded into this change. test_unclassified_4xx_remains_unknown documents that and covers 404/408/413/422/429.

Tests

  • 400 → USER + canned detail across four body shapes: vendor envelope, Bedrock envelope, raw edge HTML, empty body.
  • The provider body is never quoted back — asserted over detail, str(error) and repr(error).
  • A gateway ProblemDetails detail takes precedence over the canned text.
  • The canned detail is not wrapped in the generic "An unexpected error occurred" prefix.
  • End-to-end through the LLM node for both the new client (UiPathAPIError) and the legacy raw openai.BadRequestError path.
  • Unclassified 4xx statuses stay UNKNOWN.

uv run pytest tests/agent/test_llm.py tests/agent/react/test_llm_node.py — 54 passed. ruff check, ruff format, the httpx-client AST linter, and mypy all clean.

Drive-by

Fixed a stale file reference in the develop-agent-module skill: exceptions/licensing.pyexceptions/llm.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UZ5LHC1Xhp1zBRfYnYENwv

An LLM gateway 400 was falling into the unclassified branch of `_classify`,
so it reached telemetry as HTTP_ERROR / UNKNOWN with `detail` set to the
HTTP reason phrase — the two words "Bad Request". That is the largest slice
of the fleet's two-word failure messages (PC-5002), and it hides causes the
customer can actually fix, such as the `max_tokens=65535` that Agent Builder
itself wrote into the model settings.

400 now maps to LLM_PROVIDER_BAD_REQUEST / USER. Where the gateway supplies
a first-party ProblemDetails `detail`, that still wins; otherwise the error
carries a canned message pointing at the agent's model settings. The
provider body is deliberately not read out — it may carry customer PII and
is already recorded on the tenant-scoped LLM call span.

404 is left in UNKNOWN on purpose: every LLM-gateway 404 seen in prod over
30 days was a missing or unreachable deployment (BYO relay not connected,
Azure DeploymentNotFound, a retired Bedrock model), which is Deployment
rather than User, and deserves its own decision.

Also fixes a stale file reference in the develop-agent-module skill
(`exceptions/licensing.py` → `exceptions/llm.py`).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ5LHC1Xhp1zBRfYnYENwv
Copilot AI lite review requested due to automatic review settings September 7, 2026 20:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is well-scoped, avoids provider-body leakage by construction, and is backed by targeted unit + node-level tests that pin the new 400 classification and fallback-detail behavior.

Pull request overview

This PR improves how LLM-gateway HTTP 400 responses are mapped into AgentRuntimeError so customers see an actionable, provider-safe message instead of the unhelpful HTTP reason phrase (“Bad Request”), while keeping other 4xx statuses classified as UNKNOWN per the stated scope.

Changes:

  • Classifies HTTP 400 as LLM_PROVIDER_BAD_REQUEST with UiPathErrorCategory.USER and a stable canned detail when the gateway provides no ProblemDetails detail.
  • Extends _classify() to return a fallback_detail so only explicitly-handled statuses avoid reason-phrase fallback.
  • Adds/updates tests to pin 400 behavior (including “no provider body leakage”) and to ensure other 4xx remain UNKNOWN; updates the agent module skill doc reference.
File summaries
File Description
src/uipath_langchain/agent/exceptions/llm.py Adds 400 classification with provider-safe fallback detail and threads fallback_detail through raise_for_provider_http_error().
src/uipath_langchain/agent/exceptions/exceptions.py Introduces AgentRuntimeErrorCode.LLM_PROVIDER_BAD_REQUEST.
tests/agent/test_llm.py Expands contract tests for 400 mapping, detail precedence, and “no provider body quoted back”; pins unclassified 4xx behavior.
tests/agent/react/test_llm_node.py Adds end-to-end node-level assertions for 400 mapping for both new client and legacy OpenAI exception paths.
.claude/skills/develop-agent-module/SKILL.md Fixes the referenced exceptions module path for LLM provider errors.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@sonarqubecloud

sonarqubecloud Bot commented Sep 7, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants