diff --git a/.github/workflows/platform.yml b/.github/workflows/platform.yml index 9bc5360de..d650dc8ce 100644 --- a/.github/workflows/platform.yml +++ b/.github/workflows/platform.yml @@ -12,7 +12,13 @@ on: - 'infrastructure/cdk.context.json' - 'infrastructure/bin/infrastructure.ts' - 'infrastructure/bootstrap-assets/**' + # Staged into the mcp-sandbox bucket at synth (mcp-sandbox-bucket-construct). + # Without this a viewer change deploys NOWHERE: frontend-deploy.yml does not + # ship it, so the push goes green and the old asset stays live. + - 'infrastructure/assets/**' - 'scripts/platform/**' + # fetch-dcv-sdk.sh runs in THIS workflow, before synth. + - 'scripts/build/**' - '.github/workflows/platform.yml' permissions: @@ -128,6 +134,20 @@ jobs: # backfill. CDK_ARTIFACT_SHARE_INBOX_ENABLED: ${{ vars.CDK_ARTIFACT_SHARE_INBOX_ENABLED }} CDK_FRONTEND_CERTIFICATE_ARN: ${{ vars.CDK_FRONTEND_CERTIFICATE_ARN }} + # Hostnames Chromium refuses to navigate to during a browser session — + # the control that stops a human in a browser takeover reaching a system + # the agent must not act inside. Comma-separated, e.g. + # "instructure.com,vendor.example.com". + # + # ⚠️ There is NO built-in default: unset resolves to an empty string, + # which config.ts treats as "not configured" and leaves the list EMPTY, + # so browser sessions can reach any host and RBAC on browse_web / + # request_user_login becomes the only control. The stack ships + # fork-neutral on purpose (the hosts that matter are a per-institution + # policy call), so every environment that needs a blocklist must set + # this variable. The synth log prints the list, or warns when it is + # empty — check it after a deploy. + CDK_BROWSER_URL_BLOCKLIST: ${{ vars.CDK_BROWSER_URL_BLOCKLIST }} # MCP Apps sandbox-proxy origin (mcp-sandbox.{domain}). Without the # cert ARN the construct silently falls back to the CloudFront default # domain and creates no Route53 ALIAS — the SPA then frames a @@ -295,5 +315,14 @@ jobs: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + # The browser sign-in live view is served from the mcp-sandbox origin and + # needs the Amazon DCV Web Client SDK deployed alongside it. The SDK is a + # EULA-licensed AWS download and this repo is PUBLIC, so it is fetched and + # verified here rather than committed. Must run BEFORE the deploy: CDK + # stages assets/mcp-sandbox at synth time, so an absent SDK silently ships + # a viewer that cannot load. See docs/specs/authenticated-web-assessment.md. + - name: Fetch Amazon DCV Web Client SDK + run: bash scripts/build/fetch-dcv-sdk.sh + - name: Deploy PlatformStack run: bash scripts/platform/deploy.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index f908cdc9d..42aace095 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,94 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). +## [1.23.0] - 2026-09-20 + +The agent stops working in silence, and stops paying to re-read what it already knows. **Live turn narration** replaces the cycling placeholder with what the agent is actually doing — thinking, which tool is running, how long it took, and a one-line model-written summary of each tool batch — drained concurrently so "Using list_assignments" arrives *while* that tool runs rather than after it. **Document context offload** stops an attached PDF being re-sent on every turn: a digest replaces the bytes and a `document_read` tool pulls back the pages the model actually asks for, measured on a 60-page canary at a **109.1K → ~15K prefix drop**. A five-part **compaction overhaul** makes thresholds model-relative, bounds the summary at 8k tokens, parks cuts until the re-write is free, and offloads oversized tool results at intake. **Browser sign-in handover** lets the agent pause and hand the user a live, interactive browser to sign in to a site it cannot reach, then continue in the authenticated session — gated behind its own RBAC tool id and a MANAGED Chromium URL blocklist. **Response feedback** ships thumbs, retry-with-correction, implicit copy/continue signals and fleet-level attribution by config arm. Plus **.docx / .pptx / .csv / .xlsx previews** in a docked pane, **Agent Templates**, and a performance pass worth ~500ms off the pre-stream window. **A CDK deploy is required**, and operators who use the browser tool must set `CDK_BROWSER_URL_BLOCKLIST`. + +### 🚀 Added + +- **Browser sign-in handover (`request_user_login`)** — the agent pauses the turn and hands the user a live, interactive browser to sign in to a site it cannot reach, then continues in the authenticated session. New `browser_login_required` SSE event; the interrupt is raised by the tool itself via `ToolContext`, so the `PausedTurnSnapshot`, resume route and `PendingInterrupt` breadcrumb need no special case. While the user holds the browser the automation stream is `DISABLED` at the service, so the agent provably cannot act. Gated by `BROWSER_TAKEOVER_ENABLED` **and** its own catalog entry (`enabledByDefault: false`) (#1177, #1183) +- **Live-view viewer with no third-party code** — DCV streamed into a first-party viewer page. The event deliberately carries **no URL**: `generate_live_view_url` signs with SigV4 query auth and caps at 300s, so app-api mints one per request instead, and `assert_no_url` enforces that on the interrupt, the event and the persisted row (#1178, #1186, #1189, #1190, #1196, #1199, #1200, #1203) +- **MANAGED Chromium URL blocklist** — a per-environment hostname blocklist applied as a browser-session policy, so a human holding the browser cannot navigate to a system the agent must not act inside (#1180, #1181, #1202) +- **Document context offload** — `document_read` tool, `DocumentDigest` built at upload and persisted on `FileMetadata`, restore rehydrating stripped documents as digests, and live offload of unpinned documents once the re-write is free. Gated by `DOCUMENT_READ_ENABLED` / `DOCUMENT_OFFLOAD_ENABLED`, bucketed by `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` (#1137, #1138, #1139, #1140, #1143) +- **Per-call context ledger** — prefix split (system / tools / messages), window trims and compaction events on every `C#` cost row, surfaced through the admin cost drill-down (#1130) +- **Model-relative compaction thresholds** — floor-seeking cut with hysteresis, replacing fixed token ceilings that scaled wrong across models (#1129) +- **Bounded compaction summary + per-cut metrics** — summary capped at 8k tokens (#1131) +- **Deferred compaction apply** — cuts are parked post-turn and applied in place when the prefix re-write is free (#1132) +- **Tool-result offload at intake** — oversized tool results go to S3 with a text preview in context, `document_read` exempt (#1133) +- **Selective 1h prompt-cache TTL on the static prefix**, behind a flag and priced honestly against the 2× write premium (#1134) +- **Live turn narration (`agent_status`)** — `thinking` / `tool_start` / `tool_end` phases with Strands' own measured durations, drained concurrently with the agent stream so a status line lands while its tool is running. Deliberately no "responding" phase, and durations are live-only, never persisted (#1160, #1165) +- **Model-written tool-batch summaries (`tool_group_summary`)** — a Nova Micro side-channel task, structured like `session_title`: its own call on its own messages, so it never appends to the conversation or the cacheable prefix. Persisted as `TSUM#` rows and replayed on `GET /messages` (#1161) +- **Thinking time and turn recap** — how long the model spent thinking, and how long a finished turn took, shown in the loader's slot on the latest turn only (#1157, #1170, #1174) +- **Response feedback** — content-free thumbs on assistant messages joined to cost rows (`F#` rows), six reason buckets, and an explicit/implicit signal discriminator (#1142, #1146) +- **Retry-with-correction** — the consequence behind a thumbs down (#1148) +- **Implicit feedback signals** — copy and continue as unweighted positive signals (#1151) +- **Eval sampling** — down-thumbed turns feed AgentCore Evaluations. Opt-in at both CDK and runtime (`FEEDBACK_EVAL_SAMPLING_ENABLED`), because it sends real conversations to an AWS-managed judge (#1153) +- **Fleet-level feedback attribution** — down-thumb rate by config arm (model, tools, skills), plus a down-thumb reason split on the per-session profile (#1152, #1159) +- **File previews in a docked pane** — `.docx` and `.pptx` (uploaded or generated), `.csv` as a data grid, and `.xlsx` read server-side with openpyxl. The preview pane opens on a file the turn just created (#1119, #1121, #1122, #1123, #1136) +- **Agent Templates** — create-form prefill backed by an admin-managed template store (#1149) +- **Spreadsheet Analysis auto-enables** for a session that holds a spreadsheet, so an uploaded `.xlsx` is answerable without the user finding the toggle (#1163) +- **Turn-latency observability construct** — EMF metrics decomposing the pre-stream window into named stages (#1184, #1201) + +### ✨ Improved + +- **gzip on app-api JSON responses**, with SSE passed through un-buffered (#1125, #1127) +- **Spreadsheet previews show ten times the rows** and the scroller is no longer frozen (#1124) +- **Copy and thumbs reveal on response hover** rather than occupying the transcript permanently (#1179) +- **Agent-state orb** gains a halo and settles on finished turns (#1182) +- **Attachment turn guard** holds a turn's inline attachments to the AgentCore Memory event quota (#1145) + +### 🐛 Fixed + +- **KaTeX swallowed the prose between two currency amounts** — `$4.50 … $9.00` rendered as math. The earlier HTML-entity workaround never worked and leaked `$` into generated files; both are removed (#1128, #1135) +- **A new frontend build was not actually served** — the deploy sent no `Cache-Control`, so CloudFront served the old bundle from cache. Hashed filenames do not save you when `index.html` itself is stale (#1197) +- **The system-prompt date line carried the hour**, re-writing the prompt-cache prefix every hour for the life of every session (#1126) +- **Deleted conversations were invisible to the admin cost drill-down**, so a user's costs did not reconcile (#1120) +- **`documentTokens` understated PDFs by ~14×** — `bytes/4` ignores that Bedrock dual-encodes each PDF page as an image on top of the text layer. Every analytic built on it was low by that factor (#1144) +- **`DOCUMENT_READ_ENABLED=false` left the other two paths running**, so pulling the kill switch still evicted bytes and emitted handles for a tool that was not injected (#1147) +- **ReDoS in `document_read` pattern mode** — a valid but catastrophic regex ran unbounded. Two defences: a structural `catastrophic_pattern` check that degrades to literal search, and a scan budget (#1147) +- **Synth now fails when the uploads bucket would have no CORS rule** — a truthy-but-empty origin list synthesised green with no rule at all (#1166, #1172) +- **The tool/skill lists are awaited before a chat turn is assembled**, closing a first-turn race that re-wrote the prompt-cache prefix (#1168) +- **Browser policy fixes** — MANAGED was being sent at session level (breaking every session), the policy object key was double-prefixed, and the caller needed read on the policy object (#1173, #1175, #1176, #1183) +- **`state-*` colour steps that failed AA** in light mode for file-type chips and success text (#1118, #1119) + +### ⚠️ Changed + +- **The browser URL blocklist ships empty and is supplied per environment.** It was hardcoded to `instructure.com`, and `platform.yml` never forwarded `CDK_BROWSER_URL_BLOCKLIST` — so the list was effectively unconfigurable through the deploy pipeline and every fork inherited one institution's policy. **Set the variable for any environment that needs a blocklist**; the synth log prints the list or warns when it is empty (#1205) +- **`list_spreadsheets` and `analyze_spreadsheet` are now seeded.** They were catalogued but never written to the tool-catalog table, so a fresh deployment could not grant them to any role (#1205) + +### ⚡ Performance + +- **~500ms off the pre-stream window.** The session row was read **eight times** per turn — 445ms of a 455ms stage (#1191). boto3 clients rebuilt per call cost 21ms, ~20% of the remaining warm preamble (#1198). Quota now takes session cost from the row the preamble already read (#1193) +- **Bytecode precompiled in the app-api, inference-api and Lambda images** — `uv` does not compile by default, unlike `pip`, so first-touch import was paid at runtime (#1155, #1169) +- **CountTokens bounded on the reply path**, and the model id is no longer swapped mid-call; a throttle now costs one failed request instead of a 5.8s stall (#1158) +- **Long-term memory retrieved through a bounded client**, and the session's own summary is no longer re-fetched per message (#1164) +- **Agent cache extended to four more tool families**, and spreadsheet-analysis sessions are cached by carrying `assistant_id` in the key (#1156, #1162) +- **mermaid lazy-loaded** out of the eager scripts bundle (#1119) +- **app-api and inference-api ship code root-owned**, and app-api no longer ships `/app` twice (#1167, #1171) + +### 🔒 Security + +- **No test reaches AWS.** 25 test cases across 6 files were making real authenticated AWS calls, hidden by fail-open error handling. An off-box socket guard now blocks them, and the quarantine is burned down — the suite also runs in half the time (#1154, #1157) + +### 🏗️ Infrastructure + +- **New `browser-policy-construct`** — S3-backed MANAGED Chromium policy object for browser sessions, with `ConnectBrowserLiveViewStream` granted on `*` as AWS requires (#1180, #1194) +- **New `turn-latency-observability-construct`** — EMF metrics for the decomposed pre-stream stages (#1184) +- **New `agent-templates` table** — no GSIs (#1149) +- **`connect-src` allows `data:`/`blob:` in the MCP sandbox CSP** so DCV can load its decoder (#1192) +- **The Platform Stack deploy triggers on the assets it actually deploys** (#1187) + +### 📦 Dependencies + +- Backend: `openpyxl` 3.1.5 (new — server-side `.xlsx` reading) +- Frontend: `docx-preview` 0.4.0, `pptx-preview` 1.0.7 (new), `echarts` stubbed via a local shim + +### 🔧 CI/CD + +- **Off-box socket guard** in `tests/conftest.py`, with tiktoken warmed before the guard arms (#1154, #1157) +- **Seed/catalog parity test** — every tool in `TOOL_CATALOG` must have a row in `seed_bootstrap_data.py` (#1205) + ## [1.22.0] - 2026-09-14 The agent can stop guessing. **Clarifying questions** ship end to end: when a request is genuinely ambiguous the agent pauses the turn, the SPA renders a multiple-choice picker in the transcript, and the answer resumes that same tool call — surviving a page refresh. The tool worked from PR-2 but the model reached for it 4 times in 24 ambiguous requests; a measured system-prompt clause takes that to 24/24 while leaving clear requests at 0/18. On the admin side, the **cost drill-down** closes the gap between "top users by cost" and the per-session anatomy: an admin walks user → conversations → session profile with 15 diagnosis rules, a context trajectory chart and a copyable diagnostic JSON — all **content-free by construction**, enforced by a denylist test and a moto test that seeds content and proves none returns. Two silent data bugs are fixed: deleting a knowledge-base document mid-upload **permanently leaked its byte reservation**, and born-managed provisioning **mistook an established legacy agent for a new one** and stranded its corpus. And an `@`-mention now **binds the conversation** instead of borrowing one turn — measured on prod, 247 of 247 mentions started the conversation, so the borrow was paying an invisible tool-loss failure for a case that has never occurred. **No CDK deploy required.** One operator step: enable the Clarifying Questions tool in each existing environment's catalog — the seed skips a tool row that already exists. diff --git a/CLAUDE.MD b/CLAUDE.MD index 8ba379a7d..409f4ac71 100644 --- a/CLAUDE.MD +++ b/CLAUDE.MD @@ -64,6 +64,7 @@ npx cdk deploy {prefix}-PlatformStack | `stream_error` | Conversational error | | `oauth_required` | External MCP tool needs user consent — payload `{providerId, authorizationUrl, interruptId?}`, one event per provider emitted after `message_stop`. Two flavors. **Interrupt-driven** (`interruptId` present): `OAuthConsentHook` paused a tool call mid-turn; the SPA resumes that exact turn by POSTing the id back. **Pre-flight** (`interruptId` *absent*): the tool never registered because the MCP server refused the pre-flight `tools/list` — since the consent hook is `BeforeToolCall`, it can't fire for an unregistered tool, so without this event the tool vanishes with no explanation. Nothing is paused, so the SPA shows the Connect affordance and must NOT resume; a synthetic id would be worse than none, because the resume guard in `inference_api/chat/routes.py` 400s on unknown ids and the user would hit an error right after consenting. Pre-flight events are re-emitted each turn that rebuilds the agent (the pre-flight keeps failing until consent lands) and are deliberately not persisted as `pending_interrupt` breadcrumbs — those are keyed by interrupt id for the resume path. The SPA dedupes by `providerId` and suppresses a dismissed pre-flight prompt for the tab session. Before emitting, the runtime asks the AgentCore vault directly: a vaulted token is warmed into `oauth_token_cache` and the pre-flight retried, so a user who already consented gets the tool back instead of losing it for the life of the process | | `user_question_required` | The agent paused to ask the user structured clarifying questions — payload `{type, interruptId, toolUseId, questions}`, where each question is `{header, question, multiSelect, options: [{label, description?}]}`. Emitted after `message_stop` in the same `done` block as `oauth_required` / `tool_approval_required`, one event per pending prompt. Unlike those two the interrupt is raised by the **tool itself** (`ask_user_question`, via `ToolContext`) rather than by a `BeforeToolCall` hook; Strands routes both through `_stop_for_interrupts`, so the `PausedTurnSnapshot`, the resume route and the `PendingInterrupt` breadcrumb (`kind: "user_question"`, questions JSON-encoded) need no special case. The SPA renders a picker and resumes the same turn by POSTing an `interrupt_responses` entry whose `response` is `{answers: {"
": {selected, text}}}` — always an object, **never null**: `ToolContext.interrupt` only treats a non-None response as an answer, so a null would re-raise the interrupt forever ("Skip" sends `{skipped: true}`). The picker always offers Other + Skip, so the model must not include them as options. Cost: the tool spec is a constant in the cacheable `toolConfig` prefix and the questions travel on the SSE channel only — just the one-line formatted answer block re-enters the conversation. Gated by `ASK_USER_QUESTION_ENABLED` (default on with a kill switch); while off the tool is never registered, so the model falls back to asking in prose | +| `browser_login_required` | The agent paused so the **user** can sign in to a site it cannot reach — payload `{type, interruptId, toolUseId, sessionId, browserSessionId, browserId, viewport, deadlineAt?, targetUrl?, reason?}`. Emitted after `message_stop` in the same `done` block as `oauth_required` / `user_question_required`, one per pending handover. Raised by the tool itself (`request_user_login`, via `ToolContext`) like `ask_user_question`, so the `PausedTurnSnapshot`, the resume route and the `PendingInterrupt` breadcrumb (`kind: "browser_login"`) need no special case. `sessionId` is the **conversation** id as everywhere else; the browser is `browserSessionId` — app-api's live-view route is owner-scoped by conversation and resolves the browser server-side. **There is deliberately no URL on this event.** `generate_live_view_url` signs with SigV4 *query* auth and caps at 300s, so a minted URL is dead before a human reacts and dead again on reload; app-api mints one per request instead (`assert_no_url` enforces this on the interrupt, the event and the persisted row, rather than trusting it). `viewport` rides the event because DCV's `remoteWidth`/`remoteHeight` must match the session viewport or the stream crops — a second copy of `1280x800` in the SPA is a copy that will drift. Resume posts `{completed: true}` / `{skipped: true}` — always an object, **never null**, or the interrupt re-raises forever. While the user holds the browser the automation stream is `DISABLED` at the service, so the agent provably cannot act; the idle reaper exempts the session until `deadlineAt` and no further, which is what stops a walked-away-from takeover billing to its TTL. Cost: the tool spec is a constant in `toolConfig` for granted users (~200 tokens) and the whole round trip adds one short line to the conversation — the live view is a stream the model never sees. Gated by `BROWSER_TAKEOVER_ENABLED` (default on with a kill switch) **and** by RBAC: `request_user_login` is its own catalog entry (`enabledByDefault: false`), because RBAC granularity is one `tool_id` and an action on `browse_web` would have shipped an interactive browser in our AWS account to every user who can browse. See `docs/specs/authenticated-web-assessment.md` | | `compaction` | Backend rolled older turns into a summary on this turn — payload `{previousCheckpoint, newCheckpoint, summarizedTurns, inputTokens}`, emitted after the final `metadata` event so the badge updates first, before `done` | | `done` | Stream complete | diff --git a/README.md b/README.md index 6b49d63d3..6489ceb96 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **An open-source, production-ready Generative AI platform for institutions** *Built by Boise State University, designed for everyone.* -[![Release](https://img.shields.io/badge/Release-v1.22.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) +[![Release](https://img.shields.io/badge/Release-v1.23.0-6366f1?style=flat&logo=github&logoColor=white)](RELEASE_NOTES.md) [![Nightly](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml/badge.svg)](https://github.com/Boise-State-Development/agentcore-public-stack/actions/workflows/nightly.yml) ![Python](https://img.shields.io/badge/Python-3.13+-3776AB?style=flat&logo=python&logoColor=white) @@ -296,7 +296,7 @@ agentcore-public-stack/ See [RELEASE_NOTES.md](RELEASE_NOTES.md) for the full changelog, including new features, bug fixes, platform upgrades, and deployment notes for each release. -**Current release:** v1.22.0 +**Current release:** v1.23.0 --- diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 33fd9f27f..27449f4ca 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -1,3 +1,244 @@ +# Release Notes — v1.23.0 + +**Release Date:** September 20, 2026 +**Previous Release:** v1.22.0 (September 14, 2026) + +--- + +> 🏗️ **A CDK deploy is required.** This release adds two constructs (browser policy, turn-latency observability), a new `agent-templates` table, a CSP change for the MCP sandbox, and a synth-time guard on the uploads bucket. Deploy order is unchanged: `platform.yml` → `backend.yml` → `frontend-deploy.yml`. **No GSI operation** — the one new table is created with no indexes, so no existing table is updated. **No data backfill.** +> +> 🛠️ **Operator step — set `CDK_BROWSER_URL_BLOCKLIST` per environment.** The browser URL blocklist previously hardcoded `instructure.com` and the deploy pipeline never forwarded the variable. It now ships **empty** and is supplied per environment. Any environment that uses the browser tool and needs a blocklist must set this variable **before** deploying, or browser sessions can reach any host and the RBAC grant on `browse_web` / `request_user_login` becomes the only control. The synth log prints the list, or warns when it is empty — read it after the deploy. See Deployment notes. +> +> ⚠️ **Managed knowledge bases: this deploy applies `CDK_MANAGED_KB_MIGRATION_ENABLED` wherever it is already set to `true`.** The variable is read at deploy time, so an environment where it was set since the last platform deploy gets the Upgrade card on **this** release. Nothing migrates on its own — enrolment is a user-initiated `POST` — but the card becomes visible and users can opt their knowledge bases in one at a time. Confirm the value you want before deploying. +> +> 🔑 **`request_user_login` ships off.** Browser sign-in handover is the most restrictive default in the tool catalog: it is its own catalog entry with `enabledByDefault: false`, because while a takeover is live the user has a fully interactive Chromium running inside your AWS account with your egress. Grant it to named staff or evaluator roles, never by default. + +--- + +## Highlights + +The agent stops working in silence. **Live turn narration** replaces the cycling "Thinking…" placeholder with what is actually happening — which tool is running, how long it took, and a one-line model-written summary of each finished tool batch ("Found the Syllabus Acknowledgment assignment in BIO 101"). The detail that makes it feel live rather than retrospective is the concurrent drain: status transitions are merged into the SSE stream against a short timer instead of between agent yields, so "Using list_assignments" reaches the client **while that tool is running** rather than after its result. A three-tool browse turn used to narrate nothing for 4.5 seconds. None of it costs anything against the model — nothing the status layer produces reaches the prompt. + +And it stops paying to re-read what it already knows. **Document context offload** was the largest single cost item in the fleet: attachments were 31% of production spend, because an attached PDF is re-sent in full on every subsequent turn at the cache-write premium. Now the bytes are replaced by a structured digest and a `document_read` tool pulls back exactly the pages the model asks for. On a 60-page canary the prefix went from **109.1K tokens to roughly 15K** — an 86% drop — and the feature turned out to work *far* better than its own spec predicted, because the instrument was hiding it: `documentTokens` estimated `bytes/4` and Bedrock dual-encodes each PDF page as an image on top of the text layer, understating documents by about **14×**. That is fixed, and every analytic built on it moved with it. + +Alongside it, a five-part **compaction overhaul**: thresholds are now model-relative rather than fixed token ceilings that scaled wrong across models, the summary is bounded at 8k tokens, cuts are parked post-turn and applied in place when the prefix re-write is free, oversized tool results are offloaded to S3 at intake with a preview left in context, and the static prefix can take a selective 1-hour cache TTL — priced honestly against the 2× write premium that buys. + +**Browser sign-in handover** is the release's one genuinely new user-facing capability. When the agent hits a site it cannot reach, it pauses the turn and hands the user a live, interactive browser to sign in, then continues in the authenticated session. While the user holds it, the automation stream is `DISABLED` at the service — the agent *provably* cannot act, rather than being trusted not to. It ships off by default behind its own RBAC tool id, with a MANAGED Chromium URL blocklist as the second control. + +**Response feedback** closes the loop the cost drill-down opened: content-free thumbs joined to cost rows, six reason buckets, retry-with-correction as the consequence behind a thumbs down, implicit copy/continue signals, and fleet-level attribution that reads down-thumb rate by config arm — so "which model/tool/skill combination is actually worse" becomes a number rather than an argument. + +Rounding out the release: **`.docx`, `.pptx`, `.csv` and `.xlsx` previews** in a docked pane, **Agent Templates**, a performance pass worth about **500ms off the pre-stream window** (the session row was being read eight times per turn), and a test-integrity fix worth calling out — **25 test cases across 6 files were making real authenticated AWS calls**, hidden by fail-open error handling. An off-box socket guard now blocks them, and the suite runs in half the time. + +--- + +## Browser sign-in handover + +The agent can now pause a turn and hand the user a live, interactive browser to sign in to a site it cannot reach — then continue browsing the authenticated session. It is the answer to the class of task that used to dead-end: anything behind an institutional login, an MFA prompt, or a consent screen no automation should be clicking on a user's behalf. + +### Backend +- `agents/builtin_tools/browser/request_user_login` — raises the interrupt from the **tool itself** via `ToolContext`, the same shape as `ask_user_question`. Strands routes it through `_stop_for_interrupts`, so the `PausedTurnSnapshot`, the resume route and the `PendingInterrupt` breadcrumb (`kind: "browser_login"`) need no special case. +- New `browser_login_required` SSE event, emitted after `message_stop` alongside `oauth_required` and `user_question_required`. Resume posts `{completed: true}` / `{skipped: true}` — always an object, **never null**, or the interrupt re-raises forever. +- **The event deliberately carries no URL.** `generate_live_view_url` signs with SigV4 *query* auth and caps at 300 seconds, so a minted URL is dead before a human reacts and dead again on reload. app-api mints one per request instead, owner-scoped by conversation, and `assert_no_url` enforces the absence on the interrupt, the event and the persisted row rather than trusting it. +- While the user holds the browser the automation stream is **`DISABLED` at the service** — the agent cannot act, as a property of the system rather than a promise. The idle reaper exempts the session until `deadlineAt` and no further, which is what stops a walked-away-from takeover billing to its TTL. + +### Frontend +- A first-party sign-in viewer with **no third-party code**: DCV is streamed into a viewer page the SPA frames, sized from the `viewport` the event carries (DCV's `remoteWidth`/`remoteHeight` must match the session viewport or the stream crops, and a second copy of `1280x800` in the SPA is a copy that will drift). Full-screen, and it says plainly that it can be driven. + +### Infrastructure +- `browser-policy-construct.ts` — an S3-backed **MANAGED** Chromium policy object applied per browser session. `URLBlocklist` is a flat list of Chromium URL-filter patterns; a bare host blocks that host on every scheme, port and path. +- `ConnectBrowserLiveViewStream` is granted on `*`, as AWS requires. +- `connect-src` in the MCP sandbox CSP now allows `data:` and `blob:` so DCV can load its decoder. + +### Gating +Two independent controls, and the tool needs both. `BROWSER_TAKEOVER_ENABLED` is the kill switch; `request_user_login` is its **own catalog entry** with `enabledByDefault: false` — deliberately separate from `browse_web`, because RBAC granularity is one `tool_id` and an *action* on `browse_web` would have shipped an interactive browser in your AWS account to every user who can browse. + +--- + +## Document context offload + +An attached PDF used to be re-sent in full on every subsequent turn, at the cache-write premium. Attachments were **31% of production spend**. Now the bytes are replaced by a structured digest, and a `document_read` tool pulls back exactly the pages the model asks for. + +### Backend +- `document_read` — page ranges and pattern search over an attached document, hard-capped at 20 pages per call. +- `DocumentDigest` built at upload and persisted on `FileMetadata`: an outline plus a bounded abstract, rendered into context in place of the document. +- The **restore** path rehydrates stripped documents as digests, and the **live** path offloads unpinned documents once the prefix re-write is free. Both route through one gate, so a single flag read governs them. +- `estimate_document_tokens` — PDFs count `max(pages × PDF_PAGE_TOKEN_ESTIMATE, bytes/4)`, wired into every consumer of that quantity so a row and a decision can never disagree. + +### Measured +On a 60-page canary in dev: prefix **109.1K → ~15K**. Every mechanism fired correctly — digest-only from turn 2, a `document_read` on turn 3 pulling one page in 170ms, the slice ageing out exactly at `DOCUMENT_SLICE_MAX_TURNS`. + +### The instrument was hiding the win +`documentTokens` estimated `bytes/4`, which ignores that Bedrock **dual-encodes each PDF page as an image** on top of the text layer — understating documents by about **14×**. It reported documents as ~6% of the prefix when they were ~86%. Fixed, with every downstream analytic moved with it. + +### Hardened during validation +An independent sweep of the merged epic produced six findings, all closed before this release: the PDF token estimate above; `DOCUMENT_READ_ENABLED=false` not being coupled to the offload and rehydrate paths (pulling the one kill switch an operator would reach for left the live path still evicting bytes — strictly worse than pre-epic behaviour); **ReDoS in pattern mode**, where a valid but catastrophic regex ran unbounded; an inline byte budget sitting exactly on the quota it existed to stay under; and a soft digest token cap that XML-escaping could overshoot. + +### Rollout +`DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` defaults to **100**. The `crc32(session_id) % 100` bucket exists so an evaluation can run concurrent arms; at 100 there is no control arm, only before/after across the release boundary. Set it lower in an environment where you want both. + +--- + +## Compaction overhaul + +Five parts, shipped together, all with kill switches. + +- **Model-relative thresholds** (`COMPACTION_MODEL_RELATIVE_ENABLED`) — the cut point scales with the model's window instead of a fixed token ceiling that was right for one model and wrong for the rest. Floor-seeking, with hysteresis so a session does not oscillate across the threshold. +- **Bounded summary** (`COMPACTION_SUMMARY_MODEL_ENABLED`) — capped at 8k tokens, with per-cut metrics. +- **Deferred apply** (`COMPACTION_DEFERRED_APPLY_ENABLED`) — cuts are parked post-turn and applied in place when the prefix re-write is free, so compaction stops paying for a re-write it could have had for nothing. +- **Tool-result offload at intake** (`TOOL_RESULT_OFFLOAD_ENABLED`) — oversized tool results go to S3 with a text preview left in context. `document_read` is exempt: offloading the page slice the model just asked for would undo the read and cost a second round trip. +- **Selective 1h prompt-cache TTL** on the static prefix, behind a flag and priced against the 2× write premium a 1-hour TTL costs. + +The compaction ledger records forced cuts, floor-unreachable decisions, in-place applies and head-of-turn promotions, so a cut's cost is attributable after the fact rather than inferred. + +--- + +## Live turn narration + +### `agent_status` +`thinking` / `tool_start` / `tool_end` phases from `AgentStatusHook`, carrying Strands' own measured `durationMs` and `ok=false` for both a raised exception and a result with `status: "error"`. There is deliberately **no "responding" phase** — the SPA already knows text is streaming from the deltas, and a backend-derived duplicate of a fact the client holds first-hand would only disagree at the edges. Durations are live-only and **not persisted**: a reloaded conversation shows summaries without timings, where a client-invented number would be one the user could not trust. + +The concurrent drain is what makes it feel live. Status transitions are merged against a short timer rather than between agent yields, so a status line reaches the client while its tool is still running — a three-tool browse turn previously narrated nothing for 4.5 seconds. + +### `tool_group_summary` +A Nova Micro side-channel task structured exactly like `session_title`: its own Bedrock call on its own messages, concurrent with the agent stream, so it **never appends to the conversation** and adds nothing to the cacheable prefix. Persisted as `TSUM#` rows reusing an existing GSI — zero new infrastructure — and replayed on `GET /messages`, because the event never re-streams. Deliberately **not** written onto the message content blocks: that is the Converse payload, and a display string there would be paid at model rates on every subsequent turn. + +With `TOOL_SUMMARIES_ENABLED=false` the SPA's deterministic client-side formatter still renders ("Listed 4 assignments"), so absence is a downgrade in specificity, never a blank. + +### Turn timing +Thinking time and a turn recap, shown in the loader's slot on the latest turn only. The recap is measured from the turn, not the stream. + +--- + +## Response feedback + +### Backend +- Content-free thumbs on assistant messages, persisted as `F#` rows and joined to the turn's cost rows. Turn-class precedence is full > retrieved > digest. +- Six reason buckets on a down-thumb, plus an explicit/implicit signal discriminator. +- **Implicit signals** — copy and continue, recorded as unweighted positive signal. +- **Eval sampling** — down-thumbed turns feed AgentCore Evaluations. **Opt-in at both CDK and runtime**, and deliberately so: it sends real conversations to an AWS-managed judge, which is a scoping decision each environment makes explicitly. An unset GitHub Actions variable cannot enable it. + +### Frontend +- Copy and thumbs reveal on response hover rather than occupying the transcript permanently. +- **Fleet-level attribution** — down-thumb rate by config arm (model, tools, skills), and a down-thumb reason split on the per-session cost profile. + +--- + +## File previews + +`.docx`, `.pptx`, `.csv` and `.xlsx` render in a docked pane — uploaded or generated, and the pane opens automatically on a file the turn just created. + +- **`.docx`** via `docx-preview` 0.4.0. Office Online was evaluated and rejected; the renderer needs a detached container to mount correctly. +- **`.pptx`** via `pptx-preview` 1.0.7, with `echarts` stubbed through a local shim to keep it out of the bundle. +- **`.csv`** as a data grid, and **`.xlsx`** read server-side with `openpyxl` — which needs two passes, or formulas render blank. +- One docked rail, extracted into `DockedPaneService` so the preview and artifact panes share it. + +--- + +## Agent Templates + +A create-form prefill backed by an admin-managed template store, so a new agent can start from a curated shape rather than an empty form. New `agent-templates` table, no GSIs. + +--- + +## 🐛 Bug fixes + +- **Currency in prose rendered as math.** `$4.50 … $9.00` made KaTeX treat everything between the two amounts as a formula, swallowing the prose. An earlier HTML-entity workaround (`$` → `$`) never actually worked — `marked` strips it — and it leaked the entity into generated files, where a user would find `$` in a document the agent wrote. Both are removed and the delimiter handling is fixed at the source. +- **A new frontend build was not actually served.** The deploy sent no `Cache-Control`, so CloudFront kept serving the old bundle. Hashed filenames do not save you here: `index.html` itself is the stale object, and it is what points at the hashes. +- **The system-prompt date line carried the hour**, so the cacheable prefix was re-written every hour, for the life of every session, in every conversation. +- **Deleted conversations were invisible to the admin cost drill-down**, so a user's session list did not reconcile against their total. +- **A first-turn race re-wrote the prompt-cache prefix** — the chat turn was assembled before the tool and skill lists resolved. The turn now waits for them. +- **`DOCUMENT_READ_ENABLED=false` left two of three paths running.** It was read only at the tool-injection gate, so pulling the kill switch left restore emitting handles for a tool that was not injected, and left the live path still evicting bytes — strictly worse than pre-epic behaviour, where live bytes never left. +- **ReDoS in `document_read` pattern mode.** A valid but catastrophic pattern compiled and ran unbounded; the search runs in `asyncio.to_thread`, so the turn hung to the 600s SSE timeout. Now a structural check refuses nested unbounded quantifiers and degrades to literal search (with a note on the payload, so a literal result is never passed off as the regex one), plus a scan budget between lines and pages. **Residual by construction:** a single `re.search` cannot be interrupted, so the budget bounds the walk, never one pathological line. +- **Synth passed with no CORS rule on the uploads bucket.** A truthy-but-empty origin list (`","`) produced a green synth and no rule at all — the guard existed and the value slipped past it. +- **Browser policy fixes** — `MANAGED` was being sent at session level, which broke every session; the policy object key was double-prefixed; the caller needed read on the policy object, not just the browser; the live view connected once per `postMessage` instead of once; SigV4 parameters were sent twice; the stream socket was unsigned; the display was sized before the first frame; two live-view URLs were minted per open; and an error was painted over an already-live stream. +- **Colour steps that failed AA** in light mode — file-type chip text and success-state text both moved to the passing ramp step. + +--- + +## ⚡ Performance + +- **~500ms off the pre-stream window.** Instrumentation opened the preamble stage first and showed it was **494ms of a 641–678ms handler total**. The cause was not what anyone guessed: the session row was being read **eight times per turn**, and 445ms of a 455ms stage was DynamoDB reading one item over and over. Reading it once fixed it. A second pass found boto3 clients being rebuilt per call at 1.5ms each, ~21ms and about 20% of the 95ms the warm preamble then cost; quota now takes session cost from the row the preamble already read. +- **Bytecode precompiled** in the app-api, inference-api and Lambda images. `pip` compiles by default and `uv` does not, so first-touch import was silently being paid at runtime on the first message. +- **CountTokens bounded on the reply path**, and the model id is no longer swapped mid-call. A throttle now costs one failed request rather than a 5.8-second stall. +- **Long-term memory through a bounded client**, and the session's own summary is no longer re-fetched on every user message — it re-injected a conversation the model already held, at a lookup per message. +- **Agent cache widened** to four more injected tool families, and spreadsheet-analysis sessions became cacheable by carrying `assistant_id` in the key. That cohort is the dominant one. +- **gzip on app-api JSON**, with SSE explicitly passed through un-buffered so response headers are not delayed. +- **mermaid lazy-loaded** out of the eager scripts bundle; app-api and inference-api ship code root-owned, and app-api no longer ships `/app` twice. + +--- + +## 🔒 Security + +- **No test reaches AWS.** 25 test cases across 6 files were making **real authenticated AWS calls** against whatever credentials the runner had, hidden because the code paths fail open — a real call that succeeded looked identical to a mocked one, and a real call that failed was swallowed. An off-box socket guard in `tests/conftest.py` now blocks outbound connections, the quarantine has been burned down, and the suite runs in **half the time**. tiktoken is warmed before the guard arms. +- **The browser URL blocklist is a security control supplied from outside the repo**, so a deploy that ships an empty one now says so in the synth log rather than failing silently. + +--- + +## ⚠️ Breaking changes + +None for end users. Two configuration changes operators must act on: + +1. **`CDK_BROWSER_URL_BLOCKLIST` is now required** for any environment that needs a browser blocklist. It previously defaulted to `instructure.com` in code. See Deployment notes. +2. **`request_user_login` must be granted** to the roles that should have it. It ships `enabledByDefault: false` and is its own catalog entry. + +--- + +## 🏗️ Infrastructure + +- **`browser-policy-construct.ts`** — S3-backed MANAGED Chromium policy object for browser sessions, with `ConnectBrowserLiveViewStream` granted on `*` (AWS requires the wildcard). +- **`turn-latency-observability-construct.ts`** — EMF metrics decomposing the pre-stream window into named stages, so a latency fix is provable rather than asserted. +- **New `agent-templates` table** — created with **no GSIs**, so no existing table takes a GSI operation in this release. +- **MCP sandbox CSP** — `connect-src` allows `data:` and `blob:` for the DCV decoder. +- **Uploads bucket CORS** — synth now fails rather than producing a bucket with no CORS rule. +- **Platform Stack deploy triggers** on the assets it actually deploys. + +--- + +## 📦 Dependencies + +| Component | Package | From | To | +|---|---|---|---| +| Backend | `openpyxl` | — | 3.1.5 (new) | +| Frontend | `docx-preview` | — | 0.4.0 (new) | +| Frontend | `pptx-preview` | — | 1.0.7 (new) | +| Frontend | `echarts` | — | local stub shim | + +`echarts` is a `pptx-preview` peer that would otherwise pull a large charting library into the bundle for slides that rarely contain charts; the shim needs to be a top-level `file:` dependency to resolve. + +--- + +## 🧪 Test coverage + +**19,500+ lines** of test changes across **163 files**, including the off-box socket guard and the quarantine burn-down, 39 tests for `document_read` pattern safety, 7 for the `DOCUMENT_READ_ENABLED` coupling, and a seed/catalog parity test that fails when a tool is catalogued without a row in the bootstrap seeder. + +--- + +## 🚀 Deployment notes + +**Deploy order:** `platform.yml` (CDK) → `backend.yml` → `frontend-deploy.yml`. + +**Before deploying:** + +1. **Set `CDK_BROWSER_URL_BLOCKLIST`** on every environment that needs a browser blocklist — comma-separated hostnames, e.g. `instructure.com,vendor.example.com`. It is a GitHub Actions **environment** variable. Unset resolves to an empty list, and the RBAC grant becomes the only control. + + Chromium's `URLBlocklist` matches on **host**, not on the service behind it, so a site is only as blocked as its hostname list is complete. Prefer the registrable domain (`instructure.com` covers `.test.` and `.beta.` instances) and enumerate vendor, vanity-CNAME, regional and mobile hostnames before you consider an entry done. Do **not** block a vendor's institutional sign-in page — reaching a login page is exactly what an accessibility or VPAT review needs to do; block where it leads. + +2. **Confirm `CDK_MANAGED_KB_MIGRATION_ENABLED`** is the value you want. It is read at deploy time, so an environment where it was set to `true` since the last platform deploy gets the Upgrade card on this release. Nothing migrates on its own — enrolment is a user-initiated `POST` — but the card appears and users can opt in. + +3. **Decide `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT`.** It defaults to **100**. Leave it there to ship the feature to every session; set it lower in an environment where you want a concurrent control arm to measure against. + +**After deploying:** + +- **Read the synth log for the blocklist line.** It prints `Browser URL blocklist (N): …`, or warns that the list is empty. An empty list on an environment that expected one means the variable did not reach the synth. +- **Grant `request_user_login`** to the roles that should have browser sign-in handover. `seed_bootstrap_data.py` **skips any tool row that already exists**, so in an environment whose catalog predates this release the row is created only if it was never there; the grant is always a separate step. +- **`list_spreadsheets` and `analyze_spreadsheet`** are now in the seeder. Existing environments already have these rows (they were created by hand) and the seeder will skip them; a **fresh** deployment gets them for the first time. + +**No data backfill is required.** **No GSI operation** is performed against an existing table. + +**Kill switches.** Every feature in this release ships with one, and all default on except where noted: `BROWSER_TAKEOVER_ENABLED`, `AGENT_STATUS_ENABLED`, `AGENT_STATUS_LIVE_DRAIN_ENABLED`, `AGENT_PREPARING_PHASE_ENABLED`, `TOOL_SUMMARIES_ENABLED`, `DOCUMENT_READ_ENABLED`, `DOCUMENT_OFFLOAD_ENABLED`, `DOCUMENT_REHYDRATE_ENABLED`, `DOCUMENT_DIGEST_ENABLED`, `TOOL_RESULT_OFFLOAD_ENABLED`, `COMPACTION_MODEL_RELATIVE_ENABLED`, `COMPACTION_SUMMARY_MODEL_ENABLED`, `COMPACTION_DEFERRED_APPLY_ENABLED`, `RESPONSE_FEEDBACK_ENABLED`, `ATTACHMENT_TURN_GUARD_ENABLED`, `ATTACHMENT_TOOL_AUTOENABLE_ENABLED`, `TURN_LATENCY_METRICS_ENABLED`, `INFERENCE_WARMUP_ENABLED`. Opt-in (default **off**): `FEEDBACK_EVAL_SAMPLING_ENABLED`, `MEMORY_SUMMARY_NAMESPACE_RETRIEVAL_ENABLED`, `BEDROCK_RESPONSES_EXPLICIT_CACHE_ENABLED`. + +--- + # Release Notes — v1.22.0 **Release Date:** September 14, 2026 diff --git a/VERSION b/VERSION index 57807d6d0..a6c2798a4 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.22.0 +1.23.0 diff --git a/backend/Dockerfile.app-api b/backend/Dockerfile.app-api index 11e9d7f7c..29ca114a0 100644 --- a/backend/Dockerfile.app-api +++ b/backend/Dockerfile.app-api @@ -8,6 +8,16 @@ COPY --from=ghcr.io/astral-sh/uv:0.7.12 /uv /uvx /bin/ # Set working directory WORKDIR /app +# Compile every installed package to bytecode at build time. uv skips this by +# default, so the venv shipped with no .pyc files and every task that started +# from this image parsed the whole dependency tree (fastapi, boto3, strands, +# ...) on its first request. Unlike inference-api this is a long-lived ECS +# service, so the cost is paid per task start — deploys, scale-out, and Fargate +# task replacement — not per session. The final stage ships /app/.venv and +# /app/src root-owned, so the runtime user cannot write .pyc at all: baking them +# in at build time is the only place the bytecode can live. +ENV UV_COMPILE_BYTECODE=1 + # Install system dependencies needed for any native builds RUN apt-get update && apt-get install -y \ gcc=4:14.2.0-1 \ @@ -42,18 +52,31 @@ RUN apt-get update && apt-get install -y \ curl=8.14.1-2+deb13u* \ && rm -rf /var/lib/apt/lists/* -# Copy the virtual environment from builder +# Create the non-root user up front so the writable directory below can be +# handed to it without a recursive chown. A trailing `chown -R appuser /app` +# rewrites every file it touches into a fresh layer, so /app used to ship TWICE +# — 313MB of duplicate before this image carried bytecode, 454MB after. +RUN useradd -m -u 1000 -s /bin/bash appuser + +# Code and dependencies ship ROOT-OWNED. appuser can read and execute them and +# cannot modify them, so a process that is compromised at runtime cannot +# rewrite its own source or drop a backdoor into site-packages. +# +# This is only possible because nothing writes into the source tree any more. +# It used to: main.py resolved base_dir from __file__ (landing in /app/src/apis) +# and os.makedirs() there on every startup to back a set of StaticFiles mounts. +# That whole local-output mechanism was dead and has been deleted — generated +# output goes to S3. backend/tests/architecture/test_source_tree_not_written.py +# is the regression guard; if it fails, this image stops booting. COPY --from=builder /app/.venv /app/.venv - -# Copy application code COPY backend/src /app/src -# Create necessary directories -RUN mkdir -p /app/output /app/uploads /app/generated_images - -# Create non-root user for security -RUN useradd -m -u 1000 -s /bin/bash appuser && \ - chown -R appuser:appuser /app +# Precompile our own source the same way the venv was (see UV_COMPILE_BYTECODE +# above). `-q` keeps the layer log quiet; a syntax error still fails the build, +# which is the right time to learn about it. Runs as root, before USER, because +# /app/src is no longer writable by appuser — which is the point: the .pyc are +# baked now and can never be rewritten at runtime. +RUN python -m compileall -q /app/src # Switch to non-root user USER appuser diff --git a/backend/Dockerfile.inference-api b/backend/Dockerfile.inference-api index d03b2d5f7..0bdb95e84 100644 --- a/backend/Dockerfile.inference-api +++ b/backend/Dockerfile.inference-api @@ -8,6 +8,15 @@ COPY --from=ghcr.io/astral-sh/uv:0.7.12 /uv /uvx /bin/ WORKDIR /app +# Compile every installed package to bytecode at build time. uv skips this by +# default, so the venv shipped with no .pyc files and every fresh AgentCore +# micro-VM compiled the whole dependency tree (strands, boto3, sympy behind the +# calculator tool, ...) on the first message of a new session. The runtime user +# is non-root and each micro-VM starts from the image layer, so nothing written +# at runtime survives to the next session — the image is the only place the +# bytecode can live. See docs/specs/load-test-assessment-2026-09.md §1 fix 2. +ENV UV_COMPILE_BYTECODE=1 + # Install system dependencies RUN apt-get update && apt-get install -y \ gcc=4:14.2.0-1 \ @@ -48,12 +57,29 @@ COPY --from=builder /app/.venv /app/.venv # Copy application code COPY backend/src /app/src -# Create necessary directories -RUN mkdir -p /app/output /app/uploads /app/generated_images - -# Create non-root user for security (AgentCore Runtime best practice) -RUN useradd -m -u 1000 -s /bin/bash bedrock_agentcore && \ - chown -R bedrock_agentcore:bedrock_agentcore /app +# Precompile our own source the same way the venv was (see UV_COMPILE_BYTECODE +# above). `-q` keeps the layer log quiet; a syntax error still fails the build, +# which is the right time to learn about it. +RUN python -m compileall -q /app/src + +# Create non-root user for security (AgentCore Runtime best practice). +# Note what is NOT here any more: `chown -R bedrock_agentcore /app`. A recursive +# chown rewrites every file it touches into a fresh layer, so /app shipped +# TWICE — on this image that is the venv plus the whole source tree, duplicated +# for nothing. Code and dependencies now stay root-owned, which is both smaller +# and safer: a runtime compromise cannot rewrite the source it executes or drop +# a backdoor into site-packages. +# +# This is only possible because nothing writes into the source tree any more. +# It used to: main.py resolved base_dir from __file__ (landing in /app/src/apis) +# and os.makedirs() there on every startup, and agents/utils/config.py did the +# same one level up to hold per-session tool output. That whole local-output +# mechanism was dead — its writer fired only for two tool names that no longer +# exist — and has been deleted; generated output goes to S3. There is therefore +# no writable directory under /app at all. +# backend/tests/architecture/test_source_tree_not_written.py is the regression +# guard — if it fails, this image stops booting. +RUN useradd -m -u 1000 -s /bin/bash bedrock_agentcore # Switch to non-root user USER bedrock_agentcore diff --git a/backend/Dockerfile.kb-migration b/backend/Dockerfile.kb-migration index 587001b28..a243cb44f 100644 --- a/backend/Dockerfile.kb-migration +++ b/backend/Dockerfile.kb-migration @@ -72,6 +72,21 @@ COPY backend/src/apis/shared/kb_backend/ ${LAMBDA_TASK_ROOT}/apis/shared/kb_back # The handlers themselves. COPY backend/src/apis/app_api/kb_migration/ ${LAMBDA_TASK_ROOT}/apis/app_api/kb_migration/ +# Precompile the copied source to bytecode. /var/task is READ-ONLY at runtime, +# so a Lambda cold start can never cache the .pyc it produces — every cold start +# re-parses the whole closure. This is the smallest win of the backend images +# precisely because of the COPY-surface discipline described above: the closure +# is ~16 first-party modules, so there is not much source to parse. It is also +# one line and cannot regress, and it keeps the four Lambdas here consistent +# with kb-sync and scheduled-runs as the surface grows. The pip install above +# already compiled site-packages (pip does that by default, unlike uv), so this +# covers the gap: the code we COPY in, which arrives as raw source. `-q` keeps +# the layer log quiet; a syntax error still fails the build, which is the right +# time to learn about it. Safe to point at the whole task root here — it holds +# only our own code, since pip put the dependencies in +# /var/lang/lib/python3.12/site-packages. +RUN python -m compileall -q ${LAMBDA_TASK_ROOT} + # Default command: the dispatcher. The other three functions override CMD # through their ImageConfig, so this default is only what an unconfigured # function would run. diff --git a/backend/Dockerfile.kb-sync b/backend/Dockerfile.kb-sync index 8038e36ae..e6449efd1 100644 --- a/backend/Dockerfile.kb-sync +++ b/backend/Dockerfile.kb-sync @@ -68,6 +68,18 @@ COPY backend/src/apis/app_api/kb_sync/ ${LAMBDA_TASK_ROOT}/apis/app_api/kb_sync/ # pull FastAPI, which this image deliberately lacks. The worker uses the # oauth primitives, document/crawl services, and the crawler directly. +# Precompile the copied source to bytecode. /var/task is READ-ONLY at runtime, +# so a Lambda cold start can never cache the .pyc it produces — every cold start +# re-parses every module in the closure, and the dispatcher runs off a +# 15-minute EventBridge tick, far enough apart that most ticks land cold. The +# pip install above already compiled site-packages (pip does that by default, +# unlike uv), so this covers the gap: the code we COPY in, which arrives as raw +# source. `-q` keeps the layer log quiet; a syntax error still fails the build, +# which is the right time to learn about it. Safe to point at the whole task +# root here — it holds only our own code, since pip put the dependencies in +# /var/lang/lib/python3.12/site-packages. +RUN python -m compileall -q ${LAMBDA_TASK_ROOT} + # Default command: dispatcher. The worker function's CMD is overridden # to apis.app_api.kb_sync.worker.lambda_handler by its ImageConfig. CMD ["apis.app_api.kb_sync.dispatcher.lambda_handler"] diff --git a/backend/Dockerfile.scheduled-runs b/backend/Dockerfile.scheduled-runs index 10d99fa17..204309bf3 100644 --- a/backend/Dockerfile.scheduled-runs +++ b/backend/Dockerfile.scheduled-runs @@ -36,10 +36,19 @@ COPY backend/src/apis/shared/errors.py ${LAMBDA_TASK_ROOT}/apis/shared/errors.py # Leaf module (imports only os); reached via apis/shared/sessions/metadata.py, # which reads the COST_DIAGNOSTICS_ENABLED kill switch at aggregate-bump time. COPY backend/src/apis/shared/feature_flags.py ${LAMBDA_TASK_ROOT}/apis/shared/feature_flags.py +# Leaf module (imports boto3 lazily); reached via apis/shared/sessions/metadata.py, +# which now takes its DynamoDB handles from the shared client cache instead of +# constructing a resource per call. +COPY backend/src/apis/shared/aws_clients.py ${LAMBDA_TASK_ROOT}/apis/shared/aws_clients.py COPY backend/src/apis/shared/harness/ ${LAMBDA_TASK_ROOT}/apis/shared/harness/ COPY backend/src/apis/shared/scheduled_prompts/ ${LAMBDA_TASK_ROOT}/apis/shared/scheduled_prompts/ COPY backend/src/apis/shared/sessions_bff/ ${LAMBDA_TASK_ROOT}/apis/shared/sessions_bff/ COPY backend/src/apis/shared/sessions/ ${LAMBDA_TASK_ROOT}/apis/shared/sessions/ +# Pure-python wire contract; sessions/metadata.py imports its `assert_no_url` +# guard on the browser-takeover write path. Nothing here schedules a +# takeover, but the guard is a function-local import that would still run +# on invocation if that path were ever reached. +COPY backend/src/apis/shared/browser_takeover/ ${LAMBDA_TASK_ROOT}/apis/shared/browser_takeover/ # sessions/metadata.py lazily reaches these on its storage + EMF paths. COPY backend/src/apis/shared/storage/ ${LAMBDA_TASK_ROOT}/apis/shared/storage/ COPY backend/src/apis/shared/observability/ ${LAMBDA_TASK_ROOT}/apis/shared/observability/ @@ -48,6 +57,18 @@ COPY backend/src/apis/shared/observability/ ${LAMBDA_TASK_ROOT}/apis/shared/obse COPY backend/src/lambdas/scheduled_runs_dispatcher/dispatcher.py ${LAMBDA_TASK_ROOT}/dispatcher.py COPY backend/src/lambdas/scheduled_runs_worker/worker.py ${LAMBDA_TASK_ROOT}/worker.py +# Precompile the copied source to bytecode. /var/task is READ-ONLY at runtime, +# so a Lambda cold start can never cache the .pyc it produces — every cold start +# re-parses every module in the closure, and this dispatcher runs off a 5-minute +# EventBridge tick, which is slow enough that many ticks land cold. The pip +# install above already compiled site-packages (pip does that by default, unlike +# uv), so this covers the gap: the code we COPY in, which arrives as raw source. +# `-q` keeps the layer log quiet; a syntax error still fails the build, which is +# the right time to learn about it. Safe to point at the whole task root here — +# it holds only our own code, since pip put the dependencies in +# /var/lang/lib/python3.12/site-packages. +RUN python -m compileall -q ${LAMBDA_TASK_ROOT} + # Default command: dispatcher. The worker function's CMD is overridden # to worker.lambda_handler by its ImageConfig. CMD ["dispatcher.lambda_handler"] diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 2f96e6fb0..e3cf24bba 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agentcore-stack" -version = "1.22.0" +version = "1.23.0" requires-python = ">=3.10" description = "Multi-agent conversational AI system with AWS Bedrock AgentCore" readme = "README.md" @@ -36,6 +36,13 @@ dependencies = [ # Security: pin transitive deps to fix Dependabot alerts "pillow==12.3.0", + # Reads .xlsx workbooks into rows for the UI's spreadsheet preview + # (apis/app_api/files/sheet_preview.py). MIT, pure Python, one + # dependency (et-xmlfile). Already the library our own + # create_excel_spreadsheet tool drives inside Code Interpreter, and + # the one the RAG ingestion image uses via docling — this pin brings + # it into app-api's own closure, which it was not in before. + "openpyxl==3.1.5", "cryptography==50.0.1", "python-multipart==0.0.31", "aiohttp==3.14.3", diff --git a/backend/scripts/probe_static_prefix_ttl.py b/backend/scripts/probe_static_prefix_ttl.py new file mode 100644 index 000000000..27b313c0e --- /dev/null +++ b/backend/scripts/probe_static_prefix_ttl.py @@ -0,0 +1,169 @@ +"""Does a 1h TTL on the tools + system cachePoints pay for itself on Claude/Bedrock? + +The gate for AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h +(docs/specs/compaction-model-relative-thresholds.md §3.6, PR-5). A caching +default must never be adopted on inspection alone — #954 shipped on a wrong +premise and measured 57% more expensive live before #956 reverted it — so this +script measures the two arms against the same static prefix: + + arm 5m : tools + system points with no ttl (today's shape) + arm 1h : tools + system points with ttl "1h"; message point unchanged + +Per arm: call 1 (the write), sleep --gap-seconds, call 2 (the read-or-rewrite). +It reports cacheRead / cacheWrite per call, whether Bedrock accepted the 1h +point at all, and prices the pair at the model's own rates — 1.25x base for a +5m write, 2x base for a 1h write, 0.1x for a read — so the break-even is read +off the output rather than argued. + +What "pays" means: with a gap between 5 and 60 minutes, the 1h arm's second +call should READ the static prefix (cacheRead ≈ static tokens) where the 5m +arm re-WRITES it. Over a session the 1h arm costs +0.75x base on every static +write and saves 1.15x base on every cold-within-the-hour return; it pays when +the second event is more frequent than the first. Run this with a gap of +~420s (past 5m) and again with ~60s (inside 5m) to see both regimes. + +Read-only apart from the model invocations. ⚠️ Real spend: four calls at +~8k input tokens each plus the sleep. Nothing is written anywhere. + +Usage: + cd backend + AWS_PROFILE=dev-ai uv run python scripts/probe_static_prefix_ttl.py --gap-seconds 420 + AWS_PROFILE=dev-ai uv run python scripts/probe_static_prefix_ttl.py --gap-seconds 60 --model-id us.anthropic.claude-haiku-4-5-20251001-v1:0 + +Baseline, dev-ai us-west-2, 2026-09-16, Haiku 4.5, gap 420s: + 5m first read 0 write 6251 second read 0 write 6251 + 1h first read 0 write 6251 second read 5924 write 327 <- honored + pair $0.017197 (5m) vs $0.015130 (1h): 1h CHEAPER by 12% at this gap +Same day, gap 60s (both arms warm): + 5m second read 6251 write 0 ; 1h second read 6251 write 0 + pair $0.009289 (5m) vs $0.014446 (1h): 1h MORE EXPENSIVE by $0.005157 + = the 0.75x-base premium on the first write, nothing to recover inside 5m +""" + +from __future__ import annotations + +import argparse +import sys +import time +from typing import Any, Dict, List + +import boto3 + +# Clears every Claude family's cache minimum (4,096 on Haiku) with margin. +_SYSTEM_TEXT = ("You are a careful assistant. " * 40 + "Policy: answer briefly. ") * 20 +_TOOL_SPECS: List[Dict[str, Any]] = [ + { + "toolSpec": { + "name": f"tool_{i}", + "description": "A probe tool that does nothing useful. " * 12, + "inputSchema": {"json": {"type": "object", "properties": {"q": {"type": "string"}}}}, + } + } + for i in range(6) +] + +RATES = { # $/MTok base input, Global CRIS; adjust for the model under test + "us.anthropic.claude-haiku-4-5-20251001-v1:0": 1.10, + "global.anthropic.claude-sonnet-5": 2.00, + "us.anthropic.claude-sonnet-4-6": 3.30, +} + + +def _request(model_id: str, ttl: str | None, marker: str) -> Dict[str, Any]: + point: Dict[str, Any] = {"type": "default"} + if ttl: + point["ttl"] = ttl + return { + "modelId": model_id, + "system": [{"text": _SYSTEM_TEXT + f"\nProbe arm: {marker}."}, {"cachePoint": dict(point)}], + "toolConfig": {"tools": _TOOL_SPECS + [{"cachePoint": dict(point)}]}, + "messages": [{"role": "user", "content": [{"text": "Reply with the single word OK."}, {"cachePoint": {"type": "default"}}]}], + "inferenceConfig": {"maxTokens": 5}, + } + + +def _call(client: Any, req: Dict[str, Any]) -> Dict[str, int]: + resp = client.converse(**req) + u = resp.get("usage", {}) + return { + "input": int(u.get("inputTokens", 0)), + "read": int(u.get("cacheReadInputTokens", 0)), + "write": int(u.get("cacheWriteInputTokens", 0)), + } + + +def _price(usage: Dict[str, int], base: float, write_mult: float) -> float: + return (usage["input"] * base + usage["read"] * base * 0.1 + usage["write"] * base * write_mult) / 1e6 + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--model-id", default="us.anthropic.claude-haiku-4-5-20251001-v1:0") + ap.add_argument("--region", default="us-west-2") + ap.add_argument("--gap-seconds", type=int, default=420) + ap.add_argument("--base-rate", type=float, default=None, help="$/MTok base input; defaults from a small table") + args = ap.parse_args() + + base = args.base_rate or RATES.get(args.model_id) + if base is None: + print(f"no base rate for {args.model_id}; pass --base-rate", file=sys.stderr) + return 2 + client = boto3.client("bedrock-runtime", region_name=args.region) + + arms = [("5m", None), ("1h", "1h")] + results: Dict[str, Any] = {} + # Distinct markers per arm so the two arms never share a cache entry. + for name, ttl in arms: + marker = f"{name}-{int(time.time())}" + req = _request(args.model_id, ttl, marker) + try: + first = _call(client, req) + except Exception as e: # noqa: BLE001 + results[name] = {"error": f"{type(e).__name__}: {e}"} + continue + results[name] = {"first": first, "marker": marker} + if all("error" in r for r in results.values()): + print(results) + return 1 + + print(f"sleeping {args.gap_seconds}s so the 5m entry {'expires' if args.gap_seconds > 300 else 'stays warm'} ...") + time.sleep(args.gap_seconds) + + for name, ttl in arms: + if "error" in results[name]: + continue + # Re-send the SAME request (same marker) — identical bytes. + try: + results[name]["second"] = _call(client, _request(args.model_id, ttl, results[name]["marker"])) + except Exception as e: # noqa: BLE001 + results[name]["error"] = f"{type(e).__name__}: {e}" + + print(f"\nmodel={args.model_id} base=${base}/MTok gap={args.gap_seconds}s\n") + print(f"{'arm':<4} {'call':<7} {'input':>7} {'read':>8} {'write':>8} {'$':>10}") + total = {} + for name, ttl in arms: + r = results[name] + if "error" in r: + print(f"{name:<4} ERROR {r['error']}") + continue + mult = 2.0 if ttl == "1h" else 1.25 + cost = 0.0 + for which in ("first", "second"): + u = r[which] + c = _price(u, base, mult) + cost += c + print(f"{name:<4} {which:<7} {u['input']:>7} {u['read']:>8} {u['write']:>8} {c:>10.6f}") + total[name] = cost + print(f"{name:<4} {'pair':<7} {'':>7} {'':>8} {'':>8} {cost:>10.6f}") + if "5m" in total and "1h" in total: + delta = total["1h"] - total["5m"] + verdict = "1h CHEAPER" if delta < 0 else "1h MORE EXPENSIVE" + print(f"\n{verdict} by ${abs(delta):.6f} for this pair at a {args.gap_seconds}s gap") + second = results["1h"].get("second", {}) + if args.gap_seconds > 300 and second.get("read", 0) == 0: + print("⚠️ the 1h arm did NOT read after the gap — Bedrock may not honor ttl=1h for this model; do not enable the flag") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/backend/scripts/seed_bootstrap_data.py b/backend/scripts/seed_bootstrap_data.py index d6a273926..b89fb1dac 100644 --- a/backend/scripts/seed_bootstrap_data.py +++ b/backend/scripts/seed_bootstrap_data.py @@ -437,6 +437,26 @@ def seed_default_models( "isPublic": False, "forwardAuthToken": False, }, + { + "toolId": "request_user_login", + "displayName": "Browser Sign-In", + "description": ( + "Hand the browser to the user so they can sign in to a site the " + "agent cannot reach, then continue browsing the authenticated " + "session." + ), + "category": "browser", + # Deliberately off by default, and the most restrictive default in this + # file. While a takeover is live the user has a fully interactive + # Chromium running inside our AWS account with our egress — that is a + # capability to grant to named staff/evaluator roles, never a default. + # RBAC granularity is one tool_id, which is exactly why this is its own + # entry rather than an action on browse_web. + "enabledByDefault": False, + "protocol": "local", + "isPublic": False, + "forwardAuthToken": False, + }, { "toolId": "generate_diagram_and_validate", "displayName": "Code Interpreter", @@ -492,6 +512,44 @@ def seed_default_models( "isPublic": True, "forwardAuthToken": False, }, + { + # Spreadsheet ANALYSIS (read an uploaded .xlsx/.csv and answer questions + # about it via Code Interpreter) — distinct from the spreadsheet + # *authoring* tool below. Both ids are context-bound: enabling them + # injects the tools at runtime, see SPREADSHEET_TOOL_IDS and + # _build_spreadsheet_tools in apis/inference_api/chat/routes.py. + # + # These two rows were missing from this file until 1.23.0 while the + # Python TOOL_CATALOG carried them, so a freshly bootstrapped + # deployment got no catalog row and the tools could not be granted to + # any role — the live environments only have them because the rows + # were created by hand. test_seed_matches_tool_catalog now pins the + # two lists together so they cannot drift again. + # + # enabledByDefault mirrors the live deployments (True). Safe against + # the injected-tool cost trap because SPREADSHEET_TOOL_IDS is in + # KEY_DESCRIBED_INJECTED_TOOL_IDS: the factory closes over + # (session_id, user_id, assistant_id), all cache-key elements, so a + # default-on injected tool does NOT bypass the agent cache here. + "toolId": "list_spreadsheets", + "displayName": "List Spreadsheet Files", + "description": "List spreadsheet files available for analysis from the assistant's knowledge base or conversation attachments.", + "category": "data", + "protocol": "local", + "enabledByDefault": True, + "isPublic": True, + "forwardAuthToken": False, + }, + { + "toolId": "analyze_spreadsheet", + "displayName": "Analyze Spreadsheet", + "description": "Analyze spreadsheet data with Python in a sandboxed Code Interpreter session: filter, aggregate, compute statistics and answer questions about the contents.", + "category": "data", + "protocol": "local", + "enabledByDefault": True, + "isPublic": True, + "forwardAuthToken": False, + }, { # Single catalog entry / toggle that provisions the whole Excel # spreadsheet toolset. Enabling this one id injects create/modify/list/ diff --git a/backend/src/.env.example b/backend/src/.env.example index 087c29bf5..a0cd9ce0a 100644 --- a/backend/src/.env.example +++ b/backend/src/.env.example @@ -181,22 +181,15 @@ ENABLE_QUOTA_ENFORCEMENT=true # ============================================================================= # LOCAL DEVELOPMENT DIRECTORIES # ============================================================================= -# These directories are used ONLY for local development. -# In deployed cloud environments (Docker/Fargate), the Dockerfiles create these -# directories at build time and the Python code falls back to hardcoded defaults. -# You do NOT need to set these in production. - -# Directory for uploaded files (images, documents) -# Local-dev-only: overrides the default "uploads" directory name -UPLOAD_DIR=uploads - -# Directory for generated output files -# Local-dev-only: overrides the default "output" directory name -OUTPUT_DIR=output - -# Directory for AI-generated images -# Local-dev-only: overrides the default "generated_images" directory name -GENERATED_IMAGES_DIR=generated_images +# Nothing here any more. The services used to create output/, uploads/ and +# generated_images/ under backend/src and serve them as static mounts, with +# UPLOAD_DIR / OUTPUT_DIR / GENERATED_IMAGES_DIR documented as overriding the +# names. Those variables never did anything — nothing read them but a log line +# — and the mechanism they described was dead: the only writer fired for two +# tool names that no longer exist, and the frontend never linked to the mounts. +# +# It has all been deleted. Generated output goes to S3, and the container +# images ship /app/src and /app/.venv read-only as a result. # ============================================================================= # DATABASE CONFIGURATION @@ -871,6 +864,7 @@ DYNAMODB_USER_SETTINGS_TABLE_NAME= # CDK Deployment: Created by AppApiStack # Example: -system-prompts DYNAMODB_SYSTEM_PROMPTS_TABLE_NAME= +DYNAMODB_AGENT_TEMPLATES_TABLE_NAME= # S3 vector store bucket name (REQUIRED for RAG) # Purpose: Store vector embeddings for document retrieval diff --git a/backend/src/agents/builtin_tools/browser/__init__.py b/backend/src/agents/builtin_tools/browser/__init__.py index b3fbecdc8..0786354c4 100644 --- a/backend/src/agents/builtin_tools/browser/__init__.py +++ b/backend/src/agents/builtin_tools/browser/__init__.py @@ -1,5 +1,10 @@ """AgentCore Browser tool package.""" from .browse_tool import browse_web +from .takeover_tool import request_user_login -__all__ = ["browse_web"] +# `request_user_login` is deliberately NOT re-exported through +# `agents.builtin_tools.__all__`: `create_default_registry` registers it +# explicitly so the kill switch can withhold it, exactly as it does for +# `ask_user_question`. +__all__ = ["browse_web", "request_user_login"] diff --git a/backend/src/agents/builtin_tools/browser/browse_tool.py b/backend/src/agents/builtin_tools/browser/browse_tool.py index ffcdcf545..84184332f 100644 --- a/backend/src/agents/builtin_tools/browser/browse_tool.py +++ b/backend/src/agents/builtin_tools/browser/browse_tool.py @@ -145,7 +145,6 @@ async def browse_web( `[...document.querySelectorAll('h2')].map(e => e.innerText)`. `screenshot` — capture the viewport as an image. Expensive in tokens; only use it when you must SEE the layout. - `live_view` — get a URL where the user can watch the session. `close` — end the browser session. url: Target URL for `navigate` (http/https). selector: CSS selector for `click` and `type`. @@ -281,13 +280,14 @@ async def _dispatch( [{"image": {"format": "png", "source": {"bytes": base64.b64decode(encoded)}}}], ) - if action == "live_view": - view_url = await session_pool.live_view_url(agent) - if not view_url: - return _err("❌ No live view available for this session.") - return _ok(f"Watch the browser session here (expires shortly):\n{view_url}") + # `live_view` used to live here. It returned a presigned URL as tool-result + # text, which the model re-emits truncated at the `?` (PR #1101), and which + # expires in at most 300 seconds — dead before a human can react and dead + # again on every reload of the thread. Watching (and now driving) the + # session is `request_user_login` plus app-api's live-view route, where the + # URL is minted per request and never becomes model-visible. return _err( f"❌ Unknown action '{action}'. Valid actions: navigate, extract_text, " - "extract_links, click, type, evaluate, screenshot, live_view, close." + "extract_links, click, type, evaluate, screenshot, close." ) diff --git a/backend/src/agents/builtin_tools/browser/session_pool.py b/backend/src/agents/builtin_tools/browser/session_pool.py index d2ec789a7..dfb261751 100644 --- a/backend/src/agents/builtin_tools/browser/session_pool.py +++ b/backend/src/agents/builtin_tools/browser/session_pool.py @@ -32,6 +32,7 @@ STATE_KEY = "browser_tool" _SESSION_SUBKEY = "session" +_TAKEOVER_SUBKEY = "takeover" # AgentCore caps session_timeout at 8h; we deliberately stay near the low end. # An abandoned session bills until its TTL expires, and a browsing task that @@ -42,6 +43,22 @@ # with them. Shorter than the remote TTL so we usually release first. IDLE_REAP_SECONDS = int(os.environ.get("BROWSER_IDLE_REAP_SECONDS", 600)) +# How long a human gets to finish signing in before the takeover is abandoned. +# "Idle" during a takeover is measured in agent tool calls, of which there are +# none by construction, so without this a handed-over session would be pinned +# against the reaper until the remote TTL killed it. Kept comfortably under +# SESSION_TIMEOUT_SECONDS so we release before the service does — an abandoned +# takeover should end in a session we stopped, not one that expired under us. +TAKEOVER_DEADLINE_SECONDS = int( + os.environ.get("BROWSER_TAKEOVER_DEADLINE_SECONDS", 480) +) + +# Slack on the resume side of the deadline. The deadline governs when we stop +# *waiting*; a user who finished at 7:59 whose POST lands at 8:01 should not be +# told their sign-in expired and have the work thrown away. Only past the grace +# window do we insist the takeover lapsed. +TAKEOVER_GRACE_SECONDS = int(os.environ.get("BROWSER_TAKEOVER_GRACE_SECONDS", 60)) + DEFAULT_VIEWPORT = {"width": 1280, "height": 800} @@ -55,6 +72,25 @@ class _LiveSession: cdp: CdpSession last_used: float = field(default_factory=time.monotonic) lock: asyncio.Lock = field(default_factory=asyncio.Lock) + # Who holds the browser. While "user", the automation stream is DISABLED + # at the service — the agent provably cannot act, rather than merely + # agreeing not to. `control_deadline` is monotonic; past it the pin lapses + # and the session is reapable again. + control_state: str = "agent" + control_deadline: float = 0.0 + # The viewport this session was actually STARTED with, not whatever + # DEFAULT_VIEWPORT says now. DCV's remoteWidth/remoteHeight must match the + # real one or the stream crops, and a session can outlive a config change + # or be reconnected to from a container whose default differs. + viewport: Dict[str, int] = field( + default_factory=lambda: dict(DEFAULT_VIEWPORT) + ) + + def user_controlled(self, now: Optional[float] = None) -> bool: + """True while a human holds the browser and the deadline has not passed.""" + if self.control_state != "user": + return False + return (now if now is not None else time.monotonic()) < self.control_deadline _live: Dict[str, _LiveSession] = {} @@ -94,22 +130,92 @@ def _write_state(agent: Any, entry: Optional[Dict[str, Any]]) -> None: logger.warning("browser: session state not serializable; not persisted") +# Session-level policies are RECOMMENDED-only. The API's `type` enum accepts +# MANAGED, but the service rejects it: +# +# ValidationException ... Invalid value for parameter 'type'. +# MANAGED is not supported for session-level policies. +# +# Measured on dev 2026-09-19. Sending MANAGED here does not degrade — it fails +# `StartBrowserSession` outright, so EVERY browser session dies and `browse_web` +# stops working entirely. Do not "try" MANAGED here again. +_SESSION_POLICY_TYPE = "RECOMMENDED" + + +def _enterprise_policies() -> Optional[list]: + """The Chromium URL policy to start every browser session with. + + ⚠️ **This is NOT the security control it was designed to be, and must not + be relied on as one.** See `docs/specs/authenticated-web-assessment.md` D6. + + The intent was a MANAGED policy — mandated and un-overridable — so that a + human holding the browser during a takeover could not navigate to the LMS + and have an agent act as them. That requires `CreateBrowser`, because the + service refuses MANAGED at session level (see `_SESSION_POLICY_TYPE`). + + What a RECOMMENDED policy still buys, honestly stated: + + * It constrains the **agent**, which drives through CDP and never opens + Chromium's settings, so the blocklist holds for ordinary `browse_web`. + * It does **not** constrain a **human** in a takeover, who has a fully + interactive browser and can override a recommended policy. + + So `request_user_login` must stay ungranted until the MANAGED policy is + applied at `CreateBrowser`. The RBAC grant and `BROWSER_TAKEOVER_ENABLED` + are what hold that line in the meantime. + + Returns None when unconfigured, which starts the session with no policy at + all — correct for a local dev box with no bucket. + """ + location = os.environ.get("BROWSER_POLICY_S3", "").strip() + if not location: + return None + + if not location.startswith("s3://"): + logger.error( + "browser: BROWSER_POLICY_S3 is not an s3:// URI (%r); " + "starting sessions with NO url policy", + location, + ) + return None + + bucket, _, key = location[len("s3://"):].partition("/") + if not bucket or not key: + logger.error( + "browser: BROWSER_POLICY_S3 names no bucket/key (%r); " + "starting sessions with NO url policy", + location, + ) + return None + + return [ + { + "type": _SESSION_POLICY_TYPE, + "location": {"s3": {"bucket": bucket, "prefix": key}}, + } + ] + + async def _start_remote_session() -> Tuple[Any, str, str]: """Start an AgentCore browser session. Returns (client, identifier, id).""" from bedrock_agentcore.tools.browser_client import BrowserClient identifier = _browser_identifier() client = BrowserClient(region=_region()) + policies = _enterprise_policies() + start_kwargs: Dict[str, Any] = { + "identifier": identifier, + "session_timeout_seconds": SESSION_TIMEOUT_SECONDS, + "viewport": DEFAULT_VIEWPORT, + } + if policies: + start_kwargs["enterprise_policies"] = policies # boto3 is synchronous; keep it off the event loop. - session_id = await asyncio.to_thread( - client.start, - identifier=identifier, - session_timeout_seconds=SESSION_TIMEOUT_SECONDS, - viewport=DEFAULT_VIEWPORT, - ) + session_id = await asyncio.to_thread(client.start, **start_kwargs) logger.info( - "browser: started session %s on %s (ttl=%ss)", + "browser: started session %s on %s (ttl=%ss, url_policy=%s)", session_id, identifier, SESSION_TIMEOUT_SECONDS, + _SESSION_POLICY_TYPE.lower() if policies else "NONE", ) return client, identifier, session_id @@ -120,11 +226,23 @@ async def _connect(client: Any) -> CdpSession: async def _reap_idle() -> None: - """Close sockets unused past the idle window, stopping the remote session.""" + """Close sockets unused past the idle window, stopping the remote session. + + A session a human is driving is exempt: idleness here is measured in agent + tool calls, and during a takeover there are none by construction, so the + plain rule would reap the browser out from under someone mid-login. The + exemption is bounded by the takeover deadline rather than open-ended — an + abandoned takeover becomes ordinarily reapable once the deadline lapses, + which is what stops a walked-away-from session billing to its TTL. + + Note this runs from `acquire`, so the sweep that collects an abandoned + session is usually some *other* conversation's next browser call in the + same container. That is enough: the remote TTL is the backstop. + """ now = time.monotonic() stale = [ sid for sid, live in _live.items() - if now - live.last_used > IDLE_REAP_SECONDS + if now - live.last_used > IDLE_REAP_SECONDS and not live.user_controlled(now) ] for sid in stale: live = _live.pop(sid, None) @@ -175,7 +293,11 @@ async def acquire(agent: Any) -> _LiveSession: raise live = _LiveSession( - session_id=session_id, identifier=identifier, client=client, cdp=cdp + session_id=session_id, + identifier=identifier, + client=client, + cdp=cdp, + viewport=dict(DEFAULT_VIEWPORT), ) _live[session_id] = live _write_state( @@ -184,6 +306,9 @@ async def acquire(agent: Any) -> _LiveSession: "sessionId": session_id, "identifier": identifier, "startedAt": time.time(), + # Persisted so a reconnect in another container reports the + # viewport the browser is really running at. + "viewport": dict(DEFAULT_VIEWPORT), }, ) return live @@ -205,8 +330,13 @@ async def _try_reconnect(entry: Dict[str, Any]) -> Optional[_LiveSession]: return None logger.info("browser: reconnected to session %s", session_id) + viewport = entry.get("viewport") live = _LiveSession( - session_id=session_id, identifier=identifier, client=client, cdp=cdp + session_id=session_id, + identifier=identifier, + client=client, + cdp=cdp, + viewport=dict(viewport) if isinstance(viewport, dict) else dict(DEFAULT_VIEWPORT), ) _live[session_id] = live return live @@ -238,17 +368,193 @@ async def release(agent: Any) -> bool: return False -async def live_view_url(agent: Any) -> Optional[str]: - """Pre-signed live-view URL for the running session, if there is one.""" +# --------------------------------------------------------------------------- +# Human takeover +# +# `take_control` / `release_control` are thin wrappers over +# `UpdateBrowserStream`, which flips the automation stream DISABLED/ENABLED. +# That is a **service-side mutex**, not a convention in this code: while the +# stream is DISABLED the agent's CDP commands are refused by AWS, so "the agent +# cannot act while the human drives" is enforced somewhere we do not control. +# +# There is deliberately no URL-minting function here. The live-view URL is +# signed with SigV4 query auth and expires in at most 300 seconds, so it is +# minted per request by app-api and never travels through the agent (spec D2, +# and PR #1101's rule about presigned URLs in tool results). The `live_view` +# action that used to return one from here is gone. +# --------------------------------------------------------------------------- + + +def _describe(live: _LiveSession) -> Dict[str, Any]: + """The identifiers a takeover needs, and nothing more.""" + return { + "browserSessionId": live.session_id, + "browserId": live.identifier, + "viewport": dict(live.viewport), + "controlState": live.control_state, + } + + +async def take_control( + agent: Any, deadline_seconds: int = TAKEOVER_DEADLINE_SECONDS +) -> Dict[str, Any]: + """Hand this conversation's browser to the user. Idempotent. + + Idempotence is not a nicety here: Strands re-executes an interrupted tool + from the top when the turn resumes (see ``strands/types/interrupt.py``), so + ``request_user_login`` calls this a second time on the way back in. Taking + control of an already-user-controlled session must therefore be a cheap + no-op rather than a second ``UpdateBrowserStream`` call — and must not + silently extend the deadline, which would make an abandoned takeover + unreapable for as long as resumes kept arriving. + + Returns the session descriptor (ids + viewport + deadline). Raises if there + is no session to hand over, or if the service refuses the stream update — + the caller turns that into a tool error rather than pausing the turn. + """ + live = await acquire(agent) + now = time.monotonic() + + if live.user_controlled(now): + return {**_describe(live), "deadlineAt": _deadline_iso(live, now)} + + await asyncio.to_thread(live.client.take_control) + live.control_state = "user" + live.control_deadline = now + max(deadline_seconds, 1) + live.last_used = now + logger.info( + "browser: handed session %s to the user (deadline=%ss)", + live.session_id, deadline_seconds, + ) + return {**_describe(live), "deadlineAt": _deadline_iso(live, now)} + + +async def release_control(agent: Any) -> bool: + """Take the browser back from the user. Idempotent; True if it flipped. + + Best-effort by design: if the remote call fails the local pin is dropped + anyway, because leaving a session pinned against the reaper on the strength + of a failed API call is how an abandoned browser bills to its TTL. + """ + entry = _read_state(agent) + if not entry: + return False + session_id = entry.get("sessionId") + identifier = entry.get("identifier") + if not session_id or not identifier: + return False + + live = _live.get(session_id) + if live is not None: + if live.control_state != "user": + return False + live.control_state = "agent" + live.control_deadline = 0.0 + live.last_used = time.monotonic() + client = live.client + else: + # The resume landed in a different container from the takeover, or this + # one restarted. The stream is still DISABLED at the service, and + # nothing local knows it — so re-enable it by id rather than returning + # early. Skipping this would leave the agent permanently unable to + # drive a browser it still holds state for. + from bedrock_agentcore.tools.browser_client import BrowserClient + + client = BrowserClient(region=_region()) + client.identifier = identifier + client.session_id = session_id + + try: + await asyncio.to_thread(client.release_control) + except Exception: # noqa: BLE001 + logger.warning( + "browser: release_control failed for %s; pin dropped locally anyway", + session_id, exc_info=True, + ) + return False + logger.info("browser: took session %s back from the user", session_id) + return True + + +def has_session(agent: Any) -> bool: + """Whether this conversation already has a browser session. + + Distinct from `acquire`, which *starts* one. A takeover on a browser that + has never navigated anywhere hands the user a blank tab, so the tool checks + this first: the alternative is billing an AgentCore browser session and + pausing the turn to achieve nothing. + """ + entry = _read_state(agent) + return bool(entry and entry.get("sessionId")) + + +def read_takeover(agent: Any) -> Optional[Dict[str, Any]]: + """The in-flight takeover recorded on agent state, if any. + + Strands re-executes an interrupted tool from the top on resume, so the tool + needs to tell "first pass" from "coming back" without re-running its side + effects. This marker is that signal, and it rides `agent.state` rather than + a process-local dict on purpose: the resume can land in a different + container from the pause, and a marker that did not survive that would make + the resume look like a fresh takeover and re-take a browser the user had + already handed back. + """ + state = getattr(agent, "state", None) + if state is None: + return None + bag = state.get(STATE_KEY) or {} + entry = bag.get(_TAKEOVER_SUBKEY) + return entry if isinstance(entry, dict) else None + + +def write_takeover(agent: Any, entry: Optional[Dict[str, Any]]) -> None: + """Record (or clear) the in-flight takeover marker. `None` clears.""" + state = getattr(agent, "state", None) + if state is None: + return + bag = dict(state.get(STATE_KEY) or {}) + if entry is None: + bag.pop(_TAKEOVER_SUBKEY, None) + else: + bag[_TAKEOVER_SUBKEY] = entry + try: + state.set(STATE_KEY, bag) + except ValueError: + logger.warning("browser: takeover marker not serializable; not persisted") + + +def control_state(agent: Any) -> Optional[Dict[str, Any]]: + """Who holds this conversation's browser, if this process knows. + + None when there is no session, or when it lives in another container — + process-local by design, because the thing it guards (the in-process idle + reaper) is process-local too. + """ entry = _read_state(agent) if not entry: return None live = _live.get(entry.get("sessionId", "")) - client = live.client if live else None - if client is None: + if live is None: return None - try: - return await asyncio.to_thread(client.generate_live_view_url) - except Exception: # noqa: BLE001 - logger.debug("browser: live view url failed", exc_info=True) + now = time.monotonic() + return { + **_describe(live), + "userControlled": live.user_controlled(now), + "deadlineAt": _deadline_iso(live, now), + } + + +def _deadline_iso(live: _LiveSession, now: float) -> Optional[str]: + """The takeover deadline as a wall-clock ISO instant. + + `control_deadline` is monotonic because that is what the reaper compares + against; the client needs wall clock. Converting at read time rather than + storing both keeps the one authoritative value monotonic, so a clock change + cannot un-expire a takeover. + """ + if live.control_deadline <= 0: return None + from datetime import datetime, timedelta, timezone + + remaining = max(live.control_deadline - now, 0) + return (datetime.now(timezone.utc) + timedelta(seconds=remaining)).isoformat() diff --git a/backend/src/agents/builtin_tools/browser/takeover_tool.py b/backend/src/agents/builtin_tools/browser/takeover_tool.py new file mode 100644 index 000000000..77d942ade --- /dev/null +++ b/backend/src/agents/builtin_tools/browser/takeover_tool.py @@ -0,0 +1,286 @@ +"""`request_user_login` — hand the browser to the user so they can sign in. + +The agent hits a login wall, calls this, and the turn pauses. The user drives +the real browser through a live view, signs in with their own hands, and hands +it back; the turn resumes with the session authenticated and the agent carries +on browsing. See `docs/specs/authenticated-web-assessment.md`. + +Why this is its own tool, not a `browse_web` action +--------------------------------------------------- +RBAC granularity in this codebase is exactly one `tool_id`: a role's +`grantedTools` holds catalog keys, those roll up through parents in +`_compute_effective_permissions`, and the result becomes `enabled_tool_ids` +for `ToolFilter`. There is no sub-tool gate. An *action* on `browse_web` would +therefore ship an interactive browser inside our AWS account to everyone who +can browse, students included. As a separate catalog entry it is granted on its +own, and a user without the grant never has it registered — so it never reaches +`toolConfig`, the model cannot offer it, and that user pays zero prefix tokens +for it (spec D1). + +Why an interrupt rather than a new endpoint +------------------------------------------- +The turn genuinely must stop: the agent has nothing to do until a human +finishes, and a turn that polls for login completion burns model calls to learn +nothing. `ask_user_question` already proved the pause/snapshot/resume path, and +Strands routes tool-raised and hook-raised interrupts through the same +`_stop_for_interrupts`, so the `PausedTurnSnapshot`, the resume route and the +breadcrumb need no special case (spec D1b). + +⚠️ **Strands re-executes an interrupted tool from the top on resume** +(`strands/types/interrupt.py`), so everything above the `interrupt()` call runs +twice. Two separate things keep that safe, and they are not interchangeable: +`session_pool.take_control` is idempotent, which prevents a second +`UpdateBrowserStream` flip; the takeover marker on `agent.state` preserves the +*descriptor* — the original deadline above all — so the resume is judged +against the window the user was actually given rather than a fresh one minted +on the way back in. The marker also keeps a resume that lands in a container +which has lost the session from calling `acquire` and silently starting a +second, unauthenticated browser. + +Credentials +----------- +Nothing here accepts a secret, and nothing should be added that does. The +entire point is that the password is typed into a real browser by the person it +belongs to — a credential passed as a tool argument would land in the prompt, +in AgentCore Memory and in the cacheable prefix, and be re-read on every +subsequent turn for the life of the conversation. + +Cost +---- +The tool spec is a constant in `toolConfig` for granted users (~200 tokens, +paid once per cache write). A complete login round trip adds one short line to +the conversation: the live view is a video stream the model never sees, the +login page and the user's keystrokes never enter the prompt, and no URL travels +anywhere near the model (spec D2). +""" + +from __future__ import annotations + +import logging +from datetime import datetime, timezone +from typing import Any, Dict, Optional + +from strands import tool +from strands.types.tools import ToolContext + +from apis.shared.browser_takeover import ( + BrowserSessionRef, + assert_no_url, + format_outcome, + parse_outcome, +) +from apis.shared.feature_flags import browser_takeover_enabled + +from . import session_pool + +logger = logging.getLogger(__name__) + +# Interrupt name. Scoped by toolUseId upstream — `ToolContext._interrupt_id` +# builds `v1:tool_call:{toolUseId}:{uuid5(name)}` — so a second takeover later +# in the same turn is a distinct interrupt. +INTERRUPT_NAME = "request_user_login" + +MAX_REASON_CHARS = 200 + + +def _error(text: str) -> Dict[str, Any]: + return {"content": [{"text": text}], "status": "error"} + + +def _success(text: str) -> Dict[str, Any]: + return {"content": [{"text": text}], "status": "success"} + + +def _now() -> datetime: + return datetime.now(timezone.utc) + + +def _deadline_lapsed(deadline_at: Optional[str]) -> bool: + """Whether a recorded deadline is past, grace included. + + Unparseable reads as *not* lapsed: the failure mode of guessing "expired" + is throwing away a sign-in the user actually completed, which is strictly + worse than trusting a client that just told us it finished. + """ + if not deadline_at: + return False + try: + deadline = datetime.fromisoformat(deadline_at) + except (TypeError, ValueError): + return False + if deadline.tzinfo is None: + deadline = deadline.replace(tzinfo=timezone.utc) + elapsed = (_now() - deadline).total_seconds() + return elapsed > session_pool.TAKEOVER_GRACE_SECONDS + + +@tool(context=True) +async def request_user_login( + tool_context: ToolContext, + reason: str, +) -> Dict[str, Any]: + """Hand the browser to the user so they can sign in, then continue. + + Use this when `browse_web` has reached a page that requires authentication + — a sign-in form, a paywall, an SSO redirect, an institutional subscription + wall — and the content behind it is what the task needs. The user takes + control of the same browser you are driving, signs in themselves, and hands + it back; your next `browse_web` call continues on the authenticated page. + + Never ask the user for a username, password, one-time code or any other + credential in your reply, and never type one into a page yourself. You + cannot be given credentials and you do not need them: that is what this + tool is for. + + Navigate to the page that needs the sign-in *before* calling this, so the + user arrives where they expect to be rather than on a blank tab. + + Call it once per sign-in. If the user declines or the window lapses, say + what you could not reach and carry on with what is public — do not ask + again in the same turn. + + Args: + reason: One short sentence, shown to the user, naming what you need + signed into and why. For example: "Sign in to JSTOR so I can check + whether the search results page is screen-reader accessible." + + Returns: + Whether the user signed in, declined, or let the window lapse. + """ + if not browser_takeover_enabled(): + return _error( + "Handing the browser to the user is disabled in this environment. " + "Continue with publicly reachable pages and say what you could not " + "get to." + ) + + agent = getattr(tool_context, "agent", None) + if agent is None: + return _error("❌ Browser takeover has no agent context.") + + reason_text = (reason or "").strip()[:MAX_REASON_CHARS] + if not reason_text: + return _error( + "❌ `reason` is required: tell the user what they are signing into " + "and why, in one sentence." + ) + + tool_use_id = (tool_context.tool_use or {}).get("toolUseId", "") + + # Resume lands here too (Strands re-runs the tool from the top), so a + # marker for *this* tool_use means control was already handed over on the + # first pass. Reusing the recorded descriptor rather than re-deriving one + # is what keeps the original deadline authoritative, and what stops a + # resume in a container that has lost the session from calling `acquire` + # and starting a second, unauthenticated browser. + pending = session_pool.read_takeover(agent) + resuming = bool(pending) and pending.get("toolUseId") == tool_use_id + + if resuming: + descriptor = dict(pending.get("ref") or {}) + else: + if not session_pool.has_session(agent): + # Handing over a browser that has never navigated anywhere gives + # the user a blank tab. Erroring costs nothing; the alternative is + # billing a browser session and pausing the turn to achieve it. + return _error( + "❌ There is no browser session yet. Use `browse_web` to " + "navigate to the page that needs the sign-in first, then call " + "this so the user arrives where they expect to be." + ) + + # Read the page before handing over: once the automation stream is + # DISABLED our CDP commands are refused at the service, so this is the + # last moment we can name the page the user is about to sign into. + target_url = await _current_url(agent) + try: + descriptor = await session_pool.take_control(agent) + except Exception as exc: # noqa: BLE001 - surface the reason, not a traceback + logger.error("browser: could not hand over control: %s", exc, exc_info=True) + return _error( + f"❌ Could not hand the browser to the user: {exc}. Continue " + "without the authenticated content." + ) + + descriptor["targetUrl"] = target_url + session_pool.write_takeover( + agent, + { + "toolUseId": tool_use_id, + "reason": reason_text, + "startedAt": _now().isoformat(), + "ref": descriptor, + }, + ) + + try: + ref = BrowserSessionRef.model_validate(descriptor) + except Exception as exc: # noqa: BLE001 + logger.error("browser: takeover descriptor is unusable: %s", exc) + await session_pool.release_control(agent) + session_pool.write_takeover(agent, None) + return _error( + "❌ The browser session could not be described for handover. " + "Continue without the authenticated content." + ) + + payload: Dict[str, Any] = { + "type": "browser_login_required", + "toolUseId": tool_use_id, + "reason": reason_text, + **ref.model_dump(by_alias=True, exclude_none=True), + } + # The rule from PR #1101, enforced rather than remembered: no presigned URL + # may travel on this interrupt. app-api mints one per request (spec D2). + assert_no_url(payload) + + logger.info( + "Pausing for browser sign-in: browser_session=%s tool_use_id=%s", + ref.browser_session_id, + tool_use_id, + ) + + # Raises InterruptException on the first pass; returns the client's payload + # when the turn resumes. Strands only treats a **non-None** response as an + # answer, so the SPA must always post an object — "Skip" sends + # `{"skipped": true}`, never null, or the interrupt would re-raise forever. + response = tool_context.interrupt(name=INTERRUPT_NAME, reason=payload) + + outcome, note = parse_outcome(response) + if outcome == "completed" and _deadline_lapsed(ref.deadline_at): + # The sign-in window closed and the session was released and made + # reapable, so we cannot honestly report an authenticated browser. + logger.info( + "browser: takeover for %s reported complete past its deadline", + ref.browser_session_id, + ) + outcome = "expired" + + # Always release, whatever the outcome: leaving the automation stream + # DISABLED would brick every later `browse_web` call in the conversation. + released = await session_pool.release_control(agent) + session_pool.write_takeover(agent, None) + if outcome == "completed" and not released: + logger.warning( + "browser: could not re-enable automation for %s after sign-in", + ref.browser_session_id, + ) + + return _success(format_outcome(outcome, note)) + + +async def _current_url(agent: Any) -> Optional[str]: + """The page the browser is parked on, for the user's benefit. + + Best-effort and deliberately quiet: this is a label on a prompt, not a + correctness input, and the CDP read happens after the automation stream is + already DISABLED, so a refusal here is expected rather than exceptional. + """ + try: + live = await session_pool.acquire(agent) + value = await live.cdp.evaluate("location.href") + except Exception: # noqa: BLE001 + logger.debug("browser: could not read current url for takeover", exc_info=True) + return None + text = str(value).strip() if value else "" + return text[:500] or None diff --git a/backend/src/agents/builtin_tools/document_read_tool.py b/backend/src/agents/builtin_tools/document_read_tool.py new file mode 100644 index 000000000..637cdf113 --- /dev/null +++ b/backend/src/agents/builtin_tools/document_read_tool.py @@ -0,0 +1,201 @@ +"""``document_read`` — page-range and pattern retrieval over the session's documents. + +The recovery half of ``docs/specs/document-context-offload.md`` (§4B). A +document the user attached lives in the model's context only until history +is restored (restore strips inline bytes; see +``TurnBasedSessionManager._strip_document_bytes``) or, later, until the +offload trigger swaps it for a digest. This tool is how the model gets the +part it needs back — a PDF page range as a native ``document`` block (full +fidelity: text layer plus page images), a regex over the text layer to find +the right pages, or bounded text for text-family documents. + +Design notes +------------ +* **Gated on session state, not on the tool picker.** The tool is built for + any session that has a readable document (``_build_document_tools`` in + ``apis/inference_api/chat/routes.py``), whatever the user's RBAC grants. + The ``workspace_files`` catalog key is granted to no prod role, so gating + there would ship the recovery path dark. Its id therefore stays OUT of + ``INJECTED_TOOL_IDS`` — same reasoning as the Memory-Space tools: the + governing capability is the user's own attachment. +* Identity is captured by closure (``make_document_read_tool``), never read + from ``invocation_state``. +* One call can never re-inject a whole document: page ranges are capped by + ``max_pages`` (default 8, hard cap 20), pattern results by match count, + text by the workspace read bound. +* A pattern scan is bounded in *time* as well as in output. The regex is + model-supplied, and a nested-quantifier pattern makes Python's backtracking + engine run exponentially — measured at 14 s for a 28-character line, against + extracted PDF lines of 60–100 — with no way to cancel a running + ``re.search``. ``catastrophic_pattern`` refuses that family up front (the + pattern is searched literally, and the payload says so) and + ``DOCUMENT_READ_PATTERN_BUDGET_SECONDS`` bounds the walk for anything it + does not catch. +* **Content-free record per call** in ``AgentCoreStack/Compaction`` + (``DocumentRead``, ``DocumentReadPages``, ``DocumentReadBytes``; properties + ``mode`` / ``format``) — the same namespace as the compaction cut and + tool-result offload records, so one dashboard explains what bounded a + session's prefix. Nothing from the document is ever emitted. +* Tool results carrying a native ``document`` block are exempt from the + tool-result offloader (``core/tool_result_offload.py``): offloading the + slice the model just asked for would defeat the read. +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, Optional + +from strands import tool + +from apis.shared.files.document_read import ( + DOCUMENT_READ_HARD_MAX_PAGES, + DOCUMENT_READ_MAX_PAGES, + DocumentReadResult, + list_session_documents, + read_document, +) +from apis.shared.files.workspace import WorkspaceError, WorkspaceStorageNotConfiguredError + +logger = logging.getLogger(__name__) + +#: The tool's name — also the census / ledger key and the offloader exemption. +DOCUMENT_READ_TOOL_NAME = "document_read" + +_NO_STORAGE_MESSAGE = ( + "❌ Document storage is not configured (S3_USER_FILES_BUCKET_NAME is not set on the runtime)." +) + + +def _error(text: str) -> Dict[str, Any]: + return {"content": [{"text": text}], "status": "error"} + + +def _success(result: DocumentReadResult) -> Dict[str, Any]: + content = [{"json": result.payload}] + if result.document_block is not None: + content.append(result.document_block) + content.extend(result.extra_blocks) + return {"content": content, "status": "success"} + + +def record_document_read(result: DocumentReadResult) -> None: + """One content-free EMF record per successful read. Never raises.""" + logger.info( + "document_read: mode=%s format=%s pages=%d bytes=%d", + result.mode, result.format, result.pages_returned, result.bytes_returned, + ) + try: + from apis.shared.observability.prompt_cache import prompt_cache_observability_enabled + from apis.shared.observability.emf import emit_emf_metrics + + if not prompt_cache_observability_enabled(): + return + emit_emf_metrics( + "AgentCoreStack/Compaction", + metrics={ + "DocumentRead": 1, + "DocumentReadPages": int(result.pages_returned), + "DocumentReadBytes": int(result.bytes_returned), + }, + properties={"mode": result.mode, "format": result.format}, + units={"DocumentReadPages": "Count", "DocumentReadBytes": "Bytes"}, + ) + except Exception as e: # noqa: BLE001 + logger.debug("document_read EMF skipped: %s", e) + + +def make_document_read_tool(session_id: str, user_id: str): + """Create a ``document_read`` tool bound to the given identity.""" + if not session_id or not user_id: + raise ValueError("document_read requires a session_id and a user_id") + + @tool(name=DOCUMENT_READ_TOOL_NAME) + async def document_read( + upload_id: str = "", + page_range: str = "", + pattern: str = "", + max_pages: int = DOCUMENT_READ_MAX_PAGES, + offset: int = 0, + ) -> Any: + """Read part of a document the user attached to this conversation. + + Use this when a document you need is no longer inline (its content + was replaced by a placeholder or digest after the conversation was + restored) or when you need specific pages of a long document. + + Modes, by argument: + - No arguments: list this conversation's readable documents with + their upload_id, format and size. + - upload_id + page_range (PDF only, e.g. "4-7"): returns those pages + as an attached document at full fidelity (text and page images), + at most max_pages per call. The attachment's own page k is + original page start+k-1 — cite original page numbers. + - upload_id + pattern: case-insensitive regex over the text; returns + matching lines with page numbers (PDF) or line/paragraph numbers + (text, Word), so you can then read the right pages. Use this first + on a long document instead of paging through it. + - upload_id alone: for a PDF, the page count and a per-page snippet + index; for text and Word documents, the text itself (bounded — + continue with offset = next_offset when truncated). + + Prefer pattern then a narrow page_range over reading many pages. + Spreadsheets are read with analyze_spreadsheet and presentations + with read_powerpoint_presentation, not here. + + Args: + upload_id: The document's id (from the listing, the attachment + note, or an earlier call). Empty to list documents. + page_range: 1-indexed inclusive page range for PDFs, "start-end" + or a single page "5". + pattern: Regular expression (case-insensitive) to search for. + Do not nest one unbounded quantifier inside another — write + "X+" rather than "(X+)+" — because such a pattern can take + exponential time; it is searched as literal text instead and + the result says so. + max_pages: Cap on pages returned by page_range (default 8, hard + cap 20). + offset: Byte offset to continue a truncated text read. + + Returns: + JSON metadata, plus the requested pages as an attached document + for page_range reads. + """ + try: + if not upload_id: + listing = await list_session_documents(user_id, session_id) + listing["hint"] = ( + "Call again with upload_id plus page_range (PDF), pattern, or neither." + ) + result = DocumentReadResult(mode="list", payload=listing) + else: + result = await read_document( + user_id, + session_id, + upload_id, + page_range=page_range or None, + pattern=pattern or None, + max_pages=max_pages, + offset=offset, + ) + except WorkspaceStorageNotConfiguredError: + return _error(_NO_STORAGE_MESSAGE) + except WorkspaceError as exc: + return _error(f"❌ {exc}") + except Exception as exc: # noqa: BLE001 - surface conversationally, never raise through the loop + logger.error("document_read error: %s", exc, exc_info=True) + return _error(f"❌ Failed to read document '{upload_id}': {exc}") + + record_document_read(result) + return _success(result) + + return document_read + + +__all__ = [ + "DOCUMENT_READ_HARD_MAX_PAGES", + "DOCUMENT_READ_MAX_PAGES", + "DOCUMENT_READ_TOOL_NAME", + "make_document_read_tool", + "record_document_read", +] diff --git a/backend/src/agents/builtin_tools/excel_spreadsheet_tool.py b/backend/src/agents/builtin_tools/excel_spreadsheet_tool.py index a7c5cd5e5..bfc13e8d1 100644 --- a/backend/src/agents/builtin_tools/excel_spreadsheet_tool.py +++ b/backend/src/agents/builtin_tools/excel_spreadsheet_tool.py @@ -277,6 +277,28 @@ async def create_excel_spreadsheet( ws.append(['Q1', 100]) ws.append(['Q2', 120]) ws['B4'] = '=SUM(B2:B3)' + # Freeze the header so it stays visible while + # scrolling -- do this on EVERY sheet that has a + # header row ('A2' freezes row 1, 'B2' also freezes + # column A): + ws.freeze_panes = 'A2' + # Give every number a format. Without one, Excel + # shows raw values -- 1234567.891 instead of + # $1,234,568, and 0.1834 instead of 18.3%: + for cell in ws['B'][1:]: + cell.number_format = '#,##0' + # Common formats: '#,##0.00' (2dp), '$#,##0' + # (currency), '0.0%' (percent -- store 0.183, not + # 18.3), 'yyyy-mm-dd' (date). + + Formulas are written WITHOUT a cached result, because + openpyxl does not evaluate them. Excel fills them in on + open, but until then the cell reads as empty to + everything else -- including read_excel_spreadsheet and + the in-app preview. So when a total is meant to be read + back or shown, compute it in Python and write the value + (optionally alongside the formula on another cell): + ws['B4'] = sum(r[1] for r in rows) Example (add a second sheet + a bar chart): ws2 = wb.create_sheet('Chart') diff --git a/backend/src/agents/builtin_tools/powerpoint_presentation_tool.py b/backend/src/agents/builtin_tools/powerpoint_presentation_tool.py index 3dedddadc..d2dbf0419 100644 --- a/backend/src/agents/builtin_tools/powerpoint_presentation_tool.py +++ b/backend/src/agents/builtin_tools/powerpoint_presentation_tool.py @@ -368,6 +368,14 @@ async def create_powerpoint_presentation( Inches(8), Inches(2)).table tbl.cell(0, 0).text = 'Quarter'; tbl.cell(0, 1).text = 'Revenue' + Speaker notes -- add them to every content slide unless + the user says otherwise. They are what the presenter + actually says, so keep the detail there and the slide + itself sparse: + slide.notes_slide.notes_text_frame.text = ( + 'Revenue grew 15% on enterprise renewals; ' + 'call out the churn improvement before moving on.') + A matplotlib chart image: import matplotlib.pyplot as plt plt.figure(figsize=(8, 4.5)) diff --git a/backend/src/agents/builtin_tools/word_document_tool.py b/backend/src/agents/builtin_tools/word_document_tool.py index 3c2e45b56..eec5b9c76 100644 --- a/backend/src/agents/builtin_tools/word_document_tool.py +++ b/backend/src/agents/builtin_tools/word_document_tool.py @@ -269,6 +269,12 @@ async def create_word_document( table = doc.add_table(rows=2, cols=2) table.style = 'Light Grid Accent 1' table.rows[0].cells[0].text = 'Quarter' + # ALWAYS bold a header row this way. A table style's + # own header emphasis is conditional formatting that + # not every viewer applies, so set it on the runs: + for cell in table.rows[0].cells: + for run in cell.paragraphs[0].runs: + run.bold = True To embed a chart, save a PNG with matplotlib then insert it: import matplotlib.pyplot as plt @@ -278,6 +284,13 @@ async def create_word_document( plt.close() doc.add_picture('chart.png', width=Inches(6)) + Start a major section on a fresh page with an explicit + break -- python-docx does no layout, so a document that + should be several pages contains no page boundaries at + all unless you add them: + doc.add_page_break() + doc.add_heading('Appendix A', level=1) + document_name: File name WITHOUT extension (.docx is added automatically). Use only letters, numbers, hyphens, and underscores (e.g. "sales-report", "Q4_analysis"). diff --git a/backend/src/agents/main_agent/base_agent.py b/backend/src/agents/main_agent/base_agent.py index 281d45c27..8992769c1 100644 --- a/backend/src/agents/main_agent/base_agent.py +++ b/backend/src/agents/main_agent/base_agent.py @@ -14,6 +14,7 @@ from agents.main_agent.session import SessionFactory from agents.main_agent.session.hooks import ( AgentStatusHook, + ContextLedgerHook, ToolCensusHook, DisplayTextHook, SteeringHook, @@ -32,6 +33,8 @@ from agents.main_agent.streaming import StreamCoordinator from apis.shared.tools.scoped_ids import base_tool_id +from apis.shared.observability.build_stages import mark_stage + logger = logging.getLogger(__name__) @@ -159,12 +162,17 @@ def __init__( # prior turns; only the system-prompt date line shifts. self._construction_snapshot["system_prompt"] = system_prompt + # Sub-stages of `agent_build` (docs/specs/turn-latency-preamble.md). + # A no-op unless the inference-api turn path installed a recorder. + mark_stage("prompt") + # Initialize tool registry and filter self.tool_registry = create_default_registry() self.tool_filter = ToolFilter(self.tool_registry) # Register external MCP tool IDs from enabled tools self._register_external_mcp_tools() + mark_stage("registry") # Initialize gateway integration self.gateway_integration = GatewayIntegration() @@ -176,12 +184,16 @@ def __init__( self.session_manager = SessionFactory.create_session_manager( session_id=session_id, user_id=self.user_id, caching_enabled=self.model_config.caching_enabled ) + # Conversation restore from AgentCore Memory happens in here, so this + # is a prime suspect for the cold build and has never been timed. + mark_stage("session_mgr") # Initialize streaming coordinator self.stream_coordinator = StreamCoordinator() # Create the agent (subclass-specific) self._create_agent() + mark_stage("finalize") @abstractmethod def _create_agent(self) -> None: @@ -329,8 +341,10 @@ def _create_hooks(self) -> List: # Per-turn context-token attribution (system / tools / messages). # Best-effort; computes the breakdown on BeforeModelCallEvent and # stashes it on the agent for the stream coordinator to surface on the - # final metadata SSE event. - hooks.append(ContextAttributionHook()) + # final metadata SSE event. The session id keys a process-level memo + # of the stable split, so an Agent rebuilt for this session (cache + # bypass, @-mention, memory binding) adopts it instead of re-counting. + hooks.append(ContextAttributionHook(session_id=self.session_id)) # Live narration of what the agent is doing (model call / tool call # boundaries) plus Strands-measured per-tool durations. Held on the @@ -350,6 +364,13 @@ def _create_hooks(self) -> List: self.tool_census_hook = ToolCensusHook() hooks.append(self.tool_census_hook) + # Per-model-call context ledger: the conversation window's cumulative + # trim count and the compaction decisions taken since the previous + # call. Same shape and lifecycle as the census — read per call at + # turn end, persisted on the cost row, off with the same kill switch. + self.context_ledger_hook = ContextLedgerHook() + hooks.append(self.context_ledger_hook) + # Per-model-call prompt-cache prefix fingerprints (toolConfig / # system prompt / history hashes). Best-effort; the stream # coordinator persists them on each call's metadata row so avoidable diff --git a/backend/src/agents/main_agent/chat_agent.py b/backend/src/agents/main_agent/chat_agent.py index 2e72e608f..f29c01369 100644 --- a/backend/src/agents/main_agent/chat_agent.py +++ b/backend/src/agents/main_agent/chat_agent.py @@ -6,12 +6,15 @@ """ import logging +import os from typing import Any, AsyncGenerator, Dict, List, Optional from agents.main_agent.base_agent import BaseAgent from agents.main_agent.core import AgentFactory from agents.main_agent.skills.strands_mapping import build_skills_runtime +from apis.shared.observability.build_stages import mark_stage + logger = logging.getLogger(__name__) @@ -51,14 +54,18 @@ def _create_agent(self) -> None: """Create Strands Agent with filtered tools, hooks, and skills plugin.""" try: tools = self._build_filtered_tools() + # External MCP pre-flight lives in here — the spec's standing + # (and unverified) hypothesis for the cold build. + mark_stage("tools") hooks = self._create_hooks() + mark_stage("hooks") # Skills disclosure: the AgentSkills plugin injects the catalog + # activation tool; read_skill_file is the S3 adapter for reference # files (added after tool filtering — it is infrastructure, not an # RBAC-gated tool, and is implicitly scoped to the turn's skills). plugin, read_skill_file = build_skills_runtime(self._accessible_skill_ids) - plugins = [plugin] if plugin else None + plugins = [plugin] if plugin else [] if plugin: tools = list(tools) + [read_skill_file] logger.info( @@ -66,6 +73,30 @@ def _create_agent(self) -> None: len(self._accessible_skill_ids or []), ) + # Tool-result offload at intake (compaction PR-4): oversized tool + # results become a bounded preview + retrieval references before + # they enter the cacheable prefix. Fail-open: None when off or + # unconfigured. The plugin registers retrieve_offloaded_content + # itself — one stable spec in toolConfig, not an RBAC-gated tool, + # like read_skill_file above. + from agents.main_agent.core.tool_result_offload import build_tool_result_offloader + + offload_session = getattr(self, "session_id", None) + offload_user = getattr(self, "user_id", None) + offloader = ( + build_tool_result_offloader( + session_id=offload_session, + user_id=offload_user, + region=os.environ.get("AWS_REGION"), + ) + if offload_session and offload_user + else None + ) + if offloader is not None: + plugins.append(offloader) + plugins = plugins or None + + mark_stage("plugins") self.agent = AgentFactory.create_agent( model_config=self.model_config, system_prompt=self._system_prompt_for(tools), @@ -130,6 +161,7 @@ async def stream_async( continue_truncated: bool = False, turn_agent_id: Optional[str] = None, turn_lease: Any = None, + turn_started_at: Optional[float] = None, ) -> AsyncGenerator[str, None]: """ Stream agent responses. @@ -195,5 +227,6 @@ async def stream_async( original_message=original_message, turn_agent_id=turn_agent_id, turn_lease=turn_lease, + turn_started_at=turn_started_at, ): yield event diff --git a/backend/src/agents/main_agent/config/constants.py b/backend/src/agents/main_agent/config/constants.py index d19cf0a88..9ed45ba46 100644 --- a/backend/src/agents/main_agent/config/constants.py +++ b/backend/src/agents/main_agent/config/constants.py @@ -28,6 +28,45 @@ class EnvVars: COMPACTION_PROTECTED_TURNS = "AGENTCORE_MEMORY_COMPACTION_PROTECTED_TURNS" COMPACTION_MAX_TOOL_CONTENT_LENGTH = "AGENTCORE_MEMORY_COMPACTION_MAX_TOOL_CONTENT_LENGTH" COMPACTION_CACHE_TTL_SECONDS = "AGENTCORE_MEMORY_COMPACTION_CACHE_TTL_SECONDS" + # Model-relative thresholds (docs/specs/compaction-model-relative-thresholds.md). + # The kill switch reverts to the fixed TOKEN_THRESHOLD and the legacy + # turn-count cut; the ratios/caps shape the per-model ceiling and floor. + COMPACTION_MODEL_RELATIVE_ENABLED = "AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED" + COMPACTION_CEILING_RATIO = "AGENTCORE_MEMORY_COMPACTION_CEILING_RATIO" + COMPACTION_CEILING_CAP_TOKENS = "AGENTCORE_MEMORY_COMPACTION_CEILING_CAP_TOKENS" + COMPACTION_FLOOR_RATIO = "AGENTCORE_MEMORY_COMPACTION_FLOOR_RATIO" + COMPACTION_HARD_CEILING_RATIO = "AGENTCORE_MEMORY_COMPACTION_HARD_CEILING_RATIO" + COMPACTION_HARD_CEILING_MULTIPLIER = "AGENTCORE_MEMORY_COMPACTION_HARD_CEILING_MULTIPLIER" + # Strands conversation-manager window (messages). Our compaction owns + # history size; the SDK's default 40-message SlidingWindowConversationManager + # would otherwise slide the front of the list every turn past 40 messages + # (a prefix re-write per turn, and it moves the coordinates the compaction + # checkpoint is expressed in). Setting this to 40 restores the SDK default. + CONVERSATION_WINDOW_MESSAGES = "AGENTCORE_CONVERSATION_WINDOW_MESSAGES" + # Bounded compaction summary (spiral spec PR-2 / + # compaction-model-relative-thresholds.md §3.6). Budget in tokens; the + # re-summarize call is a Nova Micro side-channel with its own kill switch. + COMPACTION_SUMMARY_TOKEN_BUDGET = "AGENTCORE_MEMORY_COMPACTION_SUMMARY_TOKEN_BUDGET" + COMPACTION_SUMMARY_MODEL_ENABLED = "AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ENABLED" + COMPACTION_SUMMARY_MODEL_ID = "AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ID" + # Paid-when-free scheduling (thresholds spec §3.5): a cut is computed and + # persisted as PENDING post-turn and applied to the live list pre-call only + # when the prefix re-write is free (cache expired, model/agent switched) + # or unavoidable (hard ceiling). "false" applies cuts immediately (PR-1/2). + COMPACTION_DEFERRED_APPLY_ENABLED = "AGENTCORE_MEMORY_COMPACTION_DEFERRED_APPLY_ENABLED" + # Tool-result offload at intake (thresholds spec §3.6 / PR-4): oversized + # tool results are stored in S3 (user-files bucket, per-session prefix) and + # replaced in context by a bounded preview + retrieval references before + # they ever enter the cacheable prefix. Strands' vended ContextOffloader. + TOOL_RESULT_OFFLOAD_ENABLED = "AGENTCORE_TOOL_RESULT_OFFLOAD_ENABLED" + TOOL_RESULT_OFFLOAD_MAX_TOKENS = "AGENTCORE_TOOL_RESULT_OFFLOAD_MAX_TOKENS" + TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS = "AGENTCORE_TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS" + # Selective long cache TTL on the STATIC prefix (thresholds spec §3.6, + # PR-5). "1h" puts a 1-hour TTL on the tools and system cachePoints only; + # the message-level point stays at Bedrock's 5-minute default. Unset/empty + # = today's shape. An experiment arm: default OFF until the live probe + # (scripts/probe_static_prefix_ttl.py) and the cost rows say it pays. + PROMPT_CACHE_STATIC_PREFIX_TTL = "AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL" # --- Restored-history repair --- # Kill switch for the restore-time tool-pairing/alternation repair @@ -112,6 +151,50 @@ class Defaults: COMPACTION_MAX_TOOL_CONTENT_LENGTH = 500 # Bedrock prompt-cache TTL (seconds); see CompactionConfig.cache_ttl_seconds COMPACTION_CACHE_TTL_SECONDS = 300 + # Model-relative compaction policy — see + # docs/specs/compaction-model-relative-thresholds.md §3.1 for the table + # these produce. ceiling = min(window * CEILING_RATIO, CEILING_CAP_TOKENS); + # floor = ceiling * FLOOR_RATIO; hard = min(window * HARD_CEILING_RATIO, + # ceiling * HARD_CEILING_MULTIPLIER). COMPACTION_TOKEN_THRESHOLD above is + # the ceiling used when the model's window is unknown. + COMPACTION_MODEL_RELATIVE_ENABLED = True + COMPACTION_CEILING_RATIO = 0.5 + # 100k, not 200k: the 2026-09-15 replay of 20 heavy Sonnet 5 sessions + # priced a 200k/50k policy 43% above 100k/25k on the input side, because + # 36% of cache-write dollars are cold re-writes after a >5 min pause and + # their size is the context at the pause. Raise only on evidence from + # compaction_forced + the cost anatomy (spec §3.1). + COMPACTION_CEILING_CAP_TOKENS = 100_000 + COMPACTION_FLOOR_RATIO = 0.25 + COMPACTION_HARD_CEILING_RATIO = 0.7 + COMPACTION_HARD_CEILING_MULTIPLIER = 1.5 + # Effectively "never trim proactively" — compaction decides what leaves the + # prompt. Overflow recovery (reduce_context on ContextWindowOverflow) still + # works at any window size. + CONVERSATION_WINDOW_MESSAGES = 2000 + # 8k tokens ≈ 32k chars: a third of the 25k floor, so a bounded summary + # can never by itself hold a session above the ceiling (the incident's + # summary was 40k tokens against a 100k threshold). Same figure the admin + # SUMMARY_OVER_BUDGET diagnosis reads. + COMPACTION_SUMMARY_TOKEN_BUDGET = 8_000 + COMPACTION_SUMMARY_MODEL_ENABLED = True + # Same cheap model as the title and tool-batch side-channels. + COMPACTION_SUMMARY_MODEL_ID = "us.amazon.nova-micro-v1:0" + COMPACTION_DEFERRED_APPLY_ENABLED = True + # Tool-result offload gate. 4k is well under the 25k compaction floor, so a + # protected tail of a few big results can no longer hold a session above + # the ceiling on its own; the 1k preview keeps the part of a result models + # actually quote (headers, first rows, the first error). + TOOL_RESULT_OFFLOAD_ENABLED = True + TOOL_RESULT_OFFLOAD_MAX_TOKENS = 4_000 + TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS = 1_000 + TOOL_RESULT_OFFLOAD_S3_PREFIX = "compaction-offload" + # Default OFF, deliberately against the flags-default-on house style: a + # caching default adopted on inspection alone has already shipped wrong + # once (#954 measured 57% more expensive live before #956 reverted it). + # The 1h write premium is 2x base vs 1.25x at 5m, so this is a bet on the + # gap distribution that has to be measured, not read off the source. + PROMPT_CACHE_STATIC_PREFIX_TTL = "" # --- DynamoDB Tables --- DYNAMODB_QUOTA_TABLE = "UserQuotas" diff --git a/backend/src/agents/main_agent/core/agent_factory.py b/backend/src/agents/main_agent/core/agent_factory.py index 5a0b04e67..180a52697 100644 --- a/backend/src/agents/main_agent/core/agent_factory.py +++ b/backend/src/agents/main_agent/core/agent_factory.py @@ -5,13 +5,14 @@ import logging from typing import List, Optional, Any from strands import Agent +from strands.agent.conversation_manager import SlidingWindowConversationManager from strands.models import BedrockModel from strands.models.openai import OpenAIModel from strands.models.gemini import GeminiModel from strands.tools.executors import SequentialToolExecutor from agents.main_agent.core.bedrock_count_tokens import CountTokensBedrockModel from agents.main_agent.core.model_config import ModelConfig, ModelProvider -from agents.main_agent.config.constants import EnvVars +from agents.main_agent.config.constants import EnvVars, Defaults from apis.shared.models.bedrock_responses import build_bedrock_responses_model from apis.shared.models.mantle import build_mantle_model from apis.shared.models.usage_normalization import usage_normalized @@ -291,6 +292,13 @@ def create_agent( # on a NON-Anthropic model this block is passed through untouched and # Bedrock rejects the call with AccessDeniedException. # + # PR-5 (thresholds spec §3.6): the point is placed TTL-less on purpose. + # With AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h, ModelConfig sets + # CacheConfig(system_prompt_ttl="1h", tools_ttl="1h") and upstream's + # _apply_system_cache_ttl rewrites THIS point's ttl ("an explicit + # system_prompt_ttl string is honored as written"); the tools point + # gets its own. Flag unset → no ttl key anywhere → today's bytes. + # # RE-VERIFY BEFORE ANY BUMP PAST 1.55.0. This is a statement about # upstream internals and it has already rotted once. Re-check # _should_cache_system's guard, CacheConfig.system_prompt_ttl's @@ -318,9 +326,38 @@ def create_agent( tools=tools, tool_executor=SequentialToolExecutor(), session_manager=session_manager, + conversation_manager=AgentFactory.build_conversation_manager(), hooks=hooks if hooks else None, plugins=plugins if plugins else None, retry_strategy=retry_strategy, ) return agent + + @staticmethod + def build_conversation_manager() -> SlidingWindowConversationManager: + """The Strands conversation manager for the chat agent. + + Left unset, Strands installs ``SlidingWindowConversationManager()`` with + a **40-message** window and runs it after every event-loop cycle. Past + 40 messages that slides the front of ``agent.messages`` every turn, + which (a) re-writes the whole cached prefix each turn — the 2026-09-15 + prod cost audit saw fingerprint ``messageCount`` pinned at 39–41 with + every turn reading only tools+system — and (b) moves the list our + compaction checkpoint is expressed in (spiral-spec D3, ANCHOR_MISMATCH + on 14 of 20 audited sessions). History size is ``TurnBasedSessionManager``'s + job (docs/specs/compaction-model-relative-thresholds.md), so the window + is set large enough never to trim on its own. The manager is kept + (rather than ``NullConversationManager``) because its ``reduce_context`` + is the only ``ContextWindowOverflowException`` recovery in the stack, + and that path does not depend on the window size. + + ``AGENTCORE_CONVERSATION_WINDOW_MESSAGES=40`` restores the SDK default. + """ + raw = os.environ.get(EnvVars.CONVERSATION_WINDOW_MESSAGES, "").strip() + try: + window = int(raw) if raw else Defaults.CONVERSATION_WINDOW_MESSAGES + except ValueError: + window = Defaults.CONVERSATION_WINDOW_MESSAGES + window = max(2, window) + return SlidingWindowConversationManager(window_size=window, should_truncate_results=True) diff --git a/backend/src/agents/main_agent/core/bedrock_count_tokens.py b/backend/src/agents/main_agent/core/bedrock_count_tokens.py index 9852f57ce..a5fbdf564 100644 --- a/backend/src/agents/main_agent/core/bedrock_count_tokens.py +++ b/backend/src/agents/main_agent/core/bedrock_count_tokens.py @@ -1,32 +1,75 @@ -"""BedrockModel variant that makes native token counting work for our models. - -Bedrock's CountTokens API rejects cross-region inference-profile model ids -(``us.anthropic.…`` / ``eu.…`` / ``apac.…`` / ``us-gov.…``) with a misleading -``ValidationException: The provided model doesn't support counting tokens.`` — -even though on-demand invocation of the newer Claude models *requires* the -inference-profile id. CountTokens only accepts the base foundation-model id -(``anthropic.…``). The inference profile is pure cross-region routing, so the -base-id count is exact, not an approximation. - -This subclass de-prefixes the model id for the CountTokens call only; -invocation (``stream`` / ``structured_output``) keeps the profile id untouched. -Because Strands' per-turn estimate (``_estimate_input_tokens`` → -``BeforeModelCallEvent.projected_input_tokens``) routes through -``count_tokens``, making it authoritative improves proactive context-compaction -decisions in addition to feeding the context-attribution hook — both stop -relying on the chars/4 heuristic. +"""BedrockModel variant that makes native token counting work for our models, +and keeps a throttled count from ever stalling the reply. + +Two problems, one subclass. + +**Model id.** Bedrock's CountTokens API rejects cross-region inference-profile +model ids (``us.anthropic.…`` / ``eu.…`` / ``apac.…`` / ``us-gov.…``) with a +misleading ``ValidationException: The provided model doesn't support counting +tokens.`` — even though on-demand invocation of the newer Claude models +*requires* the inference-profile id. CountTokens only accepts the base +foundation-model id (``anthropic.…``). The inference profile is pure +cross-region routing, so the base-id count is exact, not an approximation. +This subclass passes the base id to CountTokens explicitly; invocation +(``stream`` / ``structured_output``) keeps the profile id untouched. It never +mutates ``self.config`` — an earlier version swapped ``config["model_id"]`` +for the duration of the call, which is only safe while a count and a stream +can never overlap on one instance. That stops being true the moment any count +runs concurrently with the model call, so the swap is gone. + +**Latency under throttling.** With ``use_native_token_count`` on, Strands +awaits one CountTokens call in front of *every* model call +(``event_loop._estimate_input_tokens`` → ``BeforeModelCallEvent +.projected_input_tokens``). CountTokens has its own request-rate quota, +separate from the model's token quota, and under load it throttles before the +model does. A throttled count is harmless in itself — Strands falls back to +the chars/4 heuristic — but the SDK's shared ``bedrock-runtime`` client +retries with backoff first, so the *reply* waited out that backoff (p95 5.8 s +measured on a 40k-user load run, docs/specs/load-test-assessment-2026-09.md +§1 fix 3). Counting therefore goes through a dedicated client with a single +attempt and a short read timeout: a throttle costs at most one failed request +before the heuristic, and a healthy count is unchanged. + +Because Strands' per-turn estimate routes through ``count_tokens``, making it +authoritative improves proactive context-compaction decisions in addition to +feeding the context-attribution hook — both stop relying on the chars/4 +heuristic. """ +import asyncio +import logging +import os import re +from typing import Any, Optional +import boto3 +from botocore.config import Config as BotocoreConfig +from botocore.exceptions import ClientError from strands.models import BedrockModel +from strands.models import bedrock as _strands_bedrock +from strands.models.model import Model from strands.types.content import Messages, SystemContentBlock +from strands.types.exceptions import ProviderTokenCountError from strands.types.tools import ToolSpec +logger = logging.getLogger(__name__) + # Cross-region inference-profile geography prefixes. Closed set per AWS — we # only strip these exact codes so a real model id is never mangled. _INFERENCE_PROFILE_PREFIX = re.compile(r"^(us|eu|apac|us-gov)\.") +# Bound on a single CountTokens attempt. Read + connect timeout, seconds. A +# healthy count answers in well under a second; anything slower is the quota +# talking, and the heuristic is a better use of the time. +COUNT_TOKENS_TIMEOUT_ENV = "COUNT_TOKENS_TIMEOUT_SECONDS" +COUNT_TOKENS_TIMEOUT_DEFAULT = 2.0 + +# Attempts per count, including the first. 1 = never retry a throttle. +COUNT_TOKENS_MAX_ATTEMPTS_ENV = "COUNT_TOKENS_MAX_ATTEMPTS" +COUNT_TOKENS_MAX_ATTEMPTS_DEFAULT = 1 + +_THROTTLE_CODES = frozenset({"ThrottlingException", "TooManyRequestsException", "ServiceQuotaExceededException"}) + def base_foundation_model_id(model_id: str) -> str: """Return the base foundation-model id for ``model_id``. @@ -39,14 +82,84 @@ def base_foundation_model_id(model_id: str) -> str: return _INFERENCE_PROFILE_PREFIX.sub("", model_id, count=1) +def count_tokens_timeout_seconds() -> float: + """Per-attempt timeout for CountTokens, from the environment.""" + raw = os.environ.get(COUNT_TOKENS_TIMEOUT_ENV, "") + try: + value = float(raw) if raw else COUNT_TOKENS_TIMEOUT_DEFAULT + except ValueError: + value = COUNT_TOKENS_TIMEOUT_DEFAULT + return value if value > 0 else COUNT_TOKENS_TIMEOUT_DEFAULT + + +def count_tokens_max_attempts() -> int: + """Attempts per CountTokens call (including the first), from the environment.""" + raw = os.environ.get(COUNT_TOKENS_MAX_ATTEMPTS_ENV, "") + try: + value = int(raw) if raw else COUNT_TOKENS_MAX_ATTEMPTS_DEFAULT + except ValueError: + value = COUNT_TOKENS_MAX_ATTEMPTS_DEFAULT + return value if value >= 1 else COUNT_TOKENS_MAX_ATTEMPTS_DEFAULT + + +def build_count_tokens_client_config() -> BotocoreConfig: + """Botocore config for the dedicated CountTokens client. + + Exposed so the bound is testable without a network: one attempt by default, + a short read/connect timeout, and the same ``strands-agents`` user-agent + marker the SDK's own client carries so the calls stay attributable. + """ + timeout = count_tokens_timeout_seconds() + return BotocoreConfig( + retries={"total_max_attempts": count_tokens_max_attempts(), "mode": "standard"}, + read_timeout=timeout, + connect_timeout=timeout, + user_agent_extra="strands-agents count-tokens", + ) + + class CountTokensBedrockModel(BedrockModel): - """``BedrockModel`` that counts tokens against the base foundation-model id. + """``BedrockModel`` that counts tokens against the base foundation-model id + through a bounded, dedicated client. See the module docstring for why the inference-profile id can't be used for - CountTokens. Everything else (invocation, config, streaming) is inherited - unchanged. + CountTokens and why the count must never wait out a retry backoff. + Everything else (invocation, config, streaming) is inherited unchanged. """ + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._count_client: Any = None + + def _get_count_client(self) -> Any: + """The dedicated CountTokens client, built on first use. + + Same region and endpoint as the SDK's invocation client, default + credential chain (the factory passes no ``boto_session``), and the + bounded config above. Lazy so a model that never counts (native + counting off, or a model on the skip list) never builds it. + """ + if self._count_client is None: + meta = self.client.meta + self._count_client = boto3.client( + "bedrock-runtime", + region_name=meta.region_name, + endpoint_url=meta.endpoint_url, + config=build_count_tokens_client_config(), + ) + return self._count_client + + async def _heuristic_count( + self, + messages: Messages, + tool_specs: Optional[list[ToolSpec]], + system_prompt: Optional[str], + system_prompt_content: Optional[list[SystemContentBlock]], + ) -> int: + """The SDK's provider-agnostic chars/4 estimate — what ``BedrockModel`` + itself falls back to.""" + return await Model.count_tokens(self, messages, tool_specs, system_prompt, system_prompt_content) + async def count_tokens( self, messages: Messages, @@ -54,27 +167,75 @@ async def count_tokens( system_prompt: str | None = None, system_prompt_content: list[SystemContentBlock] | None = None, ) -> int: - """Count tokens using the de-prefixed base model id. - - The SDK's native path reads ``self.config["model_id"]`` for the - CountTokens ``modelId`` arg, so we swap to the base id for the duration - of the count and restore it in ``finally``. Token counting and - invocation never overlap on a single model instance — the model is - per-session and the event loop awaits the estimate before streaming — - so the swap window is safe, and the profile id is always restored - before any ``stream`` call. + """Count tokens natively against the de-prefixed base model id. + + Mirrors ``BedrockModel.count_tokens`` (native-flag gate, per-model skip + cache on AccessDenied / unsupported, heuristic fallback on any failure) + with two differences: the request goes through the bounded dedicated + client, and the model id is passed to the API rather than written into + ``self.config``. """ - original_id = self.config["model_id"] - base_id = base_foundation_model_id(original_id) - if base_id == original_id: - return await super().count_tokens( - messages, tool_specs, system_prompt, system_prompt_content - ) + if self.config.get("use_native_token_count") is not True: + return await self._heuristic_count(messages, tool_specs, system_prompt, system_prompt_content) + + profile_id: str = self.config["model_id"] + base_id = base_foundation_model_id(profile_id) + if base_id in _strands_bedrock._SKIP_COUNT_TOKENS_MODELS: + return await self._heuristic_count(messages, tool_specs, system_prompt, system_prompt_content) - self.config["model_id"] = base_id try: - return await super().count_tokens( - messages, tool_specs, system_prompt, system_prompt_content + if system_prompt and system_prompt_content is None: + system_prompt_content = [{"text": system_prompt}] + + request = self.format_request(messages, tool_specs, system_prompt_content) + converse_input: dict[str, Any] = { + key: request[key] for key in ("messages", "system", "toolConfig") if key in request + } + + response = await asyncio.to_thread( + self._get_count_client().count_tokens, + modelId=base_id, + input={"converse": converse_input}, + ) + input_tokens = response.get("inputTokens") + if input_tokens is None: + raise ProviderTokenCountError("Bedrock count_tokens returned None for inputTokens") + logger.debug("model_id=<%s>, total_tokens=<%d> | native token count", base_id, input_tokens) + return int(input_tokens) + except ClientError as e: + code = e.response.get("Error", {}).get("Code") + if code == "AccessDeniedException": + logger.warning( + "model_id=<%s> | bedrock:CountTokens permission denied, falling back to heuristic estimation: %s", + base_id, + e, + ) + _strands_bedrock._SKIP_COUNT_TOKENS_MODELS.add(base_id) + elif code == "ValidationException" and "doesn't support counting tokens" in str(e): + logger.debug( + "model_id=<%s> | model does not support CountTokens, caching for future calls," + " falling back to estimation", + base_id, + ) + _strands_bedrock._SKIP_COUNT_TOKENS_MODELS.add(base_id) + elif code in _THROTTLE_CODES: + # Expected under load. Not retried by design — the heuristic + # is the bounded path and the reply must not wait. + logger.info( + "model_id=<%s> code=<%s> | CountTokens throttled, using heuristic estimate this call", + base_id, + code, + ) + else: + logger.debug( + "model_id=<%s>, error=<%s> | native token counting failed, falling back to estimation", + base_id, + e, + ) + except Exception as e: # noqa: BLE001 - counting must never break a turn + logger.debug( + "model_id=<%s>, error=<%s> | native token counting failed, falling back to estimation", + base_id, + e, ) - finally: - self.config["model_id"] = original_id + return await self._heuristic_count(messages, tool_specs, system_prompt, system_prompt_content) diff --git a/backend/src/agents/main_agent/core/model_config.py b/backend/src/agents/main_agent/core/model_config.py index 8ec486b89..f6a702582 100644 --- a/backend/src/agents/main_agent/core/model_config.py +++ b/backend/src/agents/main_agent/core/model_config.py @@ -285,6 +285,25 @@ def from_env(cls) -> "RetryConfig": ) +# Bedrock's long cache TTL. Only "1h" is a change from the default; anything +# else (unset, "5m", garbage) means "today's shape" — no ttl key on any point, +# which is what keeps the static prefix bytes identical across the flip. +LONG_CACHE_TTL = "1h" + + +def static_prefix_cache_ttl() -> Optional[str]: + """The long TTL to put on the tools + system cachePoints, or ``None``. + + Read from ``AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL`` at agent + construction (a cached agent keeps the arm it was built under). See + docs/specs/compaction-model-relative-thresholds.md §3.6 PR-5 for the + economics: 2x write premium on the static segments in exchange for + reading them, rather than re-writing them, on every 5–60 minute pause. + """ + raw = os.environ.get(EnvVars.PROMPT_CACHE_STATIC_PREFIX_TTL, "").strip().lower() + return LONG_CACHE_TTL if raw == LONG_CACHE_TTL else None + + @dataclass class ModelConfig: """Configuration for multi-provider LLM models. @@ -342,6 +361,15 @@ def get_provider(self) -> ModelProvider: # Default to configured provider return self.provider + def long_ttl_static_prefix(self) -> bool: + """True when this model's tools + system cachePoints carry the 1h TTL. + + The cost path uses it to bill the static segment's cache writes at + Bedrock's 1h premium (2x base) instead of the 5m one (1.25x) — the + correction that keeps the experiment arm's own cost rows honest. + """ + return bool(self.caching_enabled and self.bedrock_cache_points_supported() and static_prefix_cache_ttl()) + def bedrock_cache_points_supported(self) -> bool: """Whether a hand-placed Bedrock system cachePoint may be sent. @@ -471,10 +499,22 @@ def to_bedrock_config(self) -> Dict[str, Any]: # that reaches Bedrock without going through that factory. if self.caching_enabled: from strands.models import CacheConfig + + # PR-5 (thresholds spec §3.6): an explicit "1h" on the two STATIC + # points only. system_prompt_ttl as a string is "honored as + # written" by _apply_system_cache_ttl, which rewrites the TTL on + # the hand-placed, TTL-less system point AgentFactory places; + # tools_ttl as a string sets the tools point's own TTL. The + # message-level auto point carries no ttl (cache_config.ttl stays + # unset) and so stays at 5m — tools(1h) → system(1h) → messages(5m) + # is the non-increasing order Bedrock requires. Off (the default) + # emits exactly today's bytes. + supported = self.bedrock_cache_points_supported() + long_ttl = static_prefix_cache_ttl() if supported else None config["cache_config"] = CacheConfig( strategy="auto", - system_prompt_ttl=True, - tools_ttl=self.bedrock_cache_points_supported(), + system_prompt_ttl=long_ttl or True, + tools_ttl=(long_ttl or True) if supported else False, ) if self.retry_config: diff --git a/backend/src/agents/main_agent/core/system_prompt_builder.py b/backend/src/agents/main_agent/core/system_prompt_builder.py index e1c6e61e2..93ec1aa8d 100644 --- a/backend/src/agents/main_agent/core/system_prompt_builder.py +++ b/backend/src/agents/main_agent/core/system_prompt_builder.py @@ -80,8 +80,10 @@ RESPONSE GUIDELINES: - Respond using markdown. - You can ONLY use tools that are explicitly provided to you in each conversation -- When approriate, you may use KaTeX to render mathematical equations. -- Since the $ character is used to denote a variable in KaTeX, other uses of $ should be use the HTML entity $ +- When appropriate, you may use KaTeX to render mathematical equations: + $...$ or \(...\) for inline math, $$...$$ or \[...\] for display math. + Write currency as a plain $ -- "$100K" renders correctly on its own and + needs no escaping or HTML entity, in chat or in a file you generate. - When the user asks for a diagram or chart, you may use Mermaid to render it. - Available tools may change throughout the conversation based on user preferences - When multiple tools are available, select and use the most appropriate combination in the optimal order to fulfill the user's request @@ -89,18 +91,25 @@ - Always explain your reasoning when using tools - If you don't have the right tool for a task, clearly inform the user about the limitation +PREVIEWING FILES: +Every .docx and .pptx in the conversation has a "Preview" button the user +clicks to see it laid out; you cannot open it for them. When they ask to LOOK +at one ("show me this deck"), say to use that button -- never read the file +or re-create it just to show it. Reading is still right when the request is +about its CONTENT: summarize, check, answer from it, edit it. + HANDLING MISSING TOOLS: -Users can toggle individual tools on and off from the Tools section of the -model settings panel (the gear icon next to the message input). When a user -asks for something you would normally handle with a tool that isn't currently -available to you, don't just say "I can't do that." Instead: +Users can toggle individual tools on and off from Customize → Tools in the +sidebar. When a user asks for something you would normally handle with a tool +that isn't currently available to you, don't just say "I can't do that." +Instead: 1. Identify which capability they're asking for in plain language (e.g. "spreadsheet analysis", "web browsing", "Python execution", "knowledge base search"). 2. Tell them that capability isn't active in the current session and suggest - they enable the matching tool from the Tools panel in settings, then retry - the request. + they enable the matching tool from Customize → Tools in the sidebar, then + retry the request. 3. If you can offer a partial answer without the tool (e.g. explaining a formula they could run themselves), do that as a fallback — but lead with the tool suggestion so they know the better path exists. @@ -117,11 +126,10 @@ column total: > I can compute that for you, but the Spreadsheet Analysis tool isn't -> currently enabled for this conversation. Open the settings panel (gear -> icon next to the message input), enable "Spreadsheet Analysis" under -> Tools, and send the request again — I'll run the aggregation directly -> on the file. Alternatively, you can open the file in Excel and use -> `=SUM(NET_AMOUNT)` on the column. +> currently enabled. Open Customize → Tools in the sidebar, enable +> "Spreadsheet Analysis", and send the request again — I'll run the +> aggregation directly on the file. Alternatively, you can open the file +> in Excel and use `=SUM(NET_AMOUNT)` on the column. SPREADSHEET ANALYSIS — DISAMBIGUATION: When more than one spreadsheet is attached (including the assistant's @@ -191,6 +199,8 @@ def build(self, include_date: bool = True) -> str: str: Complete system prompt """ if include_date: + # This line sits in the Bedrock prompt-cache prefix, so it must be + # byte-stable within a day: date + weekday + timezone only, no hour. current_date = get_current_date_pacific() prompt = f"{self.base_prompt}\n\nCurrent date: {current_date}" logger.info(f"Built system prompt with current date: {current_date}") diff --git a/backend/src/agents/main_agent/core/tool_result_offload.py b/backend/src/agents/main_agent/core/tool_result_offload.py new file mode 100644 index 000000000..17ab9523d --- /dev/null +++ b/backend/src/agents/main_agent/core/tool_result_offload.py @@ -0,0 +1,211 @@ +""" +Tool-result offload at intake — the compaction escalation for oversized tool +results (docs/specs/compaction-model-relative-thresholds.md §3.6, PR-4). + +The compaction cut keeps the last ``protected_turns`` turns whole. When one of +those turns carries a 90k-token tool result, the protected tail alone exceeds +the floor and no cut can get the session back under the ceiling +(``compaction_floor_unreachable``). Cutting inside the tail would drop the +turn the user is working on; the right move is to keep the *reference* and +drop the *bytes*. + +Strands 1.55's vended ``ContextOffloader`` does exactly that at the cheapest +possible moment — ``AfterToolCallEvent``, before the result is appended to the +conversation. The persisted message already carries the bounded form, so a +restore reproduces it byte-for-byte and nothing in the prefix is ever +mutated after the fact (the byte-stability contract in CLAUDE.md). The model +keeps a preview and can pull any span back with ``retrieve_offloaded_content`` +(pattern / line range / full). + +What this module owns on top of the plugin: + +- **Storage**: S3 in the user-files bucket under + ``compaction-offload/{user_id}/{session_id}/`` — one namespaced storage per + agent, so references are scoped to the session by construction (a session + cannot retrieve another session's content) and a second agent instance for + the same session (an ``@``-mention) resolves the same references. +- **No eviction from the model path.** ``evict_after_cycles=None``: the + plugin's cycle-based eviction runs on ``BeforeModelCallEvent`` and would + turn a retrieval into a miss mid-conversation. Objects expire by S3 + lifecycle instead (see the file-upload construct). +- **A cheap pre-filter.** The plugin sizes every result with + ``model.count_tokens`` — a Bedrock CountTokens round trip per tool call. + ``BoundedToolResultOffloader`` estimates with chars/4 first and only lets + results near or over the gate reach the API. +- **A content-free record per offload** (``AgentCoreStack/Compaction``: + ``ToolResultOffloaded`` + token count), so the data point "how much tool + payload were we about to put in the prefix" exists. + +Documents attached by the user are NOT handled here: the digest + page-range +read tool in ``document-context-offload.md`` is the right shape for those, and +this module deliberately does not strip them. +""" + +from __future__ import annotations + +import logging +import os +from typing import Any, Optional + +from agents.main_agent.config.constants import Defaults, EnvVars +from agents.main_agent.session.compaction_policy import estimate_message_tokens + +logger = logging.getLogger(__name__) + +# Below this fraction of the gate the chars/4 estimate is trusted outright +# and CountTokens is skipped. The heuristic is ~±25% on English text and JSON; +# half the gate leaves that margin twice over. +PREFILTER_RATIO = 0.5 + +# Tools whose results are never offloaded. ``document_read`` returns the page +# slice the model just asked for as a native document block; offloading it +# to S3 and handing back a text preview would undo the read and cost a second +# round trip (docs/specs/document-context-offload.md §4B). Its own +# ``max_pages`` cap is the bound. +OFFLOAD_EXEMPT_TOOLS = frozenset({"document_read"}) + + +def tool_result_offload_enabled() -> bool: + """Default ON with a kill switch (house style): only the literal "false" disables.""" + return os.environ.get(EnvVars.TOOL_RESULT_OFFLOAD_ENABLED, "").strip().lower() != "false" + + +def _int_env(name: str, default: int) -> int: + raw = os.environ.get(name, "").strip() + try: + return int(raw) if raw else default + except ValueError: + return default + + +def _import_plugin(): + from strands.vended_plugins.context_offloader import ContextOffloader + + return ContextOffloader + + +def _exempt_tool(event: Any) -> bool: + """True when the event's tool is in ``OFFLOAD_EXEMPT_TOOLS``.""" + try: + name = (getattr(event, "tool_use", None) or {}).get("name") + except Exception: # noqa: BLE001 + return False + return name in OFFLOAD_EXEMPT_TOOLS + + +def _should_offload(tool_name: str, _token_count: int) -> bool: + """The plugin's own ``should_offload`` hook — defense in depth behind the + mixin's early return, for the path where the base class is reached.""" + return tool_name not in OFFLOAD_EXEMPT_TOOLS + + +class _OffloaderMixin: + """Behavior layered on the vended plugin; kept separate so it can be tested + against a stub base class without importing the real one.""" + + async def _handle_tool_result(self, event: Any) -> None: # type: ignore[override] + try: + result = getattr(event, "result", None) + if not isinstance(result, dict): + return + content = result.get("content") + if not isinstance(content, list): + return + if _exempt_tool(event): + return + # Cheap pre-filter: skip the CountTokens round trip for results + # that cannot be anywhere near the gate. + estimate = estimate_message_tokens({"role": "user", "content": [{"toolResult": result}]}) + if estimate < self._max_result_tokens * PREFILTER_RATIO: # type: ignore[attr-defined] + return + before = event.result + await super()._handle_tool_result(event) # type: ignore[misc] + if event.result is not before: + self._record_offload(event, estimate) + except Exception: # noqa: BLE001 - offload is never worth a failed tool call + logger.warning("tool-result offload skipped, keeping the original result", exc_info=True) + + @staticmethod + def _record_offload(event: Any, estimate: int) -> None: + tool_name = None + try: + tool_name = (getattr(event, "tool_use", None) or {}).get("name") + except Exception: # noqa: BLE001 + pass + logger.info("tool_result_offloaded: tool=%s est_tokens=%d", tool_name, estimate) + try: + from apis.shared.observability.prompt_cache import prompt_cache_observability_enabled + from apis.shared.observability.emf import emit_emf_metrics + + if prompt_cache_observability_enabled(): + emit_emf_metrics( + "AgentCoreStack/Compaction", + metrics={"ToolResultOffloaded": 1, "ToolResultOffloadedTokens": int(estimate)}, + properties={"toolName": tool_name}, + units={"ToolResultOffloadedTokens": "Count"}, + ) + except Exception as e: # noqa: BLE001 + logger.debug("offload EMF skipped: %s", e) + + +def _offloader_class(): + """``BoundedToolResultOffloader`` built lazily so importing this module never + imports the plugin (and boto3) on paths that do not use it.""" + ContextOffloader = _import_plugin() + + class BoundedToolResultOffloader(_OffloaderMixin, ContextOffloader): # type: ignore[misc, valid-type] + pass + + BoundedToolResultOffloader.__name__ = "BoundedToolResultOffloader" + return BoundedToolResultOffloader + + +def offload_prefix(user_id: str, session_id: str) -> str: + return f"{Defaults.TOOL_RESULT_OFFLOAD_S3_PREFIX}/{user_id}/{session_id}" + + +def build_tool_result_offloader( + *, + session_id: str, + user_id: str, + region: Optional[str] = None, + storage: Any = None, +) -> Optional[Any]: + """The plugin for one agent, or ``None`` when off or unconfigured (fail-open). + + ``storage`` overrides the S3 backend (tests). Returns ``None`` — never + raises — when the flag is off, the user-files bucket is not configured, + or the plugin cannot be constructed. + """ + if not tool_result_offload_enabled(): + return None + try: + max_tokens = max(1, _int_env(EnvVars.TOOL_RESULT_OFFLOAD_MAX_TOKENS, Defaults.TOOL_RESULT_OFFLOAD_MAX_TOKENS)) + preview = _int_env(EnvVars.TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS, Defaults.TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS) + preview = max(0, min(preview, max_tokens - 1)) + + if storage is None: + bucket = os.environ.get("S3_USER_FILES_BUCKET_NAME", "").strip() + if not bucket: + logger.info("tool-result offload disabled: S3_USER_FILES_BUCKET_NAME is not set") + return None + from strands.storage import S3Storage + + storage = S3Storage( + bucket, + prefix=offload_prefix(user_id, session_id), + region_name=region or os.environ.get("AWS_REGION", "us-west-2"), + ) + + cls = _offloader_class() + return cls( + storage=storage, + max_result_tokens=max_tokens, + preview_tokens=preview, + include_retrieval_tool=True, + should_offload=_should_offload, + evict_after_cycles=None, + ) + except Exception: # noqa: BLE001 + logger.warning("tool-result offload disabled: plugin construction failed", exc_info=True) + return None diff --git a/backend/src/agents/main_agent/quota/checker.py b/backend/src/agents/main_agent/quota/checker.py index 205b479b3..2834c442b 100644 --- a/backend/src/agents/main_agent/quota/checker.py +++ b/backend/src/agents/main_agent/quota/checker.py @@ -34,7 +34,8 @@ def __init__( async def check_quota( self, user: User, - session_id: Optional[str] = None + session_id: Optional[str] = None, + session_total_cost: Optional[float] = None, ) -> QuotaCheckResult: """ Check if user is within quota limits (soft + hard limits). @@ -49,6 +50,13 @@ async def check_quota( - session_cost / session_percentage_of_limit / session_notice_threshold: set only when this session alone has crossed the tier's session-notice share of the monthly limit + + ``session_total_cost`` lets a caller that has already read the session + row this turn supply its ``totalCost`` instead of making this method + read it again — the inference-api preamble does, where that read was a + measured 62ms (docs/specs/turn-latency-preamble.md). ``None`` means + "not known", and the read happens exactly as before; it is NOT a way to + say "zero". """ # Resolve user's quota tier resolved = await self.resolver.resolve_user_quota(user) @@ -167,6 +175,7 @@ async def check_quota( limit=limit, session_id=session_id, assignment_id=assignment_id, + session_total_cost=session_total_cost, ) if current_usage >= limit: # warn only @@ -216,6 +225,7 @@ async def _resolve_session_notice( limit: float, session_id: Optional[str], assignment_id: Optional[str], + session_total_cost: Optional[float] = None, ) -> dict: """Return the session-notice fields when *session_id* is a heavy one. @@ -238,15 +248,29 @@ async def _resolve_session_notice( if not session_id or threshold_usd is None: return {} - try: - from apis.shared.sessions.metadata import get_session_metadata + # A caller that already read the session row this turn can hand us its + # `totalCost` and save a 62ms GSI query (the preamble does). + # + # It supplies the value ONLY when the row actually carries the + # attribute, and `None` here means "not known" — never "zero". That + # distinction is the whole safety of this path: `get_session_metadata` + # lazily **backfills** the cost aggregates for legacy rows written + # before write-time aggregation, so treating a missing `totalCost` as + # 0.0 would skip the backfill AND silently stop producing the notice + # for exactly the long-lived conversations it exists to catch. On + # `None` we take the original read, backfill and all. + session_cost: Optional[float] = session_total_cost + if session_cost is None: + try: + from apis.shared.sessions.metadata import get_session_metadata - metadata = await get_session_metadata(session_id, user.user_id) - except Exception as e: - logger.debug("Session notice skipped (metadata read failed): %s", e) - return {} + metadata = await get_session_metadata(session_id, user.user_id) + except Exception as e: + logger.debug("Session notice skipped (metadata read failed): %s", e) + return {} + + session_cost = getattr(metadata, "total_cost", None) if metadata else None - session_cost = getattr(metadata, "total_cost", None) if metadata else None if session_cost is None: return {} diff --git a/backend/src/agents/main_agent/session/__init__.py b/backend/src/agents/main_agent/session/__init__.py index 527a5b4ec..71386ccea 100644 --- a/backend/src/agents/main_agent/session/__init__.py +++ b/backend/src/agents/main_agent/session/__init__.py @@ -1,6 +1,7 @@ """Session management modules for Strands Agent""" from .session_factory import SessionFactory -from .compaction_models import CompactionState, CompactionConfig +from .compaction_models import CompactionState, CompactionConfig, CompactionResult +from .compaction_policy import CompactionPolicy from .turn_based_session_manager import TurnBasedSessionManager from .preview_session_manager import PreviewSessionManager, is_preview_session @@ -8,6 +9,8 @@ "SessionFactory", "CompactionState", "CompactionConfig", + "CompactionResult", + "CompactionPolicy", "TurnBasedSessionManager", "PreviewSessionManager", "is_preview_session", diff --git a/backend/src/agents/main_agent/session/compaction_models.py b/backend/src/agents/main_agent/session/compaction_models.py index 3758094f3..4bcc50646 100644 --- a/backend/src/agents/main_agent/session/compaction_models.py +++ b/backend/src/agents/main_agent/session/compaction_models.py @@ -3,6 +3,9 @@ These models define the state and configuration for automatic context window compaction, which helps manage token usage in long conversations. + +Thresholds are model-relative — see ``compaction_policy.py`` and +docs/specs/compaction-model-relative-thresholds.md. """ from dataclasses import dataclass @@ -21,7 +24,7 @@ class CompactionState: a separate DynamoDB item. This simplifies storage and ensures atomic updates with session data. """ - checkpoint: int = 0 # Message index to load from (0 = load all) + checkpoint: int = 0 # Absolute message index to load from (0 = load all) summary: Optional[str] = None # Pre-computed summary for skipped messages last_input_tokens: int = 0 # Input tokens from last turn updated_at: Optional[str] = None # ISO timestamp of last update @@ -37,6 +40,26 @@ class CompactionState: # turns (the re-write is free then). It must never be derived from a # per-restore sliding window. truncation_anchor: int = 0 + # Hysteresis (spec §3.3). A cut disarms the trigger; a turn at or below + # the ceiling re-arms it. While disarmed, only the hard ceiling can force + # another cut. Legacy rows predate the field and default to armed. + armed: bool = True + # Snapshot of the policy the last cut was made under (window, ceiling, + # floor, hard ceiling, whether it was forced) — what the admin session + # profile reads to explain a compaction after the fact. + policy: Optional[Dict[str, Any]] = None + # Paid-when-free scheduling (spec §3.5). A cut is computed post-turn and + # parked here; ``apply_pending_compaction`` promotes it to ``checkpoint`` + # (and slices the live list in place) pre-call, only when the prefix + # re-write is free or unavoidable. ``checkpoint`` above is always the + # APPLIED one — what the restore slices at. + pending_checkpoint: Optional[int] = None + pending_summary: Optional[str] = None + pending_hard_ceiling: Optional[int] = None + pending_since: Optional[str] = None + # model id + agent id of the last turn; a change means the cached prefix + # is already invalid, so a pending cut can ride the same re-write. + last_prefix_key: Optional[str] = None def to_dict(self) -> Dict[str, Any]: """Convert to dictionary for DynamoDB storage.""" @@ -47,6 +70,13 @@ def to_dict(self) -> Dict[str, Any]: "updatedAt": self.updated_at, "totalSummarizedTurns": self.total_summarized_turns, "truncationAnchor": self.truncation_anchor, + "armed": self.armed, + "policy": self.policy, + "pendingCheckpoint": self.pending_checkpoint, + "pendingSummary": self.pending_summary, + "pendingHardCeiling": self.pending_hard_ceiling, + "pendingSince": self.pending_since, + "lastPrefixKey": self.last_prefix_key, } @classmethod @@ -55,6 +85,8 @@ def from_dict(cls, data: Optional[Dict[str, Any]]) -> "CompactionState": if not data: return cls() checkpoint = int(data.get("checkpoint", 0)) + armed = data.get("armed", True) + policy = data.get("policy") return cls( checkpoint=checkpoint, summary=data.get("summary"), @@ -65,6 +97,17 @@ def from_dict(cls, data: Optional[Dict[str, Any]]) -> "CompactionState": # so nothing retained by the slice is truncated (byte-stable from # the first restore under the anchor design). truncation_anchor=int(data.get("truncationAnchor", checkpoint)), + armed=bool(armed) if armed is not None else True, + policy=dict(policy) if isinstance(policy, dict) else None, + pending_checkpoint=( + int(data["pendingCheckpoint"]) if data.get("pendingCheckpoint") is not None else None + ), + pending_summary=data.get("pendingSummary"), + pending_hard_ceiling=( + int(data["pendingHardCeiling"]) if data.get("pendingHardCeiling") is not None else None + ), + pending_since=data.get("pendingSince"), + last_prefix_key=data.get("lastPrefixKey"), ) @@ -72,9 +115,10 @@ def from_dict(cls, data: Optional[Dict[str, Any]]) -> "CompactionState": class CompactionResult: """ Returned by ``TurnBasedSessionManager.update_after_turn`` when a turn - crosses the token threshold and the checkpoint advances. Carries the + crosses the ceiling and the checkpoint advances. Carries the information the frontend needs to render an inline "earlier messages - summarized" divider in the conversation. + summarized" divider in the conversation, plus the policy the cut was + made under (additive fields on the ``compaction`` SSE payload). ``summarized_turns`` is the *delta* count of turns rolled into the summary at this compaction event (not the cumulative total across @@ -84,6 +128,22 @@ class CompactionResult: new_checkpoint: int summarized_turns: int input_tokens: int + context_window: Optional[int] = None + ceiling: Optional[int] = None + floor: Optional[int] = None + hard_ceiling: Optional[int] = None + # True when the cut ran while disarmed because input reached the hard + # ceiling — the signal that the previous cut did not take. + forced: bool = False + retained_tokens_estimate: Optional[int] = None + # True when the cut was parked as pending (applied pre-call later under + # the paid-when-free rule) rather than promoted to the checkpoint now. + deferred: bool = False + + +def _env_flag_default_on(name: str) -> bool: + """House-style kill switch: unset/empty → on; only the literal "false" is off.""" + return os.environ.get(name, "").strip().lower() != "false" @dataclass @@ -94,7 +154,9 @@ class CompactionConfig: Can be loaded from environment variables or passed directly. """ enabled: bool = True - token_threshold: int = 100_000 # Trigger checkpoint when exceeded + # Ceiling used when the model's window is unknown (and the fixed + # threshold when model-relative policy is switched off). + token_threshold: int = 100_000 protected_turns: int = 3 # Recent turns to protect from truncation max_tool_content_length: int = 500 # Max chars before truncating tool output # Bedrock prompt-cache TTL. When more than this many seconds have passed @@ -102,6 +164,22 @@ class CompactionConfig: # truncations can be applied without forcing an otherwise-avoidable # prefix re-write. cache_ttl_seconds: int = 300 + # Model-relative policy (spec §3.1). See CompactionPolicy.resolve. + model_relative_enabled: bool = Defaults.COMPACTION_MODEL_RELATIVE_ENABLED + ceiling_ratio: float = Defaults.COMPACTION_CEILING_RATIO + ceiling_cap_tokens: int = Defaults.COMPACTION_CEILING_CAP_TOKENS + floor_ratio: float = Defaults.COMPACTION_FLOOR_RATIO + hard_ceiling_ratio: float = Defaults.COMPACTION_HARD_CEILING_RATIO + hard_ceiling_multiplier: float = Defaults.COMPACTION_HARD_CEILING_MULTIPLIER + # Bounded summary (spec §3.6 / spiral spec PR-2). The persisted summary is + # held at or under this many tokens (chars/4), compressed once at cut time + # by the cheap model, with newest-first truncation as the fallback. + summary_token_budget: int = Defaults.COMPACTION_SUMMARY_TOKEN_BUDGET + summary_model_enabled: bool = Defaults.COMPACTION_SUMMARY_MODEL_ENABLED + summary_model_id: str = Defaults.COMPACTION_SUMMARY_MODEL_ID + # Paid-when-free scheduling (spec §3.5). Only meaningful with the + # model-relative policy on; legacy mode always applies immediately. + deferred_apply_enabled: bool = Defaults.COMPACTION_DEFERRED_APPLY_ENABLED @classmethod def from_env(cls) -> "CompactionConfig": @@ -112,4 +190,14 @@ def from_env(cls) -> "CompactionConfig": protected_turns=int(os.environ.get(EnvVars.COMPACTION_PROTECTED_TURNS, str(Defaults.COMPACTION_PROTECTED_TURNS))), max_tool_content_length=int(os.environ.get(EnvVars.COMPACTION_MAX_TOOL_CONTENT_LENGTH, str(Defaults.COMPACTION_MAX_TOOL_CONTENT_LENGTH))), cache_ttl_seconds=int(os.environ.get(EnvVars.COMPACTION_CACHE_TTL_SECONDS, str(Defaults.COMPACTION_CACHE_TTL_SECONDS))), + model_relative_enabled=_env_flag_default_on(EnvVars.COMPACTION_MODEL_RELATIVE_ENABLED), + ceiling_ratio=float(os.environ.get(EnvVars.COMPACTION_CEILING_RATIO, str(Defaults.COMPACTION_CEILING_RATIO))), + ceiling_cap_tokens=int(os.environ.get(EnvVars.COMPACTION_CEILING_CAP_TOKENS, str(Defaults.COMPACTION_CEILING_CAP_TOKENS))), + floor_ratio=float(os.environ.get(EnvVars.COMPACTION_FLOOR_RATIO, str(Defaults.COMPACTION_FLOOR_RATIO))), + hard_ceiling_ratio=float(os.environ.get(EnvVars.COMPACTION_HARD_CEILING_RATIO, str(Defaults.COMPACTION_HARD_CEILING_RATIO))), + hard_ceiling_multiplier=float(os.environ.get(EnvVars.COMPACTION_HARD_CEILING_MULTIPLIER, str(Defaults.COMPACTION_HARD_CEILING_MULTIPLIER))), + summary_token_budget=int(os.environ.get(EnvVars.COMPACTION_SUMMARY_TOKEN_BUDGET, str(Defaults.COMPACTION_SUMMARY_TOKEN_BUDGET))), + summary_model_enabled=_env_flag_default_on(EnvVars.COMPACTION_SUMMARY_MODEL_ENABLED), + summary_model_id=os.environ.get(EnvVars.COMPACTION_SUMMARY_MODEL_ID, "").strip() or Defaults.COMPACTION_SUMMARY_MODEL_ID, + deferred_apply_enabled=_env_flag_default_on(EnvVars.COMPACTION_DEFERRED_APPLY_ENABLED), ) diff --git a/backend/src/agents/main_agent/session/compaction_policy.py b/backend/src/agents/main_agent/session/compaction_policy.py new file mode 100644 index 000000000..90567ec47 --- /dev/null +++ b/backend/src/agents/main_agent/session/compaction_policy.py @@ -0,0 +1,256 @@ +""" +Model-relative compaction policy. + +Spec: docs/specs/compaction-model-relative-thresholds.md. + +Three numbers per model, derived from the catalog's ``maxInputTokens``: + +- **ceiling** — the trigger. ``min(window * CEILING_RATIO, CEILING_CAP_TOKENS)``. + Capped on purpose: a 1M window does not mean a 500k chat prefix is a good + idea — every warm turn re-reads it and every cache bust re-writes it. +- **floor** — the target size after a cut. ``ceiling * FLOOR_RATIO``. Under + Bedrock caching a cut costs one re-write of what *survives*, so a small floor + (deeper, rarer cuts) beats a shallow one (frequent re-writes). +- **hard ceiling** — force a cut even when hysteresis says wait. + ``min(window * HARD_CEILING_RATIO, ceiling * HARD_CEILING_MULTIPLIER)``. + +``COMPACTION_TOKEN_THRESHOLD`` (the historical fixed 100k) is the ceiling when +the window is unknown. With ``COMPACTION_MODEL_RELATIVE_ENABLED=false`` the +policy degrades to exactly the legacy behavior: fixed threshold, the +``cutoffs[-protected_turns]`` turn-count cut, no hysteresis. + +The module also owns the token-aware cut selection (``choose_checkpoint``) and +the per-message estimator it uses. Estimates are calibrated against the turn's +real history token count, so the *ratio* between messages is what the +estimator has to get right, not the absolute number. +""" + +from __future__ import annotations + +import json +import logging +from dataclasses import dataclass +from typing import Any, Dict, List, Optional, Sequence, Tuple, TYPE_CHECKING + +if TYPE_CHECKING: + from .compaction_models import CompactionConfig + +logger = logging.getLogger(__name__) + +# chars/4 is the same heuristic Strands falls back to when native CountTokens +# is unavailable; good enough because the estimates are rescaled to the +# measured history size before use. +CHARS_PER_TOKEN = 4 +# Bedrock/Anthropic image tokens ~ (w*h)/750; dimensions are unknown here, so +# a flat figure that is roughly a 1000x1100 image. +IMAGE_TOKEN_ESTIMATE = 1_500 +# Per-message framing overhead (role, block boundaries). +MESSAGE_OVERHEAD_TOKENS = 4 + + +@dataclass(frozen=True) +class CompactionPolicy: + """Resolved thresholds for one turn on one model.""" + + ceiling: int + floor: Optional[int] + hard_ceiling: Optional[int] + context_window: Optional[int] + # "model_relative" (window known), "fixed" (window unknown, using the + # configured threshold), or "legacy" (kill switch off — turn-count cut). + source: str + + @property + def hysteresis_enabled(self) -> bool: + return self.floor is not None + + def to_dict(self) -> Dict[str, Any]: + return { + "ceiling": self.ceiling, + "floor": self.floor, + "hardCeiling": self.hard_ceiling, + "contextWindow": self.context_window, + "source": self.source, + } + + @classmethod + def resolve( + cls, + config: "CompactionConfig", + context_window: Optional[int], + ) -> "CompactionPolicy": + """Derive the policy for a model window under ``config``. + + ``context_window`` is the catalog's ``maxInputTokens`` for the model + that served the turn, or ``None`` when the lookup missed. + """ + window: Optional[int] = None + if context_window is not None: + try: + window = int(context_window) + except (TypeError, ValueError): + window = None + if window is not None and window <= 0: + window = None + + if not config.model_relative_enabled: + return cls( + ceiling=max(1, int(config.token_threshold)), + floor=None, + hard_ceiling=None, + context_window=window, + source="legacy", + ) + + if window is not None: + ceiling = int(min(window * config.ceiling_ratio, config.ceiling_cap_tokens)) + hard = int(min(window * config.hard_ceiling_ratio, ceiling * config.hard_ceiling_multiplier)) + source = "model_relative" + else: + ceiling = int(config.token_threshold) + hard = int(ceiling * config.hard_ceiling_multiplier) + source = "fixed" + + ceiling = max(1, ceiling) + # The floor must leave hysteresis room below the ceiling. + floor = max(0, min(ceiling - 1, int(ceiling * config.floor_ratio))) + hard = max(ceiling, hard) + return cls( + ceiling=ceiling, + floor=floor, + hard_ceiling=hard, + context_window=window, + source=source, + ) + + +# --------------------------------------------------------------------------- +# Per-message token estimate +# --------------------------------------------------------------------------- + +def _text_tokens(text: Any) -> int: + if not isinstance(text, str): + return 0 + return len(text) // CHARS_PER_TOKEN + + +def _json_tokens(value: Any) -> int: + try: + return len(json.dumps(value, ensure_ascii=False, default=str)) // CHARS_PER_TOKEN + except Exception: # noqa: BLE001 - estimator must never raise + return 0 + + +def _bytes_tokens(source: Any) -> int: + if not isinstance(source, dict): + return 0 + raw = source.get("bytes") + if isinstance(raw, (bytes, bytearray)): + return len(raw) // CHARS_PER_TOKEN + return 0 + + +def _block_tokens(block: Any) -> int: + if not isinstance(block, dict): + return _text_tokens(block) if isinstance(block, str) else 0 + if "text" in block: + return _text_tokens(block.get("text")) + if "image" in block: + return IMAGE_TOKEN_ESTIMATE + if "document" in block: + doc = block.get("document") or {} + return _bytes_tokens(doc.get("source")) if isinstance(doc, dict) else 0 + if "toolUse" in block: + tool_use = block.get("toolUse") or {} + return _json_tokens(tool_use.get("input")) + 8 + if "toolResult" in block: + result = block.get("toolResult") or {} + content = result.get("content") if isinstance(result, dict) else None + if isinstance(content, list): + return sum(_block_tokens(inner) for inner in content) + 8 + return _json_tokens(content) + 8 + if "json" in block: + return _json_tokens(block.get("json")) + if "reasoningContent" in block: + rc = block.get("reasoningContent") or {} + text = rc.get("reasoningText", {}) if isinstance(rc, dict) else {} + return _text_tokens(text.get("text") if isinstance(text, dict) else None) + if "cachePoint" in block: + return 0 + return _json_tokens(block) + + +def estimate_message_tokens(message: Dict[str, Any]) -> int: + """Heuristic token estimate for one Converse message (never raises).""" + if not isinstance(message, dict): + return 0 + content = message.get("content") + if isinstance(content, str): + return _text_tokens(content) + MESSAGE_OVERHEAD_TOKENS + if not isinstance(content, list): + return MESSAGE_OVERHEAD_TOKENS + return sum(_block_tokens(block) for block in content) + MESSAGE_OVERHEAD_TOKENS + + +# --------------------------------------------------------------------------- +# Cut selection +# --------------------------------------------------------------------------- + +def choose_checkpoint( + messages: Sequence[Dict[str, Any]], + cutoffs: Sequence[int], + protected_turns: int, + floor_tokens: int, + history_tokens: Optional[int], +) -> Tuple[int, Optional[int]]: + """Pick the cut (index into ``messages``) that lands retained history at or + below ``floor_tokens`` while keeping as much as fits. + + Returns ``(relative_cut, retained_estimate)``. ``relative_cut == 0`` means + nothing to cut. Candidates are ``cutoffs`` (tool-pair-safe turn starts) no + newer than ``cutoffs[-protected_turns]``; the last ``protected_turns`` + turns are always kept. Among candidates that satisfy the floor the + **oldest** wins. If none does — the protected tail alone exceeds the floor + — the minimum-protection cut is returned and the caller logs it (evicting + inside the tail is the offload escalation, a later PR, not a deeper cut). + + ``history_tokens`` is the measured size of the conversation portion of + the prompt this turn; per-message estimates are rescaled to sum to it so + only their ratios matter. ``None`` leaves the raw estimates as-is. + """ + protected_turns = max(0, int(protected_turns)) + if not cutoffs or len(cutoffs) <= protected_turns: + return 0, None + + newest_allowed = cutoffs[-protected_turns] if protected_turns > 0 else cutoffs[-1] + n = len(messages) + if n == 0: + return int(newest_allowed), None + + estimates = [estimate_message_tokens(m) for m in messages] + raw_total = sum(estimates) + scale = 1.0 + if history_tokens is not None and history_tokens > 0 and raw_total > 0: + scale = history_tokens / raw_total + + # suffix[i] = estimated tokens retained if we cut at i (keep messages[i:]) + suffix = [0] * (n + 1) + for i in range(n - 1, -1, -1): + suffix[i] = suffix[i + 1] + estimates[i] + + def retained(cut: int) -> int: + idx = min(max(int(cut), 0), n) + return int(round(suffix[idx] * scale)) + + candidates = [c for c in cutoffs if c <= newest_allowed] + fitting = [c for c in candidates if retained(c) <= floor_tokens] + if fitting: + cut = min(fitting) + return int(cut), retained(cut) + + logger.info( + "compaction_floor_unreachable: protected tail alone is ~%d tokens " + "(floor=%d); taking the minimum-protection cut at %d", + retained(newest_allowed), floor_tokens, newest_allowed, + ) + return int(newest_allowed), retained(newest_allowed) diff --git a/backend/src/agents/main_agent/session/compaction_summary.py b/backend/src/agents/main_agent/session/compaction_summary.py new file mode 100644 index 000000000..64076ca17 --- /dev/null +++ b/backend/src/agents/main_agent/session/compaction_summary.py @@ -0,0 +1,207 @@ +""" +Bounded compaction summary. + +Spec: docs/specs/compaction-model-relative-thresholds.md §3.6 and +docs/specs/compaction-over-threshold-cache-spiral.md PR-2 (defect D2). + +The compaction summary used to be an unbounded join of AgentCore Long-Term +Memory ``ConversationSummary`` records — a log that grows for the life of the +session (164,991 chars ≈ 40k tokens in the incident). A summary that is 40% +of the threshold guarantees compaction can never get back under it. This +module holds the persisted summary at or under a token budget: + +1. **Within budget** → unchanged. +2. **Over budget** → one call to the cheap model (Nova Micro, the same + side-channel pattern as titles and tool-batch summaries) that compresses + the records into a bounded, instruction-preserving summary. It runs once, + at checkpoint advance — the turn that already pays a prefix re-write. +3. **Model unavailable / failed / still over budget** → newest-first + truncation of the records: keep the most recent records that fit, and if + even the newest alone does not fit, keep its tail. Never oldest-first — + recent context is what the model needs. + +Whatever comes out is persisted verbatim in ``CompactionState.summary`` and +prepended byte-identically at every restore, so the byte-stability contract +is unchanged: the summary still only mutates at checkpoint advance. +""" + +from __future__ import annotations + +import asyncio +import logging +import os +from dataclasses import dataclass +from typing import List, Optional, Sequence + +from .compaction_policy import CHARS_PER_TOKEN + +logger = logging.getLogger(__name__) + +# Bound the *input* to the compression call too: a wide history of records is +# fed newest-first up to this many chars, so the side-channel's own spend is +# flat regardless of how long the session has run. +MAX_COMPRESSION_INPUT_CHARS = 120_000 +# Nova Micro's output ceiling is 5k tokens; stay under it with margin so the +# generation is not cut mid-sentence. The budget check after generation is +# what enforces the configured budget. +_MODEL_MAX_OUTPUT_TOKENS = 4_000 + +_COMPRESSION_SYSTEM_PROMPT = """You maintain the running summary of a long conversation between a user and an AI assistant. You are given the existing summary notes (oldest first). Rewrite them into ONE compact summary the assistant can continue the conversation from. + +Keep, in this order, and keep them exact: +1. Standing instructions, preferences and constraints the user gave (tone, format, length, language, things to avoid, people or systems to treat carefully). Quote them; do not paraphrase away specifics. +2. Decisions made and their reasons. +3. The current state of anything being built or edited (a document, essay, code, plan, dataset): what exists now, what is done, what is still open. +4. Open questions, pending tasks, and what the user asked for most recently. +5. Exact identifiers: file names, URLs, course or project names, IDs, numbers, dates, names of people. + +Drop: greetings, pleasantries, superseded drafts and revisions, step-by-step narration of tool calls, and anything already covered by a later note. + +Rules: +- Plain text with short headings or bullets. No preamble, no closing remark. +- Write in the language the conversation is in. +- Never invent facts; if notes conflict, the later note wins and say so briefly. +- Stay under {word_budget} words.""" + + +def approx_tokens(text: Optional[str]) -> int: + """chars/4 — the same estimate the admin SUMMARY_OVER_BUDGET diagnosis uses.""" + if not text: + return 0 + return len(text) // CHARS_PER_TOKEN + + +@dataclass(frozen=True) +class BoundedSummary: + text: Optional[str] + # "within_budget" | "model" | "truncated" | "truncated_after_model" | "empty" + outcome: str + tokens_before: int + tokens_after: int + + +def truncate_records_newest_first(records: Sequence[str], budget_tokens: int) -> Optional[str]: + """Keep the newest records that fit the budget (joined oldest→newest). + + If even the newest record alone exceeds the budget, keep its *tail*. + """ + budget_chars = max(0, int(budget_tokens)) * CHARS_PER_TOKEN + kept: List[str] = [] + used = 0 + for record in reversed([r for r in records if r]): + cost = len(record) + (2 if kept else 0) # "\n\n" joiner + if used + cost > budget_chars: + break + kept.append(record) + used += cost + if kept: + kept.reverse() + return "\n\n".join(kept) + newest = next((r for r in reversed(records) if r), None) + if not newest: + return None + if budget_chars <= 0: + return None + tail = newest[-budget_chars:] + return tail + + +def _compression_input(records: Sequence[str]) -> str: + """Records joined oldest→newest, trimmed to the input cap newest-first.""" + joined = "\n\n".join(r for r in records if r) + if len(joined) <= MAX_COMPRESSION_INPUT_CHARS: + return joined + return joined[-MAX_COMPRESSION_INPUT_CHARS:] + + +async def compress_with_model( + records: Sequence[str], + budget_tokens: int, + *, + model_id: str, + region: Optional[str] = None, +) -> Optional[str]: + """One bounded Bedrock ``converse`` call. Returns ``None`` on any failure. + + Side-channel by construction: its own messages, never ``agent.messages``. + """ + text = _compression_input(records) + if not text.strip(): + return None + try: + import boto3 + except ImportError: # pragma: no cover - dev without boto3 + return None + try: + region = region or os.environ.get("AWS_REGION", "us-west-2") + client = boto3.client("bedrock-runtime", region_name=region) + # ~0.75 words/token; aim well under the budget so the chars/4 check + # below passes with margin. + word_budget = max(150, int(budget_tokens * 0.55)) + response = await asyncio.to_thread( + client.converse, + modelId=model_id, + system=[{"text": _COMPRESSION_SYSTEM_PROMPT.replace("{word_budget}", f"{word_budget:,}")}], + messages=[{"role": "user", "content": [{"text": "Summary notes, oldest first:\n\n" + text}]}], + inferenceConfig={ + "temperature": 0.1, + "maxTokens": min(_MODEL_MAX_OUTPUT_TOKENS, max(256, int(budget_tokens))), + "topP": 0.9, + }, + ) + if response.get("stopReason") == "max_tokens": + logger.info("compaction_summary_model_truncated: generation hit the token ceiling; discarding") + return None + out = response["output"]["message"]["content"][0]["text"].strip() + return out or None + except Exception: # noqa: BLE001 - a summary is never worth an error + logger.warning("compaction_summary_model_failed: falling back to truncation", exc_info=True) + return None + + +async def bound_summary( + records: Sequence[str], + budget_tokens: int, + *, + model_enabled: bool, + model_id: str, + region: Optional[str] = None, +) -> BoundedSummary: + """Hold the summary built from ``records`` at or under ``budget_tokens``.""" + records = [r for r in records if isinstance(r, str) and r.strip()] + joined = "\n\n".join(records) if records else None + before = approx_tokens(joined) + if not joined: + return BoundedSummary(None, "empty", 0, 0) + if before <= budget_tokens: + return BoundedSummary(joined, "within_budget", before, before) + + if model_enabled: + compressed = await compress_with_model(records, budget_tokens, model_id=model_id, region=region) + if compressed is not None: + after = approx_tokens(compressed) + if after <= budget_tokens: + logger.info( + "compaction_summary_bounded: model %d -> %d tokens (budget=%d)", + before, after, budget_tokens, + ) + return BoundedSummary(compressed, "model", before, after) + # The model overshot: truncate ITS output newest-first (its tail + # holds the open items), rather than the raw records. + trimmed = truncate_records_newest_first([compressed], budget_tokens) + logger.info( + "compaction_summary_bounded: model overshot (%d > %d); tail-trimmed to %d", + after, budget_tokens, approx_tokens(trimmed), + ) + return BoundedSummary(trimmed, "truncated_after_model", before, approx_tokens(trimmed)) + outcome = "truncated_after_model" + else: + outcome = "truncated" + + truncated = truncate_records_newest_first(records, budget_tokens) + after = approx_tokens(truncated) + logger.info( + "compaction_summary_bounded: %s %d -> %d tokens (budget=%d)", + outcome, before, after, budget_tokens, + ) + return BoundedSummary(truncated, outcome, before, after) diff --git a/backend/src/agents/main_agent/session/document_context.py b/backend/src/agents/main_agent/session/document_context.py new file mode 100644 index 000000000..257148feb --- /dev/null +++ b/backend/src/agents/main_agent/session/document_context.py @@ -0,0 +1,148 @@ +"""Content-free summary of the attachments in a conversation's live context. + +The cost rows could say what a call cost and (since the context ledger) what +compaction did to the history before it, but not *how much of the context +was documents* — the one quantity ``docs/specs/document-context-offload.md`` +needs to decide whether its cost work is worth shipping (evaluation spec §4.1: +the recoverable envelope is the document share of every cold re-write, and +that share was unmeasured). This module measures it from ``agent.messages`` +at turn end, and the stream coordinator persists the numbers on each of the +turn's ``C#`` rows next to ``prefixTokens`` and ``compactionEvents``. + +Everything here is a count, a byte size or a token estimate. No filename, +title, text or document byte ever leaves this function — the MIME map is +keyed by Bedrock's ``document.format`` enum (``pdf`` / ``docx`` / …) plus +``image``. + +Token numbers are heuristics (``document_tokens.estimate_document_tokens``: +pages x the per-page image estimate for PDFs, bytes/4 otherwise; a flat figure +per image), not Bedrock counts: Bedrock reports no per-block usage. They are comparable across rows and against +``contextBreakdown.messages``, which is the measured total they are a share of. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +from agents.main_agent.session.compaction_policy import IMAGE_TOKEN_ESTIMATE +from apis.shared.files.document_tokens import estimate_document_tokens + +#: A restore-time stand-in for a document that is no longer inline — today the +#: contentless placeholder ``_strip_document_bytes`` writes; from PR-3 the +#: digest block. Both count as "the model has a reference, not the bytes". +DIGEST_MARKERS = ("[Document placeholder:", " Optional[int]: + """Byte length of an inline ``document`` / ``image`` block, or ``None`` + when the block is not one (or carries no bytes).""" + payload = block.get(key) + if not isinstance(payload, dict): + return None + raw = (payload.get("source") or {}).get("bytes") + if isinstance(raw, (bytes, bytearray)): + return len(raw) + return None + + +def _is_digest(block: Dict[str, Any]) -> bool: + text = block.get("text") + return isinstance(text, str) and text.lstrip().startswith(DIGEST_MARKERS) + + +def _is_turn_prompt(message: Dict[str, Any]) -> bool: + """The user message that carries the turn's prompt (never a tool-result + message, which is also ``role: user``).""" + if message.get("role") != "user": + return False + content = message.get("content") + if not isinstance(content, list): + return isinstance(content, str) + return not any(isinstance(b, dict) and "toolResult" in b for b in content) + + +def summarize_document_context(messages: Optional[List[Dict[str, Any]]]) -> Optional[Dict[str, Any]]: + """The attachment footprint of ``messages`` (the live conversation). + + Returns ``None`` for an empty or malformed list, otherwise a dict with: + + - ``hasDocuments`` — at least one inline attachment (document or image + bytes) is in context. + - ``documentCount`` / ``documentTokens`` — inline attachment blocks on + user prompts and their estimated token weight. + - ``documentDigests`` — digest / placeholder stand-ins in context (a turn + with digests and no inline documents answered from the digest). + - ``documentsAttached`` — attachment blocks on the most recent prompt + (an attach turn vs. a follow-up). + - ``documentSlices`` / ``documentSliceTokens`` — document blocks that + ``document_read`` returned inside tool results and that still live in + the history (the re-injected pages, bounded by the tool's cap). + - ``documentMime`` — ``{format: count}`` over the inline attachments, + ``image`` for image blocks. + """ + if not isinstance(messages, list) or not messages: + return None + + count = tokens = digests = slices = slice_tokens = 0 + mime: Dict[str, int] = {} + last_prompt_attachments = 0 + + for message in messages: + if not isinstance(message, dict): + continue + content = message.get("content") + if not isinstance(content, list): + continue + prompt = _is_turn_prompt(message) + attached_here = 0 + for block in content: + if not isinstance(block, dict): + continue + size = _inline_bytes(block, "document") + if size is not None: + count += 1 + attached_here += 1 + fmt = str((block.get("document") or {}).get("format") or "unknown") + tokens += estimate_document_tokens( + fmt, ((block.get("document") or {}).get("source") or {}).get("bytes") + ) + mime[fmt] = mime.get(fmt, 0) + 1 + continue + size = _inline_bytes(block, "image") + if size is not None: + count += 1 + attached_here += 1 + tokens += IMAGE_TOKEN_ESTIMATE + mime["image"] = mime.get("image", 0) + 1 + continue + if _is_digest(block): + digests += 1 + continue + tool_result = block.get("toolResult") + if isinstance(tool_result, dict): + for inner in tool_result.get("content") or []: + if not isinstance(inner, dict): + continue + inner_size = _inline_bytes(inner, "document") + if inner_size is not None: + slices += 1 + inner_doc = inner.get("document") or {} + slice_tokens += estimate_document_tokens( + inner_doc.get("format"), (inner_doc.get("source") or {}).get("bytes") + ) + if prompt: + last_prompt_attachments = attached_here + + return { + "hasDocuments": count > 0, + "documentCount": count, + "documentTokens": tokens, + "documentDigests": digests, + "documentsAttached": last_prompt_attachments, + "documentSlices": slices, + "documentSliceTokens": slice_tokens, + "documentMime": mime, + } + + +__all__ = ["DIGEST_MARKERS", "summarize_document_context"] diff --git a/backend/src/agents/main_agent/session/document_offload.py b/backend/src/agents/main_agent/session/document_offload.py new file mode 100644 index 000000000..c4ac06610 --- /dev/null +++ b/backend/src/agents/main_agent/session/document_offload.py @@ -0,0 +1,403 @@ +"""Document offload — swap an inline document for its digest once it is no +longer the active subject, and only when the prefix re-write is free or +unavoidable (docs/specs/document-context-offload.md §4C / §4D, PR-4). + +A document enters the cacheable prefix on the turn that attaches it and is +re-written in full on every cold turn for the life of the session — that is +the cost this spec exists to bound. The digest (PR-2) is ~1/20th of a large +PDF and the model can pull any page back with ``document_read`` (PR-1), so +after the document has had its turn the bytes can leave the prefix. + +Three rules, all enforced here as pure functions over ``agent.messages``: + +* **Pinning** (§4D) — a document stays inline while it is the active + subject: on the attach turn and the next ``pin_turns - 1`` turns; while + the incoming or previous prompt names it; while a ``document_read`` result + for it sits in the recent turns. +* **Worth it** — the document's estimated tokens are at least + ``DOCUMENT_OFFLOAD_MIN_TOKENS``; below that the re-write costs more than the + eviction saves (the ``clear_at_least`` idea). The estimate is + ``document_tokens.estimate_document_tokens`` — pages x the per-page image + estimate for a PDF, bytes/4 otherwise — *not* bytes/4 for everything, which + under-counted PDFs ~14x and held large scans below the floor forever. +* **Free or unavoidable** — decided by the session manager, which owns the + cache-gap facts: the prompt cache has expired since the last turn, the + model/agent prefix changed, or the last turn's input exceeded the + compaction ceiling. Never while the cache is live (the 72% violation the + compaction PR-3 rule fixed). + +The replacement is **the same transformation restore performs** (PR-3): +match the block to its upload row, render the persisted digest. So an +offloaded block is byte-identical to what the next cold restore would have +produced for it anyway — the offload only moves the live prefix toward the +restore shape, earlier and on purpose. A block that cannot be matched stays +inline (unlike restore, the live path is allowed to keep bytes). + +``document_read`` page slices (tool-result document blocks) are aged the +same way, on the live path and on restore: older than ``slice_turns`` turns +they become a one-line stub. Their names carry a random tail, so the stub +is a deterministic function of the block and stable across restores. + +Mutation is in place on the message dicts; ``agent.messages`` is never +rebound (the #741 alias). Never raises. +""" + +from __future__ import annotations + +import logging +import os +import re +import zlib +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Sequence, Set, Tuple + +from agents.main_agent.multimodal.file_sanitizer import FileSanitizer +from agents.main_agent.session.compaction_policy import CHARS_PER_TOKEN +from apis.shared.files.document_tokens import estimate_document_tokens + +logger = logging.getLogger(__name__) + +#: Below this estimated size an eviction is not worth the prefix re-write. +DOCUMENT_OFFLOAD_MIN_TOKENS = int(os.environ.get("DOCUMENT_OFFLOAD_MIN_TOKENS", 5_000)) +#: A document is pinned on the turn it was attached and this many turns after +#: (2 = attach turn + the next one, spec §4D). +DOCUMENT_OFFLOAD_PIN_TURNS = int(os.environ.get("DOCUMENT_OFFLOAD_PIN_TURNS", 2)) +#: ``document_read`` page slices older than this many turns are stubbed. +DOCUMENT_SLICE_MAX_TURNS = int(os.environ.get("DOCUMENT_SLICE_MAX_TURNS", 2)) +#: Shortest document-name stem that counts as "the prompt names it". +_MIN_STEM_CHARS = 4 +_KNOWN_FORMAT_SUFFIX = re.compile(r"_(pdf|docx|doc|txt|md|html|csv|xls|xlsx)(_\d+)?$", re.IGNORECASE) +_DUPLICATE_SUFFIX = re.compile(r"_\d+$") + + +# --------------------------------------------------------------------------- +# Flags +# --------------------------------------------------------------------------- + + +def document_offload_enabled() -> bool: + """Default ON with a kill switch: only the literal "false" disables.""" + return os.environ.get("DOCUMENT_OFFLOAD_ENABLED", "").strip().lower() != "false" + + +def rollout_percent() -> int: + raw = os.environ.get("DOCUMENT_OFFLOAD_ROLLOUT_PERCENT", "").strip() + try: + return max(0, min(100, int(raw))) if raw else 100 + except ValueError: + return 100 + + +def session_bucket(session_id: str) -> int: + """Stable 0–99 bucket for a session (crc32, not ``hash`` — which is + salted per process). The evaluation spec's randomized per-session arm.""" + return zlib.crc32((session_id or "").encode("utf-8")) % 100 + + +def offload_enabled_for(session_id: Optional[str]) -> bool: + """Kill switch, the ``document_read`` gate, and the rollout bucket + together. Sessions below the percent are the treated arm; the rest keep + today's inline-forever behavior. + + ``DOCUMENT_READ_ENABLED=false`` disables this path too. The live offload + is the one place bytes leave the prefix *optionally* — restore has to drop + them, this does not — so evicting a document while its only recovery path + is switched off would be strictly worse than the pre-offload world, in + which live bytes never left. Spec §5: "a digest that points at a tool + nobody has is no better than today's placeholder." + """ + from apis.shared.feature_flags import document_read_enabled + + if not document_offload_enabled() or not session_id: + return False + if not document_read_enabled(): + return False + return session_bucket(session_id) < rollout_percent() + + +# --------------------------------------------------------------------------- +# Reading the conversation (content-free outputs only) +# --------------------------------------------------------------------------- + + +def _is_prompt(message: Dict[str, Any]) -> bool: + if not isinstance(message, dict) or message.get("role") != "user": + return False + content = message.get("content") + if isinstance(content, str): + return True + if not isinstance(content, list): + return False + return not any(isinstance(b, dict) and "toolResult" in b for b in content) + + +def prompt_indices(messages: Sequence[Dict[str, Any]]) -> List[int]: + """Indices of the user prompts — one per turn.""" + return [i for i, m in enumerate(messages) if _is_prompt(m)] + + +def _text_of(message_or_prompt: Any) -> str: + if isinstance(message_or_prompt, str): + return message_or_prompt + if isinstance(message_or_prompt, dict): + message_or_prompt = message_or_prompt.get("content") + if isinstance(message_or_prompt, str): + return message_or_prompt + if isinstance(message_or_prompt, list): + return " ".join( + b.get("text", "") for b in message_or_prompt if isinstance(b, dict) and isinstance(b.get("text"), str) + ) + return "" + + +def name_stem(block_name: str) -> str: + """``"BBR Policy_pdf_2"`` → ``"BBR Policy"`` — what a user would type.""" + stem = _KNOWN_FORMAT_SUFFIX.sub("", block_name) + stem = _DUPLICATE_SUFFIX.sub("", stem) + return stem.strip() + + +def _inline_document(block: Any) -> Optional[Tuple[str, str, int]]: + """``(name, format, size)`` for a document block with inline bytes.""" + if not isinstance(block, dict): + return None + doc = block.get("document") + if not isinstance(doc, dict): + return None + raw = (doc.get("source") or {}).get("bytes") if isinstance(doc.get("source"), dict) else None + if not isinstance(raw, (bytes, bytearray)): + return None + return str(doc.get("name", "")), str(doc.get("format", "")), len(raw) + + +def _inline_document_bytes(block: Any) -> Optional[bytes]: + """The block's raw bytes, for weighing it (``estimate_document_tokens``).""" + if not isinstance(block, dict): + return None + doc = block.get("document") + if not isinstance(doc, dict): + return None + raw = (doc.get("source") or {}).get("bytes") if isinstance(doc.get("source"), dict) else None + return raw if isinstance(raw, (bytes, bytearray)) else None + + +def pinned_document_names( + messages: Sequence[Dict[str, Any]], + incoming_prompt: Any = None, + *, + pin_turns: int = DOCUMENT_OFFLOAD_PIN_TURNS, +) -> Set[str]: + """Block names that must stay inline this turn (spec §4D).""" + pinned: Set[str] = set() + prompts = prompt_indices(messages) + recent_start = prompts[-pin_turns] if len(prompts) >= pin_turns else (prompts[0] if prompts else len(messages)) + if pin_turns <= 0: + recent_start = len(messages) + + # Every document name in the conversation, to test against prompt text. + all_names: Set[str] = set() + read_names: Set[str] = set() + for idx, message in enumerate(messages): + content = message.get("content") if isinstance(message, dict) else None + if not isinstance(content, list): + continue + for block in content: + inline = _inline_document(block) + if inline: + all_names.add(inline[0]) + if idx >= recent_start: + pinned.add(inline[0]) # attach turn + the next pin_turns-1 + continue + if idx >= recent_start and isinstance(block, dict) and isinstance(block.get("toolResult"), dict): + for inner in block["toolResult"].get("content") or []: + payload = inner.get("json") if isinstance(inner, dict) else None + if isinstance(payload, dict) and isinstance(payload.get("filename"), str): + read_names.add(FileSanitizer.sanitize_filename(payload["filename"])) + + # Named in the incoming prompt or the previous one. + texts = [_text_of(incoming_prompt)] + if prompts: + texts.append(_text_of(messages[prompts[-1]])) + haystack = " ".join(texts).lower() + for name in all_names: + stem = name_stem(name).lower() + if len(stem) >= _MIN_STEM_CHARS and stem in haystack: + pinned.add(name) + base = _DUPLICATE_SUFFIX.sub("", name) + if name in read_names or base in read_names: + pinned.add(name) + return pinned + + +@dataclass +class Candidate: + message_index: int + block_index: int + name: str + format: str + size: int + #: Estimated prefix weight — pages x the per-page image estimate for PDFs, + #: bytes/4 otherwise. This, not ``size``, is what the ``min_tokens`` floor + #: and the ledger's ``documentTokens`` are measured against, so the row and + #: the eviction decision always read the same quantity. + tokens: int = 0 + + +def candidate_documents( + messages: Sequence[Dict[str, Any]], + pinned: Set[str], + *, + min_tokens: int = DOCUMENT_OFFLOAD_MIN_TOKENS, +) -> List[Candidate]: + """Inline documents that are unpinned and large enough to be worth it.""" + out: List[Candidate] = [] + for mi, message in enumerate(messages): + content = message.get("content") if isinstance(message, dict) else None + if not isinstance(content, list): + continue + for bi, block in enumerate(content): + inline = _inline_document(block) + if not inline: + continue + name, fmt, size = inline + if name in pinned: + continue + tokens = estimate_document_tokens(fmt, _inline_document_bytes(block)) + if tokens < min_tokens: + continue + out.append(Candidate(mi, bi, name, fmt, size, tokens)) + return out + + +# --------------------------------------------------------------------------- +# Mutations (in place) +# --------------------------------------------------------------------------- + + +@dataclass +class OffloadResult: + offloaded: int = 0 + evicted_tokens: int = 0 + digest_tokens: int = 0 + skipped_unmatched: int = 0 + slices_aged: int = 0 + slice_tokens: int = 0 + upload_ids: List[str] = field(default_factory=list) + + +def offload_documents( + messages: List[Dict[str, Any]], + candidates: Sequence[Candidate], + *, + session_id: Optional[str], + user_id: Optional[str], +) -> OffloadResult: + """Replace each candidate block with its digest, in place, using the + restore path's own matcher and renderer so the bytes equal what a cold + restore would produce. Unmatched candidates stay inline. Never raises.""" + from agents.main_agent.session.document_rehydration import digest_for, load_session_documents, match_document + from apis.shared.feature_flags import document_read_enabled + from apis.shared.files.document_digest import render_digest + + result = OffloadResult() + if not candidates: + return result + try: + rows = load_session_documents(session_id or "", user_id) + except Exception: # noqa: BLE001 + logger.warning("document_offload: upload rows unavailable; leaving documents inline", exc_info=True) + result.skipped_unmatched = len(candidates) + return result + + used: Set[str] = set() + for cand in candidates: + try: + content = messages[cand.message_index]["content"] + block = content[cand.block_index] + inline = _inline_document(block) + if not inline or inline[0] != cand.name: + continue # the list moved under us; skip rather than guess + meta = match_document(cand.name, cand.format, cand.size, rows, used) + if meta is None: + result.skipped_unmatched += 1 + continue + raw = block["document"]["source"]["bytes"] + digest = digest_for(meta, bytes(raw)) + if digest is None: + result.skipped_unmatched += 1 + continue + text = render_digest( + digest, filename=meta.filename, upload_id=meta.upload_id, + include_handle=document_read_enabled(), + ) + content[cand.block_index] = {"text": text} + used.add(meta.upload_id) + result.offloaded += 1 + result.evicted_tokens += cand.tokens + result.digest_tokens += len(text) // CHARS_PER_TOKEN + result.upload_ids.append(meta.upload_id) + except Exception: # noqa: BLE001 - leave this block inline + logger.warning("document_offload: one block skipped", exc_info=True) + result.skipped_unmatched += 1 + return result + + +def slice_stub(name: str) -> str: + return f"[Retrieved pages placeholder: name={name} — call document_read again to reload them]" + + +def age_document_slices( + messages: List[Dict[str, Any]], + *, + max_turns: int = DOCUMENT_SLICE_MAX_TURNS, +) -> Tuple[int, int]: + """Stub ``document_read`` page slices older than ``max_turns`` turns, in + place. Returns ``(slices, estimated tokens evicted)``.""" + prompts = prompt_indices(messages) + if max_turns <= 0: + cutoff = len(messages) + elif len(prompts) >= max_turns: + cutoff = prompts[-max_turns] + else: + return 0, 0 + count = tokens = 0 + for idx in range(min(cutoff, len(messages))): + message = messages[idx] + content = message.get("content") if isinstance(message, dict) else None + if not isinstance(content, list): + continue + for block in content: + if not isinstance(block, dict) or not isinstance(block.get("toolResult"), dict): + continue + inner_content = block["toolResult"].get("content") + if not isinstance(inner_content, list): + continue + for ii, inner in enumerate(inner_content): + inline = _inline_document(inner) + if not inline: + continue + name, fmt, _size = inline + tokens += estimate_document_tokens(fmt, _inline_document_bytes(inner)) + inner_content[ii] = {"text": slice_stub(name)} + count += 1 + return count, tokens + + +__all__ = [ + "Candidate", + "DOCUMENT_OFFLOAD_MIN_TOKENS", + "DOCUMENT_OFFLOAD_PIN_TURNS", + "DOCUMENT_SLICE_MAX_TURNS", + "OffloadResult", + "age_document_slices", + "candidate_documents", + "document_offload_enabled", + "name_stem", + "offload_documents", + "offload_enabled_for", + "pinned_document_names", + "prompt_indices", + "rollout_percent", + "session_bucket", + "slice_stub", +] diff --git a/backend/src/agents/main_agent/session/document_rehydration.py b/backend/src/agents/main_agent/session/document_rehydration.py new file mode 100644 index 000000000..d6cd54187 --- /dev/null +++ b/backend/src/agents/main_agent/session/document_rehydration.py @@ -0,0 +1,249 @@ +"""Rehydrate stripped documents on restore — digest plus a live handle, not a +contentless placeholder (docs/specs/document-context-offload.md §4E, PR-3). + +Restore must drop inline document bytes from history: Bedrock rejects any +request where two document blocks share a name across the conversation, and +a re-attached file collides with the copy already in history. Until PR-3 the +replacement was ``[Document placeholder: name=…, format=…, original_size=…]`` +— zero content — so a returning user's document was silently gone (the 14% +same-file re-upload rate). This module replaces the bytes with the document's +``DocumentDigest`` rendered as a ```` block that names the +``upload_id`` the ``document_read`` tool (PR-1) reads pages back from. + +How a block finds its file: the session's upload rows (``SessionIndex``, one +query per restore that has a document to rehydrate, none otherwise) are +matched on the sanitized filename — the same sanitizer ``PromptBuilder`` used +when the block was built, allowing for its ``_2`` / ``_3`` duplicate suffix — +and the byte size. A block with no matching row (a direct base64 attachment +that never went through the upload flow, or a deleted file) keeps today's +placeholder, so nothing here can be worse than before. + +Digests missing from the row (uploads that predate PR-2, agent-written files) +are built lazily from the bytes that are *right there* in the restored +message — the outline only, no model call on the restore path — and +persisted so the next restore renders the same bytes. The upload-path build +(PR-2) may later overwrite an outline-only digest with one that has an +abstract; that changes the rendered block once, which is one prefix +re-write, accepted and recorded (``document_rehydrated`` carries the digest +tokens, so the change is visible). + +When ``DOCUMENT_READ_ENABLED=false`` has taken the tool away the digest is +still rendered — it is strictly more than the placeholder — but without the +``upload_id`` handle, so the model is never invited to call a tool it does +not have. The live offload path stops entirely in that case; see +``document_offload.offload_enabled_for``. + +Everything is synchronous: it runs inside ``TurnBasedSessionManager.initialize`` +under the Strands agent constructor. It never raises — every failure falls +back to the placeholder for that block. +""" + +from __future__ import annotations + +import copy +import logging +import os +import re +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Set + +from agents.main_agent.multimodal.file_sanitizer import FileSanitizer +from agents.main_agent.session.compaction_policy import CHARS_PER_TOKEN +from apis.shared.files.document_tokens import estimate_document_tokens + +logger = logging.getLogger(__name__) + +_DUPLICATE_SUFFIX = re.compile(r"_(\d+)$") + + +def document_rehydration_enabled() -> bool: + """Default ON with a kill switch (house style): ``DOCUMENT_REHYDRATE_ENABLED=false`` + restores today's placeholder behavior without touching PR-1/PR-2.""" + return os.environ.get("DOCUMENT_REHYDRATE_ENABLED", "").strip().lower() != "false" + + +def placeholder_text(name: str, fmt: str, size: int) -> str: + """The pre-PR-3 stand-in, kept byte-identical for blocks that cannot be + matched to an upload (restore output must be stable across restores).""" + return f"[Document placeholder: name={name}, format={fmt}, original_size={size} bytes]" + + +@dataclass +class RehydrationResult: + messages: List[Dict[str, Any]] + rehydrated: int = 0 + stripped: int = 0 + digest_tokens: int = 0 + stripped_tokens: int = 0 + lazy_digests: int = 0 + upload_ids: List[str] = field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Matching +# --------------------------------------------------------------------------- + + +def _base_name(block_name: str) -> str: + """``"policy_2"`` → ``"policy"`` (PromptBuilder's duplicate suffix).""" + return _DUPLICATE_SUFFIX.sub("", block_name) + + +def match_document( + name: str, + fmt: str, + size: int, + candidates: List[Any], + used: Set[str], +) -> Optional[Any]: + """The upload row a document block came from, or ``None``. + + Preference order: same sanitized filename **and** same byte size; then + same filename; then same format and size (a renamed re-upload). Rows + already claimed by another block in this restore are skipped, so two + copies of one file map to two rows, newest first. + """ + sanitize = FileSanitizer.sanitize_filename + wanted = {name, _base_name(name)} + fresh = [c for c in candidates if c.upload_id not in used] + by_name = [c for c in fresh if sanitize(c.filename) in wanted] + for c in by_name: + if c.size_bytes == size: + return c + if by_name: + return by_name[0] + for c in fresh: + if c.size_bytes == size and (c.file_format or "") == fmt: + return c + return None + + +# --------------------------------------------------------------------------- +# Data access (sync — see module docstring) +# --------------------------------------------------------------------------- + + +def load_session_documents(session_id: str, user_id: Optional[str]) -> List[Any]: + """This user's READY document-class uploads in the session, newest first.""" + from apis.shared.files.document_read import is_document_class + from apis.shared.files.models import FileStatus + from apis.shared.files.repository import get_file_upload_repository + + rows = get_file_upload_repository().list_session_files_sync(session_id, status=FileStatus.READY) + return [ + meta for meta in rows + if (not user_id or meta.user_id == user_id) and is_document_class(meta.mime_type, meta.filename) + ] + + +def digest_for(meta: Any, raw: bytes) -> Optional[Any]: + """The row's ready ``DocumentDigest``, or one built now from ``raw`` + (outline only) and persisted best-effort. ``None`` if neither works.""" + from apis.shared.files.document_digest import DocumentDigest, estimate_tokens, extract_outline, render_digest + from apis.shared.files.repository import get_file_upload_repository + + existing = DocumentDigest.from_item(getattr(meta, "digest", None)) + if existing is not None and existing.status == "ready": + return existing + + fmt = meta.file_format + if not fmt: + return None + digest = extract_outline(fmt, raw) + digest.tokens = estimate_tokens(render_digest(digest, filename=meta.filename, upload_id=meta.upload_id)) + try: + get_file_upload_repository().update_file_digest_sync(meta.user_id, meta.upload_id, digest.to_item()) + except Exception: # noqa: BLE001 - the digest still serves this restore + logger.debug("Lazy digest not persisted for upload %s", meta.upload_id, exc_info=True) + return digest + + +# --------------------------------------------------------------------------- +# The pass +# --------------------------------------------------------------------------- + + +def rehydrate_documents( + messages: List[Dict[str, Any]], + *, + session_id: Optional[str], + user_id: Optional[str], +) -> RehydrationResult: + """Replace every inline document block with its digest block (matched to + an upload row) or, failing that, the placeholder. Never raises.""" + from apis.shared.files.document_digest import render_digest + + from apis.shared.feature_flags import document_read_enabled + + out = copy.deepcopy(messages) + result = RehydrationResult(messages=out) + enabled = document_rehydration_enabled() and bool(session_id) + # Restore has to drop the bytes either way (Bedrock rejects duplicate + # document names), so a digest is still strictly better than the + # placeholder when the tool is off — but it must not advertise a handle + # the model cannot use. + handle = document_read_enabled() + candidates: Optional[List[Any]] = None + used: Set[str] = set() + + for msg in out: + content = msg.get("content", []) if isinstance(msg, dict) else None + if not isinstance(content, list): + continue + for idx, block in enumerate(content): + if not isinstance(block, dict) or "document" not in block: + continue + doc = block["document"] if isinstance(block["document"], dict) else {} + source = doc.get("source") or {} + if not isinstance(source, dict) or "bytes" not in source: + continue + raw = source.get("bytes", b"") + size = len(raw) if isinstance(raw, (bytes, bytearray)) else 0 + name = str(doc.get("name", "unknown")) + fmt = str(doc.get("format", "unknown")) + + replacement: Optional[str] = None + if enabled: + try: + if candidates is None: + candidates = load_session_documents(session_id, user_id) + meta = match_document(name, fmt, size, candidates, used) + if meta is not None: + digest = digest_for(meta, bytes(raw) if isinstance(raw, (bytes, bytearray)) else b"") + if digest is not None: + was_lazy = not (isinstance(getattr(meta, "digest", None), dict) and meta.digest.get("status") == "ready") + replacement = render_digest( + digest, filename=meta.filename, upload_id=meta.upload_id, + include_handle=handle, + ) + used.add(meta.upload_id) + result.rehydrated += 1 + result.digest_tokens += len(replacement) // CHARS_PER_TOKEN + result.lazy_digests += 1 if was_lazy else 0 + result.upload_ids.append(meta.upload_id) + except Exception: # noqa: BLE001 - fall back to the placeholder for this block + logger.warning("Document rehydration failed for one block; using placeholder", exc_info=True) + if candidates is None: + candidates = [] # do not retry the lookup for every block + + if replacement is None: + replacement = placeholder_text(name, fmt, size) + result.stripped += 1 + # Same estimator the row and the offloader use, so the + # ``document_stripped`` ledger event is comparable to + # ``document_offload`` / ``document_rehydrated``. + result.stripped_tokens += estimate_document_tokens(fmt, raw) + content[idx] = {"text": replacement} + + return result + + +__all__ = [ + "RehydrationResult", + "digest_for", + "document_rehydration_enabled", + "load_session_documents", + "match_document", + "placeholder_text", + "rehydrate_documents", +] diff --git a/backend/src/agents/main_agent/session/hooks/__init__.py b/backend/src/agents/main_agent/session/hooks/__init__.py index 1080adc59..d57545246 100644 --- a/backend/src/agents/main_agent/session/hooks/__init__.py +++ b/backend/src/agents/main_agent/session/hooks/__init__.py @@ -2,6 +2,7 @@ from agents.main_agent.session.hooks.agent_status import AgentStatusHook from agents.main_agent.session.hooks.context_attribution import ContextAttributionHook +from agents.main_agent.session.hooks.context_ledger import ContextLedgerHook from agents.main_agent.session.hooks.display_text import DisplayTextHook from agents.main_agent.session.hooks.oauth_consent import OAuthConsentHook from agents.main_agent.session.hooks.prefix_fingerprint import PrefixFingerprintHook @@ -13,6 +14,7 @@ __all__ = [ "AgentStatusHook", "ContextAttributionHook", + "ContextLedgerHook", "DisplayTextHook", "OAuthConsentHook", "PrefixFingerprintHook", diff --git a/backend/src/agents/main_agent/session/hooks/context_attribution.py b/backend/src/agents/main_agent/session/hooks/context_attribution.py index 8edd55759..7a7df0901 100644 --- a/backend/src/agents/main_agent/session/hooks/context_attribution.py +++ b/backend/src/agents/main_agent/session/hooks/context_attribution.py @@ -21,17 +21,39 @@ ``systemTokens`` / ``toolTokens`` are stable across a session (the tool overhead is constant as the conversation grows — verified), so they are -computed once per agent at cold start (two extra CountTokens calls) and cached; -every turn afterward is pure arithmetic against the free, authoritative -``projected_input_tokens``. +computed once per agent at cold start (two extra CountTokens calls, plus a +once-per-model probe baseline — see ``_probe_baseline``) and cached; every +turn afterward is pure arithmetic against the free, authoritative +``projected_input_tokens``. ``count(system only)`` is really +``count(probe + system) - count(probe)``: Bedrock refuses an empty message +list, so a bare system count silently degraded to the chars/4 heuristic. + +**Why the split is not computed while an attachment is in context.** +``toolTokens`` is a *residual* between two independently sourced numbers — +``full`` (Strands' projection for the upcoming request) and ``no_tools`` (our own +CountTokens call) — so any disagreement between them about how a content block +is counted lands wholly in it. Bedrock understands a PDF page as an image *and* +a text layer; when the two sources do not agree on that, the document's entire +weight is attributed to tools. Measured on dev 2026-09-16 (session +``61de2256``): a call reported ``toolTokens`` of **106,756** where the session's +real tools prefix was **12,516** — a difference of 94,240 against a document +measured at ~94,485, i.e. the whole document. The split is therefore skipped on +any turn whose context carries inline document or image bytes, and taken on a +later clean turn instead. An absent ``prefixTokens`` reads "not tracked" (the +ledger's convention); a wrong one silently corrupts every share computed from +it. Best-effort: any failure is swallowed so context attribution can never break a model call. For non-Bedrock models ``count_tokens`` falls back to a heuristic, so the numbers are approximate there. """ +import hashlib +import json import logging -from typing import Any, Optional +import threading +from collections import OrderedDict +from typing import Any, Dict, Optional, Tuple from strands.hooks import BeforeModelCallEvent, HookProvider, HookRegistry @@ -41,6 +63,130 @@ _SPLIT_ATTR = "_context_attribution_split" # cached stable {systemTokens, toolTokens} _BREAKDOWN_ATTR = "_context_attribution_breakdown" # latest per-turn breakdown dict +# Process-level memo of the stable split, keyed by *session and configuration* +# rather than by ``Agent`` instance. The instance attribute above is enough +# only while one Agent serves a session for its whole life; it does not, for +# every session whose injected tools keep it out of the agent cache (the +# spreadsheet-analysis family — see `apis/shared/tools/injected.py`), for +# `@`-mention turns, and for any Memory-Space binding. Those rebuild the Agent +# every turn and, without this memo, paid the two cold-start CountTokens calls +# every turn as well — on top of the SDK's own pre-call count, which is the +# "counted three times per turn" of docs/specs/load-test-assessment-2026-09.md +# §1 fix 1. The key carries a digest of the system prompt and of the full tool +# specs, so any configuration change that would move the split misses cleanly. +_SPLIT_MEMO_MAX = 512 +_split_memo: "OrderedDict[Tuple[str, str, str], Dict[str, int]]" = OrderedDict() +_split_memo_lock = threading.Lock() + + +def clear_split_memo() -> None: + """Drop every memoised split (tests, and any admin reset path).""" + with _split_memo_lock: + _split_memo.clear() + + +def _digest(payload: Any) -> str: + try: + raw = json.dumps(payload, sort_keys=True, default=str) + except (TypeError, ValueError): + raw = repr(payload) + return hashlib.md5(raw.encode("utf-8")).hexdigest()[:16] + + +def _memo_key(session_id: str, agent: Any) -> Optional[Tuple[str, str, str]]: + """``(session, prompt digest, tool-spec digest)`` or None if any part is + unavailable — in which case the memo is simply not consulted.""" + try: + prompt_payload = getattr(agent, "_system_prompt_content", None) or getattr(agent, "system_prompt", None) + specs = agent.tool_registry.get_all_tool_specs() + except Exception: # noqa: BLE001 - never let key construction break a turn + return None + return (session_id, _digest(prompt_payload), _digest(specs)) + + +def _memo_get(key: Tuple[str, str, str]) -> Optional[Dict[str, int]]: + with _split_memo_lock: + split = _split_memo.get(key) + if split is not None: + _split_memo.move_to_end(key) + return dict(split) if split is not None else None + + +def _memo_put(key: Tuple[str, str, str], split: Dict[str, int]) -> None: + with _split_memo_lock: + _split_memo[key] = dict(split) + _split_memo.move_to_end(key) + while len(_split_memo) > _SPLIT_MEMO_MAX: + _split_memo.popitem(last=False) + + +# The fixed user message the system prompt is counted against. Its own weight +# (message scaffolding + the two-letter text) is a per-model constant, so it is +# measured once per model id per process and subtracted. Keep it short and +# never change it casually: a different probe changes every systemTokens +# figure that follows, so the ledger's shares stop being comparable across the +# deploy. +_PROBE_MESSAGES = [{"role": "user", "content": [{"text": "hi"}]}] +_probe_baselines: Dict[str, int] = {} +_probe_lock = threading.Lock() + + +def clear_probe_baselines() -> None: + """Drop the per-model probe weights (tests).""" + with _probe_lock: + _probe_baselines.clear() + + +def _probe_model_key(model: Any) -> Optional[str]: + """Memo key for the probe baseline: the model id when the model exposes + one, else None (count every time — a test double, not a Bedrock model).""" + config = getattr(model, "config", None) + if isinstance(config, dict): + model_id = config.get("model_id") + if isinstance(model_id, str) and model_id: + return model_id + return None + + +async def _probe_baseline(model: Any) -> int: + """Token weight of ``_PROBE_MESSAGES`` alone on ``model``, memoised per model id.""" + key = _probe_model_key(model) + if key is not None: + with _probe_lock: + cached = _probe_baselines.get(key) + if cached is not None: + return cached + baseline = int(await model.count_tokens(messages=list(_PROBE_MESSAGES))) + if key is not None: + with _probe_lock: + _probe_baselines[key] = baseline + return baseline + + +def _has_inline_attachment(messages: Any) -> bool: + """Whether any message carries inline ``document`` / ``image`` bytes. + + The condition under which ``toolTokens`` cannot be trusted — see the module + docstring. Cheap: a walk over content blocks, no decoding.""" + if not isinstance(messages, list): + return False + for message in messages: + content = message.get("content") if isinstance(message, dict) else None + if not isinstance(content, list): + continue + for block in content: + if not isinstance(block, dict): + continue + for key in ("document", "image"): + payload = block.get(key) + if isinstance(payload, dict): + source = payload.get("source") + if isinstance(source, dict) and isinstance( + source.get("bytes"), (bytes, bytearray) + ): + return True + return False + def get_context_breakdown(agent: Any) -> Optional[dict]: """Return the latest context breakdown stashed on ``agent``, or ``None``. @@ -50,8 +196,37 @@ def get_context_breakdown(agent: Any) -> Optional[dict]: return getattr(agent, _BREAKDOWN_ATTR, None) +def get_prefix_token_split(agent: Any) -> Optional[Dict[str, int]]: + """The stable ``{"system": n, "tools": n}`` split for this agent, or ``None``. + + Persisted on each call's cost row (as ``prefixTokens``) so the static + prefix a session carries — and which part of it is tool schemas — is a + stored fact rather than a scan-and-guess. Same numbers the SSE breakdown + reports; this just reads the cached split without re-counting. + """ + split = getattr(agent, _SPLIT_ATTR, None) + if not isinstance(split, dict): + return None + try: + return { + "system": int(split.get("systemTokens") or 0), + "tools": int(split.get("toolTokens") or 0), + } + except (TypeError, ValueError): + return None + + class ContextAttributionHook(HookProvider): - """Compute the system / tools / messages token breakdown each turn.""" + """Compute the system / tools / messages token breakdown each turn. + + ``session_id`` enables the process-level split memo (see the module + comment): a rebuilt Agent for the same session and configuration adopts + the split its predecessor measured instead of re-counting. Without it the + split lives on the Agent instance only. + """ + + def __init__(self, session_id: Optional[str] = None) -> None: + self._session_id = session_id or None def register_hooks(self, registry: HookRegistry, **kwargs: Any) -> None: registry.add_callback(BeforeModelCallEvent, self._on_before_model_call) @@ -70,12 +245,35 @@ async def _compute(self, event: BeforeModelCallEvent) -> None: full = event.projected_input_tokens split = getattr(agent, _SPLIT_ATTR, None) + memo_key = _memo_key(self._session_id, agent) if (split is None and self._session_id) else None + if split is None and memo_key is not None: + split = _memo_get(memo_key) + if split is not None: + # A predecessor Agent for this session + configuration already + # measured it; adopt without spending two CountTokens calls. + setattr(agent, _SPLIT_ATTR, split) + logger.debug("Context attribution split adopted from session memo") + if split is None and _has_inline_attachment(agent.messages): + # Untrustworthy residual (see module docstring) — leave the split + # uncomputed and try again on a turn without inline bytes. + logger.debug("Context attribution deferred: inline attachment in context") + return if split is None: - system_tokens = await model.count_tokens( - messages=[], + # Bedrock CountTokens rejects an empty message list ("A + # conversation must start with a user message" — verified live + # against dev 2026-09-18), and Strands swallows that into the + # chars/4 heuristic, so `count(messages=[])` was never the + # authoritative system count it looked like. Count the system + # prompt against a fixed probe user message and subtract the + # probe's own weight, which is a per-model constant measured once + # per process. + probe_only = await _probe_baseline(model) + system_with_probe = await model.count_tokens( + messages=list(_PROBE_MESSAGES), system_prompt=system_prompt, system_prompt_content=system_prompt_content, ) + system_tokens = max(0, system_with_probe - probe_only) # system + the current conversation, WITHOUT tools — so the # difference from `full` captures tool schemas + the tool-use # scaffolding (present only when tools and messages coexist). @@ -99,6 +297,8 @@ async def _compute(self, event: BeforeModelCallEvent) -> None: "toolTokens": max(0, full - no_tools), } setattr(agent, _SPLIT_ATTR, split) + if memo_key is not None: + _memo_put(memo_key, split) if full is None: # No authoritative total this turn — can't place the messages diff --git a/backend/src/agents/main_agent/session/hooks/context_ledger.py b/backend/src/agents/main_agent/session/hooks/context_ledger.py new file mode 100644 index 000000000..85f6d276b --- /dev/null +++ b/backend/src/agents/main_agent/session/hooks/context_ledger.py @@ -0,0 +1,181 @@ +"""Hook that records, per model call, what happened to the context *before* it. + +Two content-free facts the cost anatomy could not otherwise answer from stored +data, both found missing during the 2026-09-15 prod cost audit: + +- ``windowRemovedMessages`` — the conversation manager's cumulative + ``removed_message_count`` at the moment of the call. A rise between two + consecutive rows means the message list was trimmed in between (Strands' + sliding window sliding, or a compaction slice), which re-writes the cached + prefix. Distinguishing "pure window slide" sessions from "compaction spiral" + sessions took an hour of fingerprint reading; with this field it is one + query. +- ``compactionEvents`` — the compaction decisions the session manager made + since the previous call (restore-time slice applied, checkpoint advanced, + and whatever a future scheduling policy records: forced cut, floor + unreachable). Each carries the summary's token size at that moment, which + is what proves a summary cap shrank summaries without another scan. +- ``documentReads`` — how many ``document_read`` retrievals the call + requested and how many pages / bytes they pulled back into context + (docs/specs/document-context-offload.md §6.1). With the per-row document + context fields the coordinator adds, this is what says whether a turn + answered from the full document, from a digest, or from retrieved pages. + +Per-turn, per-model-call, held on the agent wrapper exactly like the tool +census: the stream coordinator reads ``ledger_for_call(idx)`` at turn end and +persists it on that call's ``C#`` cost row. Gated by the same +``COST_DIAGNOSTICS_ENABLED`` kill switch; while off the callbacks return +immediately and nothing is written, so a row without the fields reads +"not tracked", never "0". +""" + +from __future__ import annotations + +import copy +import logging +from typing import Any, Dict, List, Optional + +from strands.hooks import ( + AfterToolCallEvent, + BeforeInvocationEvent, + BeforeModelCallEvent, + HookProvider, + HookRegistry, +) + +from apis.shared.feature_flags import cost_diagnostics_enabled + +logger = logging.getLogger(__name__) + +#: A call never carries more events than this; a runaway recorder must not +#: grow a cost row without bound. +_MAX_EVENTS_PER_CALL = 8 + +#: The document-retrieval tool whose results the ledger tallies +#: (``documentReads``): calls, pages and bytes the model pulled back into +#: context. Read from the tool result's own ``json`` block — numbers only. +DOCUMENT_READ_TOOL_NAME = "document_read" + + +def _document_read_counts(event: Any) -> Optional[Dict[str, int]]: + """``{"calls": 1, "pages": n, "bytes": b}`` for a successful + ``document_read`` result, ``None`` for anything else. Only the result's + numeric metadata is read; the document block itself is never inspected.""" + tool_use = getattr(event, "tool_use", None) or {} + if not isinstance(tool_use, dict) or tool_use.get("name") != DOCUMENT_READ_TOOL_NAME: + return None + result = getattr(event, "result", None) + if not isinstance(result, dict) or result.get("status") != "success": + return None + pages = 0 + size = 0 + for block in result.get("content") or []: + if not isinstance(block, dict): + continue + payload = block.get("json") + if isinstance(payload, dict): + try: + pages += int(payload.get("pages_returned") or 0) + except (TypeError, ValueError): + pass + document = block.get("document") + if isinstance(document, dict): + raw = (document.get("source") or {}).get("bytes") + if isinstance(raw, (bytes, bytearray)): + size += len(raw) + return {"calls": 1, "pages": pages, "bytes": size} + + +def _removed_message_count(agent: Any) -> Optional[int]: + manager = getattr(agent, "conversation_manager", None) + value = getattr(manager, "removed_message_count", None) + if value is None: + return None + try: + return int(value) + except (TypeError, ValueError): + return None + + +def _drain_compaction_events(agent: Any) -> List[Dict[str, Any]]: + """Take the session manager's pending compaction events, if it keeps any. + + Strands stores the manager as ``agent._session_manager``; ours exposes + ``drain_compaction_events``. Anything else (tests, other managers) yields + an empty list. + """ + manager = getattr(agent, "_session_manager", None) + drain = getattr(manager, "drain_compaction_events", None) + if not callable(drain): + return [] + try: + events = drain() + except Exception as e: # noqa: BLE001 - a ledger must never break a turn + logger.debug("Context ledger could not drain compaction events: %s", e) + return [] + return list(events or [])[:_MAX_EVENTS_PER_CALL] + + +class ContextLedgerHook(HookProvider): + """Per-turn, per-model-call context ledger: ``{cycle: {...}}``.""" + + def __init__(self) -> None: + self._cycle = 0 + self._ledger: Dict[int, Dict[str, Any]] = {} + + def register_hooks(self, registry: HookRegistry, **kwargs: Any) -> None: + registry.add_callback(BeforeInvocationEvent, self._on_turn_start) + registry.add_callback(BeforeModelCallEvent, self._on_before_model_call) + registry.add_callback(AfterToolCallEvent, self._on_after_tool_call) + + def _on_after_tool_call(self, event: AfterToolCallEvent) -> None: + """Tally ``document_read`` retrievals against the model call that + requested them (same cycle attribution as the tool census: a tool + running during cycle N was requested by call N). The pages come back + into context for call N+1; the row of call N says the model asked.""" + if not cost_diagnostics_enabled(): + return + try: + counts = _document_read_counts(event) + if counts is None: + return + entry = self._ledger.setdefault(self._cycle, {}) + reads = entry.setdefault("documentReads", {"calls": 0, "pages": 0, "bytes": 0}) + for key, value in counts.items(): + reads[key] = reads.get(key, 0) + value + except Exception as e: # noqa: BLE001 - a ledger must never break a turn + logger.debug("Context ledger skipped a tool result: %s", e) + + def ledger_for_call(self, call_index: int) -> Optional[Dict[str, Any]]: + """The ledger entry for model call ``call_index`` (0-based), or + ``None`` when nothing was recorded — or the diagnostics are off. + + Returns a copy so the caller can hand it to the persistence layer + without aliasing per-turn state. + """ + if not cost_diagnostics_enabled(): + return None + entry = self._ledger.get(call_index + 1) + return copy.deepcopy(entry) if entry else None + + def _on_turn_start(self, event: BeforeInvocationEvent) -> None: + self._cycle = 0 + self._ledger = {} + + def _on_before_model_call(self, event: BeforeModelCallEvent) -> None: + self._cycle += 1 + if not cost_diagnostics_enabled(): + return + try: + agent = event.agent + entry: Dict[str, Any] = {} + removed = _removed_message_count(agent) + if removed is not None: + entry["windowRemovedMessages"] = removed + events = _drain_compaction_events(agent) + if events: + entry["compactionEvents"] = events + if entry: + self._ledger[self._cycle] = entry + except Exception as e: # noqa: BLE001 - a ledger must never break a turn + logger.debug("Context ledger skipped a call: %s", e) diff --git a/backend/src/agents/main_agent/session/session_factory.py b/backend/src/agents/main_agent/session/session_factory.py index 02ecf7e9e..6bb12856e 100644 --- a/backend/src/agents/main_agent/session/session_factory.py +++ b/backend/src/agents/main_agent/session/session_factory.py @@ -22,6 +22,18 @@ SESSION_ASYNC_PERSISTENCE_ENABLED_ENV = "AGENTCORE_SESSION_ASYNC_PERSISTENCE_ENABLED" +MEMORY_SUMMARY_RETRIEVAL_ENV = "MEMORY_SUMMARY_NAMESPACE_RETRIEVAL_ENABLED" + + +def summary_namespace_retrieval_enabled() -> bool: + """Whether the current session's own summary is retrieved on every user + message and prepended to it. Off unless ``=true``: it re-injects a + conversation the model already holds, at a lookup per message. The + compaction path still reads summaries where they matter + (`TurnBasedSessionManager._retrieve_session_summaries`).""" + return os.environ.get(MEMORY_SUMMARY_RETRIEVAL_ENV, "").strip().lower() == "true" + + def session_async_persistence_enabled() -> bool: """Whether AgentCore Memory writes are offloaded off the event loop. @@ -222,9 +234,20 @@ def _create_cloud_session_manager( ) logger.info(f" • Facts namespace: {facts_namespace}") - if summary_id: + if summary_id and summary_namespace_retrieval_enabled(): # Session summaries (condensed conversation context for the current session) - # Note: Summary namespace includes sessionId since summaries are per-session + # Note: Summary namespace includes sessionId since summaries are per-session. + # + # Off by default. This namespace holds AgentCore's summary of THIS + # session, and the SDK retrieves it on every user message and + # prepends it to that message — a second copy of a conversation + # the model already has in context, paid as input tokens on every + # turn, plus one RetrieveMemoryRecords call per message against a + # 30/s account quota. The summary is genuinely useful once the + # conversation has been compacted, and that path reads it directly + # (`_retrieve_session_summaries` at checkpoint advance), so nothing + # is lost by not retrieving it per message. See + # docs/specs/load-test-assessment-2026-09.md P2-F. summary_namespace = f"/strategies/{summary_id}/actors/{{actorId}}/sessions/{{sessionId}}" retrieval_config[summary_namespace] = RetrievalConfig( top_k=top_k, diff --git a/backend/src/agents/main_agent/session/turn_based_session_manager.py b/backend/src/agents/main_agent/session/turn_based_session_manager.py index b889a0630..174d964df 100644 --- a/backend/src/agents/main_agent/session/turn_based_session_manager.py +++ b/backend/src/agents/main_agent/session/turn_based_session_manager.py @@ -10,7 +10,11 @@ - Stage 1: Tool content truncation — applied only below the persisted truncation anchor, which moves at checkpoint advances or when the Bedrock prompt cache has already expired between turns -- Stage 2: Checkpoint + Summary — triggered when token threshold exceeded +- Stage 2: Checkpoint + Summary — triggered when the turn's context exceeds + the model-relative ceiling (compaction_policy.py; the cut lands retained + history at the policy floor, and a cut disarms the trigger until the + context drops back under the ceiling — spec: + docs/specs/compaction-model-relative-thresholds.md) Byte-stability contract: between compaction-state changes, restoring the same stored history must produce byte-identical ``agent.messages``. Bedrock prompt @@ -29,20 +33,80 @@ import logging import os from datetime import datetime, timezone -from typing import Optional, Dict, Any, List, TYPE_CHECKING +from typing import Optional, Dict, Any, List, Tuple, TYPE_CHECKING -from agents.main_agent.config.constants import EnvVars +from agents.main_agent.config.constants import Defaults, EnvVars from bedrock_agentcore.memory.integrations.strands.session_manager import AgentCoreMemorySessionManager from bedrock_agentcore.memory.integrations.strands.config import AgentCoreMemoryConfig from .compaction_models import CompactionState, CompactionConfig, CompactionResult +from .compaction_policy import CompactionPolicy, choose_checkpoint +from .compaction_summary import bound_summary if TYPE_CHECKING: from strands.agent.agent import Agent logger = logging.getLogger(__name__) +# Bound on a single RetrieveMemoryRecords attempt from the per-message +# long-term memory retrieval hook (see `_get_retrieval_client`). Seconds. +MEMORY_RETRIEVAL_TIMEOUT_ENV = "MEMORY_RETRIEVAL_TIMEOUT_SECONDS" +MEMORY_RETRIEVAL_TIMEOUT_DEFAULT = 2.0 +# Attempts per retrieval, including the first. 1 = never retry a throttle. +MEMORY_RETRIEVAL_MAX_ATTEMPTS_ENV = "MEMORY_RETRIEVAL_MAX_ATTEMPTS" +MEMORY_RETRIEVAL_MAX_ATTEMPTS_DEFAULT = 1 + + +def memory_retrieval_timeout_seconds() -> float: + """Per-attempt timeout for long-term memory retrieval, from the environment.""" + raw = os.environ.get(MEMORY_RETRIEVAL_TIMEOUT_ENV, "") + try: + value = float(raw) if raw else MEMORY_RETRIEVAL_TIMEOUT_DEFAULT + except ValueError: + value = MEMORY_RETRIEVAL_TIMEOUT_DEFAULT + return value if value > 0 else MEMORY_RETRIEVAL_TIMEOUT_DEFAULT + + +def memory_retrieval_max_attempts() -> int: + """Attempts per long-term memory retrieval (including the first).""" + raw = os.environ.get(MEMORY_RETRIEVAL_MAX_ATTEMPTS_ENV, "") + try: + value = int(raw) if raw else MEMORY_RETRIEVAL_MAX_ATTEMPTS_DEFAULT + except ValueError: + value = MEMORY_RETRIEVAL_MAX_ATTEMPTS_DEFAULT + return value if value >= 1 else MEMORY_RETRIEVAL_MAX_ATTEMPTS_DEFAULT + +#: Compaction decisions the per-call ledger will record. Reserved kinds exist +#: so a scheduling policy can report them without a schema change. +#: Compaction-ledger event kinds. The ``document_*`` kinds are the document +#: lifecycle (docs/specs/document-context-offload.md §6.1): ``document_stripped`` +#: — restore replaced inline documents with contentless placeholders (the +#: defect PR-3 fixes; recorded from PR-1 so its reach is measured before the +#: fix lands); ``document_rehydrated`` — restore replaced them with a digest +#: plus a live ``document_read`` handle (PR-3); ``document_offload`` — the +#: post-turn trigger swapped an inline document for its digest, with the +#: cache gap at the moment it fired (PR-4). +#: ``truncation_anchor`` — the restore-time anchor slid forward because the +#: prompt cache had already expired (offload spec PR-5): carries the gap, so +#: "truncation applied while the cache was live" is a row query, not a scan. +COMPACTION_EVENT_KINDS = frozenset({ + "applied", "checkpoint", "forced", "floor_unreachable", + "document_stripped", "document_rehydrated", "document_offload", + "truncation_anchor", +}) +_MAX_PENDING_COMPACTION_EVENTS = 8 + + +def _approx_tokens(text: Any) -> int: + """~4 chars/token; the same estimate the admin profile uses for summaries.""" + if not text: + return 0 + try: + return len(text) // 4 + except TypeError: + return 0 + class TurnBasedSessionManager(AgentCoreMemorySessionManager): """ @@ -102,7 +166,20 @@ def __init__( # Cached data for checkpoint calculation self._valid_cutoff_indices: List[int] = [] self._all_messages_for_summary: List[Dict] = [] + # Compaction decisions taken since the last model call, drained by + # ``ContextLedgerHook`` onto that call's cost row. Bounded; content-free. + self._pending_compaction_events: List[Dict[str, Any]] = [] self._total_message_count_at_init: int = 0 + # Absolute index (into the stored history) of ``agent.messages[0]``. + # The restore slice sets it to the applied checkpoint; the persisted + # checkpoint is always ``_live_offset + ``, + # so cuts computed over the live list and the slice applied at restore + # share one coordinate system (spec §3.4). + self._live_offset: int = 0 + # model|agent key stamped at the head of each turn by the coordinator; + # persisted at turn end so the next head-of-turn can tell whether the + # cached prefix is already invalid (spec §3.5). + self._current_prefix_key: Optional[str] = None # Session control self.cancelled = False @@ -148,6 +225,119 @@ def append_message(self, message: Dict, agent: "Agent", **kwargs: Any) -> None: super().append_message(filtered_message, agent, **kwargs) self.message_count += 1 + # ------------------------------------------------------------------ + # Long-term memory retrieval, bounded + # ------------------------------------------------------------------ + + def _get_retrieval_client(self) -> Any: + """A dedicated ``bedrock-agentcore`` data-plane client for + ``RetrieveMemoryRecords``, built on first use. + + The SDK's ``MemoryClient`` serves reads and writes from one boto client + with default retries. Writes (``CreateEvent``) should keep retrying — + a dropped message is a corrupted conversation. Reads should not: the + retrieval hook runs on ``MessageAddedEvent`` and is awaited before the + model call, so under load a throttled ``RetrieveMemoryRecords`` (30/s + per account by default) put boto's retry backoff on the reply's + first-token latency, exactly the CountTokens mechanism from + docs/specs/load-test-assessment-2026-09.md §1 fix 3. One attempt and + a short timeout: a throttle costs one failed request and the turn + simply runs without long-term context, which is what a miss means. + """ + if getattr(self, "_retrieval_client", None) is None: + import boto3 + from botocore.config import Config as BotocoreConfig + + timeout = memory_retrieval_timeout_seconds() + self._retrieval_client = boto3.client( + "bedrock-agentcore", + region_name=self.region_name, + config=BotocoreConfig( + retries={"total_max_attempts": memory_retrieval_max_attempts(), "mode": "standard"}, + read_timeout=timeout, + connect_timeout=timeout, + ), + ) + return self._retrieval_client + + def retrieve_customer_context(self, event: Any) -> None: + """Retrieve long-term memory for the last user message and prepend it. + + Same contract as the SDK method this overrides (namespaces resolved + from ``retrieval_config``, relevance filter, results wrapped in + ```` and inserted as the first content block of the last + user message so the user's own words stay last) — only the client + differs, see :meth:`_get_retrieval_client`. Registered by the SDK's + ``register_hooks`` in both sync and async mode via + ``self.retrieve_customer_context``, so the override is picked up. + """ + messages = event.agent.messages + if not messages or messages[-1].get("role") != "user": + return None + content = messages[-1].get("content") + if not content or "text" not in content[0]: + return None + retrieval_config = getattr(self.config, "retrieval_config", None) + if not retrieval_config: + return None + + user_query = messages[-1]["content"][0]["text"] + client = self._get_retrieval_client() + + def retrieve_for_namespace(namespace: str, cfg: Any) -> List[str]: + resolved = namespace.format( + actorId=self.config.actor_id, + sessionId=self.config.session_id, + memoryStrategyId=getattr(cfg, "strategy_id", None) or "", + ) + response = client.retrieve_memory_records( + memoryId=self.config.memory_id, + namespacePath=resolved, + searchCriteria={"searchQuery": user_query, "topK": cfg.top_k}, + ) + records = response.get("memoryRecordSummaries", []) + if getattr(cfg, "relevance_score", None): + records = [r for r in records if r.get("score", 0.0) >= cfg.relevance_score] + items: List[str] = [] + for record in records: + text = (record.get("content") or {}).get("text", "") if isinstance(record, dict) else "" + text = text.strip() if isinstance(text, str) else "" + if text: + items.append(text) + return items + + try: + from concurrent.futures import ThreadPoolExecutor, as_completed + + all_context: List[str] = [] + with ThreadPoolExecutor() as executor: + futures = { + executor.submit(retrieve_for_namespace, ns, cfg): ns + for ns, cfg in retrieval_config.items() + } + for future in as_completed(futures): + try: + all_context.extend(future.result()) + except Exception as e: # noqa: BLE001 - one namespace failing must not sink the rest + code = getattr(e, "response", {}).get("Error", {}).get("Code") if hasattr(e, "response") else None + if code in ("ThrottlingException", "TooManyRequestsException", "ServiceQuotaExceededException"): + logger.info( + "memory retrieval throttled namespace=%s; turn proceeds without long-term context", + futures[future], + ) + else: + logger.warning("memory retrieval failed namespace=%s: %s", futures[future], e) + + if all_context: + tag = getattr(self.config, "context_tag", "user_context") + event.agent.messages[-1]["content"].insert( + 0, {"text": f"<{tag}>{chr(10).join(all_context)}"} + ) + logger.info("Retrieved %s customer context items", len(all_context)) + except Exception as e: # noqa: BLE001 - retrieval must never break a turn + logger.error("Failed to retrieve customer context: %s", e) + return None + def initialize(self, agent: "Agent", **kwargs: Any) -> None: """ Initialize agent with two-feature compaction. @@ -234,6 +424,20 @@ def initialize(self, agent: "Agent", **kwargs: Any) -> None: except Exception as e: logger.warning(f"Document byte stripping failed, continuing: {e}", exc_info=True) + # Age document_read page slices on restore exactly as the live path + # does (offload PR-4), so a cold restore and a warm agent agree on + # which slices are still inline. Pure function of the history's turn + # structure, so the output is stable across restores. + try: + from agents.main_agent.session.document_offload import age_document_slices, offload_enabled_for + + if offload_enabled_for(getattr(self.config, "session_id", None)): + aged, aged_tokens = age_document_slices(agent.messages) + if aged: + logger.debug("Aged %d document_read slice(s) (~%d tokens) in restored history", aged, aged_tokens) + except Exception as e: # noqa: BLE001 + logger.warning(f"Document slice ageing failed, continuing: {e}", exc_info=True) + # Drop empty/unrecognized content blocks from restored history. The # write side runs `_filter_empty_text` in `append_message`, but history # restored from AgentCore Memory bypasses it — a single block that comes @@ -261,6 +465,7 @@ def initialize(self, agent: "Agent", **kwargs: Any) -> None: self.compaction_state = CompactionState() self._valid_cutoff_indices = [] self._all_messages_for_summary = [] + self._live_offset = 0 # Repair tool-use/tool-result pairing and role alternation on the FINAL # restored list — after compaction slicing/truncation — so it is always @@ -310,6 +515,11 @@ def _apply_compaction(self, agent: "Agent") -> None: # if update_after_turn actually advances the checkpoint) self._all_messages_for_summary = all_messages[:] + # The gap this restore ran at, measured before anything below stamps + # updated_at, so the ledger can say whether a restore-time truncation + # (or slice) landed inside the prompt-cache TTL. + restore_gap_seconds = self._seconds_since(self.compaction_state.updated_at) + # Advance the truncation anchor only while the prompt cache is # already cold — the prefix re-write is unavoidable then, so pending # truncations are free. Persisted before use so subsequent restores @@ -334,6 +544,9 @@ def _apply_compaction(self, agent: "Agent") -> None: else: messages_to_process = all_messages + # The live list now starts at this absolute index. + self._live_offset = offset + # Truncate only messages strictly below the anchor (absolute index), # translated into post-slice coordinates via the checkpoint offset. truncation_count = 0 @@ -348,6 +561,19 @@ def _apply_compaction(self, agent: "Agent") -> None: agent.messages = messages_to_process + if checkpoint > 0 and offset > 0: + self.record_compaction_event( + "applied", + checkpoint=checkpoint, + summaryTokens=_approx_tokens(self.compaction_state.summary), + retainedMessages=len(agent.messages), + truncatedToolResults=truncation_count, + # The gap at restore time (offload spec PR-5): a restore-time + # slice or truncation inside the TTL is a rebuild while the + # cache was live — a rebuild's cost, and now countable. + cacheGapSeconds=restore_gap_seconds if restore_gap_seconds is not None else -1, + ) + logger.info( f"Compaction initialized: stage={stage}, " f"original={self._total_message_count_at_init}, " @@ -381,12 +607,35 @@ def _maybe_advance_truncation_anchor(self) -> None: if sliding_anchor <= max(state.truncation_anchor, state.checkpoint): return + # Measured BEFORE the save below stamps updated_at, so the row shows + # the gap the decision was made on. This is the offload spec's PR-5: + # the July 2026 audit found 72% of truncations firing inside the TTL + # and could only say so by correlating timestamps by hand; the guard + # is now the ``_cache_window_expired`` check above, and this event is + # what proves it holds — a ``truncation_anchor`` row with + # ``cacheGapSeconds`` under the TTL is a regression. + gap_seconds = self._seconds_since(state.updated_at) + previous_anchor = state.truncation_anchor logger.info( - "Truncation anchor advance (cache expired): %d -> %d", - state.truncation_anchor, sliding_anchor, + "Truncation anchor advance (cache expired, gap=%ss): %d -> %d", + gap_seconds, previous_anchor, sliding_anchor, ) state.truncation_anchor = sliding_anchor self._save_compaction_state(state) + self.record_compaction_event( + "truncation_anchor", + anchorFrom=previous_anchor, + anchorTo=sliding_anchor, + cacheGapSeconds=gap_seconds if gap_seconds is not None else -1, + ) + self._emit_emf( + { + "TruncationAnchorAdvanced": 1, + "TruncationAnchorCacheGapSeconds": int(gap_seconds or 0), + }, + {"anchorFrom": previous_anchor, "anchorTo": sliding_anchor}, + {"TruncationAnchorCacheGapSeconds": "Seconds"}, + ) @staticmethod def _cache_window_expired(updated_at: Optional[str], ttl_seconds: int) -> bool: @@ -407,6 +656,25 @@ def _cache_window_expired(updated_at: Optional[str], ttl_seconds: int) -> bool: last = last.replace(tzinfo=timezone.utc) return (datetime.now(timezone.utc) - last).total_seconds() > ttl_seconds + def _record_ledger_event(self, kind: str, **fields: Any) -> None: + """Hand a compaction decision to the per-call compaction ledger, if present. + + The cost-diagnostics ledger (``record_compaction_event`` / + ``drain_compaction_events`` + ``ContextLedgerHook``) lands whatever is + recorded here on the NEXT model call's ``C#`` cost row as + ``compactionEvents``, next to ``windowRemovedMessages`` and the prefix + token split — the evidence the summary cap and the scheduling rule are + judged on. Resolved by attribute so this is a no-op on a build without + the ledger; fields are ints. Never raises. + """ + recorder = getattr(self, "record_compaction_event", None) + if not callable(recorder): + return + try: + recorder(kind, **{k: int(v) for k, v in fields.items() if isinstance(v, (int, float)) and not isinstance(v, bool)}) + except Exception as e: # noqa: BLE001 + logger.debug("compaction ledger event skipped: %s", e) + # ========================================================================= # Compaction State Persistence # ========================================================================= @@ -564,6 +832,48 @@ def _adopt_persisted_compaction_state(self) -> None: ) self.compaction_state = persisted + # ------------------------------------------------------------------ + # Compaction event ledger (content-free; persisted per model call) + # ------------------------------------------------------------------ + + def record_compaction_event(self, kind: str, **fields: Any) -> None: + """Queue a compaction decision for the next model call's cost row. + + ``kind`` is one of the ``COMPACTION_EVENT_KINDS`` — ``applied`` (the + restore-time slice ran), ``checkpoint`` (a new checkpoint was cut + post-turn), ``forced`` and ``floor_unreachable`` (reserved for the + scheduling policy: a cut taken at the hard ceiling because the previous + one did not take, and a cut that could not reach its floor because the + protected tail alone exceeds it). ``fields`` are numbers only — + ``summaryTokens`` in particular is what proves a summary cap works + without another table scan. Anything else is dropped here so the + ledger can never carry content. + + No-op when ``COST_DIAGNOSTICS_ENABLED`` is off, so a row without the + field reads "not tracked", never "0". Bounded so a runaway caller + cannot grow a cost row. + """ + from apis.shared.feature_flags import cost_diagnostics_enabled + + if not cost_diagnostics_enabled(): + return + if kind not in COMPACTION_EVENT_KINDS: + logger.debug("Ignoring unknown compaction event kind %r", kind) + return + if len(self._pending_compaction_events) >= _MAX_PENDING_COMPACTION_EVENTS: + return + event: Dict[str, Any] = {"kind": kind} + for key, value in fields.items(): + if isinstance(value, bool) or not isinstance(value, (int, float)): + continue + event[key] = int(value) + self._pending_compaction_events.append(event) + + def drain_compaction_events(self) -> List[Dict[str, Any]]: + """Return and clear the queued events (called by ``ContextLedgerHook``).""" + events, self._pending_compaction_events = self._pending_compaction_events, [] + return events + def _save_compaction_state(self, state: CompactionState, record_event: bool = False) -> None: """Save compaction state to DynamoDB session metadata. @@ -731,38 +1041,101 @@ async def update_after_turn( self, input_tokens: int, current_messages: Optional[List[Dict]] = None, + context_window: Optional[int] = None, + history_tokens: Optional[int] = None, ) -> Optional[CompactionResult]: """ Update compaction state after a turn completes. - Called by StreamCoordinator with input token count from model response. - Triggers checkpoint creation when token threshold exceeded. + Called by StreamCoordinator with the turn's cache-inclusive input token + count. Resolves the model-relative policy (ceiling / floor / hard + ceiling — spec §3.1), applies the hysteresis rule (§3.3) and, when a + cut is due, chooses a floor-seeking checkpoint (§3.2) in absolute + coordinates (§3.4). Persists the new checkpoint + summary; the slice + itself is applied at the next restore by ``_apply_compaction``. Returns a ``CompactionResult`` when the checkpoint advances on this turn so the caller can emit a ``compaction`` SSE event; otherwise returns ``None``. - ``current_messages`` is the agent's live message list. When provided, - the cutoff cache is re-derived from it so compaction works even when - AgentCoreMemory loads messages via hooks (skipping the initialize-time - prime path). + Args: + input_tokens: cache-inclusive input tokens of the turn's last call. + current_messages: the agent's live message list. When provided, + the cutoff cache is re-derived from it so compaction works even + when AgentCoreMemory loads messages via hooks (skipping the + initialize-time prime path). + context_window: the model's ``maxInputTokens`` from the catalog, + or ``None`` when unknown (falls back to the fixed threshold). + history_tokens: measured size of the conversation portion of the + prompt (the ``messages`` partition of the context breakdown); + calibrates the per-message estimates. ``None`` → use + ``input_tokens``, which biases the cut slightly deeper. """ + # In-process "when did the last turn end" for the document-offload + # cache-gap decision when compaction (whose state stamps updated_at + # every turn) is off. Stamped before the early return on purpose. + self._last_turn_completed_at = datetime.now(timezone.utc).isoformat() + if not self.compaction_config or not self.compaction_config.enabled: return None # Re-read persisted state before touching it — on EVERY turn, not just # the first. See ``_adopt_persisted_compaction_state`` (#751). self._adopt_persisted_compaction_state() + state = self.compaction_state + state.last_input_tokens = input_tokens + if self._current_prefix_key: + state.last_prefix_key = self._current_prefix_key + + policy = CompactionPolicy.resolve(self.compaction_config, context_window) - self.compaction_state.last_input_tokens = input_tokens + if input_tokens <= policy.ceiling: + if policy.hysteresis_enabled and not state.armed: + logger.info( + "compaction_rearmed: input=%d <= ceiling=%d (source=%s)", + input_tokens, policy.ceiling, policy.source, + ) + state.armed = True + self._save_compaction_state(state) + return None - if input_tokens <= self.compaction_config.token_threshold: - self._save_compaction_state(self.compaction_state) + at_hard_ceiling = policy.hard_ceiling is not None and input_tokens >= policy.hard_ceiling + # "Forced" is the spiral signal: a cut that ran while DISARMED because + # the hard ceiling was reached. An armed cut above the hard ceiling is + # just a large ordinary cut. + forced = policy.hysteresis_enabled and not state.armed and at_hard_ceiling + if state.pending_checkpoint is not None and policy.hysteresis_enabled: + # A cut is already parked. Cutting deeper now would be the spiral; + # the head of the next turn applies it (hard ceiling included). + logger.info( + "compaction_pending_waiting: input=%d > ceiling=%d, pending_checkpoint=%d, hard=%s", + input_tokens, policy.ceiling, state.pending_checkpoint, policy.hard_ceiling, + ) + self._save_compaction_state(state) + return None + if policy.hysteresis_enabled and not state.armed and not at_hard_ceiling: + # The previous cut has not been observed to take effect yet (the + # slice lands at the next restore) — cutting again now is exactly + # the spiral. Wait for the context to drop under the ceiling, or + # for the hard ceiling. + logger.info( + "compaction_disarmed_noop: input=%d > ceiling=%d, hard=%d (source=%s)", + input_tokens, policy.ceiling, policy.hard_ceiling, policy.source, + ) + self._save_compaction_state(state) return None + if forced: + logger.warning( + "compaction_forced: input=%d >= hard_ceiling=%d while disarmed — " + "the previous cut did not bring the context under the ceiling " + "(summary too large, or slice not yet applied on this agent)", + input_tokens, policy.hard_ceiling, + ) logger.info( f"Threshold exceeded: {input_tokens:,} > " - f"{self.compaction_config.token_threshold:,}" + f"{policy.ceiling:,} (floor={policy.floor}, hard={policy.hard_ceiling}, " + f"window={policy.context_window}, source={policy.source})" ) # Refresh cutoff cache from the agent's current messages — at this @@ -780,10 +1153,11 @@ async def update_after_turn( if not self._valid_cutoff_indices: logger.info("No valid cutoff points cached, skipping checkpoint update") - self._save_compaction_state(self.compaction_state) + self._save_compaction_state(state) return None - total_turns = len(self._valid_cutoff_indices) + cutoffs = self._valid_cutoff_indices + total_turns = len(cutoffs) protected_turns = self.compaction_config.protected_turns if total_turns <= protected_turns: @@ -791,52 +1165,145 @@ async def update_after_turn( f"Only {total_turns} turns available (need > {protected_turns}), " f"keeping all messages" ) - self._save_compaction_state(self.compaction_state) + self._save_compaction_state(state) + return None + + # Choose the cut in LIVE-LIST coordinates, then translate to absolute. + messages = self._all_messages_for_summary + retained_estimate: Optional[int] = None + if policy.floor is None: + # Legacy (kill switch): keep the last N turns, whatever their size. + relative_cut = cutoffs[-protected_turns] + else: + relative_cut, retained_estimate = choose_checkpoint( + messages, + cutoffs, + protected_turns, + policy.floor, + history_tokens if history_tokens is not None else input_tokens, + ) + + if relative_cut <= 0: + # Everything already fits under the floor — the excess is system + # prompt / tools, which a history cut cannot fix. + logger.info( + "compaction_nothing_to_cut: retained≈%s <= floor=%s with input=%d", + retained_estimate, policy.floor, input_tokens, + ) + self._save_compaction_state(state) return None - new_checkpoint = self._valid_cutoff_indices[-protected_turns] - current_checkpoint = self.compaction_state.checkpoint + new_checkpoint = self._live_offset + relative_cut + current_checkpoint = state.checkpoint if new_checkpoint <= current_checkpoint: - self._save_compaction_state(self.compaction_state) + self._save_compaction_state(state) return None - logger.info(f"Checkpoint update: {current_checkpoint} -> {new_checkpoint}") + logger.info( + "compaction_cut: checkpoint %d -> %d (live_offset=%d, relative_cut=%d, " + "retained≈%s, floor=%s, ceiling=%d, hard=%s, window=%s, forced=%s)", + current_checkpoint, new_checkpoint, self._live_offset, relative_cut, + retained_estimate, policy.floor, policy.ceiling, policy.hard_ceiling, + policy.context_window, forced, + ) + # Per-call compaction ledger: the two decisions the spec asks to see + # on the anatomy — a cut that ran while disarmed, and a cut that could + # not reach the floor because the protected tail alone exceeds it. + if forced: + self._record_ledger_event("forced", checkpoint=new_checkpoint, inputTokens=input_tokens) + if policy.floor is not None and retained_estimate is not None and retained_estimate > policy.floor: + self._record_ledger_event( + "floor_unreachable", + checkpoint=new_checkpoint, inputTokens=input_tokens, retainedTokens=retained_estimate, + ) # Count turns rolled into the summary on THIS event (delta, not - # cumulative) — each inline divider stands on its own. + # cumulative) — each inline divider stands on its own. In absolute + # coordinates: turn starts at or after the previous checkpoint (they + # were retained by the last slice) and before the new one. summarized_turns = sum( - 1 for idx in self._valid_cutoff_indices - if current_checkpoint < idx <= new_checkpoint + 1 for idx in cutoffs + if current_checkpoint <= self._live_offset + idx < new_checkpoint ) - # Retrieve or generate summary for compacted messages + # Retrieve or generate the summary for the retired messages, then + # hold it at the budget (spec §3.6 / spiral spec PR-2). Compression + # runs here — once, at checkpoint advance, the turn that already pays + # a prefix re-write — and the result is persisted verbatim so every + # restore prepends identical bytes. summaries = self._retrieve_session_summaries() if summaries: - summary = "\n\n".join(summaries) + records = list(summaries) + summary_source = "ltm" else: - messages_to_summarize = self._all_messages_for_summary[:new_checkpoint] - summary = self._generate_fallback_summary(messages_to_summarize) - - self.compaction_state.checkpoint = new_checkpoint - # The anchor rides the checkpoint: everything the slice retains stays - # byte-identical until the next compaction-state change, so the single - # mutation (slice + summary) is paid with exactly one cache re-write. - self.compaction_state.truncation_anchor = max( - self.compaction_state.truncation_anchor, new_checkpoint + fallback = self._generate_fallback_summary(messages[:relative_cut]) + records = [fallback] if fallback else [] + summary_source = "fallback" + bounded = await bound_summary( + records, + self.compaction_config.summary_token_budget, + model_enabled=self.compaction_config.summary_model_enabled, + model_id=self.compaction_config.summary_model_id, + region=self.region_name, ) - self.compaction_state.summary = summary + summary = bounded.text + + deferred = bool(self.compaction_config.deferred_apply_enabled and policy.hysteresis_enabled) + if deferred: + # Park the cut. The bytes the model sees do not change until + # ``apply_pending_compaction`` decides the re-write is free or + # unavoidable (spec §3.5). + state.pending_checkpoint = new_checkpoint + state.pending_summary = summary + state.pending_hard_ceiling = policy.hard_ceiling + state.pending_since = datetime.now(timezone.utc).isoformat() + else: + state.checkpoint = new_checkpoint + # The anchor rides the checkpoint: everything the slice retains stays + # byte-identical until the next compaction-state change, so the single + # mutation (slice + summary) is paid with exactly one cache re-write. + state.truncation_anchor = max(state.truncation_anchor, new_checkpoint) + state.summary = summary # Running total persisted alongside the rest of the compaction state # so a refresh can rehydrate the end-of-conversation summary indicator. - self.compaction_state.total_summarized_turns += summarized_turns + state.total_summarized_turns += summarized_turns + if policy.hysteresis_enabled: + state.armed = False + state.policy = { + **policy.to_dict(), + "forced": forced, + "inputTokens": input_tokens, + "retainedTokensEstimate": retained_estimate, + # Summary provenance — what the admin profile and the cost + # anatomy read to explain a cut without reading the conversation. + "summarySource": summary_source, + "summaryOutcome": bounded.outcome, + "summaryTokensBefore": bounded.tokens_before, + "summaryTokensAfter": bounded.tokens_after, + "summaryTokenBudget": self.compaction_config.summary_token_budget, + "deferred": deferred, + } # This save is the compaction event itself — count it. - self._save_compaction_state(self.compaction_state, record_event=True) + self._save_compaction_state(state, record_event=True) + self._emit_compaction_metrics(state, policy, forced, retained_estimate, bounded, deferred) + # Routed through the defensive seam rather than calling the recorder + # directly: same no-op-without-a-ledger contract as every other cut + # decision. Queued when the cut is *decided* — when ``deferred`` the + # bytes do not move until ``apply_pending_compaction`` runs. + self._record_ledger_event( + "checkpoint", + checkpoint=new_checkpoint, + summaryTokens=_approx_tokens(summary), + summarizedTurns=summarized_turns, + inputTokens=input_tokens, + ) logger.info( f"Compaction checkpoint set: {new_checkpoint}, " f"summary_length={len(summary) if summary else 0}, " f"summarized_turns={summarized_turns}, " - f"total_summarized_turns={self.compaction_state.total_summarized_turns}" + f"total_summarized_turns={state.total_summarized_turns}" ) return CompactionResult( @@ -844,6 +1311,350 @@ async def update_after_turn( new_checkpoint=new_checkpoint, summarized_turns=summarized_turns, input_tokens=input_tokens, + context_window=policy.context_window, + ceiling=policy.ceiling, + floor=policy.floor, + hard_ceiling=policy.hard_ceiling, + forced=forced, + retained_tokens_estimate=retained_estimate, + deferred=deferred, + ) + + # ========================================================================= + # Paid-when-free application (spec §3.5) + # ========================================================================= + + def apply_pending_compaction(self, agent: "Agent", *, prefix_key: Optional[str] = None) -> Optional[str]: + """Head-of-turn: apply a parked cut to the live list if the re-write is free. + + Called by the stream coordinator before the first model call of every + turn, on cached and freshly restored agents alike. Promotes + ``pending_checkpoint`` to ``checkpoint`` and slices ``agent.messages`` + **in place** (slice assignment, never rebinding — the #741 alias) when + one of these holds, in this order: + + - ``cache_expired`` — more than ``cache_ttl_seconds`` since the last + turn: the Bedrock entry is gone and the next call re-writes the + prefix anyway, so the cut rides for free; + - ``prefix_changed`` — ``prefix_key`` (model|agent) differs from the + last turn's: the cached prefix is already invalid; + - ``hard_ceiling`` — the last turn's input reached the hard ceiling + the cut was computed under: waiting is no longer affordable. + + Otherwise the cut keeps waiting (``compaction_pending_waiting``). Returns + the reason applied, or ``None``. Never raises. + """ + if not self.compaction_config or not self.compaction_config.enabled: + return None + try: + self._adopt_persisted_compaction_state() + state = self.compaction_state + if state is None: + return None + + previous_key = state.last_prefix_key + self._current_prefix_key = prefix_key + if state.pending_checkpoint is None: + return None + + config = self.compaction_config + gap_seconds = self._seconds_since(state.updated_at) + reason: Optional[str] = None + if self._cache_window_expired(state.updated_at, config.cache_ttl_seconds): + reason = "cache_expired" + elif prefix_key and previous_key and prefix_key != previous_key: + reason = "prefix_changed" + elif state.pending_hard_ceiling is not None and state.last_input_tokens >= state.pending_hard_ceiling: + reason = "hard_ceiling" + + if reason is None: + logger.info( + "compaction_pending_waiting: pending_checkpoint=%d, gap=%ss, last_input=%d, hard=%s", + state.pending_checkpoint, gap_seconds, state.last_input_tokens, state.pending_hard_ceiling, + ) + return None + + messages = getattr(agent, "messages", None) + if not isinstance(messages, list): + return None + applied = self._apply_pending_in_place(messages, state) + if not applied: + # Unusable pending (would drop the whole live list): clear it + # rather than leave a cut that can never land. + state.pending_checkpoint = None + state.pending_summary = None + state.pending_hard_ceiling = None + state.pending_since = None + self._save_compaction_state(state) + return None + + promoted = state.pending_checkpoint + state.checkpoint = promoted + state.truncation_anchor = max(state.truncation_anchor, promoted) + state.summary = state.pending_summary + state.pending_checkpoint = None + state.pending_summary = None + state.pending_hard_ceiling = None + pending_since = state.pending_since + state.pending_since = None + state.policy = { + **(state.policy or {}), + "applied": reason, + "appliedAt": datetime.now(timezone.utc).isoformat(), + "cacheGapSeconds": gap_seconds, + "pendingSince": pending_since, + } + self._save_compaction_state(state) + # Per-call compaction ledger: the apply is the moment the bytes + # the model sees change, so it is the event the anatomy marks. + self._record_ledger_event( + "applied", + checkpoint=promoted, + summaryTokens=len(state.summary or "") // 4, + retainedMessages=len(messages), + cacheGapSeconds=gap_seconds or 0, + # Distinguishes the head-of-turn promotion from the restore + # slice's own "applied" event (both are real byte changes). + promoted=1, + ) + logger.info( + "compaction_applied: reason=%s checkpoint=%d gap=%ss live_len=%d (%s)", + reason, promoted, gap_seconds, len(messages), + "rewrite_forced" if reason == "hard_ceiling" else "rewrite_scheduled", + ) + self._emit_emf( + { + "CompactionApplied": 1, + "CompactionAppliedForced": 1 if reason == "hard_ceiling" else 0, + "CompactionCacheGapSeconds": int(gap_seconds or 0), + }, + {"applyReason": reason, "checkpoint": promoted}, + {"CompactionCacheGapSeconds": "Seconds"}, + ) + return reason + except Exception as e: # noqa: BLE001 - never break a turn + logger.warning(f"apply_pending_compaction skipped: {e}", exc_info=True) + return None + + # ========================================================================= + # Document offload (offload spec §4C / §4D, PR-4) + # ========================================================================= + + def apply_document_offload(self, agent: "Agent", *, prompt: Any = None) -> Optional[str]: + """Head-of-turn: swap unpinned, large inline documents for their digests + — and stub aged ``document_read`` page slices — when the prefix + re-write is free or unavoidable. + + Runs right after ``apply_pending_compaction`` on every turn. The + eligibility rules (pinning, minimum size) live in + ``document_offload.py``; this method owns the cache-gap decision, the + same one the parked cut uses: + + - ``cache_expired`` — more than ``cache_ttl_seconds`` since the last + turn (the Bedrock entry is gone; the next call re-writes anyway); + - ``prefix_changed`` — this turn's model|agent key differs from the + last turn's (the cached prefix is already invalid); + - ``over_ceiling`` — the last turn's input exceeded the compaction + ceiling (waiting is no longer affordable; the eviction is what + brings the prefix down). + + Otherwise nothing moves (``document_offload_waiting``). The replacement + is the restore path's own transformation, so the live block equals + what a cold restore would produce. Records ``document_offload`` on the + ledger with the cache gap. Never raises; returns the reason applied. + """ + from agents.main_agent.session.document_offload import ( + DOCUMENT_OFFLOAD_MIN_TOKENS, + DOCUMENT_OFFLOAD_PIN_TURNS, + DOCUMENT_SLICE_MAX_TURNS, + age_document_slices, + candidate_documents, + offload_documents, + offload_enabled_for, + pinned_document_names, + ) + + session_id = getattr(self.config, "session_id", None) + if not offload_enabled_for(session_id): + return None + try: + messages = getattr(agent, "messages", None) + if not isinstance(messages, list) or not messages: + return None + pinned = pinned_document_names(messages, prompt, pin_turns=DOCUMENT_OFFLOAD_PIN_TURNS) + candidates = candidate_documents(messages, pinned, min_tokens=DOCUMENT_OFFLOAD_MIN_TOKENS) + has_slices = any( + isinstance(b, dict) and isinstance(b.get("toolResult"), dict) + for m in messages if isinstance(m, dict) and isinstance(m.get("content"), list) + for b in m["content"] + ) + if not candidates and not has_slices: + return None + + reason, gap_seconds = self._document_offload_reason() + if reason is None: + if candidates: + logger.info( + "document_offload_waiting: candidates=%d gap=%ss (cache live, prefix unchanged, under ceiling)", + len(candidates), gap_seconds, + ) + return None + + result = offload_documents(messages, candidates, session_id=session_id, user_id=self.user_id) + result.slices_aged, result.slice_tokens = age_document_slices(messages, max_turns=DOCUMENT_SLICE_MAX_TURNS) + if not result.offloaded and not result.slices_aged: + return None + + logger.info( + "document_offload: reason=%s documents=%d evicted≈%d tok digests≈%d tok slices=%d gap=%ss unmatched=%d", + reason, result.offloaded, result.evicted_tokens, result.digest_tokens, + result.slices_aged, gap_seconds, result.skipped_unmatched, + ) + self._record_ledger_event( + "document_offload", + documents=result.offloaded, + documentTokens=result.evicted_tokens, + digestTokens=result.digest_tokens, + slices=result.slices_aged, + sliceTokens=result.slice_tokens, + cacheGapSeconds=gap_seconds if gap_seconds is not None else -1, + ) + self._emit_emf( + { + "DocumentOffloaded": result.offloaded, + "DocumentOffloadedTokens": int(result.evicted_tokens), + "DocumentSlicesAged": result.slices_aged, + "DocumentOffloadCacheGapSeconds": int(gap_seconds or 0), + }, + {"offloadReason": reason}, + {"DocumentOffloadedTokens": "Count", "DocumentOffloadCacheGapSeconds": "Seconds"}, + ) + return reason + except Exception as e: # noqa: BLE001 - never break a turn + logger.warning(f"apply_document_offload skipped: {e}", exc_info=True) + return None + + def _document_offload_reason(self) -> Tuple[Optional[str], Optional[int]]: + """``(reason, cache gap seconds)`` — why a prefix re-write is free or + unavoidable right now, or ``(None, gap)`` while the cache is live. + + Reads the compaction state when compaction is on (``updated_at`` is + stamped every turn, ``last_prefix_key`` / ``last_input_tokens`` too) + and the in-process last-turn timestamp otherwise, so the gate works + with compaction disabled as well. Conservative on any doubt. + """ + config = self.compaction_config + state = self.compaction_state if (config and config.enabled) else None + ttl = config.cache_ttl_seconds if config else Defaults.COMPACTION_CACHE_TTL_SECONDS + last_turn_at = (state.updated_at if state else None) or getattr(self, "_last_turn_completed_at", None) + gap = self._seconds_since(last_turn_at) + if self._cache_window_expired(last_turn_at, ttl): + return "cache_expired", gap + if state is not None: + current_key = getattr(self, "_current_prefix_key", None) + if current_key and state.last_prefix_key and current_key != state.last_prefix_key: + return "prefix_changed", gap + ceiling = (state.policy or {}).get("ceiling") if isinstance(state.policy, dict) else None + if ceiling is None: + try: + ceiling = CompactionPolicy.resolve(config, None).ceiling + except Exception: # noqa: BLE001 + ceiling = None + if ceiling and state.last_input_tokens and state.last_input_tokens > int(ceiling): + return "over_ceiling", gap + return None, gap + + def _apply_pending_in_place(self, messages: List[Dict], state: CompactionState) -> bool: + """Slice the live list at the pending checkpoint, in place. + + ``messages[0]`` sits at absolute index ``_live_offset``; the pending + checkpoint is absolute. Produces the same bytes ``_apply_compaction`` + would derive from stored history with the promoted state (slice, then + the summary prepended to the new first user message), so a later cold + restore matches the live prefix. + """ + pending = state.pending_checkpoint + if pending is None: + return False + k = pending - self._live_offset + if k <= 0: + # Live list already starts at/after the cut (e.g. a restore that + # sliced there). Nothing to remove; promotion still records it. + return True + if k >= len(messages): + logger.warning( + "compaction pending checkpoint %d is beyond the live list (offset=%d, len=%d); dropping it", + pending, self._live_offset, len(messages), + ) + return False + head = messages[k] + if state.pending_summary: + head = self._prepend_summary_to_first_message([head], state.pending_summary)[0] + messages[:] = [head] + messages[k + 1:] + self._live_offset = pending + return True + + @staticmethod + def _seconds_since(updated_at: Optional[str]) -> Optional[int]: + if not updated_at: + return None + try: + last = datetime.fromisoformat(updated_at.replace("Z", "+00:00")) + except (ValueError, TypeError): + return None + if last.tzinfo is None: + last = last.replace(tzinfo=timezone.utc) + return int((datetime.now(timezone.utc) - last).total_seconds()) + + @staticmethod + def _emit_emf(metrics: Dict[str, Any], properties: Dict[str, Any], units: Optional[Dict[str, str]] = None) -> None: + """One content-free EMF record in ``AgentCoreStack/Compaction``. Never raises. + + ``PROMPT_CACHE_OBSERVABILITY_ENABLED=false`` silences it with the rest + of the cost observability layer. + """ + try: + from apis.shared.observability.prompt_cache import prompt_cache_observability_enabled + from apis.shared.observability.emf import emit_emf_metrics + + if not prompt_cache_observability_enabled(): + return + emit_emf_metrics("AgentCoreStack/Compaction", metrics=metrics, properties=properties, units=units or {}) + except Exception as e: # noqa: BLE001 + logger.debug("Compaction EMF skipped: %s", e) + + def _emit_compaction_metrics(self, state, policy, forced, retained_estimate, bounded, deferred=False) -> None: + """One record per cut: how often cuts fire, how often they are forced + (the spiral detector), how big the summary is against its budget, how + deep cuts land, and whether the cut was parked for a free turn.""" + self._emit_emf( + { + "CompactionCut": 1, + "CompactionForced": 1 if forced else 0, + "CompactionDeferred": 1 if deferred else 0, + "CompactionInputTokens": int(state.last_input_tokens or 0), + "CompactionRetainedTokens": int(retained_estimate or 0), + "CompactionSummaryTokens": int(bounded.tokens_after or 0), + "CompactionSummaryOverBudget": 1 if bounded.tokens_before > bounded.tokens_after else 0, + # The protected tail alone exceeded the floor: the residual + # case after intake offload (attachments, sub-gate results). + "CompactionFloorUnreachable": ( + 1 if (policy.floor is not None and retained_estimate is not None and retained_estimate > policy.floor) else 0 + ), + }, + { + "policySource": policy.source, + "contextWindow": policy.context_window, + "ceiling": policy.ceiling, + "floor": policy.floor, + "summaryOutcome": bounded.outcome, + "summaryTokensBefore": bounded.tokens_before, + }, + { + "CompactionInputTokens": "Count", + "CompactionRetainedTokens": "Count", + "CompactionSummaryTokens": "Count", + }, ) # ========================================================================= @@ -970,7 +1781,8 @@ def _sanitize_restored_content_blocks(self, messages: List[Dict]) -> List[Dict]: return sanitized def _strip_document_bytes(self, messages: List[Dict]) -> List[Dict]: - """Replace document content blocks' inline bytes with a text placeholder. + """Replace document content blocks' inline bytes with their digest — or, + when the block cannot be matched to an upload, a text placeholder. Called unconditionally on every session restore — independent of whether compaction is enabled. Document blocks with ``source.bytes`` must never @@ -978,42 +1790,45 @@ def _strip_document_bytes(self, messages: List[Dict]) -> List[Dict]: document blocks share the same sanitized name across the conversation (ValidationException: "Messages can't contain duplicate document names"). + Since PR-3 of the offload spec the replacement is a ```` + block carrying the document's outline, abstract and ``upload_id``, so the + model still knows what the document says and can pull any page back with + ``document_read`` (see ``document_rehydration.py``). Blocks with no + upload row keep the contentless placeholder, exactly as before. + Images are handled the same way inside ``_truncate_tool_contents``, but that method is gated on compaction being enabled. This one is not. - The ``[Attached files: …]`` text marker already present in the user - message preserves the reference for the model without re-sending bytes. + Two content-free ledger events record what happened on the next cost + row: ``document_rehydrated`` (documents and their digest tokens) and + ``document_stripped`` (documents that fell back to the placeholder and + the tokens they were). The second going to zero is PR-3's gate. """ - stripped_messages = copy.deepcopy(messages) - strip_count = 0 - - for msg in stripped_messages: - content = msg.get("content", []) - if not isinstance(content, list): - continue - for block_idx, block in enumerate(content): - if not isinstance(block, dict) or "document" not in block: - continue - doc_data = block["document"] - source = doc_data.get("source", {}) - # Only replace blocks that carry inline bytes — s3Location - # blocks (enhancement #401) have no bytes to strip and are - # safe to leave as-is since they don't accumulate in history. - if "bytes" not in source: - continue - doc_name = doc_data.get("name", "unknown") - doc_format = doc_data.get("format", "unknown") - original_bytes = source.get("bytes", b"") - original_size = len(original_bytes) if isinstance(original_bytes, bytes) else 0 - content[block_idx] = { - "text": f"[Document placeholder: name={doc_name}, format={doc_format}, original_size={original_size} bytes]" - } - strip_count += 1 - - if strip_count > 0: - logger.debug(f"Stripped inline bytes from {strip_count} document block(s) in history") + from agents.main_agent.session.document_rehydration import rehydrate_documents - return stripped_messages + result = rehydrate_documents( + messages, + session_id=getattr(self.config, "session_id", None), + user_id=self.user_id, + ) + if result.rehydrated: + logger.info( + "Rehydrated %d document block(s) as digests (%d built lazily) in restored history", + result.rehydrated, result.lazy_digests, + ) + self.record_compaction_event( + "document_rehydrated", + documents=result.rehydrated, + documentTokens=result.digest_tokens, + ) + if result.stripped: + logger.debug(f"Stripped inline bytes from {result.stripped} unmatched document block(s) in history") + self.record_compaction_event( + "document_stripped", + documents=result.stripped, + documentTokens=result.stripped_tokens, + ) + return result.messages # ========================================================================= # Tool-pairing / role-alternation repair (restore-time safety net) diff --git a/backend/src/agents/main_agent/streaming/__init__.py b/backend/src/agents/main_agent/streaming/__init__.py index 239770f04..0e81a1f62 100644 --- a/backend/src/agents/main_agent/streaming/__init__.py +++ b/backend/src/agents/main_agent/streaming/__init__.py @@ -1,10 +1,8 @@ """Streaming coordination for Strands Agent""" from .stream_coordinator import StreamCoordinator from .stream_processor import process_agent_stream -from .tool_result_processor import ToolResultProcessor __all__ = [ "StreamCoordinator", "process_agent_stream", - "ToolResultProcessor", ] diff --git a/backend/src/agents/main_agent/streaming/event_formatter.py b/backend/src/agents/main_agent/streaming/event_formatter.py deleted file mode 100644 index c096ecc4a..000000000 --- a/backend/src/agents/main_agent/streaming/event_formatter.py +++ /dev/null @@ -1,177 +0,0 @@ -import json -import logging -from typing import Dict, Any, List, Tuple -from .tool_result_processor import ToolResultProcessor - -logger = logging.getLogger(__name__) - -class StreamEventFormatter: - """Handles formatting of streaming events for SSE""" - - @staticmethod - def format_sse_event(event_data: dict) -> str: - """Format event data as Server-Sent Event with proper JSON serialization""" - try: - return f"data: {json.dumps(event_data)}\n\n" - except (TypeError, ValueError) as e: - # Fallback for non-serializable objects - return f"data: {json.dumps({'type': 'error', 'message': f'Serialization error: {str(e)}'})}\n\n" - - @staticmethod - def extract_final_result_data(final_result) -> Tuple[List[Dict[str, str]], str]: - """Extract images and text from final result with simplified logic""" - images = [] - result_text = str(final_result) - - try: - if hasattr(final_result, 'message') and hasattr(final_result.message, 'content'): - content = final_result.message.content - text_parts = [] - - for item in content: - if isinstance(item, dict): - if "text" in item: - text_parts.append(item["text"]) - elif "image" in item and "source" in item["image"]: - # Simple image extraction - image_data = item["image"] - images.append({ - "format": image_data.get("format", "png"), - "data": image_data["source"].get("data", "") - }) - - if text_parts: - result_text = " ".join(text_parts) - - except Exception as e: - logger.warning("Failed to extract final result data: %s", e) - - return images, result_text - - @staticmethod - def create_init_event() -> str: - """Create initialization event""" - return StreamEventFormatter.format_sse_event({ - "type": "init", - "message": "Initializing..." - }) - - @staticmethod - def create_reasoning_event(reasoning_text: str) -> str: - """Create reasoning event""" - return StreamEventFormatter.format_sse_event({ - "type": "reasoning", - "text": reasoning_text, - "step": "thinking" - }) - - @staticmethod - def create_response_event(text: str) -> str: - """Create response event""" - return StreamEventFormatter.format_sse_event({ - "type": "response", - "text": text, - "step": "answering" - }) - - @staticmethod - def create_tool_use_event(tool_use: Dict[str, Any]) -> str: - """Create tool use event""" - return StreamEventFormatter.format_sse_event({ - "type": "tool_use", - "toolUseId": tool_use.get("toolUseId"), - "name": tool_use.get("name"), - "input": tool_use.get("input", {}) - }) - - @staticmethod - def create_tool_result_event( - tool_result: Dict[str, Any], - session_id: str = None, - tool_name: str = None - ) -> str: - """ - Create tool result event - uses ToolResultProcessor for processing. - - Args: - tool_result: The tool result dictionary - session_id: Optional session ID for file storage - tool_name: Optional tool name for specialized processing - """ - # Process the tool result using the dedicated processor - result_text, result_images = ToolResultProcessor.process_tool_result( - tool_result=tool_result, - session_id=session_id, - tool_name=tool_name - ) - - # Build and return the event - event = StreamEventFormatter._build_tool_result_event( - tool_result, result_text, result_images - ) - - return event - - @staticmethod - def _build_tool_result_event(tool_result: Dict[str, Any], result_text: str, result_images: List[Dict[str, str]]) -> str: - """Build the final tool result event""" - tool_result_data = { - "type": "tool_result", - "toolUseId": tool_result.get("toolUseId"), - "result": result_text - } - - if result_images: - tool_result_data["images"] = result_images - - # Include metadata if present (e.g., browserSessionId for Live View) - if "metadata" in tool_result: - tool_result_data["metadata"] = tool_result["metadata"] - - return StreamEventFormatter.format_sse_event(tool_result_data) - - @staticmethod - def create_complete_event(message: str, images: List[Dict[str, str]] = None, usage: Dict[str, Any] = None) -> str: - """Create completion event with optional token usage metrics""" - completion_data = { - "type": "complete", - "message": message - } - if images: - completion_data["images"] = images - if usage: - completion_data["usage"] = usage - - return StreamEventFormatter.format_sse_event(completion_data) - - @staticmethod - def create_error_event(error_message: str) -> str: - """Create error event""" - return StreamEventFormatter.format_sse_event({ - "type": "error", - "message": error_message - }) - - @staticmethod - def create_thinking_event(message: str = "Processing your request...") -> str: - """Create thinking event""" - return StreamEventFormatter.format_sse_event({ - "type": "thinking", - "message": message - }) - - @staticmethod - def create_progress_event(progress_data: Dict[str, Any]) -> str: - """Create progress event for tool execution""" - return StreamEventFormatter.format_sse_event({ - "type": "tool_progress", - "toolId": progress_data.get("toolId"), - "sessionId": progress_data.get("sessionId"), - "step": progress_data.get("step"), - "message": progress_data.get("message"), - "progress": progress_data.get("progress"), - "timestamp": progress_data.get("timestamp"), - "metadata": progress_data.get("metadata", {}) - }) - - diff --git a/backend/src/agents/main_agent/streaming/stream_coordinator.py b/backend/src/agents/main_agent/streaming/stream_coordinator.py index 477b9707d..52f98af2d 100644 --- a/backend/src/agents/main_agent/streaming/stream_coordinator.py +++ b/backend/src/agents/main_agent/streaming/stream_coordinator.py @@ -15,6 +15,11 @@ get_prefix_fingerprint, reset_prefix_fingerprints, ) +from agents.main_agent.session.hooks.context_attribution import get_prefix_token_split +from apis.shared.feature_flags import ( + agent_status_live_drain_enabled, + cost_diagnostics_enabled, +) from apis.shared.errors import ( ConversationalErrorEvent, ErrorCode, @@ -26,6 +31,28 @@ logger = logging.getLogger(__name__) +# How long the status merge waits on the agent stream before looking at the +# hook again. Small enough that "Running list_assignments" lands while that +# tool is actually running; large enough that a silent stretch costs a handful +# of wakeups a second and nothing else. +_STATUS_POLL_SECONDS = 0.1 + + +class _StatusFrame: + """A ready-to-emit ``agent_status`` SSE travelling with the agent's events. + + The merge yields these alongside the processed agent events so the + coordinator keeps ONE loop over ONE stream. A class rather than a tagged + dict because the loop body reads `event.get("type")` on everything else — + a dict would have to be excluded by a value check, and a frame whose text + happened to look like an event type would be a very unpleasant bug. + """ + + __slots__ = ("sse",) + + def __init__(self, sse: str) -> None: + self.sse = sse + class _CooperativeStopSignal(Exception): """Internal: a user Stop was observed mid-stream (cooperative cancellation). @@ -222,6 +249,7 @@ async def stream_response( original_message: Optional[str] = None, turn_agent_id: Optional[str] = None, turn_lease: Any = None, + turn_started_at: Optional[float] = None, ) -> AsyncGenerator[str, None]: """ Stream agent responses with proper lifecycle management @@ -272,6 +300,33 @@ async def stream_response( if session_manager is not None: session_manager.turn_lease = turn_lease + # Paid-when-free compaction (spec §3.5): if a cut is parked, apply it + # to the live list now — before the first model call — only when the + # prefix re-write is free (cache expired, model/agent switched) or + # unavoidable (hard ceiling). Runs on cached and freshly restored + # agents alike; the session manager decides, this just supplies the + # model|agent key. Best-effort: never blocks the turn. + if session_manager is not None and hasattr(session_manager, "apply_pending_compaction"): + try: + _model_for_key = getattr(getattr(main_agent_wrapper, "model_config", None), "model_id", None) + session_manager.apply_pending_compaction( + agent, prefix_key=f"{_model_for_key}|{turn_agent_id or 'default'}" + ) + except Exception as e: # noqa: BLE001 + logger.warning(f"apply_pending_compaction failed, continuing: {e}") + + # Document offload (offload spec §4C, PR-4): in the same head-of-turn + # slot, swap unpinned large documents for their digests and stub aged + # document_read slices — only when the re-write is free or + # unavoidable, decided by the session manager on the same cache-gap + # facts. The incoming prompt is passed so a document the user just + # named stays pinned. Best-effort: never blocks the turn. + if session_manager is not None and hasattr(session_manager, "apply_document_offload"): + try: + session_manager.apply_document_offload(agent, prompt=prompt) + except Exception as e: # noqa: BLE001 + logger.warning(f"apply_document_offload failed, continuing: {e}") + # Likewise a pause armed by a previous turn: if the user abandoned an # OAuth/tool-approval consent and just typed again, the still-armed # interrupt state makes Strands reject this turn's prompt outright. @@ -396,8 +451,30 @@ async def stream_response( # Get raw agent stream agent_stream = agent.stream_async(prompt) - # Process through new stream processor and format as SSE - async for event in process_agent_stream(agent_stream): + # Process through new stream processor and format as SSE. + # + # The status merge sits between the two so a transition recorded + # while the agent stream is SILENT still reaches the client — see + # `_merge_agent_status`. With its kill switch off this is the bare + # `process_agent_stream(...)` the loop has always consumed, and no + # `_StatusFrame` is ever produced. + processed_stream: AsyncGenerator[Any, None] = process_agent_stream( + agent_stream + ) + if agent_status_live_drain_enabled(): + processed_stream = self._merge_agent_status( + processed_stream, main_agent_wrapper, session_id + ) + + async for event in processed_stream: + # A status transition the merge picked up mid-silence. It is + # already a formatted SSE frame and describes nothing the rest + # of this body reasons about (no message index, no metadata, no + # persistence), so it passes straight through. + if isinstance(event, _StatusFrame): + yield event.sse + continue + # Cooperative stop. A user Stop arms a cancel on the session's # single-flight lease; the inference-api heartbeat observes it # and flips ``session_manager.cancelled``. Because a client @@ -625,6 +702,12 @@ async def stream_response( user_id=user_id, ): yield sse + for sse in await self._extract_browser_login_required_events( + agent, + session_id=session_id, + user_id=user_id, + ): + yield sse for sse in self._extract_preflight_consent_events(user_id): yield sse @@ -656,6 +739,25 @@ async def stream_response( # Add end-to-end latency to metrics for consistency final_metadata["metrics"]["latencyMs"] = int((stream_end_time - stream_start_time) * 1000) + # The same number, named for what it means, so the live + # stream and a reloaded conversation agree. `latencyMs` + # here is the whole turn, but the PERSISTED + # `endToEndLatency` prefers the provider's API-call + # time — so a client reading that field would show one + # number live and a smaller one after refresh. + # + # Measured from `turn_started_at` — the moment the + # invocation reached the container — NOT from + # `stream_start_time`, which is when THIS generator + # began. Those diverged the moment the agent build was + # deferred into the stream (PR-3): the build now runs + # before `stream_response` is ever iterated, so + # `stream_start_time` excludes it. Measured on dev, a + # turn the user waited 7.8s for reported 2.1s. + final_metadata["turnDurationMs"] = int( + (stream_end_time - (turn_started_at or stream_start_time)) * 1000 + ) + # Cost: sum the FINAL usage of each assistant message in # this turn and price it. We deliberately price each # message independently and sum, instead of pricing @@ -667,6 +769,9 @@ async def stream_response( # persisted (one C# record per assistant message). if main_agent_wrapper and hasattr(main_agent_wrapper, "model_config"): model_id = main_agent_wrapper.model_config.model_id + # PR-5: when the static prefix carries the 1h TTL, + # its cache writes are billed at the 1h premium. + long_ttl_static_tokens = self._long_ttl_static_prefix_tokens(main_agent_wrapper, agent) try: turn_total = 0.0 turn_input_cost = 0.0 @@ -680,6 +785,7 @@ async def stream_response( msg_cost = await self._calculate_streaming_cost( model_id=model_id, usage=msg_usage, + long_ttl_static_prefix_tokens=long_ttl_static_tokens, ) if msg_cost is None: continue @@ -771,9 +877,18 @@ async def stream_response( if total_input_tokens > 0: try: current_messages = getattr(agent, "messages", None) + # Model-relative policy inputs: the catalog + # window (same lookup the badge uses — the + # catalog is cached) and the measured size of + # the conversation portion of the prompt. + # See docs/specs/compaction-model-relative-thresholds.md. + turn_context_window = await self._resolve_context_window(main_agent_wrapper) + history_tokens = self._history_tokens_from_breakdown(agent) compaction_result = await session_manager.update_after_turn( total_input_tokens, current_messages=current_messages, + context_window=turn_context_window, + history_tokens=history_tokens, ) logger.info(f" Compaction state updated: {total_input_tokens:,} input tokens") if compaction_result is not None: @@ -783,6 +898,14 @@ async def stream_response( "newCheckpoint": compaction_result.new_checkpoint, "summarizedTurns": compaction_result.summarized_turns, "inputTokens": compaction_result.input_tokens, + # Additive policy fields (the SPA + # validator ignores unknown keys). + "contextWindow": compaction_result.context_window, + "ceiling": compaction_result.ceiling, + "floor": compaction_result.floor, + "hardCeiling": compaction_result.hard_ceiling, + "forced": compaction_result.forced, + "retainedTokensEstimate": compaction_result.retained_tokens_estimate, } yield f"event: compaction\ndata: {json.dumps(compaction_payload)}\n\n" except Exception as e: @@ -1001,9 +1124,13 @@ async def stream_response( # Live narration: the status hook records model-call and # tool-call boundaries from inside Strands' event loop, which # has no route to the SSE stream. Drained here, before the - # event it precedes is yielded, so "Using list_assignments" - # reaches the client while that tool is actually running - # rather than after its result. + # event it precedes is yielded. + # + # With the live drain on, `_merge_agent_status` has usually + # taken these already and this finds nothing — it is kept + # because it is the ONLY drain when that kill switch is off, + # and because a transition recorded in the gap between the + # merge's last poll and this event still lands in order. for status_sse in self._drain_agent_status_events( main_agent_wrapper, session_id ): @@ -1181,6 +1308,9 @@ async def stream_response( # cost row carries the tools that call requested. None when the # wrapper has no hook (tests, older agents) or the census is off. tool_census_hook = getattr(main_agent_wrapper, "tool_census_hook", None) + # Same discipline for the context ledger (window trims + + # compaction decisions per call). + context_ledger_hook = getattr(main_agent_wrapper, "context_ledger_hook", None) # Build list of metadata storage tasks for parallel execution metadata_tasks = [] @@ -1232,11 +1362,29 @@ async def stream_response( agent=main_agent_wrapper, # Use wrapper instead of internal agent citations=citations_for_message, # Pass citations for persistence call_index=idx, # Nth model call of this turn (prefix fingerprint lookup) + # Turn-level, so only the LAST message carries it. + # The per-message `endToEndLatency` cannot stand in: + # it prefers the provider's own API-call time, so + # summing it across a turn silently drops tool + # execution and the pre-stream agent build — a turn + # the user watched for 9s would read as 3s. + turn_duration_ms=( + int( + (stream_end_time - (turn_started_at or stream_start_time)) + * 1000 + ) + if idx == len(message_ids_to_store) - 1 + else None + ), turn_agent_id=turn_agent_id, # Which Agent ran this turn (#756) tool_calls=( tool_census_hook.tally_for_call(idx) if tool_census_hook is not None else None ), + context_ledger=( + context_ledger_hook.ledger_for_call(idx) + if context_ledger_hook is not None else None + ), ) ) @@ -1722,6 +1870,7 @@ async def _persist_paused_turn_snapshot( inference_params=dict(inference_params) if inference_params else None, mantle_api_mode=snapshot_source.get("mantle_api_mode"), mantle_region=snapshot_source.get("mantle_region"), + assistant_id=snapshot_source.get("assistant_id"), captured_at=now.isoformat(), expires_at=(now + timedelta(hours=1)).isoformat(), ) @@ -2025,6 +2174,128 @@ async def _extract_user_question_required_events( ) return events + async def _extract_browser_login_required_events( + self, + agent: Any, + session_id: Optional[str] = None, + user_id: Optional[str] = None, + ) -> List[str]: + """Yield one SSE-formatted `browser_login_required` event per pending + ``request_user_login`` interrupt, and project the browser session onto + the metadata row so app-api can mint a live view for it. + + Structurally the same as + :meth:`_extract_user_question_required_events` — a tool-raised + interrupt Strands routes through ``_stop_for_interrupts``, so the + ``PausedTurnSnapshot`` written on this same ``done`` event covers it. + + Two things are specific to this flavor: + + * **The conversation id is added here, not in the tool.** The tool + knows only which *browser* session it handed over; + ``sessionId`` on the event means the conversation, as it does on + every other SSE event, and this is the layer that has it. + * **The D4 projection is written here too**, for the same reason: the + write is keyed by conversation and owner, which the tool does not + know. It is best-effort — without it the prompt still renders and + the turn still resumes on skip; the user just has no working viewer. + + Nothing in either payload is a URL, and + :func:`apis.shared.browser_takeover.assert_no_url` enforces that rather + than trusting it. + """ + from apis.shared.browser_takeover import ( + BrowserLoginRequiredEvent, + BrowserSessionRef, + assert_no_url, + encode_ref, + ) + from apis.shared.sessions.metadata import ( + add_pending_interrupt, + set_browser_session, + ) + from apis.shared.sessions.models import PendingInterrupt + + interrupt_state = getattr(agent, "_interrupt_state", None) + if not interrupt_state or not getattr(interrupt_state, "activated", False): + return [] + + events: List[str] = [] + for interrupt in interrupt_state.interrupts.values(): + reason = interrupt.reason or {} + if not isinstance(reason, dict) or reason.get("type") != "browser_login_required": + continue + + try: + ref = BrowserSessionRef.model_validate(reason) + except Exception as e: # noqa: BLE001 - never break the stream + logger.warning( + "Browser-login interrupt carries an unusable session ref " + "(id=%s): %s", + interrupt.id, e, + ) + continue + + tool_use_id = reason.get("toolUseId", "") + prompt_reason = reason.get("reason") + + if session_id and user_id: + try: + await set_browser_session( + session_id=session_id, + user_id=user_id, + ref=ref.model_dump(by_alias=True, exclude_none=True), + ) + except Exception as e: + logger.error( + "Failed to project browser_session for %s: %s", + session_id, e, exc_info=True, + ) + + try: + await add_pending_interrupt( + session_id=session_id, + user_id=user_id, + interrupt=PendingInterrupt( + interrupt_id=interrupt.id, + kind="browser_login", + tool_use_id=tool_use_id, + tool_name="request_user_login", + browser_session=encode_ref(ref), + reason=prompt_reason, + created_at=datetime.now(timezone.utc).isoformat(), + ), + ) + except Exception as e: + logger.error( + "Failed to persist browser_login pending_interrupt %s: %s", + interrupt.id, e, exc_info=True, + ) + + event = BrowserLoginRequiredEvent( + interrupt_id=interrupt.id, + tool_use_id=tool_use_id, + session_id=session_id or "", + browser_session_id=ref.browser_session_id, + browser_id=ref.browser_id, + viewport=ref.viewport, + deadline_at=ref.deadline_at, + target_url=ref.target_url, + reason=prompt_reason, + # Same origin MCP Apps are framed from, and for the same + # reasons: its CloudFront function locks `frame-ancestors` to + # the SPA and composes `connect-src` from `?csp=`, which is how + # the viewer is allowed to open DCV's WebSocket. Empty when the + # sandbox origin is not deployed — the SPA then shows the + # prompt without a viewer rather than framing nothing. + sandbox_origin=os.environ.get( + "AGENTCORE_MCP_APPS_SANDBOX_ORIGIN", "" + ).strip(), + ) + assert_no_url(event.model_dump(by_alias=True, exclude_none=True)) + events.append(event.to_sse_format()) + return events + async def _extract_artifact_events( self, session_id: Optional[str], @@ -2380,6 +2651,96 @@ def _drain_steering_events( ) return events + async def _merge_agent_status( + self, + events: AsyncGenerator[Dict[str, Any], None], + main_agent_wrapper: Any, + session_id: str, + ) -> AsyncGenerator[Any, None]: + """Yield the agent's events, interleaved with status transitions as they happen. + + WHAT THIS FIXES + --------------- + ``_drain_agent_status_events`` is called from the coordinator's emit + loop, which only regains control when the agent stream yields. During + tool execution the agent stream yields NOTHING, so a ``tool_start`` sat + in the hook's queue for exactly the silence it existed to explain and + arrived bundled with its own ``tool_end``. Measured on a three-tool + browse turn: the indicator read "Thinking" for all 4.5s and never named + a tool. The SPA worked around it by deriving the running tool from the + content stream instead — correct, but it leaves every OTHER phase + (model call, batch shape) unreachable. + + HOW + --- + Race the agent stream's next event against a short timer. On a timeout, + drain the hook and yield whatever it recorded; on an event, drain first + (so a status still precedes the event it describes, exactly as before) + and then yield the event. + + The in-flight ``__anext__`` is deliberately kept across timeouts rather + than re-requested: an async generator cannot have two ``__anext__`` + calls outstanding, and re-creating it would drop events. + + ON EARLY EXIT + ------------- + The coordinator abandons this stream on several paths (cooperative + stop, max_tokens, a conversational error). Closing an async generator + runs the ``finally`` below, which cancels the in-flight ``__anext__``. + That throws ``CancelledError`` into ``process_agent_stream`` at its + yield point and unwinds it — the cleanup-on-cancellation path that + generator already documents and already took when the coordinator + consumed it directly. + + Best-effort in both directions: with no hook (voice, tests) or a failing + drain this degrades to a plain pass-through of the agent stream. + """ + iterator = events.__aiter__() + pending: Optional[asyncio.Future] = None + try: + while True: + if pending is None: + pending = asyncio.ensure_future(iterator.__anext__()) + + done, _ = await asyncio.wait({pending}, timeout=_STATUS_POLL_SECONDS) + + # On EVERY pass, including the ones where the agent stream + # produced nothing — which is the entire point of this merge. + for sse in self._drain_agent_status_events( + main_agent_wrapper, session_id + ): + yield _StatusFrame(sse) + + if not done: + continue + + completed, pending = pending, None + try: + event = completed.result() + except StopAsyncIteration: + # A transition recorded in the same instant the stream + # ended still belongs to this turn — most often the final + # `tool_end` of the last batch. + for sse in self._drain_agent_status_events( + main_agent_wrapper, session_id + ): + yield _StatusFrame(sse) + return + + yield event + finally: + if pending is not None and not pending.done(): + pending.cancel() + try: + await pending + except (asyncio.CancelledError, StopAsyncIteration): + pass + except Exception: # noqa: BLE001 - the turn is already ending + logger.debug( + "Agent stream raised while cancelling the status merge", + exc_info=True, + ) + def _drain_agent_status_events( self, main_agent_wrapper: Any, session_id: str ) -> List[str]: @@ -2565,6 +2926,46 @@ def _format_sse_event(self, event: Dict[str, Any]) -> str: logger.error(f"Failed to serialize event: {e}") return f"event: error\ndata: {json.dumps({'error': f'Serialization error: {str(e)}'})}\n\n" + @staticmethod + async def _resolve_context_window(main_agent_wrapper: Any) -> Optional[int]: + """The serving model's ``maxInputTokens`` from the catalog, or ``None``. + + Feeds the model-relative compaction policy. Best-effort: a miss means + the policy falls back to the fixed threshold, never an error. + """ + model_config = getattr(main_agent_wrapper, "model_config", None) + model_id = getattr(model_config, "model_id", None) + if not model_id: + return None + try: + from apis.shared.costs.pricing_config import get_model_by_model_id + + record = await get_model_by_model_id(model_id) + value = getattr(record, "max_input_tokens", None) if record is not None else None + return int(value) if value else None + except Exception as e: # noqa: BLE001 - never let a lookup break the turn + logger.debug(f"Skipping contextWindow lookup for compaction: {e}") + return None + + @staticmethod + def _history_tokens_from_breakdown(agent: Any) -> Optional[int]: + """The ``messages`` partition of this turn's context breakdown, or ``None``. + + Calibrates the compaction policy's per-message estimates against the + measured size of the conversation portion of the prompt. + """ + try: + from agents.main_agent.session.hooks.context_attribution import get_context_breakdown + + breakdown = get_context_breakdown(agent) + for partition in (breakdown or {}).get("partitions", []) or []: + if isinstance(partition, dict) and partition.get("key") == "messages": + tokens = partition.get("tokens") + return int(tokens) if tokens is not None else None + except Exception as e: # noqa: BLE001 + logger.debug(f"Skipping history-token calibration: {e}") + return None + def _log_cache_metrics(self, usage: Dict[str, Any], session_id: str) -> None: """ Log cache performance metrics for monitoring and optimization. @@ -2817,6 +3218,8 @@ async def _store_message_metadata( call_index: Optional[int] = None, turn_agent_id: Optional[str] = None, tool_calls: Optional[Dict[str, Dict[str, int]]] = None, + context_ledger: Optional[Dict[str, Any]] = None, + turn_duration_ms: Optional[int] = None, ) -> None: """ Store message-level metadata (token usage, latency, model info, citations) @@ -2940,7 +3343,13 @@ async def _store_message_metadata( # Calculate cost if we have both usage and pricing if token_usage and pricing_snapshot: - cost_result = self._calculate_message_cost(usage=accumulated_metadata.get("usage", {}), pricing=pricing_snapshot) + cost_result = self._calculate_message_cost( + usage=accumulated_metadata.get("usage", {}), + pricing=pricing_snapshot, + long_ttl_static_prefix_tokens=self._long_ttl_static_prefix_tokens( + agent, getattr(agent, "agent", None) + ), + ) if cost_result is not None: cost = cost_result @@ -2968,6 +3377,15 @@ async def _store_message_metadata( ) if context_window is not None: metadata_kwargs["contextWindow"] = context_window + # PR-5 experiment arm marker (extra field via extra="allow"), + # so the cost anatomy can split 1h-static-prefix turns from + # the 5m baseline without guessing from the write:read shape. + try: + if agent is not None and getattr(agent, "model_config", None) is not None and \ + getattr(agent.model_config, "long_ttl_static_prefix", lambda: False)(): + metadata_kwargs["staticPrefixTtl"] = "1h" + except Exception: # noqa: BLE001 + pass # Prompt-cache prefix fingerprints for this model call # (extra field via extra="allow"; persisted on the cost row @@ -2996,6 +3414,50 @@ async def _store_message_metadata( if tool_calls: metadata_kwargs["toolCalls"] = tool_calls + # Context ledger for this call: the conversation window's + # cumulative trim count (a rise between consecutive rows is a + # trim, i.e. a prefix re-write) and the compaction decisions + # taken since the previous call, each with the summary's + # token size. Plus the agent's stable prefix split (system / + # tools tokens) so "how big is the static prefix, and how much + # of it is tool schemas" is a stored fact. All numbers. + if context_ledger: + removed = context_ledger.get("windowRemovedMessages") + if removed is not None: + metadata_kwargs["windowRemovedMessages"] = removed + events = context_ledger.get("compactionEvents") + if events: + metadata_kwargs["compactionEvents"] = events + # document_read retrievals this call requested (calls / + # pages / bytes) — numbers read off the tool's own + # metadata, never the document. + reads = context_ledger.get("documentReads") + if reads: + metadata_kwargs["documentReads"] = reads + if strands_agent is not None and cost_diagnostics_enabled(): + prefix_tokens = get_prefix_token_split(strands_agent) + if prefix_tokens: + metadata_kwargs["prefixTokens"] = prefix_tokens + # The attachment footprint of the live context: inline + # documents (count, estimated tokens, format mix), digest + # stand-ins, and retrieved page slices. Flat fields so a + # query can split rows by hasDocuments / documentDigests + # without reading the conversation + # (docs/specs/document-context-offload.md §6.1). + try: + from agents.main_agent.session.document_context import summarize_document_context + + footprint = summarize_document_context(getattr(strands_agent, "messages", None)) + if footprint: + metadata_kwargs.update(footprint) + except Exception as doc_err: # noqa: BLE001 - never block the cost row + logger.debug(f"Skipping document context summary: {doc_err}") + + # Turn-level: present only on the turn's last message, which + # is where the SPA anchors the end-of-turn recap. + if turn_duration_ms is not None: + metadata_kwargs["turn_duration_ms"] = turn_duration_ms + message_metadata = MessageMetadata(**metadata_kwargs) # Store metadata @@ -3052,6 +3514,35 @@ def _extract_model_version(self, model_id: str) -> Optional[str]: return part.split(":")[0] return None + @staticmethod + def _long_ttl_static_prefix_tokens(main_agent_wrapper: Any, strands_agent: Any) -> Optional[int]: + """Size of the tools + system segment when it carries the 1h cache TTL, else ``None``. + + PR-5 (thresholds spec §3.6): Bedrock bills a 1h cache write at 2x base, + not the 1.25x the catalog's cacheWritePricePerMtok carries, and usage + does not split writes by TTL. The context-attribution breakdown knows + the static segment's size, and the read count tells whether it was + written this call (see CostCalculator.calculate_message_cost). + """ + try: + model_config = getattr(main_agent_wrapper, "model_config", None) + predicate = getattr(model_config, "long_ttl_static_prefix", None) + if not callable(predicate) or not predicate(): + return None + from agents.main_agent.session.hooks.context_attribution import get_context_breakdown + + breakdown = get_context_breakdown(strands_agent) if strands_agent is not None else None + if not breakdown: + return None + total = 0 + for partition in breakdown.get("partitions", []) or []: + if isinstance(partition, dict) and partition.get("key") in ("system", "tools"): + total += int(partition.get("tokens") or 0) + return total or None + except Exception as e: # noqa: BLE001 + logger.debug(f"long-TTL static prefix size unavailable: {e}") + return None + async def _get_pricing_snapshot(self, model_id: str) -> Optional[Dict[str, Any]]: """ Get pricing snapshot from managed models database @@ -3080,7 +3571,12 @@ async def _get_pricing_snapshot(self, model_id: str) -> Optional[Dict[str, Any]] logger.error(f"Failed to get pricing snapshot for {model_id}: {e}") return None - def _calculate_message_cost(self, usage: Dict[str, Any], pricing: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + def _calculate_message_cost( + self, + usage: Dict[str, Any], + pricing: Optional[Dict[str, Any]], + long_ttl_static_prefix_tokens: Optional[int] = None, + ) -> Optional[Dict[str, Any]]: """ Calculate message cost from usage and pricing @@ -3103,7 +3599,9 @@ def _calculate_message_cost(self, usage: Dict[str, Any], pricing: Optional[Dict[ else: pricing_dict = pricing - total_cost, breakdown = CostCalculator.calculate_message_cost(usage, pricing_dict) + total_cost, breakdown = CostCalculator.calculate_message_cost( + usage, pricing_dict, long_ttl_static_prefix_tokens=long_ttl_static_prefix_tokens + ) return { "total": total_cost, "inputCost": breakdown.input_cost, @@ -3116,7 +3614,12 @@ def _calculate_message_cost(self, usage: Dict[str, Any], pricing: Optional[Dict[ logger.error(f"Failed to calculate message cost: {e}") return None - async def _calculate_streaming_cost(self, model_id: str, usage: Dict[str, Any]) -> Optional[Dict[str, Any]]: + async def _calculate_streaming_cost( + self, + model_id: str, + usage: Dict[str, Any], + long_ttl_static_prefix_tokens: Optional[int] = None, + ) -> Optional[Dict[str, Any]]: """ Calculate cost for streaming response to send to client in real-time. @@ -3151,7 +3654,7 @@ async def _calculate_streaming_cost(self, model_id: str, usage: Dict[str, Any]) ) # Calculate cost using the calculator - return self._calculate_message_cost(usage, pricing) + return self._calculate_message_cost(usage, pricing, long_ttl_static_prefix_tokens=long_ttl_static_prefix_tokens) except Exception as e: logger.warning(f"Failed to calculate streaming cost: {e}") diff --git a/backend/src/agents/main_agent/streaming/tool_result_processor.py b/backend/src/agents/main_agent/streaming/tool_result_processor.py deleted file mode 100644 index 3060fc7b3..000000000 --- a/backend/src/agents/main_agent/streaming/tool_result_processor.py +++ /dev/null @@ -1,375 +0,0 @@ -""" -Tool Result Processor - -Handles post-processing of tool results including: -- Image extraction from various formats -- Base64 file saving to disk -- JSON content processing -- Text cleaning for display - -This is a separate concern from stream processing and should be called -by the router/service layer after receiving tool results. -""" - -import json -import base64 -import os -import re -from typing import Dict, Any, List, Tuple - - -class ToolResultProcessor: - """Processes tool results for image extraction and file operations""" - - @staticmethod - def process_tool_result( - tool_result: Dict[str, Any], - session_id: str = None, - tool_name: str = None - ) -> Tuple[str, List[Dict[str, str]]]: - """ - Process a tool result to extract content and images. - - Args: - tool_result: The tool result dictionary with content field - session_id: Session ID for file storage - tool_name: Name of the tool that generated this result - - Returns: - Tuple of (processed_text, list_of_images) - - Example: - text, images = ToolResultProcessor.process_tool_result( - tool_result={"content": [{"text": "Result"}]}, - session_id="abc123", - tool_name="run_python_code" - ) - """ - # Handle case where entire tool_result might be a JSON string - if isinstance(tool_result, str): - try: - tool_result = json.loads(tool_result) - except json.JSONDecodeError: - tool_result = { - "toolUseId": "unknown", - "content": [{"text": str(tool_result)}] - } - - # Extract all content (text and images) and process Base64 - result_text, result_images = ToolResultProcessor._extract_all_content(tool_result) - - # Handle storage based on tool type (if session_id provided) - if session_id and tool_name: - tool_use_id = tool_result.get("toolUseId") - result_text = ToolResultProcessor._handle_tool_storage( - tool_use_id, tool_name, result_text, session_id - ) - - return result_text, result_images - - @staticmethod - def _extract_all_content(tool_result: Dict[str, Any]) -> Tuple[str, List[Dict[str, str]]]: - """Extract text content and images from tool result""" - # Extract basic content from MCP format - result_text, result_images = ToolResultProcessor._extract_basic_content(tool_result) - - # Process JSON content for screenshots and additional images - json_images, cleaned_text = ToolResultProcessor._process_json_content(result_text) - result_images.extend(json_images) - - return cleaned_text, result_images - - @staticmethod - def _extract_basic_content(tool_result: Dict[str, Any]) -> Tuple[str, List[Dict[str, str]]]: - """Extract basic text and image content from MCP format""" - result_text = "" - result_images = [] - - # Handle case where content might be a JSON string - if "content" in tool_result and isinstance(tool_result["content"], str): - try: - parsed_content = json.loads(tool_result["content"]) - tool_result = tool_result.copy() - tool_result["content"] = parsed_content - except json.JSONDecodeError: - # Content is not valid JSON; leave it as a plain string - pass - - if "content" in tool_result: - content = tool_result["content"] - - for item in content: - if isinstance(item, dict): - if "text" in item: - text_content = item["text"] - - # Check if this text is actually a JSON-stringified MCP response - if text_content.strip().startswith('{"status":') and '"content":[' in text_content: - try: - parsed_mcp = json.loads(text_content) - - # Replace the current tool_result with the parsed MCP response - if "content" in parsed_mcp and isinstance(parsed_mcp["content"], list): - # Recursively process the unwrapped content - for unwrapped_item in parsed_mcp["content"]: - if isinstance(unwrapped_item, dict): - if "text" in unwrapped_item: - result_text += unwrapped_item["text"] - elif "image" in unwrapped_item and "source" in unwrapped_item["image"]: - image_source = unwrapped_item["image"]["source"] - image_data = "" - - if "data" in image_source: - image_data = image_source["data"] - elif "bytes" in image_source: - if isinstance(image_source["bytes"], bytes): - image_data = base64.b64encode( - image_source["bytes"] - ).decode('utf-8') - else: - image_data = str(image_source["bytes"]) - - if image_data: - result_images.append({ - "format": unwrapped_item["image"].get("format", "png"), - "data": image_data - }) - - # Skip the normal text processing since we handled the unwrapped content - continue - except json.JSONDecodeError: - # Fall through to normal text processing - pass - - # Normal text processing (if not unwrapped) - result_text += text_content - - elif "image" in item: - if "source" in item["image"]: - image_source = item["image"]["source"] - image_data = "" - - if "data" in image_source: - image_data = image_source["data"] - elif "bytes" in image_source: - if isinstance(image_source["bytes"], bytes): - image_data = base64.b64encode( - image_source["bytes"] - ).decode('utf-8') - else: - image_data = str(image_source["bytes"]) - - if image_data: - result_images.append({ - "format": item["image"].get("format", "png"), - "data": image_data - }) - - return result_text, result_images - - @staticmethod - def _process_json_content(result_text: str) -> Tuple[List[Dict[str, str]], str]: - """Process JSON content to extract screenshots and clean text""" - try: - parsed_result = json.loads(result_text) - extracted_images = ToolResultProcessor._extract_images_from_json_response(parsed_result) - - if extracted_images: - cleaned_text = ToolResultProcessor._clean_result_text_for_display( - result_text, parsed_result - ) - return extracted_images, cleaned_text - else: - return [], result_text - - except (json.JSONDecodeError, TypeError): - return [], result_text - - @staticmethod - def _extract_images_from_json_response(response_data: Dict[str, Any]) -> List[Dict[str, str]]: - """Extract images from any JSON tool response automatically""" - images = [] - - if isinstance(response_data, dict): - # Support common image field patterns - image_fields = ['screenshot', 'image', 'diagram', 'chart', 'visualization', 'figure'] - - for field in image_fields: - if field in response_data and isinstance(response_data[field], dict): - img_data = response_data[field] - - # Handle new lightweight screenshot format (Nova Act optimized) - if img_data.get("available") and "description" in img_data: - # This is the new optimized format - no actual image data - print(f"📷 Found optimized screenshot reference: {img_data.get('description')}") - continue - - # Handle legacy format with actual base64 data - elif "data" in img_data and "format" in img_data: - images.append({ - "format": img_data["format"], - "data": img_data["data"] - }) - - # Preserve existing images array - if "images" in response_data and isinstance(response_data["images"], list): - images.extend(response_data["images"]) - - return images - - @staticmethod - def _clean_result_text_for_display(original_text: str, parsed_result: dict) -> str: - """Clean result text by removing large image data but keeping other information""" - try: - import copy - - # Create a copy to avoid modifying the original - cleaned_result = copy.deepcopy(parsed_result) - - # Remove large image data fields but keep metadata - image_fields = ['screenshot', 'image', 'diagram', 'chart', 'visualization', 'figure'] - - for field in image_fields: - if field in cleaned_result and isinstance(cleaned_result[field], dict): - if "data" in cleaned_result[field]: - # Keep format and size info, remove the large base64 data - data_size = len(cleaned_result[field]["data"]) - cleaned_result[field] = { - "format": cleaned_result[field].get("format", "unknown"), - "size": f"{data_size} characters", - "note": "Image data extracted and displayed separately" - } - - # Return the cleaned JSON string - return json.dumps(cleaned_result, indent=2) - - except Exception as e: - # If cleaning fails, return the original - print(f"Warning: Failed to clean result text: {e}") - return original_text - - @staticmethod - def _handle_tool_storage( - tool_use_id: str, - tool_name: str, - result_text: str, - session_id: str - ) -> str: - """Process Base64 downloads for Python MCP tools""" - # Only process for Python MCP tools - if tool_name not in ['run_python_code', 'finalize_document']: - return result_text - - try: - processed_text, file_info = ToolResultProcessor._handle_python_mcp_base64( - tool_use_id, result_text, session_id - ) - if file_info: - print(f"📁 Processed {len(file_info)} files for {tool_use_id}") - return processed_text - except Exception as e: - print(f"⚠️ Error processing Base64 downloads: {e}") - - return result_text - - @staticmethod - def _handle_python_mcp_base64( - tool_use_id: str, - result_text: str, - session_id: str - ) -> Tuple[str, List[Dict[str, Any]]]: - """ - Intercept Base64 file data from Python MCP results and save to local files. - - Returns: - Tuple of (processed_text_without_base64, file_info_list) - """ - from agents.utils.config import Config - - file_info = [] - processed_text = result_text - - try: - # Pattern to match Base64 data URLs with optional filename attribute - base64_pattern = r'data:([^;]+);base64,([A-Za-z0-9+/=\s]+?)' - - def process_base64_match(match): - custom_filename = match.group(1) # May be None if not provided - mime_type = match.group(2) - base64_data = match.group(3) - - try: - # Decode Base64 data (strip whitespace first) - clean_base64 = base64_data.replace('\n', '').replace('\r', '').replace(' ', '') - file_data = base64.b64decode(clean_base64) - - # Determine file extension from MIME type - extension_map = { - 'application/zip': '.zip', - 'text/plain': '.txt', - 'application/json': '.json', - 'text/csv': '.csv', - 'image/png': '.png', - 'image/jpeg': '.jpg', - 'application/vnd.openxmlformats-officedocument.wordprocessingml.document': '.docx' - } - extension = extension_map.get(mime_type, '.bin') - - # Use custom filename if provided, otherwise generate one - if custom_filename: - filename = custom_filename - else: - filename = f"python_output_{len(file_info) + 1}{extension}" - - # Create output directory using provided session_id - try: - session_output_dir = Config.get_session_output_dir(session_id) - tool_dir = os.path.join(session_output_dir, tool_use_id) - os.makedirs(tool_dir, exist_ok=True) - except Exception as dir_error: - print(f"❌ Error creating directory: {dir_error}") - return match.group(0) - - # Save file - try: - file_path = os.path.join(tool_dir, filename) - with open(file_path, 'wb') as f: - f.write(file_data) - - # Create download URL (relative to output dir, served from /output/) - relative_path = os.path.relpath(file_path, Config.get_output_dir()) - download_url = f"/output/{relative_path}" - - file_info.append({ - 'filename': filename, - 'mime_type': mime_type, - 'size': len(file_data), - 'download_url': download_url, - 'local_path': file_path - }) - - print(f"💾 Saved Base64 file: {filename} ({len(file_data)} bytes) -> {file_path}") - - # Replace Base64 data with file-specific message - file_size_kb = len(file_data) / 1024 - if mime_type == 'application/vnd.openxmlformats-officedocument.wordprocessingml.document': - return f"✅ Document saved: **{filename}** ({file_size_kb:.1f} KB) - [Download]({download_url})" - elif mime_type == 'application/zip': - return f"📁 Files saved as ZIP archive: **{filename}** ({file_size_kb:.1f} KB) - [Download]({download_url})" - else: - return f"✅ File saved: **{filename}** ({file_size_kb:.1f} KB) - [Download]({download_url})" - except Exception as save_error: - print(f"❌ Error saving file: {save_error}") - return match.group(0) - - except Exception as e: - print(f"❌ Error processing Base64 data: {e}") - return match.group(0) - - # Process all Base64 matches - processed_text = re.sub(base64_pattern, process_base64_match, result_text) - - except Exception as e: - print(f"❌ Error in _handle_python_mcp_base64: {e}") - - return processed_text, file_info diff --git a/backend/src/agents/main_agent/tools/tool_catalog.py b/backend/src/agents/main_agent/tools/tool_catalog.py index 639f230b2..eca3bc1c0 100644 --- a/backend/src/agents/main_agent/tools/tool_catalog.py +++ b/backend/src/agents/main_agent/tools/tool_catalog.py @@ -91,6 +91,14 @@ def to_dict(self) -> dict: icon="globe-alt", ), + "request_user_login": ToolMetadata( + tool_id="request_user_login", + name="Browser Sign-In", + description="Hand the browser to the user so they can sign in to a site the agent cannot reach, then continue browsing the authenticated session.", + category=ToolCategory.SEARCH, + icon="key", + ), + # --- Built-in Tools (Code Interpreter) --- "generate_diagram_and_validate": ToolMetadata( tool_id="generate_diagram_and_validate", diff --git a/backend/src/agents/main_agent/tools/tool_registry.py b/backend/src/agents/main_agent/tools/tool_registry.py index 972cd7e9d..13d60c3c3 100644 --- a/backend/src/agents/main_agent/tools/tool_registry.py +++ b/backend/src/agents/main_agent/tools/tool_registry.py @@ -93,7 +93,11 @@ def create_default_registry() -> ToolRegistry: from strands_tools.calculator import calculator from agents import local_tools, builtin_tools from agents.builtin_tools.ask_user_question import ask_user_question - from apis.shared.feature_flags import ask_user_question_enabled + from agents.builtin_tools.browser import request_user_login + from apis.shared.feature_flags import ( + ask_user_question_enabled, + browser_takeover_enabled, + ) registry = ToolRegistry() @@ -113,5 +117,12 @@ def create_default_registry() -> ToolRegistry: if ask_user_question_enabled(): registry.register_tool("ask_user_question", ask_user_question) + # Same reasoning, and one more: this one hands a human an interactive + # browser inside our AWS account, so it has both a kill switch here and its + # own catalog entry granted per role (spec D1). Registration only makes it + # *grantable*; a role that has not been granted it still never sees it. + if browser_takeover_enabled(): + registry.register_tool("request_user_login", request_user_login) + logger.info(f"Default registry created with {registry.get_tool_count()} tools") return registry diff --git a/backend/src/agents/main_agent/utils/timezone.py b/backend/src/agents/main_agent/utils/timezone.py index 9b6ef363b..d454891ba 100644 --- a/backend/src/agents/main_agent/utils/timezone.py +++ b/backend/src/agents/main_agent/utils/timezone.py @@ -22,10 +22,21 @@ def get_current_date_pacific() -> str: """ - Get current date and hour in US Pacific timezone (America/Los_Angeles) + Get the current calendar date in US Pacific timezone (America/Los_Angeles). + + The result is rendered into the system prompt, which is the head of the + Bedrock prompt-cache prefix (see the prompt-cache contract in CLAUDE.md). + It therefore deliberately contains **no hour**: the string is byte-stable + for a whole Pacific day, so the cached prefix is re-written once per day + instead of at every hour boundary. Earlier versions appended ``HH:00`` and + a 2026-09 prod cost audit attributed ~2.6% of cache-write spend to the + resulting hourly ``systemPromptHash`` flips. If a flow ever needs the + time of day, put it in the user turn (or another turn-scoped message), + never back in this prefix. Returns: - str: Formatted date string with timezone (e.g., "2024-01-15 (Monday) 14:00 PST") + str: Date, weekday and timezone abbreviation + (e.g., "2024-01-15 (Monday) PST") """ try: if TIMEZONE_AVAILABLE: @@ -44,12 +55,12 @@ def get_current_date_pacific() -> str: # Get timezone abbreviation (PST/PDT) tz_abbr = now.strftime("%Z") - return now.strftime(f"%Y-%m-%d (%A) %H:00 {tz_abbr}") + return now.strftime(f"%Y-%m-%d (%A) {tz_abbr}") else: # Fallback to UTC if no timezone library available now = datetime.now(timezone.utc) - return now.strftime("%Y-%m-%d (%A) %H:00 UTC") + return now.strftime("%Y-%m-%d (%A) UTC") except Exception as e: logger.warning(f"Failed to get Pacific time: {e}, using UTC") now = datetime.now(timezone.utc) - return now.strftime("%Y-%m-%d (%A) %H:00 UTC") + return now.strftime("%Y-%m-%d (%A) UTC") diff --git a/backend/src/agents/utils/config.py b/backend/src/agents/utils/config.py deleted file mode 100644 index 8a7497c03..000000000 --- a/backend/src/agents/utils/config.py +++ /dev/null @@ -1,42 +0,0 @@ -""" -Configuration management for AgentCore -""" -from pathlib import Path - - -class Config: - """Configuration class for AgentCore paths and settings""" - - @staticmethod - def get_base_dir() -> Path: - """Get the base directory (backend/src/)""" - # Get path to src/ directory (parent of agents/) - return Path(__file__).parent.parent.parent - - @staticmethod - def get_output_dir() -> Path: - """Get the output directory path (agentcore/output)""" - output_dir = Config.get_base_dir() / "output" - output_dir.mkdir(exist_ok=True) - return output_dir - - @staticmethod - def get_session_output_dir(session_id: str) -> Path: - """Get the session-specific output directory (agentcore/output/session_id)""" - session_dir = Config.get_output_dir() / session_id - session_dir.mkdir(parents=True, exist_ok=True) - return session_dir - - @staticmethod - def get_uploads_dir() -> Path: - """Get the uploads directory path (agentcore/uploads)""" - uploads_dir = Config.get_base_dir() / "uploads" - uploads_dir.mkdir(exist_ok=True) - return uploads_dir - - @staticmethod - def get_generated_images_dir() -> Path: - """Get the generated images directory path (agentcore/generated_images)""" - images_dir = Config.get_base_dir() / "generated_images" - images_dir.mkdir(exist_ok=True) - return images_dir diff --git a/backend/src/apis/app_api/admin/agent_templates/__init__.py b/backend/src/apis/app_api/admin/agent_templates/__init__.py new file mode 100644 index 000000000..0dde0c460 --- /dev/null +++ b/backend/src/apis/app_api/admin/agent_templates/__init__.py @@ -0,0 +1 @@ +"""Admin CRUD API for the agent template catalog.""" diff --git a/backend/src/apis/app_api/admin/agent_templates/routes.py b/backend/src/apis/app_api/admin/agent_templates/routes.py new file mode 100644 index 000000000..e6292cd59 --- /dev/null +++ b/backend/src/apis/app_api/admin/agent_templates/routes.py @@ -0,0 +1,132 @@ +"""Admin API routes for agent template management. + +Every endpoint requires the ``admin.agent_templates`` scope. Non-admin users +read the enabled catalog through the public ``GET /templates`` endpoint, which +returns the client ``TemplateDraft`` shape and never exposes the catalog/audit +fields. +""" + +import logging + +from fastapi import APIRouter, Depends, HTTPException, Query, status + +from apis.shared.auth import User, require_admin_scope +from apis.shared.agent_templates.models import ( + AgentTemplateAdminListResponse, + AgentTemplateAdminResponse, + AgentTemplateCreate, + AgentTemplateUpdate, +) +from apis.shared.agent_templates.service import get_agent_templates_service + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/agent-templates", tags=["admin-agent-templates"]) + +# Every route in this package is guarded by this one scope, so the permission +# boundary is the package boundary. Enforced by +# tests/architecture/test_admin_scope_coverage.py. +require_agent_templates_admin = require_admin_scope("admin.agent_templates") + + +@router.get( + "/", + response_model=AgentTemplateAdminListResponse, + summary="List all agent templates", +) +async def list_agent_templates( + enabled_only: bool = Query(False, description="Filter to enabled templates only"), + admin_user: User = Depends(require_agent_templates_admin), +) -> AgentTemplateAdminListResponse: + """List all templates. Admin sees both enabled and disabled.""" + service = get_agent_templates_service() + templates = await service.list_templates(enabled_only=enabled_only) + return AgentTemplateAdminListResponse( + templates=[AgentTemplateAdminResponse.from_template(t) for t in templates], + total=len(templates), + ) + + +@router.get( + "/{template_id}", + response_model=AgentTemplateAdminResponse, + summary="Get an agent template", +) +async def get_agent_template( + template_id: str, + admin_user: User = Depends(require_agent_templates_admin), +) -> AgentTemplateAdminResponse: + service = get_agent_templates_service() + template = await service.get_template(template_id) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Agent template '{template_id}' not found", + ) + return AgentTemplateAdminResponse.from_template(template) + + +@router.post( + "/", + response_model=AgentTemplateAdminResponse, + status_code=status.HTTP_201_CREATED, + summary="Create an agent template", +) +async def create_agent_template( + data: AgentTemplateCreate, + admin_user: User = Depends(require_agent_templates_admin), +) -> AgentTemplateAdminResponse: + try: + service = get_agent_templates_service() + template = await service.create_template(data, created_by=admin_user.email) + return AgentTemplateAdminResponse.from_template(template) + except ValueError as e: + # Duplicate id (slug collision) or validation failure. + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=str(e), + ) + + +@router.patch( + "/{template_id}", + response_model=AgentTemplateAdminResponse, + summary="Update an agent template", +) +async def update_agent_template( + template_id: str, + updates: AgentTemplateUpdate, + admin_user: User = Depends(require_agent_templates_admin), +) -> AgentTemplateAdminResponse: + try: + service = get_agent_templates_service() + template = await service.update_template(template_id, updates) + if not template: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Agent template '{template_id}' not found", + ) + return AgentTemplateAdminResponse.from_template(template) + except ValueError as e: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail=str(e), + ) + + +@router.delete( + "/{template_id}", + status_code=status.HTTP_204_NO_CONTENT, + summary="Delete an agent template", +) +async def delete_agent_template( + template_id: str, + admin_user: User = Depends(require_agent_templates_admin), +) -> None: + service = get_agent_templates_service() + deleted = await service.delete_template(template_id) + if not deleted: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"Agent template '{template_id}' not found", + ) diff --git a/backend/src/apis/app_api/admin/costs/models.py b/backend/src/apis/app_api/admin/costs/models.py index ec5fc5ab2..de9588f54 100644 --- a/backend/src/apis/app_api/admin/costs/models.py +++ b/backend/src/apis/app_api/admin/costs/models.py @@ -106,6 +106,56 @@ class PrefixFingerprints(BaseModel): message_count: Optional[int] = Field(None, alias="messageCount") +class PrefixTokens(BaseModel): + """The agent's stable static prefix, split: system prompt vs tool schemas.""" + model_config = ConfigDict(populate_by_name=True) + + system: int = 0 + tools: int = 0 + + +class CompactionEvent(BaseModel): + """One compaction decision recorded before a model call (numbers only). + + ``kind``: ``applied`` (restore-time slice ran), ``checkpoint`` (a new + checkpoint was cut after the previous turn), ``forced`` / ``floor_unreachable`` + (reserved for the scheduling policy). ``summaryTokens`` is the summary's + size at that moment — the number a summary cap has to move. + """ + model_config = ConfigDict(populate_by_name=True, extra="allow") + + kind: str + checkpoint: Optional[int] = None + summary_tokens: Optional[int] = Field(None, alias="summaryTokens") + summarized_turns: Optional[int] = Field(None, alias="summarizedTurns") + retained_messages: Optional[int] = Field(None, alias="retainedMessages") + truncated_tool_results: Optional[int] = Field(None, alias="truncatedToolResults") + input_tokens: Optional[int] = Field(None, alias="inputTokens") + # Document lifecycle kinds (`document_stripped` / `document_rehydrated` / + # `document_offload`): how many documents the event touched, their + # estimated token weight, and — for an offload — the prompt-cache gap at + # the moment it fired (an offload while the cache is live is the + # regression the trigger must never produce). + documents: Optional[int] = None + document_tokens: Optional[int] = Field(None, alias="documentTokens") + cache_gap_seconds: Optional[int] = Field(None, alias="cacheGapSeconds") + # `document_offload` only: what the evicted documents were replaced with, + # and how many document_read page slices were aged in the same pass. + digest_tokens: Optional[int] = Field(None, alias="digestTokens") + slices: Optional[int] = None + slice_tokens: Optional[int] = Field(None, alias="sliceTokens") + + +class DocumentReads(BaseModel): + """``document_read`` retrievals one model call requested: calls, pages + returned as native document blocks, and their byte size.""" + model_config = ConfigDict(populate_by_name=True) + + calls: int = 0 + pages: int = 0 + bytes: int = 0 + + class SessionCallRow(BaseModel): """One model call within a session's cost anatomy.""" model_config = ConfigDict(populate_by_name=True) @@ -144,6 +194,31 @@ class SessionCallRow(BaseModel): prefix_fingerprints: Optional[PrefixFingerprints] = Field( None, alias="prefixFingerprints" ) + # Context ledger (optional; absent on rows written before it shipped or + # while COST_DIAGNOSTICS_ENABLED=false). + prefix_tokens: Optional[PrefixTokens] = Field(None, alias="prefixTokens") + # The conversation window's cumulative trimmed-message count at this call. + window_removed_messages: Optional[int] = Field(None, alias="windowRemovedMessages") + # Messages trimmed since the previous ledger-bearing call — derived, so a + # reader does not have to diff consecutive rows. A positive value means + # the prefix changed before this call. + window_trimmed: Optional[int] = Field(None, alias="windowTrimmed") + compaction_events: Optional[List[CompactionEvent]] = Field(None, alias="compactionEvents") + # Document context at this call (optional; absent on rows written before + # it shipped or with diagnostics off). `hasDocuments` + `documentDigests` + # classify the call: full document inline, digest only, or neither. + # `documentTokens` is the compaction estimator's heuristic (bytes/4, flat + # per image), comparable across rows; `documentMime` is keyed by Bedrock's + # format enum plus `image` — never a filename. + has_documents: Optional[bool] = Field(None, alias="hasDocuments") + document_count: Optional[int] = Field(None, alias="documentCount") + document_tokens: Optional[int] = Field(None, alias="documentTokens") + document_digests: Optional[int] = Field(None, alias="documentDigests") + documents_attached: Optional[int] = Field(None, alias="documentsAttached") + document_slices: Optional[int] = Field(None, alias="documentSlices") + document_slice_tokens: Optional[int] = Field(None, alias="documentSliceTokens") + document_mime: Optional[Dict[str, int]] = Field(None, alias="documentMime") + document_reads: Optional[DocumentReads] = Field(None, alias="documentReads") class SessionCostAnatomy(BaseModel): @@ -306,6 +381,11 @@ class UserSessionSummary(BaseModel): tool_call_count: Optional[int] = Field(None, alias="toolCallCount") tool_error_count: Optional[int] = Field(None, alias="toolErrorCount") compaction_count: Optional[int] = Field(None, alias="compactionCount") + compaction_applied_count: Optional[int] = Field(None, alias="compactionAppliedCount") + compaction_forced_count: Optional[int] = Field(None, alias="compactionForcedCount") + compaction_floor_unreachable_count: Optional[int] = Field( + None, alias="compactionFloorUnreachableCount" + ) diagnosis_count: int = Field(0, alias="diagnosisCount") top_diagnosis_severity: Optional[str] = Field(None, alias="topDiagnosisSeverity") @@ -327,6 +407,12 @@ class UserSessionsResponse(BaseModel): # Sessions excluded because their cost is unrecorded (they are still listed, # with costKnown=False, when sort != "cost"; under cost-sort they trail). unknown_cost_count: int = Field(0, alias="unknownCostCount") + # Soft-deleted conversations in the list (status="deleted"). Listed, not + # hidden: a delete removes the row from the user's sidebar, not its cost + # rows or its share of `userPeriodCost`, so an audit that dropped them + # could not account for the period total. + deleted_session_count: int = Field(0, alias="deletedSessionCount") + deleted_session_cost: float = Field(0.0, alias="deletedSessionCost") class AttachmentProfile(BaseModel): @@ -336,6 +422,10 @@ class AttachmentProfile(BaseModel): count: int = 0 total_bytes: int = Field(0, alias="totalBytes") by_mime: Dict[str, int] = Field(default_factory=dict, alias="byMime") + # DocumentDigest coverage: uploads with a ready digest and the rendered + # token estimate they would cost in context (offload spec §4A). + digested: int = 0 + digest_tokens: int = Field(0, alias="digestTokens") class ContextTrajectoryPoint(BaseModel): @@ -350,6 +440,10 @@ class ContextTrajectoryPoint(BaseModel): cost: Optional[float] = None # Per-call tool census when recorded (PR-3): tool name -> calls tool_calls: Optional[Dict[str, int]] = Field(None, alias="toolCalls") + # Context ledger when recorded: messages trimmed before this call, and the + # kinds of compaction decision taken before it. + window_trimmed: Optional[int] = Field(None, alias="windowTrimmed") + compaction: Optional[List[str]] = None class FingerprintChanges(BaseModel): @@ -369,6 +463,90 @@ class ToolCensusEntry(BaseModel): errors: int = 0 +class FeedbackCounts(BaseModel): + """Thumbs on one bucket of calls: ``up`` / ``down`` are counts of live + feedback rows, never the text of anything.""" + model_config = ConfigDict(populate_by_name=True) + + up: int = 0 + down: int = 0 + + +class FeedbackByTurnClass(BaseModel): + """Feedback split by the call's document turn class (document-context + offload spec §6.1): *full* (``hasDocuments``), *retrieved* + (``documentReads.pages > 0``), *digestOnly* (``documentDigests > 0``), + else *none* — in that precedence, since a retrieving call still holds + the digest. ``n`` per class is ``up + down``; the down-thumb rate is + ``down / n``.""" + model_config = ConfigDict(populate_by_name=True) + + full: FeedbackCounts = Field(default_factory=FeedbackCounts) + digest_only: FeedbackCounts = Field(default_factory=FeedbackCounts, alias="digestOnly") + retrieved: FeedbackCounts = Field(default_factory=FeedbackCounts) + none: FeedbackCounts = Field(default_factory=FeedbackCounts) + + +class ImplicitSignalCounts(BaseModel): + """Implicit signals (spec §10) as *messages touched* per kind — a message + copied three times counts once here. Kept apart from the thumbs; the two + have different base rates and are never summed.""" + model_config = ConfigDict(populate_by_name=True) + + copied: int = 0 + continued: int = 0 + + +class EvaluatorAggregate(BaseModel): + """Mean judged score for one evaluator over this session's sampled thumbs.""" + model_config = ConfigDict(populate_by_name=True) + + n: int = 0 + mean: float = 0.0 + + +class FeedbackEvaluations(BaseModel): + """What the eval sampler (spec §11 PR-4) concluded about this session's + down-thumbs: how many were judged, the mean per evaluator, and for + ``tool_failed`` thumbs whether the call's tool census corroborated them. + Scores and counts only — the judge's explanation is never stored.""" + model_config = ConfigDict(populate_by_name=True) + + judged: int = 0 + by_evaluator: Dict[str, EvaluatorAggregate] = Field(default_factory=dict, alias="byEvaluator") + tool_failures_reported: int = Field(0, alias="toolFailuresReported") + tool_failures_corroborated: int = Field(0, alias="toolFailuresCorroborated") + + +class FeedbackProfile(BaseModel): + """The outcome signal joined to the session's cost rows. ``byTurnClass`` + is ``None`` when no cost row carries the turn-class fields (they arrive + with #1137; rows written before it have none) — "not tracked", not zero. + ``unjoined`` counts thumbs whose message has no cost row at all (the row + expired, or the call was never recorded).""" + model_config = ConfigDict(populate_by_name=True) + + up: int = 0 + down: int = 0 + by_turn_class: Optional[FeedbackByTurnClass] = Field(None, alias="byTurnClass") + # Down-thumb reason codes, ``{code: count}`` over the closed set in + # ``FEEDBACK_REASONS`` — the same split the fleet view reports (#1152), + # here for the one conversation an admin has drilled into: the fleet says + # *how much*, this says *why this session*. A code, never free text. + reasons: Dict[str, int] = Field(default_factory=dict) + unjoined: int = 0 + # Down-thumbs the user followed with a retry-with-correction, and what + # that rework cost: the thumbed call(s) plus the retry turn's calls + # (response-feedback spec §7 "rework cost"). ``None`` when no retry has + # a cost row to price. + retried: int = 0 + rework_usd: Optional[float] = Field(None, alias="reworkUsd") + # Implicit signals, or None when the session has none. + implicit: Optional[ImplicitSignalCounts] = None + # Judged down-thumbs, or None when the sampler has not touched this session. + evaluations: Optional[FeedbackEvaluations] = None + + class DataCoverage(BaseModel): """Which optional signals this session actually has, so the UI can say "not tracked" instead of rendering an honest-looking zero.""" @@ -378,6 +556,12 @@ class DataCoverage(BaseModel): compaction_count: bool = Field(False, alias="compactionCount") fingerprints: bool = False cost: bool = False + prefix_tokens: bool = Field(False, alias="prefixTokens") + window_trim: bool = Field(False, alias="windowTrim") + compaction_events: bool = Field(False, alias="compactionEvents") + # Any F# row, or a session rollup written while diagnostics were on. + feedback: bool = False + documents: bool = False class SessionProfile(BaseModel): @@ -408,3 +592,26 @@ class SessionProfile(BaseModel): diagnoses: List[SessionDiagnosis] = Field(default_factory=list) data_coverage: DataCoverage = Field(default_factory=DataCoverage, alias="dataCoverage") + # Latest recorded static prefix split (system prompt vs tool schemas). + prefix_tokens: Optional[PrefixTokens] = Field(None, alias="prefixTokens") + # How many calls in this session were preceded by a window trim, and the + # messages the window has removed in total (last ledger-bearing call). + window_trim_calls: int = Field(0, alias="windowTrimCalls") + window_removed_messages: Optional[int] = Field(None, alias="windowRemovedMessages") + # Compaction decisions by kind across the session's calls. + compaction_event_counts: Dict[str, int] = Field( + default_factory=dict, alias="compactionEventCounts" + ) + # The summary's token size at the most recent compaction decision. + last_summary_tokens: Optional[int] = Field(None, alias="lastSummaryTokens") + # Thumbs up/down joined to the cost rows by (sessionId, messageId). + feedback: FeedbackProfile = Field(default_factory=FeedbackProfile) + # Document lifecycle across the session's calls: how many calls ran with + # the full document inline vs. a digest only (the digest-vs-full turn + # shares), the largest estimated document footprint seen, and what + # document_read pulled back in total. + full_document_calls: int = Field(0, alias="fullDocumentCalls") + digest_only_calls: int = Field(0, alias="digestOnlyCalls") + peak_document_tokens: Optional[int] = Field(None, alias="peakDocumentTokens") + document_read_calls: int = Field(0, alias="documentReadCalls") + document_read_pages: int = Field(0, alias="documentReadPages") diff --git a/backend/src/apis/app_api/admin/costs/service.py b/backend/src/apis/app_api/admin/costs/service.py index be484cadb..9fa754967 100644 --- a/backend/src/apis/app_api/admin/costs/service.py +++ b/backend/src/apis/app_api/admin/costs/service.py @@ -10,6 +10,7 @@ from datetime import datetime, timedelta, timezone from typing import Any, Dict, Optional, List +from apis.shared.sessions.models import FEEDBACK_REASONS from apis.shared.storage.dynamodb_storage import DynamoDBStorage from .diagnoses import ( CHARS_PER_TOKEN, @@ -20,9 +21,18 @@ top_severity, ) from .models import ( + CompactionEvent, + DocumentReads, + PrefixTokens, AttachmentProfile, ContextTrajectoryPoint, DataCoverage, + FeedbackByTurnClass, + FeedbackCounts, + EvaluatorAggregate, + FeedbackEvaluations, + FeedbackProfile, + ImplicitSignalCounts, FingerprintChanges, SessionDiagnosis, SessionProfile, @@ -75,6 +85,155 @@ def _record_cost(record: Dict[str, Any]) -> Optional[float]: return _as_float(raw) +def turn_class(record: Dict[str, Any]) -> Optional[str]: + """Turn class of one ``C#`` row from its document-context fields (spec + §6.1): ``full`` (``hasDocuments``), ``retrieved`` (``documentReads.pages + > 0``), ``digestOnly`` (``documentDigests > 0``), else ``none``. + + Precedence is full > retrieved > digestOnly: a call that pulled pages back + still holds the digest, so testing the digest first would leave the + retrieved arm — the one the quality gate is about — permanently empty. + ``None`` when the row carries none of the fields (written before #1137 + or with diagnostics off), so the caller says "not tracked", not "none". + """ + has_documents = record.get("hasDocuments") + digests = record.get("documentDigests") + reads = record.get("documentReads") + if has_documents is None and digests is None and reads is None: + return None + if has_documents: + return "full" + pages = _as_int(reads.get("pages")) if isinstance(reads, dict) else _as_int(reads) + if (pages or 0) > 0: + return "retrieved" + if (_as_int(digests) or 0) > 0: + return "digestOnly" + return "none" + + +def _join_feedback( + records: List[Dict[str, Any]], + feedback_rows: List[Dict[str, Any]], +) -> FeedbackProfile: + """Join ``F#`` rows to ``C#`` rows on ``messageId`` and bucket by turn + class. Pure; the profile's numbers, never any content. Explicit thumbs + only (``signal`` absent or ``"explicit"``).""" + by_message: Dict[int, Dict[str, Any]] = {} + for record in records: + message_id = _as_int(record.get("messageId")) + if message_id is not None: + # The last call of a multi-call turn is the one the user thumbed; + # rows share a messageId only across the turn's tool round trips + # and later rows have the fuller context, so last write wins. + by_message[message_id] = record + + any_turn_class = any(turn_class(r) is not None for r in records) + buckets = FeedbackByTurnClass() if any_turn_class else None + profile = FeedbackProfile() + implicit_messages: Dict[str, set] = {"copy": set(), "continue": set()} + evaluations = FeedbackEvaluations() + evaluator_sums: Dict[str, List[float]] = {} + # Every cost row per assistant message index, for pricing rework. + cost_by_message: Dict[int, float] = {} + for record in records: + message_id = _as_int(record.get("messageId")) + if message_id is not None: + cost_by_message[message_id] = cost_by_message.get(message_id, 0.0) + (_record_cost(record) or 0.0) + rework_total: Optional[float] = None + for row in feedback_rows: + # Explicit thumbs only below — implicit signals (spec §10) share the + # row family but answer a different question and are counted apart. + if row.get("signal") not in (None, "explicit"): + if row.get("signal") == "implicit" and row.get("kind") in implicit_messages: + message_id = _as_int(row.get("messageId")) + if message_id is not None: + implicit_messages[row["kind"]].add(message_id) + continue + value = _as_int(row.get("value")) + if value not in (1, -1): + continue + if value == 1: + profile.up += 1 + else: + profile.down += 1 + reason = row.get("reason") + # Closed set only: the write path types ``reason`` as a Literal, so + # an unknown code here means a row from a future schema. + if isinstance(reason, str) and reason in FEEDBACK_REASONS: + profile.reasons[reason] = profile.reasons.get(reason, 0) + 1 + message_id = _as_int(row.get("messageId")) + verdict = row.get("evaluation") + if isinstance(verdict, dict): + evaluations.judged += 1 + for evaluator, score in (verdict.get("scores") or {}).items(): + value = _as_float(score.get("value")) if isinstance(score, dict) else None + if value is not None: + evaluator_sums.setdefault(str(evaluator), []).append(value) + if verdict.get("reason") == "tool_failed": + evaluations.tool_failures_reported += 1 + if verdict.get("toolFailureCorroborated") is True: + evaluations.tool_failures_corroborated += 1 + retry_id = _as_int(row.get("retryMessageId")) + if value == -1 and retry_id is not None: + profile.retried += 1 + rework = _rework_cost(cost_by_message, message_id, retry_id) + if rework is not None: + rework_total = (rework_total or 0.0) + rework + record = by_message.get(message_id) if message_id is not None else None + if record is None: + profile.unjoined += 1 + continue + if buckets is None: + continue + klass = turn_class(record) or "none" + bucket: FeedbackCounts = { + "full": buckets.full, + "digestOnly": buckets.digest_only, + "retrieved": buckets.retrieved, + }.get(klass, buckets.none) + if value == 1: + bucket.up += 1 + else: + bucket.down += 1 + profile.by_turn_class = buckets + profile.rework_usd = round(rework_total, 6) if rework_total is not None else None + if any(implicit_messages.values()): + profile.implicit = ImplicitSignalCounts( + copied=len(implicit_messages["copy"]), + continued=len(implicit_messages["continue"]), + ) + if evaluations.judged: + evaluations.by_evaluator = { + name: EvaluatorAggregate(n=len(values), mean=round(sum(values) / len(values), 4)) + for name, values in sorted(evaluator_sums.items()) + } + profile.evaluations = evaluations + return profile + + +def _rework_cost( + cost_by_message: Dict[int, float], + thumbed_message_id: Optional[int], + retry_message_id: int, +) -> Optional[float]: + """Dollars spent on a down-thumbed answer plus its retry: the thumbed + message's call rows, plus the retry turn's assistant rows. The retry is + a *user* message (no cost row); its turn's assistant messages are the + consecutive indexes after it — a gap means the next user message. ``None`` + when neither side has a cost row to price.""" + found = False + total = 0.0 + if thumbed_message_id is not None and thumbed_message_id in cost_by_message: + total += cost_by_message[thumbed_message_id] + found = True + index = retry_message_id + 1 + while index in cost_by_message: + total += cost_by_message[index] + found = True + index += 1 + return total if found else None + + def _context_tokens(record: Dict[str, Any]) -> int: """True context occupancy of one call: uncached input + cached prefix + newly cached tokens (Bedrock reports the three disjointly).""" @@ -86,6 +245,114 @@ def _context_tokens(record: Dict[str, Any]) -> int: ) +from dataclasses import dataclass as _dataclass, field as _field + + +@_dataclass +class _CallLedger: + """The context-ledger fields of one cost row, decoded and diffed.""" + + prefix_tokens: Optional[PrefixTokens] = None + removed: Optional[int] = None + trimmed: Optional[int] = None + events: List[CompactionEvent] = _field(default_factory=list) + #: The row's document context fields, decoded (``None`` = not tracked). + documents: Optional[Dict[str, Any]] = None + document_reads: Optional[DocumentReads] = None + + +_DOCUMENT_INT_FIELDS = ( + "documentCount", "documentTokens", "documentDigests", "documentsAttached", + "documentSlices", "documentSliceTokens", +) + + +def _call_documents(record: Dict[str, Any]) -> Optional[Dict[str, Any]]: + """The row's document context (``hasDocuments`` and the counts), or + ``None`` when the row predates the fields. Ints coerced, the format map + kept as ``{format: count}``.""" + if "hasDocuments" not in record: + return None + out: Dict[str, Any] = {"hasDocuments": bool(record.get("hasDocuments"))} + for key in _DOCUMENT_INT_FIELDS: + value = _as_int(record.get(key)) + if value is not None: + out[key] = value + mime = record.get("documentMime") + if isinstance(mime, dict): + out["documentMime"] = { + str(k): (_as_int(v) or 0) for k, v in mime.items() + } + return out + + +def _call_document_reads(record: Dict[str, Any]) -> Optional[DocumentReads]: + raw = record.get("documentReads") + if not isinstance(raw, dict): + return None + return DocumentReads( + calls=_as_int(raw.get("calls")) or 0, + pages=_as_int(raw.get("pages")) or 0, + bytes=_as_int(raw.get("bytes")) or 0, + ) + + +def _document_row_fields(ledger: "_CallLedger") -> Dict[str, Any]: + """``SessionCallRow`` kwargs for the row's document context — empty when + the row predates the fields, so they render as null ("not tracked").""" + fields: Dict[str, Any] = {} + docs = ledger.documents + if docs is not None: + fields["has_documents"] = docs.get("hasDocuments") + fields["document_count"] = docs.get("documentCount") + fields["document_tokens"] = docs.get("documentTokens") + fields["document_digests"] = docs.get("documentDigests") + fields["documents_attached"] = docs.get("documentsAttached") + fields["document_slices"] = docs.get("documentSlices") + fields["document_slice_tokens"] = docs.get("documentSliceTokens") + fields["document_mime"] = docs.get("documentMime") + if ledger.document_reads is not None: + fields["document_reads"] = ledger.document_reads + return fields + + +def _call_ledger(record: Dict[str, Any], previous_removed: Optional[int]) -> _CallLedger: + """Decode a cost row's ``prefixTokens`` / ``windowRemovedMessages`` / + ``compactionEvents`` / document context and derive ``trimmed`` (messages + removed since the previous ledger-bearing row). Absent fields stay + ``None`` — "not tracked", never 0 — and malformed ones are ignored rather + than raised. + """ + ledger = _CallLedger() + ledger.documents = _call_documents(record) + ledger.document_reads = _call_document_reads(record) + raw_prefix = record.get("prefixTokens") + if isinstance(raw_prefix, dict): + try: + ledger.prefix_tokens = PrefixTokens( + system=int(raw_prefix.get("system") or 0), + tools=int(raw_prefix.get("tools") or 0), + ) + except (TypeError, ValueError): + ledger.prefix_tokens = None + removed = _as_int(record.get("windowRemovedMessages")) + if removed is not None: + ledger.removed = removed + ledger.trimmed = ( + max(removed - previous_removed, 0) if previous_removed is not None else 0 + ) + raw_events = record.get("compactionEvents") + if isinstance(raw_events, list): + for entry in raw_events: + if not isinstance(entry, dict) or not entry.get("kind"): + continue + try: + ledger.events.append(CompactionEvent(**entry)) + except (TypeError, ValueError): + continue + return ledger + + class AdminCostService: """Service for admin cost dashboard operations.""" @@ -614,11 +881,15 @@ async def get_session_cost_anatomy(self, session_id: str) -> SessionCostAnatomy: wasted_usd = 0.0 agent_switch_misses = 0 agent_switch_usd = 0.0 + previous_removed: Optional[int] = None for record in records: token_usage = record.get("tokenUsage") or {} model_info = record.get("modelInfo") or {} fingerprints_raw = record.get("prefixFingerprints") + ledger = _call_ledger(record, previous_removed) + if ledger.removed is not None: + previous_removed = ledger.removed # cost is a breakdown dict ({"total": ...}) on the streaming path # or a bare float on the legacy path. @@ -675,6 +946,11 @@ async def get_session_cost_anatomy(self, session_id: str) -> SessionCostAnatomy: PrefixFingerprints(**fingerprints_raw) if isinstance(fingerprints_raw, dict) else None ), + prefix_tokens=ledger.prefix_tokens, + window_removed_messages=ledger.removed, + window_trimmed=ledger.trimmed, + compaction_events=ledger.events or None, + **_document_row_fields(ledger), )) cache_traffic = total_cache_read + total_cache_write @@ -800,6 +1076,11 @@ def _session_summary( tool_call_count=_as_int(row.get("toolCallCount")), tool_error_count=_as_int(row.get("toolErrorCount")), compaction_count=_as_int(row.get("compactionCount")), + compaction_applied_count=_as_int(row.get("compactionAppliedCount")), + compaction_forced_count=_as_int(row.get("compactionForcedCount")), + compaction_floor_unreachable_count=_as_int( + row.get("compactionFloorUnreachableCount") + ), diagnosis_count=len(findings), top_diagnosis_severity=top_severity(findings), ) @@ -831,15 +1112,23 @@ async def get_user_sessions( period = None if all_time else (period or self._get_current_period()) active_since = self._get_period_date_range(period)[0] if period else None + # Deleted conversations stay in the list: their cost rows and their + # share of the period total outlive the delete, so hiding them left + # a user's spend unaccounted for (one $3.77 row against $20.32). rows = await self.storage.get_user_session_diagnostics( user_id=user_id, active_since=active_since, + include_deleted=True, ) user_period_cost = await self._user_period_cost(user_id, period) threshold = compaction_token_threshold() summaries: List[UserSessionSummary] = [] for row in rows: + if row.get("deleted") or row.get("status") == "deleted": + # Legacy tombstones carry `deleted` without the status flip; + # normalise so the page has one signal to render. + row["status"] = "deleted" share = self._share(_as_float(row.get("totalCost")), user_period_cost) findings = run_diagnoses(self._row_facts(row, threshold, share)) summaries.append(self._session_summary(row, findings, share)) @@ -860,9 +1149,11 @@ def recent_key(s: UserSessionSummary) -> str: ) unknown = sum(1 for s in summaries if not s.cost_known) + deleted = [s for s in summaries if s.status == "deleted"] + deleted_cost = sum(s.total_cost for s in deleted if s.total_cost is not None) logger.info( - f"User sessions: {len(summaries)} rows ({unknown} unknown-cost), " - f"returning {min(limit, len(summaries))}" + f"User sessions: {len(summaries)} rows ({unknown} unknown-cost, " + f"{len(deleted)} deleted), returning {min(limit, len(summaries))}" ) return UserSessionsResponse( user_id=user_id, @@ -871,6 +1162,8 @@ def recent_key(s: UserSessionSummary) -> str: sessions=summaries[:limit], total=len(summaries), unknown_cost_count=unknown, + deleted_session_count=len(deleted), + deleted_session_cost=round(deleted_cost, 6), ) async def _attachment_profile(self, session_id: str) -> AttachmentProfile: @@ -883,15 +1176,36 @@ async def _attachment_profile(self, session_id: str) -> AttachmentProfile: return AttachmentProfile() by_mime: Counter = Counter() total_bytes = 0 + digested = digest_tokens = 0 for item in stats: by_mime[item.get("mimeType") or "unknown"] += 1 total_bytes += _as_int(item.get("sizeBytes")) or 0 + digest = item.get("digest") + if isinstance(digest, dict) and digest.get("status") == "ready": + digested += 1 + digest_tokens += _as_int(digest.get("tokens")) or 0 return AttachmentProfile( count=len(stats), total_bytes=total_bytes, by_mime=dict(by_mime), + digested=digested, + digest_tokens=digest_tokens, ) + async def _feedback_rows(self, session_id: str) -> List[Dict[str, Any]]: + """The session's ``F#`` rows. Best-effort: a storage fork without the + reader, or a transient error, yields none — the profile then falls + back to the session rollups and reports coverage honestly.""" + reader = getattr(self.storage, "get_session_feedback_rows", None) + if reader is None: + return [] + try: + rows = await reader(session_id) + except Exception as e: # noqa: BLE001 - feedback is one signal of several + logger.debug("Feedback rows unavailable for session: %s", e) + return [] + return list(rows or []) + async def get_session_profile(self, session_id: str) -> Optional[SessionProfile]: """The content-free diagnostic profile of one conversation, or ``None`` when the session has no metadata row. @@ -908,6 +1222,7 @@ async def get_session_profile(self, session_id: str) -> Optional[SessionProfile] records = await self.storage.get_session_cost_records(session_id) attachments = await self._attachment_profile(session_id) + feedback_rows = await self._feedback_rows(session_id) user_period_cost = ( await self._user_period_cost(user_id, self._get_current_period()) if user_id else None @@ -924,6 +1239,17 @@ async def get_session_profile(self, session_id: str) -> Optional[SessionProfile] read_total = write_total = 0 any_fingerprints = any_census = False previous_fp: Optional[Dict[str, Any]] = None + any_prefix_tokens = any_window = any_compaction_events = False + prefix_tokens: Optional[PrefixTokens] = None + previous_removed: Optional[int] = None + last_removed: Optional[int] = None + window_trim_calls = 0 + compaction_event_counts: Counter = Counter() + last_summary_tokens: Optional[int] = None + any_documents = False + full_document_calls = digest_only_calls = 0 + document_read_calls = document_read_pages = 0 + peak_document_tokens: Optional[int] = None for index, record in enumerate(records): usage = record.get("tokenUsage") or {} @@ -969,6 +1295,34 @@ async def get_session_profile(self, session_id: str) -> Optional[SessionProfile] slot.calls += calls slot.errors += errors + ledger = _call_ledger(record, previous_removed) + if ledger.prefix_tokens is not None: + any_prefix_tokens = True + prefix_tokens = ledger.prefix_tokens + if ledger.removed is not None: + any_window = True + previous_removed = last_removed = ledger.removed + if ledger.trimmed: + window_trim_calls += 1 + if ledger.events: + any_compaction_events = True + for event in ledger.events: + compaction_event_counts[event.kind] += 1 + if event.summary_tokens is not None: + last_summary_tokens = event.summary_tokens + if ledger.documents is not None: + any_documents = True + if ledger.documents.get("hasDocuments"): + full_document_calls += 1 + elif (ledger.documents.get("documentDigests") or 0) > 0: + digest_only_calls += 1 + doc_tokens = ledger.documents.get("documentTokens") + if doc_tokens is not None: + peak_document_tokens = max(peak_document_tokens or 0, doc_tokens) + if ledger.document_reads is not None: + any_documents = True + document_read_calls += ledger.document_reads.calls + document_read_pages += ledger.document_reads.pages trajectory.append(ContextTrajectoryPoint( call_index=index, timestamp=record.get("timestamp", ""), @@ -977,6 +1331,8 @@ async def get_session_profile(self, session_id: str) -> Optional[SessionProfile] model_id=model_id, cost=_record_cost(record), tool_calls=point_tool_calls, + window_trimmed=ledger.trimmed, + compaction=[e.kind for e in ledger.events] or None, )) # Cache totals: the rows are authoritative when present, else the @@ -1007,6 +1363,15 @@ async def get_session_profile(self, session_id: str) -> Optional[SessionProfile] facts.tool_call_count = sum(e.calls for e in census.values()) facts.tool_error_count = sum(e.errors for e in census.values()) + # Outcome signal: thumbs joined to the calls they rate. The rows are + # authoritative when present; the session rollups cover thumbs whose + # rows expired (they share the C# TTL, so this is rare). + feedback = _join_feedback(records, feedback_rows) + if not feedback_rows: + feedback.up = _as_int(row.get("thumbsUp")) or 0 + feedback.down = _as_int(row.get("thumbsDown")) or 0 + feedback_tracked = bool(feedback_rows) or row.get("thumbsUp") is not None + findings = run_diagnoses(facts) summary = self._session_summary(row, findings, share) if any_census and summary.tool_call_count is None: @@ -1039,6 +1404,35 @@ async def get_session_profile(self, session_id: str) -> Optional[SessionProfile] compaction_count=row.get("compactionCount") is not None, fingerprints=any_fingerprints, cost=total_cost is not None, + prefix_tokens=any_prefix_tokens, + window_trim=any_window, + compaction_events=( + any_compaction_events or row.get("compactionAppliedCount") is not None + ), + feedback=feedback_tracked, + documents=any_documents or row.get("fullDocumentCalls") is not None, + ), + feedback=feedback, + prefix_tokens=prefix_tokens, + window_trim_calls=window_trim_calls, + window_removed_messages=last_removed, + compaction_event_counts=dict(compaction_event_counts), + last_summary_tokens=last_summary_tokens, + # Rows are authoritative when present; the session rollups cover + # calls whose rows have expired (365-day TTL) or a session read + # without its rows. + full_document_calls=( + full_document_calls if any_documents else (_as_int(row.get("fullDocumentCalls")) or 0) + ), + digest_only_calls=( + digest_only_calls if any_documents else (_as_int(row.get("digestOnlyCalls")) or 0) + ), + peak_document_tokens=peak_document_tokens, + document_read_calls=( + document_read_calls if any_documents else (_as_int(row.get("documentReadCalls")) or 0) + ), + document_read_pages=( + document_read_pages if any_documents else (_as_int(row.get("documentReadPages")) or 0) ), ) diff --git a/backend/src/apis/app_api/admin/feedback/__init__.py b/backend/src/apis/app_api/admin/feedback/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/src/apis/app_api/admin/feedback/fleet.py b/backend/src/apis/app_api/admin/feedback/fleet.py new file mode 100644 index 000000000..e3f74737b --- /dev/null +++ b/backend/src/apis/app_api/admin/feedback/fleet.py @@ -0,0 +1,240 @@ +"""Fleet-level feedback attribution — response-feedback spec §7 (the +"config-dimension attribution" the spec calls the prize) and the compaction +spec §7.2 ("outcome signal joined to compaction"). + +The question this answers is deliberately not *"what is our quality score"*. +Spec §9 forbids that number existing, because an absolute rate over a +self-selected few percent of turns is noise that reads like a KPI. What it +answers is **"does the down-thumb rate differ between config arms we chose?"** +— model, agent switch, distance from a compaction cut, document turn class. + +Three rules from §9, enforced here rather than promised: + +* **Every arm carries its own ``n``.** A rate without one is not reportable. +* **Arms below the coverage floor are flagged** (:data:`DEFAULT_MINIMUM_N`), + so a one-of-one arm renders greyed rather than as a finding. The spec's + open question 3 said to pick that number before the panel exists; this is + the pick, and it is an env-tunable constant rather than a magic literal. +* **No single-number rate exists at the top level.** ``totals`` carries up + and down counts and nothing derives a headline score from them. That + absence is the design. + +Everything here is a pure function over rows that already passed the +content-free projections, so no conversation content can reach it. +""" + +from __future__ import annotations + +import os +from dataclasses import dataclass, field +from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple + +from apis.app_api.admin.costs.service import turn_class + +#: Below this many thumbs an arm's rate is not reportable. Picked, not +#: derived: 20 is the point where one extra down-thumb moves the rate by +#: five points rather than fifty, which is the coarsest resolution worth +#: showing an admin. Tune per environment with ``FEEDBACK_ARM_MINIMUM_N``. +DEFAULT_MINIMUM_N = 20 + +#: Buckets for "how far was this call from the last compaction cut" — the +#: §7.2 question. Distance is in *model calls*, not turns, because the ``C#`` +#: rows are per call and a turn may be several of them. +COMPACTION_BUCKETS: Tuple[str, ...] = ("never", "same call", "1-3 calls", "4+ calls") + + +def minimum_n() -> int: + """The coverage floor, env-tunable.""" + raw = os.environ.get("FEEDBACK_ARM_MINIMUM_N", "").strip() + try: + value = int(raw) if raw else DEFAULT_MINIMUM_N + except ValueError: + return DEFAULT_MINIMUM_N + return max(1, value) + + +@dataclass +class ArmCounts: + """One arm of one dimension: the thumbs that landed on calls with it.""" + + up: int = 0 + down: int = 0 + + @property + def n(self) -> int: + return self.up + self.down + + def add(self, value: int) -> None: + if value == 1: + self.up += 1 + elif value == -1: + self.down += 1 + + def to_dict(self, floor: int) -> Dict[str, Any]: + n = self.n + return { + "up": self.up, + "down": self.down, + "n": n, + # Rate is None below the floor rather than a number nobody should + # quote — the UI has nothing to render misleadingly. + "downRate": round(self.down / n, 4) if n >= floor and n else None, + "belowFloor": n < floor, + } + + +@dataclass +class SessionIndex: + """One session's cost rows, indexed for the joins this module makes.""" + + #: messageId -> the last ``C#`` row carrying it (a turn's tool round trips + #: share a messageId; the last row has the fullest context, matching the + #: per-session profile's own rule). + by_message: Dict[int, Dict[str, Any]] = field(default_factory=dict) + #: messageId -> that row's position in the session's chronological order. + position: Dict[int, int] = field(default_factory=dict) + #: Ordered positions at which a compaction event was recorded. + compaction_positions: List[int] = field(default_factory=list) + + +def index_session(records: Sequence[Dict[str, Any]]) -> SessionIndex: + """Index one session's ``C#`` rows, assumed chronological (the storage + reader queries the GSI with ``ScanIndexForward=True``).""" + index = SessionIndex() + for position, record in enumerate(records): + message_id = _as_int(record.get("messageId")) + if message_id is not None: + index.by_message[message_id] = record + index.position[message_id] = position + events = record.get("compactionEvents") + if isinstance(events, list) and events: + index.compaction_positions.append(position) + return index + + +def calls_since_compaction(index: SessionIndex, message_id: int) -> str: + """Which :data:`COMPACTION_BUCKETS` bucket this message sits in. + + "never" means no cut had happened in this session at or before the call — + the control arm the §7.2 question needs, and the reason a session that + was never compacted still contributes to the comparison. + """ + position = index.position.get(message_id) + if position is None: + return "never" + prior = [p for p in index.compaction_positions if p <= position] + if not prior: + return "never" + distance = position - prior[-1] + if distance == 0: + return "same call" + if distance <= 3: + return "1-3 calls" + return "4+ calls" + + +def build_fleet_report( + thumbs: Sequence[Dict[str, Any]], + records_by_session: Dict[str, Sequence[Dict[str, Any]]], + *, + window: Dict[str, Any], + floor: Optional[int] = None, + truncated: bool = False, + sessions_omitted: int = 0, +) -> Dict[str, Any]: + """Aggregate thumbs into per-dimension arms. + + ``thumbs`` are explicit feedback rows (implicit signals never carry the + queue keys, so they cannot appear here). ``records_by_session`` holds the + ``C#`` rows of the sessions that were joined — a thumb whose session is + absent, or whose message has no cost row, counts toward the totals and + toward ``coverage.unjoined``, never silently into an arm. + """ + floor = floor if floor is not None else minimum_n() + indexes = {sid: index_session(rows) for sid, rows in records_by_session.items()} + + totals = ArmCounts() + reasons: Dict[str, int] = {} + arms: Dict[str, Dict[str, ArmCounts]] = { + "model": {}, + "agentSwitch": {}, + "callsSinceCompaction": {}, + "turnClass": {}, + } + joined = 0 + sessions_seen = set() + + for row in thumbs: + if row.get("signal") not in (None, "explicit"): + continue + value = _as_int(row.get("value")) + if value not in (1, -1): + continue + totals.add(value) + session_id = str(row.get("sessionId") or "") + if session_id: + sessions_seen.add(session_id) + if value == -1: + reason = row.get("reason") + key = reason if isinstance(reason, str) and reason else "unspecified" + reasons[key] = reasons.get(key, 0) + 1 + + message_id = _as_int(row.get("messageId")) + index = indexes.get(session_id) + record = index.by_message.get(message_id) if index and message_id is not None else None + if record is None: + continue + joined += 1 + + model_id = (record.get("modelInfo") or {}).get("modelId") + if model_id: + arms["model"].setdefault(str(model_id), ArmCounts()).add(value) + switched = "switched agent" if record.get("agentSwitched") else "same agent" + arms["agentSwitch"].setdefault(switched, ArmCounts()).add(value) + arms["callsSinceCompaction"].setdefault( + calls_since_compaction(index, message_id), ArmCounts() + ).add(value) + klass = turn_class(record) + if klass is not None: + arms["turnClass"].setdefault(klass, ArmCounts()).add(value) + + n_thumbs = totals.n + return { + "window": window, + # Counts, never a derived headline rate — see the module docstring. + "totals": {"up": totals.up, "down": totals.down, "thumbs": n_thumbs}, + "coverage": { + "thumbs": n_thumbs, + "joined": joined, + "unjoined": n_thumbs - joined, + "joinRate": round(joined / n_thumbs, 4) if n_thumbs else None, + "sessionsWithFeedback": len(sessions_seen), + "sessionsJoined": len(records_by_session), + "sessionsOmitted": sessions_omitted, + "truncated": truncated, + }, + "arms": { + dimension: _render_arm(counts, floor, dimension) + for dimension, counts in arms.items() + }, + "reasons": dict(sorted(reasons.items(), key=lambda kv: (-kv[1], kv[0]))), + "minimumN": floor, + } + + +def _render_arm(counts: Dict[str, ArmCounts], floor: int, dimension: str) -> List[Dict[str, Any]]: + """Arms as a list, ordered so a reader compares like with like: the + compaction dimension keeps its natural distance order, everything else + leads with the largest sample.""" + if dimension == "callsSinceCompaction": + ordered: Iterable[str] = [k for k in COMPACTION_BUCKETS if k in counts] + else: + ordered = sorted(counts, key=lambda k: (-counts[k].n, k)) + return [{"key": key, **counts[key].to_dict(floor)} for key in ordered] + + +def _as_int(value: Any) -> Optional[int]: + try: + return int(value) + except (TypeError, ValueError): + return None diff --git a/backend/src/apis/app_api/admin/feedback/models.py b/backend/src/apis/app_api/admin/feedback/models.py new file mode 100644 index 000000000..1458a4e24 --- /dev/null +++ b/backend/src/apis/app_api/admin/feedback/models.py @@ -0,0 +1,116 @@ +"""Admin models for the feedback eval-sampling surface (spec §11 PR-4). +Content-free: ids, codes, scores, counts. No conversation text, and never +the judge's explanation (the content-policy walk covers this module).""" + +from __future__ import annotations + +from typing import Dict, List, Optional + +from pydantic import BaseModel, ConfigDict, Field + + +class EvaluatorScore(BaseModel): + model_config = ConfigDict(populate_by_name=True) + + value: float + rating: Optional[str] = None + n: int = 1 + tokens: int = 0 + + +class FeedbackVerdict(BaseModel): + """The judged result stored on a thumb row.""" + model_config = ConfigDict(populate_by_name=True) + + reason: str = "none" + evaluators: List[str] = Field(default_factory=list) + scores: Dict[str, EvaluatorScore] = Field(default_factory=dict) + tool_failure_corroborated: Optional[bool] = Field(None, alias="toolFailureCorroborated") + + +class DownThumbQueueItem(BaseModel): + """One recent down-thumb as the sampler's queue sees it.""" + model_config = ConfigDict(populate_by_name=True) + + session_id: str = Field(..., alias="sessionId") + message_id: int = Field(..., alias="messageId") + reason: Optional[str] = None + updated_at: str = Field("", alias="updatedAt") + retry_message_id: Optional[int] = Field(None, alias="retryMessageId") + evaluated_at: Optional[str] = Field(None, alias="evaluatedAt") + evaluation: Optional[FeedbackVerdict] = None + + +class DownThumbQueueResponse(BaseModel): + model_config = ConfigDict(populate_by_name=True) + + items: List[DownThumbQueueItem] + pending: int = Field(0, description="Items in this page not yet judged") + sampling_enabled: bool = Field(False, alias="samplingEnabled") + + +class FleetArm(BaseModel): + """One arm of one dimension. ``downRate`` is deliberately ``None`` below + the coverage floor: there is then no number for anyone to quote.""" + model_config = ConfigDict(populate_by_name=True) + + key: str + up: int = 0 + down: int = 0 + n: int = 0 + down_rate: Optional[float] = Field(None, alias="downRate") + below_floor: bool = Field(False, alias="belowFloor") + + +class FleetWindow(BaseModel): + model_config = ConfigDict(populate_by_name=True) + + start: str + end: str + days: int + + +class FleetCoverage(BaseModel): + """How much of the window the arms actually rest on. Spec §9: every + response carries its n and its coverage, so a comparison can be judged + rather than taken.""" + model_config = ConfigDict(populate_by_name=True) + + thumbs: int = 0 + joined: int = 0 + unjoined: int = 0 + join_rate: Optional[float] = Field(None, alias="joinRate") + sessions_with_feedback: int = Field(0, alias="sessionsWithFeedback") + sessions_joined: int = Field(0, alias="sessionsJoined") + sessions_omitted: int = Field(0, alias="sessionsOmitted") + truncated: bool = False + + +class FleetTotals(BaseModel): + """Counts only. There is deliberately no fleet-wide rate field here — + see `fleet.py` and spec §9.""" + model_config = ConfigDict(populate_by_name=True) + + up: int = 0 + down: int = 0 + thumbs: int = 0 + + +class FleetFeedbackResponse(BaseModel): + """Down-thumb rate by config arm across the fleet (spec §7).""" + model_config = ConfigDict(populate_by_name=True) + + window: FleetWindow + totals: FleetTotals + coverage: FleetCoverage + arms: Dict[str, List[FleetArm]] + reasons: Dict[str, int] = Field(default_factory=dict) + minimum_n: int = Field(20, alias="minimumN") + + +class SamplingRunResponse(BaseModel): + model_config = ConfigDict(populate_by_name=True) + + accepted: bool + limit: int + note: str diff --git a/backend/src/apis/app_api/admin/feedback/routes.py b/backend/src/apis/app_api/admin/feedback/routes.py new file mode 100644 index 000000000..ec6633ba9 --- /dev/null +++ b/backend/src/apis/app_api/admin/feedback/routes.py @@ -0,0 +1,208 @@ +"""Admin feedback routes — the eval-sampling queue (spec §11 PR-4). + + GET /admin/feedback/evaluations recent down-thumbs + any verdict + POST /admin/feedback/evaluations/run judge up to `limit` of them, offline + +Scope: ``admin.costs`` — the judge spends tokens and the verdicts sit beside +the cost rows. The run is a background task (the SDK waits on span +ingestion; minutes, not milliseconds) and 404s while +``FEEDBACK_EVAL_SAMPLING_ENABLED`` is off, per the flag's docstring. +""" + +from __future__ import annotations + +import asyncio +import logging +from datetime import datetime, timedelta, timezone +from typing import Any, Dict, List, Optional + +from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query + +from apis.shared.auth import User, require_admin_scope +from apis.shared.feature_flags import feedback_eval_sampling_enabled +from apis.shared.storage.dynamodb_storage import DynamoDBStorage + +from .fleet import build_fleet_report + +from .models import ( + DownThumbQueueItem, + DownThumbQueueResponse, + FleetFeedbackResponse, + SamplingRunResponse, +) + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/feedback", tags=["admin-feedback"]) +require_feedback_admin = require_admin_scope("admin.costs") + + +def get_storage() -> DynamoDBStorage: + return DynamoDBStorage() + + +def get_judge(): + """The AgentCore Evaluations adapter. A dependency so tests inject a fake.""" + from apis.shared.feedback_eval.sampler import AgentCoreJudge + + return AgentCoreJudge() + + +#: Hard caps on one fleet query. The window is a GSI range read, but the +#: join is one query per session, so an unbounded month would fan out +#: without limit. Exceeding either is reported as `coverage.truncated` / +#: `sessionsOmitted` rather than silently under-counted. +MAX_THUMBS_PER_WINDOW = 2000 +MAX_SESSIONS_JOINED = 300 +#: Concurrent per-session cost-row reads. +SESSION_FETCH_CONCURRENCY = 16 + + +@router.get("/fleet", response_model=FleetFeedbackResponse, response_model_by_alias=True) +async def get_fleet_feedback( + days: int = Query(30, ge=1, le=90, description="Trailing window in days"), + current_user: User = Depends(require_feedback_admin), + storage: DynamoDBStorage = Depends(get_storage), +): + """Down-thumb rate by config arm across the fleet — model, agent switch, + distance from a compaction cut, document turn class. + + Content-free end to end: the thumbs arrive through the projected reader + (no user id), the cost rows through the anatomy's own projection, and + nothing but counts leaves. Per spec §9 every arm carries its ``n``, arms + under the floor report no rate at all, and there is no fleet-wide + "quality score" field. + """ + end = datetime.now(timezone.utc) + start = end - timedelta(days=days) + window = {"start": start.isoformat(), "end": end.isoformat(), "days": days} + + try: + thumbs = await storage.get_feedback_in_window( + start=window["start"], end=window["end"], limit=MAX_THUMBS_PER_WINDOW, + ) + except Exception: + logger.error("Error reading the feedback window", exc_info=True) + raise HTTPException(status_code=500, detail="Failed to read feedback window") + + truncated = len(thumbs) >= MAX_THUMBS_PER_WINDOW + session_ids: List[str] = [] + for row in thumbs: + sid = str(row.get("sessionId") or "") + if sid and sid not in session_ids: + session_ids.append(sid) + omitted = max(0, len(session_ids) - MAX_SESSIONS_JOINED) + session_ids = session_ids[:MAX_SESSIONS_JOINED] + + records_by_session = await _load_session_records(storage, session_ids) + report = build_fleet_report( + thumbs, + records_by_session, + window=window, + truncated=truncated, + sessions_omitted=omitted, + ) + logger.info( + "Fleet feedback: %d thumbs over %dd, %d sessions joined (%d omitted)", + report["totals"]["thumbs"], days, len(records_by_session), omitted, + ) + return FleetFeedbackResponse(**report) + + +async def _load_session_records( + storage: DynamoDBStorage, session_ids: List[str] +) -> Dict[str, List[Dict[str, Any]]]: + """Cost rows for each session, in bounded-concurrency batches. A session + that fails to read is dropped, not fatal: its thumbs then count toward + `coverage.unjoined`, which is the honest result.""" + out: Dict[str, List[Dict[str, Any]]] = {} + for offset in range(0, len(session_ids), SESSION_FETCH_CONCURRENCY): + batch = session_ids[offset:offset + SESSION_FETCH_CONCURRENCY] + results = await asyncio.gather( + *(storage.get_session_cost_records(sid) for sid in batch), + return_exceptions=True, + ) + for sid, result in zip(batch, results): + if isinstance(result, Exception): + logger.debug("Fleet join: session rows unavailable") + continue + out[sid] = list(result or []) + return out + + +@router.get("/evaluations", response_model=DownThumbQueueResponse, response_model_by_alias=True) +async def list_down_thumb_queue( + limit: int = Query(50, ge=1, le=200), + current_user: User = Depends(require_feedback_admin), + storage: DynamoDBStorage = Depends(get_storage), +): + """Recent down-thumbs across the fleet, newest first, with the verdict + where one exists. Content-free by projection.""" + try: + rows = await storage.get_recent_down_thumbs(limit=limit) + except Exception: + logger.error("Error listing the down-thumb queue", exc_info=True) + raise HTTPException(status_code=500, detail="Failed to list feedback queue") + items = [] + for row in rows: + try: + items.append(DownThumbQueueItem(**row)) + except Exception: # noqa: BLE001 - a malformed row is skipped, not fatal + continue + return DownThumbQueueResponse( + items=items, + pending=sum(1 for i in items if not i.evaluated_at), + sampling_enabled=feedback_eval_sampling_enabled(), + ) + + +@router.post("/evaluations/run", response_model=SamplingRunResponse, status_code=202, response_model_by_alias=True) +async def run_eval_sampling( + background: BackgroundTasks, + limit: int = Query(10, ge=1, le=50), + current_user: User = Depends(require_feedback_admin), + storage: DynamoDBStorage = Depends(get_storage), + judge=Depends(get_judge), +): + """Judge up to ``limit`` recent, not-yet-judged down-thumbs in the + background. 202 immediately; results appear on the queue list and the + session profiles as they land.""" + if not feedback_eval_sampling_enabled(): + raise HTTPException(status_code=404, detail="Not found") + + def cost_row_lookup(session_id: str, message_id: int) -> Optional[Dict[str, Any]]: + # Sync lookup for tool-failure corroboration: the call's C# row. + try: + from boto3.dynamodb.conditions import Key + + response = storage.sessions_metadata_table.query( + IndexName="SessionLookupIndex", + KeyConditionExpression=Key("GSI_PK").eq(f"SESSION#{session_id}") & Key("GSI_SK").begins_with("C#"), + ) + for item in response.get("Items", []): + try: + if int(item.get("messageId")) == message_id: + return storage._convert_decimal_to_float(item) + except (TypeError, ValueError): + continue + except Exception: # noqa: BLE001 - corroboration is best-effort + return None + return None + + async def task() -> None: + from apis.shared.feedback_eval.sampler import run_sampling_batch + + try: + await run_sampling_batch( + storage.sessions_metadata_table, judge, limit=limit, cost_row_lookup=cost_row_lookup, + ) + except Exception: # noqa: BLE001 - background; nothing to return to + logger.error("eval sampling batch failed", exc_info=True) + + background.add_task(task) + logger.info("Admin queued an eval sampling batch (limit=%d)", limit) + return SamplingRunResponse( + accepted=True, + limit=limit, + note="Judging runs in the background; the SDK waits for span ingestion, so allow a few minutes.", + ) diff --git a/backend/src/apis/app_api/admin/routes.py b/backend/src/apis/app_api/admin/routes.py index 2e4956931..4a310f4b7 100644 --- a/backend/src/apis/app_api/admin/routes.py +++ b/backend/src/apis/app_api/admin/routes.py @@ -923,6 +923,11 @@ async def get_managed_model_roles( router.include_router(costs_router) +# ========== Include Feedback Eval-Sampling Subrouter ========== +from .feedback.routes import router as feedback_admin_router + +router.include_router(feedback_admin_router) + # ========== Include User Admin Subrouter ========== from .users.routes import router as users_router @@ -1000,6 +1005,11 @@ async def get_managed_model_roles( router.include_router(system_prompts_admin_router) +# ========== Include Agent Templates Admin Subrouter ========== +from .agent_templates.routes import router as agent_templates_admin_router + +router.include_router(agent_templates_admin_router) + # ========== Include Announcements Admin Subrouter (conditional) ========== # Default ON with a kill switch. While ANNOUNCEMENTS_ENABLED=false the admin # authoring API is unmounted so the surface 404s, but the data and code remain diff --git a/backend/src/apis/app_api/agent_templates/__init__.py b/backend/src/apis/app_api/agent_templates/__init__.py new file mode 100644 index 000000000..7c5a5788f --- /dev/null +++ b/backend/src/apis/app_api/agent_templates/__init__.py @@ -0,0 +1 @@ +"""Public read endpoint for the agent template catalog (picker data source).""" diff --git a/backend/src/apis/app_api/agent_templates/routes.py b/backend/src/apis/app_api/agent_templates/routes.py new file mode 100644 index 000000000..778d75b88 --- /dev/null +++ b/backend/src/apis/app_api/agent_templates/routes.py @@ -0,0 +1,45 @@ +"""User-facing read endpoint for agent templates. + +Any signed-in user who can create an agent fetches the enabled templates to +render the "Start from a template" picker. The response is the client +``TemplateCatalogEntry[]`` shape (``{ draft, pitch }`` per entry, camelCase), +so the existing picker / reconcile / form-population path is unchanged — only +the picker's data *source* moves from the hardcoded array to this endpoint. + +Returns ``[]`` (never an error) when the catalog is empty, so a fork that has +not authored any templates degrades to a sensible empty state. + +Admin writes go through ``/admin/agent-templates``. +""" + +import logging + +from fastapi import APIRouter, Depends + +from apis.shared.auth import User, get_current_user_from_session +from apis.shared.agent_templates.models import ( + PublicTemplateListResponse, + TemplateCatalogEntryResponse, +) +from apis.shared.agent_templates.service import get_agent_templates_service + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/templates", tags=["agent-templates"]) + + +@router.get( + "/", + response_model=PublicTemplateListResponse, + summary="List available agent templates", +) +async def list_enabled_agent_templates( + current_user: User = Depends(get_current_user_from_session), +) -> PublicTemplateListResponse: + """Return all enabled templates as picker catalog entries, in display order.""" + service = get_agent_templates_service() + templates = await service.list_templates(enabled_only=True) + return PublicTemplateListResponse( + templates=[TemplateCatalogEntryResponse.from_template(t) for t in templates], + total=len(templates), + ) diff --git a/backend/src/apis/app_api/files/routes.py b/backend/src/apis/app_api/files/routes.py index beb6ccc55..29a093418 100644 --- a/backend/src/apis/app_api/files/routes.py +++ b/backend/src/apis/app_api/files/routes.py @@ -18,6 +18,7 @@ PresignResponse, CompleteUploadResponse, PreviewUrlResponse, + SheetPreviewResponse, TextSnippetResponse, ThumbnailResponse, FileListResponse, @@ -34,6 +35,7 @@ FileNotFoundError, FileUploadError, ) +from .sheet_preview import WorkbookTooLargeError, WorkbookUnreadableError from .thumbnails import ThumbnailRenderError, ThumbnailUnsupportedError from apis.shared.security.log_sanitize import scrub_log @@ -227,6 +229,64 @@ async def get_text_snippet( ) +@router.get("/{upload_id}/sheet-preview", response_model=SheetPreviewResponse) +async def get_sheet_preview( + upload_id: str, + user: User = Depends(get_current_user_from_session), + service: FileUploadService = Depends(get_file_upload_service), +): + """ + Read an .xlsx workbook into rows for the UI's data grid. + + The other previews (.docx, .pptx, .csv) hand the browser a presigned + URL and parse the bytes client-side. Spreadsheets cannot: the npm + build of SheetJS is frozen on a release with unfixed advisories, and + ExcelJS raises on any workbook holding a native chart — which is what + `create_excel_spreadsheet` produces. So the workbook is read here and + only values cross the wire. + + Values only. No fills, fonts, borders, merges or charts; download and + open the file for those. + + Status codes: + - 200: Sheets read (possibly truncated — see `truncated` on each). + - 404: File not found, not owned by the caller, or not readable. + - 413: Workbook is past the reader's size cap. + - 415: MIME type is not a readable workbook (the UI should not have + offered a preview; `.xls` lands here). + - 422: File present but unreadable (corrupt, encrypted, not OOXML). + """ + try: + return await service.get_sheet_preview(user.user_id, upload_id) + + except FileNotFoundError: + raise HTTPException( + status_code=status.HTTP_404_NOT_FOUND, + detail=f"File {upload_id} not found or not owned by you", + ) + + except ThumbnailUnsupportedError as e: + raise HTTPException( + status_code=status.HTTP_415_UNSUPPORTED_MEDIA_TYPE, + detail=str(e), + ) + + except WorkbookTooLargeError: + raise HTTPException( + status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE, + detail="This workbook is too large to preview. Download it to open in Excel.", + ) + + except WorkbookUnreadableError: + # Deliberately not echoing openpyxl's message: it names internal + # XML parts and tells the user nothing they can act on. + logger.warning("Workbook could not be parsed") + raise HTTPException( + status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, + detail="This workbook could not be read. It may be corrupt or password-protected.", + ) + + @router.get("/{upload_id}/thumbnail", response_model=ThumbnailResponse) async def get_thumbnail( upload_id: str, diff --git a/backend/src/apis/app_api/files/service.py b/backend/src/apis/app_api/files/service.py index 6dcd151ab..b392002de 100644 --- a/backend/src/apis/app_api/files/service.py +++ b/backend/src/apis/app_api/files/service.py @@ -9,7 +9,7 @@ import logging import uuid from datetime import datetime, timedelta, timezone -from typing import List, Optional +from typing import List, Optional, Set import boto3 from botocore.config import Config @@ -25,6 +25,8 @@ PresignResponse, CompleteUploadResponse, PreviewUrlResponse, + SheetPreviewResponse, + SHEET_PREVIEW_MIME_TYPES, TextSnippetResponse, ThumbnailResponse, THUMBNAIL_SUPPORTED_MIME_TYPES, @@ -34,6 +36,13 @@ is_allowed_mime_type, is_presentation_file, ALLOWED_MIME_TYPES, + MAX_FILES_PER_MESSAGE, +) +from .sheet_preview import ( + MAX_WORKBOOK_BYTES, + WorkbookTooLargeError, + WorkbookUnreadableError, + read_workbook_preview, ) from .thumbnails import ( ThumbnailRenderer, @@ -120,6 +129,11 @@ class FileUploadService: # Stored alongside the original so cleanup happens with the file. THUMBNAIL_KEY_NAME = "_thumb.png" + # Strong refs to fire-and-forget digest builds. The event loop only holds + # weak references to tasks, so a bare `create_task` can be collected + # mid-run (the trap web_sources/deletion_service.py documents). + _digest_tasks: Set["asyncio.Task[None]"] = set() + def __init__( self, repository: Optional[FileUploadRepository] = None, @@ -174,9 +188,10 @@ def __init__( "FILE_UPLOAD_MAX_SIZE_BYTES_PRESENTATION", 25 * 1024 * 1024 # 25MB ) ) - self.max_files_per_message = max_files_per_message or int( - os.environ.get("FILE_UPLOAD_MAX_FILES_PER_MESSAGE", 5) - ) + # Single source of truth is the shared constant (the inference API + # enforces it per message; see ``_apply_message_file_cap``). Kept on + # the service so callers can read the effective limit. + self.max_files_per_message = max_files_per_message or MAX_FILES_PER_MESSAGE self.user_quota_bytes = user_quota_bytes or int( os.environ.get("FILE_UPLOAD_USER_QUOTA_BYTES", 1024 * 1024 * 1024) # 1GB ) @@ -355,6 +370,13 @@ async def complete_upload( # Increment quota await self.repository.increment_quota(user_id, file_meta.size_bytes) + # DocumentDigest, off the request path (offload spec §4A / PR-2): + # documents get an outline + abstract persisted on their row so a + # later turn can carry the digest instead of the bytes. Non-documents + # (spreadsheets, decks, images) and the kill switch skip it. The + # response does not wait for it. + self._schedule_digest(file_meta) + logger.info("Completed file upload") return CompleteUploadResponse( @@ -540,6 +562,77 @@ async def get_text_snippet( mime_type=file_meta.mime_type, ) + # ========================================================================= + # Spreadsheet preview + # ========================================================================= + + async def get_sheet_preview( + self, user_id: str, upload_id: str + ) -> SheetPreviewResponse: + """Read an .xlsx into rows the UI can draw in its data grid. + + The workbook never reaches the browser. Unlike the `.docx`, + `.pptx` and `.csv` previews — which fetch the bytes through a + presigned URL and parse them client-side — there is no + client-side spreadsheet reader we are willing to ship, so the + parse happens here and only values cross the wire. + + Args: + user_id: The owner's user ID + upload_id: The upload identifier + + Returns: + SheetPreviewResponse with one entry per visible worksheet + + Raises: + FileNotFoundError: not found, not owned, or not ready + ThumbnailUnsupportedError: MIME type is not a readable workbook + WorkbookTooLargeError: past the reader's byte cap + WorkbookUnreadableError: corrupt, encrypted, or not OOXML + """ + file_meta = await self.repository.get_file(user_id, upload_id) + if not file_meta: + raise FileNotFoundError(f"File {upload_id} not found") + + if file_meta.status != FileStatus.READY: + raise FileNotFoundError( + f"File {upload_id} is not ready (status: {file_meta.status})" + ) + + if file_meta.mime_type not in SHEET_PREVIEW_MIME_TYPES: + raise ThumbnailUnsupportedError( + f"No spreadsheet reader for {file_meta.mime_type}" + ) + + # Checked before the download so an oversized workbook costs a + # metadata read rather than a transfer into memory. + if file_meta.size_bytes > MAX_WORKBOOK_BYTES: + raise WorkbookTooLargeError(file_meta.size_bytes) + + try: + response = self._s3_client.get_object( + Bucket=self.bucket_name, + Key=file_meta.s3_key, + ) + data = response["Body"].read() + except ClientError as e: + logger.warning( + f"Failed to read workbook {scrub_log(upload_id)}: {scrub_log(e)}" + ) + raise FileNotFoundError(f"File {upload_id} could not be read") + + # openpyxl is CPU-bound and blocking, so it runs off the event + # loop. A 20 MB workbook parses for long enough to stall every + # other request on this worker if it does not. + sheets = await asyncio.to_thread(read_workbook_preview, data) + + return SheetPreviewResponse( + upload_id=upload_id, + filename=file_meta.filename, + sheets=sheets, + truncated=any(sheet.truncated for sheet in sheets), + ) + # ========================================================================= # Thumbnails # ========================================================================= @@ -675,6 +768,56 @@ async def _render_and_store_thumbnail( f"({len(png_bytes)} bytes)" ) + # ========================================================================= + # DocumentDigest (docs/specs/document-context-offload.md §4A, PR-2) + # ========================================================================= + + def _schedule_digest(self, file_meta: FileMetadata) -> Optional["asyncio.Task[None]"]: + """Queue a digest build for a document upload; ``None`` when skipped. + + Skipped for non-documents, when ``DOCUMENT_DIGEST_ENABLED=false``, or + when no event loop is running (a synchronous caller). Never raises. + """ + try: + from apis.shared.files.document_digest import document_digest_enabled + from apis.shared.files.document_read import is_document_class + + if not document_digest_enabled(): + return None + if not is_document_class(file_meta.mime_type, file_meta.filename): + return None + task = asyncio.get_running_loop().create_task(self._generate_and_store_digest(file_meta)) + except Exception: # noqa: BLE001 - scheduling must never fail the upload + logger.warning("Document digest not scheduled", exc_info=True) + return None + self._digest_tasks.add(task) + task.add_done_callback(self._digest_tasks.discard) + return task + + async def _generate_and_store_digest(self, file_meta: FileMetadata) -> None: + """Read the original, build the digest, persist it. Never raises.""" + from apis.shared.files.document_digest import build_digest, record_digest + + try: + response = await asyncio.to_thread( + self._s3_client.get_object, Bucket=self.bucket_name, Key=file_meta.s3_key + ) + raw = await asyncio.to_thread(response["Body"].read) + digest = await build_digest( + raw=raw, + mime_type=file_meta.mime_type, + filename=file_meta.filename, + upload_id=file_meta.upload_id, + ) + record_digest(digest) + stored = await self.repository.update_file_digest( + file_meta.user_id, file_meta.upload_id, digest.to_item() + ) + if stored is None: + logger.info("Document digest discarded: file row no longer exists") + except Exception: # noqa: BLE001 - a digest is never worth an error + logger.warning("Document digest build failed", exc_info=True) + def _delete_thumbnail_object(self, file_meta: FileMetadata) -> None: """ Best-effort delete of the thumbnail sibling. diff --git a/backend/src/apis/app_api/files/sheet_preview.py b/backend/src/apis/app_api/files/sheet_preview.py new file mode 100644 index 000000000..22e01e155 --- /dev/null +++ b/backend/src/apis/app_api/files/sheet_preview.py @@ -0,0 +1,366 @@ +"""Read an .xlsx workbook into rows of display strings for the UI's grid. + +This is the server-side half of the spreadsheet preview. The browser gets +no renderer and no workbook bytes — just headers and rows it draws in the +same grid the `.csv` preview uses. + +**Why this runs here and not in the browser.** Every client-side option +was rejected on its own terms: the npm build of SheetJS is frozen on a +2022 release with unfixed advisories, and ExcelJS — the only base for a +maintained grid renderer — raises on any workbook containing a native +chart, which is exactly what `create_excel_spreadsheet` produces. Both of +those are *renderers*, reproducing Excel's own layout. A grid of values +needs only a reader, and openpyxl is already the library our own +spreadsheet tools drive inside Code Interpreter. + +**What this deliberately is not.** No fills, fonts, borders, merges, +column widths or charts. Download-and-open remains the path for anyone +who needs fidelity. Cells arrive as strings, already formatted for +display, because the alternative is shipping a type tag per cell and +re-implementing the same formatting decisions in TypeScript. +""" + +import logging +from datetime import date, datetime, time, timedelta +from decimal import Decimal +from io import BytesIO +from typing import Any, Iterable, Optional + +from openpyxl import load_workbook +from openpyxl.utils import get_column_letter + +from apis.shared.files.models import SheetPreview + +logger = logging.getLogger(__name__) + + +# Workbook bytes we are willing to pull into memory and parse. Well above +# anything the spreadsheet tools generate, and below the point where a +# request would threaten the container. +MAX_WORKBOOK_BYTES = 25 * 1024 * 1024 + +# Per-sheet caps. +# +# This is NOT a limit on what the grid can draw. The viewer virtualises +# with `cdk-virtual-scroll-viewport`, which recycles row elements, so the +# DOM is bounded no matter how many rows it is handed — the browser-side +# `.csv` reader hands it 50,000 quite happily. The limit here is the +# server: every row crosses the wire as JSON in one response, and +# openpyxl has to parse it first. +# +# Measured on a 20-column sheet (two-pass read, JSON serialised): +# +# cap parse JSON gzip +# 500 0.04s 0.06 MB 0.02 MB +# 5,000 0.53s 0.68 MB 0.15 MB +# 50,000 3.80s 7.17 MB 1.51 MB +# +# There is no gzip middleware on app-api, so the JSON column is what +# actually goes over the wire. 5,000 buys ten times the rows for half a +# second; 50,000 costs nearly four seconds of parse and a 7 MB response, +# which is past what a preview should spend. +# +# Fetching later pages on scroll was considered and rejected: openpyxl's +# read-only mode is a streaming parser, so `min_row` does not seek. +# Reaching row 45,000 of a 50,000-row sheet measured 2.63s against 2.97s +# for a *complete* pass — page cost grows with offset, and scrolling a +# whole sheet page-by-page would cost ~119s of CPU where one pass costs +# 3s. For this format, reading once and sending more is strictly better +# than reading repeatedly and sending less. +MAX_ROWS_PER_SHEET = 5_000 +MAX_COLUMNS_PER_SHEET = 64 + +# Sheets read from one workbook. +MAX_SHEETS = 12 + +# Global cell budget across every sheet, so a workbook of many wide +# sheets cannot multiply the per-sheet caps into a huge response. At +# roughly 12 bytes of JSON per cell this bounds the body near 1.8 MB. +MAX_TOTAL_CELLS = 150_000 + + +class WorkbookTooLargeError(Exception): + """The workbook is past `MAX_WORKBOOK_BYTES`.""" + + def __init__(self, size_bytes: int): + self.size_bytes = size_bytes + super().__init__( + f"Workbook is {size_bytes} bytes, over the {MAX_WORKBOOK_BYTES} limit" + ) + + +class WorkbookUnreadableError(Exception): + """The bytes are not a workbook openpyxl can read.""" + + +def read_workbook_preview(data: bytes) -> list[SheetPreview]: + """Read every visible sheet of an .xlsx into display-ready rows. + + Raises: + WorkbookTooLargeError: past the byte cap. + WorkbookUnreadableError: corrupt, encrypted, or not OOXML. + """ + if len(data) > MAX_WORKBOOK_BYTES: + raise WorkbookTooLargeError(len(data)) + + # Two passes over the same bytes, and both are needed. + # + # `data_only=True` yields the value Excel last cached for a formula + # cell — and `None` when nothing was cached, which is the normal case + # for a workbook openpyxl itself wrote, since openpyxl does no + # evaluation. Every formula in a workbook our own tools generated + # reads back as None, so a values-only preview would show blanks + # exactly where the totals are. + # + # `data_only=False` yields the formula text for those cells. Showing + # "=SUM(B2:B10)" is both more useful than an empty cell and more + # honest: the file really does not carry that number yet. + values = _load(data, data_only=True) + # The formula pass is opened lazily. It doubles the parse, and most + # sheets need it for nothing — it only earns its cost on a sheet that + # actually came back with an empty cell. + formulas = None + + def formula_sheet_at(index: int): + nonlocal formulas + if formulas is None: + formulas = _load(data, data_only=False) + sheets = formulas.worksheets + return sheets[index] if index < len(sheets) else None + + try: + sheets: list[SheetPreview] = [] + budget = MAX_TOTAL_CELLS + + for index, value_sheet in enumerate(values.worksheets): + if index >= MAX_SHEETS: + break + # Hidden sheets are hidden for a reason and Excel does not + # show them either. A preview that surfaced scratch sheets + # would misrepresent the workbook. + if getattr(value_sheet, "sheet_state", "visible") != "visible": + continue + + sheet, used = _read_sheet(value_sheet, index, formula_sheet_at, budget) + budget -= used + sheets.append(sheet) + if budget <= 0: + break + + return sheets + finally: + # `read_only` workbooks hold the zip open until closed. + values.close() + if formulas is not None: + formulas.close() + + +def _load(data: bytes, *, data_only: bool): + """Open the workbook in streaming mode, or say it is unreadable. + + `read_only=True` streams rows instead of building the whole object + graph, which is what makes a cap-and-stop read cheap on a large + sheet. `keep_links=False` avoids resolving external workbook + references — we never show them, and resolving them is work. + """ + try: + return load_workbook( + BytesIO(data), + read_only=True, + data_only=data_only, + keep_links=False, + ) + except Exception as e: # openpyxl raises a wide range on bad input + raise WorkbookUnreadableError(str(e)) from e + + +def _read_sheet( + value_sheet, index: int, formula_sheet_at, budget: int +) -> tuple[SheetPreview, int]: + """Read one worksheet into a `SheetPreview`, stopping at the caps. + + `formula_sheet_at` opens the second, formula-bearing pass on demand. + Returns the preview and how much of the cell budget it consumed. + """ + # Spend the cell budget against the sheet's OWN width, not the + # maximum a sheet is allowed to be. Dividing by MAX_COLUMNS_PER_SHEET + # charged a seven-column sheet as though it were sixty-four columns + # wide and cut its rows by an order of magnitude for no reason. + estimated_width = _peek_width(value_sheet) or MAX_COLUMNS_PER_SHEET + row_limit = min(MAX_ROWS_PER_SHEET, max(1, budget // estimated_width)) + + value_rows = _take_rows(value_sheet, row_limit + 1) + + # Only pay for the formula pass when this sheet has a gap to fill. An + # empty cell is either genuinely empty or a formula with no cached + # value, and the two are indistinguishable from the values pass — but + # a sheet with no empty cells at all cannot be hiding a formula. + formula_rows: list[tuple[Any, ...]] = [] + if any(cell is None for row in value_rows for cell in row): + formula_sheet = formula_sheet_at(index) + if formula_sheet is not None: + formula_rows = _take_rows(formula_sheet, row_limit + 1) + + merged = [ + _merge_row(value_rows[i], formula_rows[i] if i < len(formula_rows) else ()) + for i in range(len(value_rows)) + ] + + # Trailing empty columns are an artifact of a sheet's used range + # reaching further than its data — a stray format on an empty cell is + # enough. Dropping them keeps the grid from opening with a screenful + # of blanks. + width = 0 + for row in merged: + for i in range(len(row) - 1, -1, -1): + if row[i] != "": + width = max(width, i + 1) + break + + hit_column_cap = width > MAX_COLUMNS_PER_SHEET + width = min(width, MAX_COLUMNS_PER_SHEET) + + # `max_row` is what the file claims, which is the honest denominator + # for "showing 500 of N" even though we stopped reading early. + declared_rows = value_sheet.max_row or 0 + + if width == 0 or not merged: + return ( + SheetPreview( + name=str(value_sheet.title), + headers=[], + rows=[], + total_rows=max(0, declared_rows - 1) if declared_rows else 0, + truncated=False, + truncated_by=None, + ), + 0, + ) + + hit_row_cap = len(merged) > row_limit + body_source = merged[1 : row_limit + 1] + + headers = _normalize_headers(_fit(merged[0], width)) + rows = [_fit(row, width) for row in body_source] + + total_rows = max(len(rows), declared_rows - 1 if declared_rows else 0) + + return ( + SheetPreview( + name=str(value_sheet.title), + headers=headers, + rows=rows, + total_rows=total_rows, + truncated=hit_row_cap or hit_column_cap, + truncated_by="columns" if hit_column_cap else ("rows" if hit_row_cap else None), + ), + len(rows) * width, + ) + + +def _peek_width(sheet) -> int: + """Columns the sheet's header row occupies, for budgeting. + + Cheap: `max_column` comes from the sheet's declared dimensions and + needs no row scan. It can overstate the real width when a stray + format stretches the used range, which only makes the budget more + conservative, never less. + """ + try: + return min(int(sheet.max_column or 0), MAX_COLUMNS_PER_SHEET) + except Exception: + return 0 + + +def _take_rows(sheet, limit: int) -> list[tuple[Any, ...]]: + """Pull at most `limit` rows, stopping the stream rather than reading + the whole sheet to throw most of it away.""" + out: list[tuple[Any, ...]] = [] + try: + for row in sheet.iter_rows(values_only=True): + out.append(row) + if len(out) >= limit: + break + except Exception as e: + # A sheet can fail mid-stream on a malformed shared-string or + # date. Keep whatever was read — a partial preview beats none. + logger.warning("Stopped reading sheet early: %s", e) + return out + + +def _merge_row(value_row: Iterable[Any], formula_row: Iterable[Any]) -> list[str]: + """Format one row, falling back to formula text where no value was cached.""" + values = list(value_row) + formulas = list(formula_row) + out: list[str] = [] + for i, value in enumerate(values): + if value is None and i < len(formulas): + candidate = formulas[i] + if isinstance(candidate, str) and candidate.startswith("="): + out.append(candidate) + continue + out.append(_format_cell(value)) + return out + + +def _fit(row: list[str], width: int) -> list[str]: + """Pad or clip a row to the sheet's grid width.""" + if len(row) == width: + return row + if len(row) > width: + return row[:width] + return row + [""] * (width - len(row)) + + +def _normalize_headers(row: list[str]) -> list[str]: + """Column labels, with the spreadsheet letter standing in for a blank.""" + return [h.strip() or get_column_letter(i + 1) for i, h in enumerate(row)] + + +def _format_cell(value: Any) -> str: + """Render one cell value the way it should read in the grid. + + The formatting rules a spreadsheet applies (currency, percentages, + thousands separators) live in the cell's `number_format`, which + `read_only` + `values_only` does not carry. Rather than half-apply + them, values are rendered plainly and predictably. The one thing + worth care is float noise: a cell holding 0.1 + 0.2 must not preview + as 0.30000000000000004. + """ + if value is None: + return "" + if isinstance(value, bool): + # Checked before int: bool is a subclass of int in Python, and + # Excel shows these as TRUE/FALSE. + return "TRUE" if value else "FALSE" + if isinstance(value, int): + return str(value) + if isinstance(value, (float, Decimal)): + return _format_number(float(value)) + if isinstance(value, datetime): + # openpyxl returns a datetime for every date-formatted cell, so a + # plain date arrives with a midnight time that was never in the + # file. Showing it would invent precision. + if value.hour == value.minute == value.second == value.microsecond == 0: + return value.date().isoformat() + return value.isoformat(sep=" ") + if isinstance(value, date): + return value.isoformat() + if isinstance(value, time): + return value.isoformat() + if isinstance(value, timedelta): + return str(value) + return str(value) + + +def _format_number(value: float) -> str: + """Format a float without exposing binary-float noise.""" + if value != value or value in (float("inf"), float("-inf")): + return str(value) + if value.is_integer() and abs(value) < 1e15: + return str(int(value)) + # 15 significant digits is the most a float carries; formatting to it + # collapses 0.30000000000000004 to 0.3 while leaving genuine + # precision alone. + text = f"{value:.15g}" + return text diff --git a/backend/src/apis/app_api/main.py b/backend/src/apis/app_api/main.py index 2755a4876..99907f938 100644 --- a/backend/src/apis/app_api/main.py +++ b/backend/src/apis/app_api/main.py @@ -17,7 +17,6 @@ load_dotenv(dotenv_path=env_path, override=True) from fastapi import FastAPI -from fastapi.staticfiles import StaticFiles from fastapi.middleware.cors import CORSMiddleware from contextlib import asynccontextmanager import logging @@ -83,17 +82,6 @@ async def lifespan(app: FastAPI): logger.info("=== AgentCore Public Stack API Starting ===") logger.info("Agent execution engine initialized") - # Create output directories if they don't exist - base_dir = Path(__file__).parent.parent - output_dir = os.path.join(base_dir, "output") - uploads_dir = os.path.join(base_dir, "uploads") - generated_images_dir = os.path.join(base_dir, "generated_images") - - os.makedirs(output_dir, exist_ok=True) - os.makedirs(uploads_dir, exist_ok=True) - os.makedirs(generated_images_dir, exist_ok=True) - logger.info("Output directories ready") - yield # Application is running # Shutdown @@ -160,8 +148,10 @@ async def lifespan(app: FastAPI): # Starlette `add_middleware` prepends, so the LAST-added middleware is # outermost. Request-side order is therefore the reverse of the call order # below: -# request: SessionRefresh → CSRF → AgentCoreContext → CORS → router -# response: router → CORS → AgentCoreContext → CSRF → SessionRefresh +# request: ProxiedRedirect → GZip → SessionRefresh → CSRF → +# AgentCoreContext → CORS → router +# response: router → CORS → AgentCoreContext → CSRF → SessionRefresh → +# GZip → ProxiedRedirect # This is the order we need: SessionRefresh has to populate # `state.bff_session` before CSRF reads it. from apis.shared.middleware.csrf import CSRFMiddleware @@ -171,6 +161,29 @@ async def lifespan(app: FastAPI): app.add_middleware(SessionRefreshMiddleware) logger.info("Added BFF session-refresh + CSRF middlewares (dormant until cookie present)") +# gzip the JSON surface. Nothing compressed app-api responses before this: +# CloudFront's `/api/*` behaviour is deliberately `compress: false` so the +# edge never buffers a `text/event-stream`, which leaves compression to the +# origin — the only layer that knows a response's content type rather than +# guessing from its path. Measured on this repo's own payload shapes at +# `compresslevel=6`: ~3.0x on a conversation-history response, ~3.9x on a +# 5,000-row spreadsheet-shaped one. Level 9 (Starlette's default) buys 3% +# more for 4x the CPU on the large case, so 6 — zlib's own default — it is. +# +# `StreamSafeGZipMiddleware` passes `text/event-stream` and already-encoded +# bodies through untouched *and un-buffered*; see its module docstring for +# why the second half of that matters on the chat path. +# +# Sits one layer inside ProxiedRedirect so it compresses everything the app +# emits — including the error bodies CSRF and SessionRefresh return — while +# leaving ProxiedRedirect genuinely outermost. The two never collide: +# ProxiedRedirect reads and rewrites only `Location`, on responses (3xx) +# whose bodies are empty or below the compression threshold anyway. +from apis.shared.middleware.compression import StreamSafeGZipMiddleware + +app.add_middleware(StreamSafeGZipMiddleware, minimum_size=500, compresslevel=6) +logger.info("Added gzip compression middleware (SSE and pre-encoded bodies excluded)") + # Outermost middleware: repair `Location` headers on redirects this app # generates for itself (Starlette's `redirect_slashes`, chiefly). Behind # CloudFront those come out as `http://api./` — the internal @@ -217,6 +230,7 @@ async def lifespan(app: FastAPI): from apis.app_api.user_menu_links.routes import router as user_menu_links_router from apis.app_api.announcements.routes import router as announcements_router from apis.app_api.system_prompts.routes import router as system_prompts_router +from apis.app_api.agent_templates.routes import router as agent_templates_router from apis.app_api.runs.routes import router as runs_router from apis.app_api.schedules.routes import router as schedules_router @@ -256,6 +270,7 @@ async def lifespan(app: FastAPI): app.include_router(user_menu_links_router) # Public read of admin-managed user-menu links app.include_router(announcements_router) # Feature announcements feed + ack; 404s while ANNOUNCEMENTS_ENABLED off app.include_router(system_prompts_router) # Public read of admin-managed system prompts +app.include_router(agent_templates_router) # Public read of admin-managed agent templates (create-agent picker) app.include_router(runs_router) # Headless "Run now" + grant lifecycle (scheduled-runs PR-1; SCHEDULED_RUNS_ENABLED + RBAC gated at runtime) app.include_router(schedules_router) # Schedule CRUD (scheduled-runs B1; inert — SCHEDULED_RUNS_ENABLED + RBAC gated, nothing fires yet) @@ -289,26 +304,6 @@ async def lifespan(app: FastAPI): app.include_router(shared_artifacts_router) logger.info("Artifact render-token and sharing routes enabled") -# Mount static file directories for serving generated content -# These are created by tools (visualization, code interpreter, etc.) -# Use parent directory (src/) as base -base_dir = Path(__file__).parent.parent -output_dir = os.path.join(base_dir, "output") -uploads_dir = os.path.join(base_dir, "uploads") -generated_images_dir = os.path.join(base_dir, "generated_images") - -if os.path.exists(output_dir): - app.mount("/output", StaticFiles(directory=output_dir), name="output") - logger.info(f"Mounted static files: /output -> {output_dir}") - -if os.path.exists(uploads_dir): - app.mount("/uploads", StaticFiles(directory=uploads_dir), name="uploads") - logger.info(f"Mounted static files: /uploads -> {uploads_dir}") - -if os.path.exists(generated_images_dir): - app.mount("/generated_images", StaticFiles(directory=generated_images_dir), name="generated_images") - logger.info(f"Mounted static files: /generated_images -> {generated_images_dir}") - if __name__ == "__main__": import uvicorn # Watch the full backend/src tree so edits to shared modules outside diff --git a/backend/src/apis/app_api/messages/models.py b/backend/src/apis/app_api/messages/models.py index 44c0f4723..fa93ef496 100644 --- a/backend/src/apis/app_api/messages/models.py +++ b/backend/src/apis/app_api/messages/models.py @@ -117,6 +117,64 @@ class Citation(BaseModel): text: str = Field(..., description="Relevant text excerpt from the document") +#: Reason codes a down-thumb may carry — the six buckets of +#: ``docs/specs/response-feedback.md`` §6, each of which routes to an +#: evaluator or an ops signal. A closed enum, never free text: the row is +#: content-free by construction so it can sit beside the ``C#`` cost row and +#: be read by the admin profile without reading the conversation. The spec's +#: "something else → free text" is deliberately not here; that hand-off is +#: the existing Agent report dialog (spec §3), which already has moderation. +FEEDBACK_REASONS = ("wrong", "instructions", "length", "tool_failed", "outdated", "other") +FeedbackReason = Literal["wrong", "instructions", "length", "tool_failed", "outdated", "other"] + + +#: Implicit signals (response-feedback spec §10): denser than thumbs, no UI +#: cost, written to the same ``F#`` family under ``signal: "implicit"`` and +#: never summed with them. ``copy`` = the response was copied out; +#: ``continue`` = a truncated / interrupted response was resumed. Edit-and- +#: resend has no affordance in the SPA yet; abandonment is deferred (its +#: base rate is indistinguishable from a satisfied user going quiet). +IMPLICIT_SIGNAL_KINDS = ("copy", "continue") +ImplicitSignalKind = Literal["copy", "continue"] + + +class ImplicitSignalRequest(BaseModel): + """Body of ``POST /sessions/{id}/messages/{messageId}/signals``.""" + + model_config = ConfigDict(populate_by_name=True) + + kind: ImplicitSignalKind = Field(..., description="Which implicit signal fired (closed enum)") + + +class MessageFeedback(BaseModel): + """One user's thumb on one assistant message (``F#`` row, see + ``apis.shared.sessions.metadata``). ``value`` is +1 (up) or -1 (down); + ``reason`` is an optional code from ``FEEDBACK_REASONS``.""" + + model_config = ConfigDict(populate_by_name=True) + + value: Literal[1, -1] = Field(..., description="+1 for thumbs up, -1 for thumbs down") + reason: Optional[FeedbackReason] = Field(None, description="Optional reason code (never free text)") + retry_message_id: Optional[int] = Field( + None, alias="retryMessageId", ge=0, + description="Index of the user message sent as a retry-with-correction after this thumb (content-free link)", + ) + updated_at: str = Field(..., alias="updatedAt", description="ISO timestamp of the latest thumb") + + +class MessageFeedbackRequest(BaseModel): + """Body of ``PUT /sessions/{id}/messages/{messageId}/feedback``.""" + + model_config = ConfigDict(populate_by_name=True) + + value: Literal[1, -1] = Field(..., description="+1 for thumbs up, -1 for thumbs down") + reason: Optional[FeedbackReason] = Field(None, description="Optional reason code (never free text)") + retry_message_id: Optional[int] = Field( + None, alias="retryMessageId", ge=0, + description="Set when the user sent a retry-with-correction: that user message's index", + ) + + class MessageMetadata(BaseModel): """Metadata associated with a single message""" @@ -129,8 +187,10 @@ class MessageMetadata(BaseModel): cost: Optional[Union[float, Dict[str, float]]] = Field(None, description="Cost for this message — either a total float (legacy) or a breakdown dict with total, inputCost, outputCost, cacheReadCost, cacheWriteCost") citations: Optional[List[Dict[str, str]]] = Field(None, description="RAG citations for this message (stored as dicts for flexible JSON storage)") display_text: Optional[str] = Field(None, alias="displayText", description="Original user message text before RAG augmentation (for clean UI display)") - # Note: Feedback will be added in future implementation - # feedback: Optional[Feedback] = None + # One user's thumb on this message, merged from the ``F#`` row on read + # (see ``apis.shared.sessions.metadata``). Content-free: a ±1, a timestamp + # and an optional reason code. + feedback: Optional[MessageFeedback] = Field(None, description="User thumbs up/down on this assistant message") class Message(BaseModel): diff --git a/backend/src/apis/app_api/sessions/routes.py b/backend/src/apis/app_api/sessions/routes.py index 47c01fee4..953d48e4e 100644 --- a/backend/src/apis/app_api/sessions/routes.py +++ b/backend/src/apis/app_api/sessions/routes.py @@ -3,7 +3,7 @@ Provides endpoints for managing session metadata. """ -from fastapi import APIRouter, HTTPException, Depends, Query, Response, BackgroundTasks, status +from fastapi import APIRouter, HTTPException, Depends, Path, Query, Response, BackgroundTasks, status from typing import Optional import logging from apis.shared.sessions.models import ( @@ -18,7 +18,17 @@ BulkDeleteSessionsRequest, BulkDeleteSessionsResponse, BulkDeleteSessionResult, - MessagesListResponse + MessagesListResponse, + MessageFeedback, + MessageFeedbackRequest, + ImplicitSignalRequest, + BrowserLiveViewResponse, +) +from apis.shared.sessions.feedback import ( + SessionNotOwned, + delete_message_feedback, + put_message_feedback, + record_implicit_signal, ) from apis.shared.sessions.messages import get_messages from apis.shared.sessions.metadata import ( @@ -35,7 +45,11 @@ from apis.app_api.shares.service import get_share_service from apis.app_api.artifacts.service import get_artifact_share_service from apis.shared.auth.dependencies import get_current_user_from_session -from apis.shared.feature_flags import mid_turn_steering_enabled +from apis.shared.feature_flags import ( + response_feedback_enabled, + mid_turn_steering_enabled, + browser_takeover_enabled, +) from apis.shared.auth.models import User from apis.shared.system_prompts.service import get_system_prompts_service @@ -662,6 +676,113 @@ async def get_session_messages_endpoint( ) +def _require_message_feedback() -> None: + """404 while ``RESPONSE_FEEDBACK_ENABLED=false`` — the surface does not exist.""" + if not response_feedback_enabled(): + raise HTTPException(status_code=404, detail="Not found") + + +@router.put( + "/{session_id}/messages/{message_id}/feedback", + response_model=MessageFeedback, + response_model_by_alias=True, + response_model_exclude_none=True, +) +async def put_message_feedback_endpoint( + session_id: str, + message_id: int = Path(..., ge=0, description="0-based message index"), + body: MessageFeedbackRequest = ..., + current_user: User = Depends(get_current_user_from_session), +): + """Thumb an assistant message up (+1) or down (-1), optionally with a + reason code. Idempotent per (user, message): a second click replaces the + first. Content-free by construction — the body is a closed enum, so no + text can be stored (``apis.shared.sessions.feedback``). + + ``message_id`` is the message's 0-based index in the conversation — the + trailing number of the SPA's ``msg-{sessionId}-{index}`` id, and the + ``messageId`` the message's cost row carries. ``retryMessageId`` in the + body links the user message sent as a retry-with-correction; it is kept + across later thumbs on the same message. + """ + _require_message_feedback() + try: + return await put_message_feedback( + session_id=session_id, + user_id=current_user.user_id, + message_id=message_id, + value=body.value, + reason=body.reason, + retry_message_id=body.retry_message_id, + ) + except SessionNotOwned: + raise HTTPException(status_code=404, detail=f"Session not found: {session_id}") + except ValueError as e: + raise HTTPException(status_code=422, detail=str(e)) + except RuntimeError as e: + # No metadata table configured (local dev without DynamoDB). + logger.warning("Message feedback unavailable: %s", scrub_log(str(e))) + raise HTTPException(status_code=503, detail="Message feedback is not available") + except Exception: + logger.error("Error storing message feedback", exc_info=True) + raise HTTPException(status_code=500, detail="Failed to store message feedback") + + +@router.delete("/{session_id}/messages/{message_id}/feedback", status_code=204) +async def delete_message_feedback_endpoint( + session_id: str, + message_id: int = Path(..., ge=0, description="0-based message index"), + current_user: User = Depends(get_current_user_from_session), +): + """Withdraw this user's thumb on a message. 204 whether or not one existed.""" + _require_message_feedback() + try: + await delete_message_feedback( + session_id=session_id, + user_id=current_user.user_id, + message_id=message_id, + ) + except SessionNotOwned: + raise HTTPException(status_code=404, detail=f"Session not found: {session_id}") + except RuntimeError as e: + logger.warning("Message feedback unavailable: %s", scrub_log(str(e))) + raise HTTPException(status_code=503, detail="Message feedback is not available") + except Exception: + logger.error("Error deleting message feedback", exc_info=True) + raise HTTPException(status_code=500, detail="Failed to delete message feedback") + return Response(status_code=204) + + +@router.post("/{session_id}/messages/{message_id}/signals", status_code=204) +async def record_implicit_signal_endpoint( + session_id: str, + message_id: int = Path(..., ge=0, description="0-based message index"), + body: ImplicitSignalRequest = ..., + current_user: User = Depends(get_current_user_from_session), +): + """Record an implicit signal (``copy`` / ``continue``) on an assistant + message — response-feedback spec §10. Fire-and-forget from the SPA: + always 204 once accepted, never a reason to show the user anything. + Content-free: the body is a closed enum.""" + _require_message_feedback() + try: + await record_implicit_signal( + session_id=session_id, + user_id=current_user.user_id, + message_id=message_id, + kind=body.kind, + ) + except SessionNotOwned: + raise HTTPException(status_code=404, detail=f"Session not found: {session_id}") + except RuntimeError as e: + logger.warning("Message feedback unavailable: %s", scrub_log(str(e))) + raise HTTPException(status_code=503, detail="Message feedback is not available") + except Exception: + logger.error("Error recording implicit signal", exc_info=True) + raise HTTPException(status_code=500, detail="Failed to record signal") + return Response(status_code=204) + + @router.post("/{session_id}/interrupt", status_code=204) async def signal_turn_interrupted_endpoint( session_id: str, @@ -892,3 +1013,77 @@ async def dismiss_pending_interrupt_endpoint( status_code=500, detail=f"Failed to dismiss interrupt: {str(e)}", ) + + +@router.post( + "/{session_id}/browser/live-view", + response_model=BrowserLiveViewResponse, + response_model_by_alias=True, +) +async def mint_browser_live_view_endpoint( + session_id: str, + current_user: User = Depends(get_current_user_from_session), +): + """Mint a short-lived Live View URL for this conversation's browser session. + + The client sends **only** the conversation id. This route looks the browser + session up server-side from the metadata row's `browserSession` projection + (spec D4) and signs a fresh URL per call, which is what makes a sign-in that + takes twenty minutes work against a signature that lives 300 seconds. + + POST rather than GET on purpose: the response carries a live SigV4 + signature, and a GET invites it into browser history, referrer headers and + access logs. Nothing sensitive goes in the path or query either way. + + Ownership is the metadata read itself — `get_session_metadata` is + user-scoped through the GSI, so another user's conversation is + indistinguishable from a missing one, and both are 404. + + Status codes: + * 404 — flag off, no such conversation for this user, or no browser + session on it. All three are "this surface does not exist for you". + * 409 — the conversation names a browser session the service will no + longer stream (ended, timed out, stopped). Distinct from 404 because + the SPA should say "the session ended", not "no viewer here". + + Lives on app-api, not inference-api: the Runtime data plane proxies only + `/invocations` and `/ping`, so this would 404 in cloud from there. + """ + if not browser_takeover_enabled(): + raise HTTPException(status_code=404, detail="Not found") + + from .services.browser_live_view import ( + LiveViewUnavailable, + mint_live_view, + summarize_for_log, + ) + + user_id = current_user.user_id + logger.info("POST /sessions/.../browser/live-view") + + metadata = await get_session_metadata(session_id, user_id) + if not metadata: + raise HTTPException(status_code=404, detail="Not found") + + ref = metadata.browser_session + if not ref: + raise HTTPException( + status_code=404, + detail="This conversation has no browser session to view.", + ) + + try: + minted = await mint_live_view(ref) + except LiveViewUnavailable as exc: + logger.info( + "browser live view unavailable for %s (%s): %s", + scrub_log(session_id), summarize_for_log(ref), exc.message, + ) + raise HTTPException(status_code=exc.code, detail=exc.message) + except Exception: + # Deliberately generic: the exception text from a signing failure can + # contain the partially-built URL. + logger.error("Error minting browser live view", exc_info=True) + raise HTTPException(status_code=500, detail="Failed to mint live view") + + return BrowserLiveViewResponse.model_validate(minted) diff --git a/backend/src/apis/app_api/sessions/services/browser_live_view.py b/backend/src/apis/app_api/sessions/services/browser_live_view.py new file mode 100644 index 000000000..3ad7f9a5a --- /dev/null +++ b/backend/src/apis/app_api/sessions/services/browser_live_view.py @@ -0,0 +1,144 @@ +"""Mint a short-lived Live View URL for a conversation's browser session. + +Spec D2 (`docs/specs/authenticated-web-assessment.md`). The agent never sees +this URL and it is never persisted; it is minted here, per request, against the +browser session the conversation's metadata row names (the D4 projection +written in PR 1). + +Why a route rather than a value on the SSE event +------------------------------------------------ +`generate_live_view_url` signs with `SigV4QueryAuth` — the signature lives in +the query string — and caps at **300 seconds**. So a URL minted when the turn +paused is dead before a human has found their password manager, dead again on +every reload of the thread, and (per PR #1101) re-emitted truncated at the `?` +if it ever passes through a tool result. Minting per request is the only shape +that survives a twenty-minute login. + +Why app-api +----------- +The AgentCore Runtime data plane proxies only `/invocations` and `/ping`, so a +route on inference-api would 404 in cloud no matter how well it worked locally +(CLAUDE.md, "Inference API boundary"). app-api also owns the session-metadata +row and the cookie session, which is what makes the ownership check real. +""" + +from __future__ import annotations + +import asyncio +import logging +import os +from datetime import datetime, timedelta, timezone +from typing import Any, Dict, Optional + +logger = logging.getLogger(__name__) + +# The service's own hard cap (`MAX_LIVE_VIEW_PRESIGNED_URL_TIMEOUT`). Asking +# for more raises rather than clamping, so we clamp here. +MAX_EXPIRY_SECONDS = 300 + +LIVE_VIEW_EXPIRY_SECONDS = min( + int(os.environ.get("BROWSER_LIVE_VIEW_EXPIRY_SECONDS", MAX_EXPIRY_SECONDS)), + MAX_EXPIRY_SECONDS, +) + +DEFAULT_VIEWPORT = {"width": 1280, "height": 800} + + +class LiveViewUnavailable(Exception): + """No live view can be minted, with a reason the caller maps to a status. + + ``code`` mirrors the HTTP status the route should return: + 404 when the conversation has no browser session at all, 409 when it names + one the service can no longer stream. + """ + + def __init__(self, message: str, code: int = 404) -> None: + super().__init__(message) + self.message = message + self.code = code + + +def _region() -> str: + return os.environ.get("AWS_REGION", "us-west-2") + + +def _viewport(ref: Dict[str, Any]) -> Dict[str, int]: + """The session's real viewport, falling back to the default. + + Carried through rather than re-declared in the viewer, because DCV's + `remoteWidth`/`remoteHeight` must match the browser session's viewport or + the stream crops — and a session can outlive a config change. + """ + raw = ref.get("viewport") + if isinstance(raw, dict): + try: + return {"width": int(raw["width"]), "height": int(raw["height"])} + except (KeyError, TypeError, ValueError): + logger.warning("browser live view: unusable viewport %r; using default", raw) + return dict(DEFAULT_VIEWPORT) + + +async def mint_live_view(ref: Dict[str, Any]) -> Dict[str, Any]: + """Return `{url, expiresAt, viewport, controlState}` for a browser session. + + `ref` is the conversation's `browserSession` projection. Raises + :class:`LiveViewUnavailable` when it is missing fields or the service + refuses — an expired or stopped session is the common case and reads as + 409, not as an error worth a 500. + """ + browser_session_id = ref.get("browserSessionId") + browser_id = ref.get("browserId") + if not browser_session_id or not browser_id: + raise LiveViewUnavailable( + "This conversation has no browser session to view.", code=404 + ) + + from bedrock_agentcore.tools.browser_client import BrowserClient + + client = BrowserClient(region=_region()) + client.identifier = browser_id + client.session_id = browser_session_id + + requested_at = datetime.now(timezone.utc) + try: + # boto3 is synchronous; keep it off the event loop. + url = await asyncio.to_thread( + client.generate_live_view_url, LIVE_VIEW_EXPIRY_SECONDS + ) + except Exception as exc: # noqa: BLE001 - the reason is the useful part + logger.info( + "browser live view: could not sign %s (%s)", browser_session_id, exc + ) + raise LiveViewUnavailable( + "The browser session is no longer available. It may have ended or " + "timed out.", + code=409, + ) from exc + + if not url: + raise LiveViewUnavailable( + "The browser session returned no live view.", code=409 + ) + + # Deliberately computed from when we asked, not from when the client reads + # it: the viewer refreshes against this instant, and a value derived later + # would drift past the signature's real expiry. + expires_at = requested_at + timedelta(seconds=LIVE_VIEW_EXPIRY_SECONDS) + + return { + "url": url, + "expiresAt": expires_at.isoformat(), + "viewport": _viewport(ref), + "controlState": ref.get("controlState") or "agent", + } + + +def summarize_for_log(ref: Optional[Dict[str, Any]]) -> str: + """A log-safe description of a browser session reference. + + Never includes the minted URL: it carries a live SigV4 signature, and a + signed URL in CloudWatch is a credential in CloudWatch. + """ + if not ref: + return "none" + return f"browser={ref.get('browserId')} session={ref.get('browserSessionId')}" diff --git a/backend/src/apis/app_api/skills/routes.py b/backend/src/apis/app_api/skills/routes.py index e528f1b92..f1bd5a34f 100644 --- a/backend/src/apis/app_api/skills/routes.py +++ b/backend/src/apis/app_api/skills/routes.py @@ -44,6 +44,7 @@ from apis.shared.skills.access import resolve_accessible_skill_ids from apis.shared.skills.bundle import slugify_skill_name from apis.shared.skills.models import ( + SKILL_DESCRIPTION_MAX_LENGTH, SkillDefinition, SkillResourceRef, SkillResourcesResponse, @@ -178,7 +179,7 @@ async def update_skill_preferences( class MySkillResponse(BaseModel): - """One skill the caller authored, as shown on the My Skills page.""" + """One skill the caller authored, as shown under Customize → Skills.""" skill_id: str = Field(..., alias="skillId") display_name: str = Field(..., alias="displayName") @@ -230,7 +231,7 @@ class CreateMySkillRequest(BaseModel): """ display_name: str = Field(..., alias="displayName", max_length=200) - description: str = Field(..., max_length=2000) + description: str = Field(..., max_length=SKILL_DESCRIPTION_MAX_LENGTH) instructions: str = "" allowed_tools: List[str] = Field(default_factory=list, alias="allowedTools") skill_metadata: Dict = Field(default_factory=dict, alias="skillMetadata") @@ -248,7 +249,9 @@ class UpdateMySkillRequest(BaseModel): """ display_name: Optional[str] = Field(None, alias="displayName", max_length=200) - description: Optional[str] = Field(None, max_length=2000) + description: Optional[str] = Field( + None, max_length=SKILL_DESCRIPTION_MAX_LENGTH + ) instructions: Optional[str] = None allowed_tools: Optional[List[str]] = Field(None, alias="allowedTools") skill_metadata: Optional[Dict] = Field(None, alias="skillMetadata") diff --git a/backend/src/apis/app_api/system_prompts/routes.py b/backend/src/apis/app_api/system_prompts/routes.py index 6a9e73c83..96782bf24 100644 --- a/backend/src/apis/app_api/system_prompts/routes.py +++ b/backend/src/apis/app_api/system_prompts/routes.py @@ -1,7 +1,8 @@ """User-facing read endpoint for system prompts. -Signed-in users fetch the list of enabled prompts to display in the -conversation settings panel. The prompt_text is never returned here — +Signed-in users fetch the list of enabled prompts to display in whichever +picker surfaces them (the composer settings drawer that used to was removed +with the Customize surface). The prompt_text is never returned here — only the name and description. Admin writes go through /admin/system-prompts. """ diff --git a/backend/src/apis/inference_api/chat/routes.py b/backend/src/apis/inference_api/chat/routes.py index eadd580fc..923dcb35b 100644 --- a/backend/src/apis/inference_api/chat/routes.py +++ b/backend/src/apis/inference_api/chat/routes.py @@ -10,6 +10,7 @@ import asyncio import json import logging +from collections import OrderedDict from typing import AsyncGenerator, Optional, Union from fastapi import APIRouter, Depends, HTTPException, status @@ -26,11 +27,17 @@ ) from apis.inference_api.runtime_health import ping_payload from apis.shared.feature_flags import ( + agent_preparing_phase_enabled, agents_enabled, + attachment_turn_guard_enabled, mid_turn_steering_enabled, skills_enabled, ) from apis.shared.files.file_resolver import get_file_resolver +from apis.shared.files.models import ( + INLINE_ATTACHMENTS_MAX_TOTAL_BYTES, + MAX_FILES_PER_MESSAGE, +) from apis.shared.models.managed_models import list_managed_models from apis.shared.quota import ( QuotaExceededEvent, @@ -50,7 +57,7 @@ ) from apis.shared.sessions.metadata import ( ensure_session_metadata_exists, - session_owned_by_other_user, + load_session_meta, ) from apis.shared.tools.injected import ( ARTIFACT_TOOL_IDS, @@ -74,6 +81,7 @@ from .agent_binding_policy import binds_conversation from .models import FileContent, InvocationRequest from .service import generate_conversation_title, get_agent +from .turn_timing import TurnPrelude from .system_prompt_resolver import ( append_active_prompt, resolve_active_prompt_text, @@ -751,10 +759,226 @@ def _build_memory_tools(agent_memory, user_id: str, user_email: str) -> list: return tools +# ============================================================ +# Document Read Tool Injection (docs/specs/document-context-offload.md §4B) +# ============================================================ + +#: Sessions known to carry a readable document. The gate is one DynamoDB +#: query per turn otherwise; a positive answer is memoized because it is +#: monotonic in practice (an upload stays unless the user deletes it, and a +#: stale tool on a session whose files were deleted just returns "not found"). +#: Negative answers are never memoized — the next turn may be the upload. +_DOCUMENT_SESSIONS: "OrderedDict[str, bool]" = OrderedDict() +_DOCUMENT_SESSIONS_MAX = 10_000 + + +def _remember_document_session(session_id: str) -> None: + _DOCUMENT_SESSIONS[session_id] = True + _DOCUMENT_SESSIONS.move_to_end(session_id) + while len(_DOCUMENT_SESSIONS) > _DOCUMENT_SESSIONS_MAX: + _DOCUMENT_SESSIONS.popitem(last=False) + + +async def _session_has_documents( + session_id: str, + user_id: str, + turn_upload_ids: list | None = None, +) -> bool: + """Whether ``document_read`` should exist on this turn. + + True when this turn attaches uploads (their metadata rows already exist, + so no query is needed), when the session was seen carrying a document + earlier in this process, or when the session's upload rows include at + least one readable document (PDF, Word, text, markdown, HTML — not + spreadsheets, decks or images, which have other paths). Fail-closed on + error: a turn without the tool is today's behavior, never a broken turn. + """ + if turn_upload_ids: + _remember_document_session(session_id) + return True + if _DOCUMENT_SESSIONS.get(session_id): + return True + try: + from apis.shared.files.document_read import session_has_documents + + present = await session_has_documents(user_id, session_id) + except Exception: # noqa: BLE001 - the gate must never fail a turn + logger.warning("document_read gate lookup failed; tool not injected this turn", exc_info=True) + return False + if present: + _remember_document_session(session_id) + return present + + +async def _document_tools_gate( + session_id: str, + user_id: str, + turn_upload_ids: list | None = None, +) -> bool: + """The single answer to "does this turn carry ``document_read``" — the + builder and the resume path's cache key both read it, so the two can + never disagree (a disagreement orphans a paused agent).""" + from apis.shared.feature_flags import document_read_enabled + + if not document_read_enabled(): + return False + if not session_id or not user_id: + return False + return await _session_has_documents(session_id, user_id, turn_upload_ids) + + +async def _build_document_tools( + session_id: str, + user_id: str, + turn_upload_ids: list | None = None, +) -> list: + """Context-bound ``document_read`` for a session that has a readable attachment. + + **Not gated on ``enabled_tools``** — the governing capability is the user's + own attachment, exactly as the Memory-Space tools are governed by an + Agent's binding. Its id stays out of ``INJECTED_TOOL_IDS``. Kill switch: + ``DOCUMENT_READ_ENABLED=false``. The gate's answer also feeds the agent + cache key (``has_document_tools``), so an agent cached before the first + upload is never served without the tool afterwards. + """ + if not await _document_tools_gate(session_id, user_id, turn_upload_ids): + return [] + + from agents.builtin_tools.document_read_tool import make_document_read_tool + + tools = [make_document_read_tool(session_id, user_id)] + logger.info("Created document_read tool (session has a readable document)") + return tools + + # ============================================================ # Attachment Partitioning (#206) # ============================================================ +# ============================================================ +# Spreadsheet Analysis auto-enable on attachment +# (docs/specs/load-test-assessment-2026-09.md P2-E) +# ============================================================ + +#: Sessions known to hold a spreadsheet attachment. Same contract as +#: ``_DOCUMENT_SESSIONS``: one query per turn otherwise, positive answers +#: memoized because an upload stays unless the user deletes it, negative +#: answers never memoized because the next turn may be the upload. +_TABULAR_SESSIONS: "OrderedDict[str, bool]" = OrderedDict() +_TABULAR_SESSIONS_MAX = 10_000 + + +def _remember_tabular_session(session_id: str) -> None: + _TABULAR_SESSIONS[session_id] = True + _TABULAR_SESSIONS.move_to_end(session_id) + while len(_TABULAR_SESSIONS) > _TABULAR_SESSIONS_MAX: + _TABULAR_SESSIONS.popitem(last=False) + + +async def _session_has_tabular( + session_id: str, + user_id: str, + turn_has_tabular: bool = False, +) -> bool: + """Whether this session's turns need the Spreadsheet Analysis tools. + + True when this turn attaches a CSV/XLSX (already partitioned, no query), + when the session was seen holding one earlier in this process, or when + the session's READY uploads include one. Sticky by design: the answer + feeds ``enabled_tools`` and therefore the agent-cache key, so it must + not flip between the attach turn and the follow-up. Fail-closed on + error: a turn without the tools is today's behavior, never a broken turn. + """ + if turn_has_tabular: + _remember_tabular_session(session_id) + return True + if _TABULAR_SESSIONS.get(session_id): + return True + if not session_id or not user_id: + return False + try: + from apis.shared.files.document_read import session_has_tabular_files + + present = await session_has_tabular_files(user_id, session_id) + except Exception: # noqa: BLE001 - the gate must never fail a turn + logger.warning("spreadsheet auto-enable lookup failed; tools not injected this turn", exc_info=True) + return False + if present: + _remember_tabular_session(session_id) + return present + + +def _with_auto_enabled_tools(enabled_tools: list | None, auto_ids: list[str]) -> list | None: + """``enabled_tools`` plus ``auto_ids`` not already present, appended in the + order given. Returns the same object when there is nothing to add, so a + caller that passed ``None`` still passes ``None`` and every consumer of the + list (cache key, builders, guidance, ToolFilter) sees one value.""" + if not auto_ids: + return enabled_tools + current = list(enabled_tools or []) + missing = [tool_id for tool_id in auto_ids if tool_id not in current] + if not missing: + return enabled_tools + return current + missing + + +async def _auto_enabled_attachment_tool_ids( + current_user: User, + session_id: str, + user_id: str, + turn_has_tabular: bool = False, +) -> list[str]: + """The Spreadsheet Analysis ids this turn should carry regardless of the + picker, in a fixed order: every id in ``SPREADSHEET_TOOL_IDS`` the + caller's RBAC grant admits, when the session holds a spreadsheet. + + Enables, never grants: ``can_access_tool`` is the same predicate the + picker and Agent bindings answer to (role grant ∪ public tools). A user + whose roles do not carry the tool gets today's behavior — the attachment + note tells them the tool is not available to their account. + """ + from apis.shared.feature_flags import attachment_tool_autoenable_enabled + + if not attachment_tool_autoenable_enabled(): + return [] + if not await _session_has_tabular(session_id, user_id, turn_has_tabular): + return [] + role_service = get_app_role_service() + allowed: list[str] = [] + for tool_id in sorted(SPREADSHEET_TOOL_IDS): + try: + if await role_service.can_access_tool(current_user, tool_id): + allowed.append(tool_id) + except Exception: # noqa: BLE001 - an RBAC lookup failure must not fail the turn + logger.warning("RBAC check for %s failed; not auto-enabling", tool_id, exc_info=True) + if allowed: + logger.info( + "Auto-enabled %s for a session with a spreadsheet attachment (session=%s)", + allowed, scrub_log(session_id), + ) + return allowed + + +async def _apply_attachment_tool_autoenable( + enabled_tools: list | None, + current_user: User, + session_id: str, + user_id: str, + turn_has_tabular: bool = False, +) -> list | None: + """``enabled_tools`` for this turn with the attachment auto-enable applied. + + The single seam every ``get_agent`` caller on the invocation path goes + through, so the main turn and the MCP App dispatch paths compute the same + effective list — and therefore the same agent-cache slot — for a session + holding a spreadsheet. + """ + auto_ids = await _auto_enabled_attachment_tool_ids( + current_user, session_id, user_id, turn_has_tabular=turn_has_tabular + ) + return _with_auto_enabled_tools(enabled_tools, auto_ids) + + def _estimate_decoded_size(file: "FileContent") -> int: """Estimate decoded byte size of a base64-encoded FileContent payload. @@ -821,6 +1045,110 @@ def _partition_attachments( return inline, tabular, presentations, oversized +def _apply_message_file_cap( + direct_files: list, + upload_ids: list, + max_files: int, +) -> tuple[list, list, list, int]: + """Hold a message to ``max_files`` attachments across both request paths. + + Returns ``(direct_files, upload_ids, dropped_names, dropped_total)``. + Direct ``files`` come first (they are already in the request body), then + ``file_upload_ids`` fill whatever budget remains. Attachment order is + kept, so the first N the user attached are the N that survive. + + The cap is applied to the upload IDs *before* they are resolved: the old + resolver default truncated silently after the fact, and letting every ID + through just to name the losers would fan out one S3 read per ID a client + chose to send. IDs beyond the budget are therefore counted, not named — + ``dropped_names`` holds the direct files (names known) and + ``dropped_total`` counts both. ``max_files <= 0`` disables the cap. + """ + if max_files <= 0: + return direct_files, upload_ids, [], 0 + + kept_direct = direct_files[:max_files] + dropped_names = [f.filename for f in direct_files[max_files:]] + id_budget = max(0, max_files - len(kept_direct)) + kept_ids = upload_ids[:id_budget] + dropped_total = len(dropped_names) + (len(upload_ids) - len(kept_ids)) + return kept_direct, kept_ids, dropped_names, dropped_total + + +def _apply_inline_byte_budget( + inline: list, + max_total_bytes: int, +) -> tuple[list, list, int]: + """Hold the inline set (documents *and* images) to one message's byte + budget. Returns ``(kept, over_budget, requested_bytes)``. + + Why this exists: the turn's inline attachments are persisted as one + AgentCore Memory event, and past ~7.5 MB of raw bytes that write fails + with ``SessionException`` — a hole in history, not a degraded turn. See + ``INLINE_ATTACHMENTS_MAX_TOTAL_BYTES`` for the derivation. + + Policy — first-fit in attachment order: walk the files as the user + attached them, keep each one that still fits, and move any that would + push the running total over the budget to ``over_budget``. Earlier + attachments win, and a later, smaller file that still fits rides along + rather than being punished for a large neighbour. Order within both + lists is the attachment order, so the marker text and the guidance note + are deterministic (they land in the cacheable prefix on later turns). + + Images count toward the budget: they are part of the same message and + the same event, even though the per-file document gate skips them. + ``max_total_bytes <= 0`` disables the budget. + """ + requested = sum(_estimate_decoded_size(f) for f in inline) + if max_total_bytes <= 0: + return list(inline), [], requested + + kept: list = [] + over: list = [] + running = 0 + for file in inline: + size = _estimate_decoded_size(file) + if running + size > max_total_bytes: + over.append(file) + continue + running += size + kept.append(file) + return kept, over, requested + + +def _emit_attachment_over_quota_metric( + requested_bytes: int, + cap_bytes: int, + inline_count: int, + dropped_count: int, +) -> None: + """One content-free EMF record in ``AgentCoreStack/Compaction`` when a + turn's inline attachments had to be trimmed to the byte budget. Never + raises. ``AttachmentTurnOverQuota`` carries the requested bytes so the + rate *and* the size distribution of over-quota turns are measurable + (spec §4E put the rate at ~1.3–1.4% of attachment turns from a proxy; + this is the direct count). + """ + try: + from apis.shared.observability.emf import emit_emf_metrics + from apis.shared.observability.prompt_cache import prompt_cache_observability_enabled + + if not prompt_cache_observability_enabled(): + return + emit_emf_metrics( + "AgentCoreStack/Compaction", + metrics={"AttachmentTurnOverQuota": requested_bytes}, + properties={ + "capBytes": cap_bytes, + "inlineFileCount": inline_count, + "droppedFileCount": dropped_count, + }, + units={"AttachmentTurnOverQuota": "Bytes"}, + ) + except Exception as e: # noqa: BLE001 + logger.debug("AttachmentTurnOverQuota EMF skipped: %s", e) + + def _attachment_marker_names(all_files: list, oversized_inline: list) -> list: """Filenames for the ``[Attached files: …]`` marker on the user message. @@ -852,10 +1180,21 @@ def _build_attachment_guidance( diverted_presentations: list, oversized_inline: list, enabled_tools: list | None, + over_budget: list | None = None, + dropped_over_count_names: list[str] | None = None, + dropped_over_count_total: int = 0, + max_files: int = 0, ) -> str: """Return a short markdown addendum describing how attachments will be handled, to append to the user's message so the agent (and the user) both understand why a file isn't inline. + + ``oversized_inline`` is the per-file case (the file itself is too big; + the fix is a smaller file). ``over_budget`` is the aggregate case (each + file is fine, together they exceed one message's budget; the fix is a + follow-up message). They get separate sentences because the remedy + differs. ``dropped_over_count_*`` describe files beyond the per-message + count cap: names where known (direct ``files``), a count otherwise. """ parts: list[str] = [] @@ -872,11 +1211,15 @@ def _build_attachment_guidance( f"to run aggregations or lookups._" ) else: + # Reached only when the auto-enable did not apply: the caller's + # roles do not carry the tool, or the kill switch is set. Say so + # without sending them to a toggle that may not be there. parts.append( - f"_Attached spreadsheet(s) {names} can't be read inline at " - f"this size. To analyze them, enable **Spreadsheet Analysis** " - f"in the Tools section of the settings panel (gear icon next " - f"to the message input), then re-send your message._" + f"_Attached spreadsheet(s) {names} can't be read inline. " + f"Analyzing them needs the **Spreadsheet Analysis** tool, which " + f"isn't available in this conversation — if it is listed under " + f"Customize → Tools in the sidebar, enable it and re-send your " + f"message; if it isn't, your account doesn't have access to it._" ) if diverted_presentations: @@ -895,9 +1238,9 @@ def _build_attachment_guidance( else: parts.append( f"_Attached presentation(s) {names} can't be read inline. To " - f"work with them, enable **PowerPoint Presentations** in the " - f"Tools section of the settings panel (gear icon next to the " - f"message input), then re-send your message._" + f"work with them, enable **PowerPoint Presentations** under " + f"Customize → Tools in the sidebar, then re-send your " + f"message._" ) if oversized_inline: @@ -908,6 +1251,32 @@ def _build_attachment_guidance( f"and use the Spreadsheet Analysis tool._" ) + if over_budget: + names = ", ".join(f"`{f.filename}`" for f in over_budget) + parts.append( + f"_Attached file(s) {names} were skipped because this message's " + f"attachments together exceed the combined size limit for a " + f"single message. Send them in a follow-up message._" + ) + + if dropped_over_count_total > 0: + limit = f"{max_files} file" + ("s" if max_files != 1 else "") + if dropped_over_count_names: + names = ", ".join(f"`{n}`" for n in dropped_over_count_names) + unnamed = dropped_over_count_total - len(dropped_over_count_names) + tail = f" and {unnamed} more" if unnamed > 0 else "" + parts.append( + f"_Only the first {limit} per message are attached; " + f"{names}{tail} were not. Send them in a follow-up message._" + ) + else: + noun = "file was" if dropped_over_count_total == 1 else "files were" + parts.append( + f"_Only the first {limit} per message are attached; " + f"{dropped_over_count_total} more {noun} not. " + f"Send them in a follow-up message._" + ) + return "\n\n".join(parts) @@ -1290,6 +1659,17 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g user_id = current_user.user_id auth_token = current_user.raw_token + # Where the pre-stream time goes. Everything between here and the + # `StreamingResponse` return happens with NO channel open to the client — + # measured at 3.75s on a warm turn — so this is the only way to see which + # stage owns it. Pure timing: nothing reaches the model. + # See `turn_timing.py` and docs/specs/agent-state-feedback.md. + prelude = TurnPrelude() + # Whether this turn's agent is built inside the stream (PR-3). Recorded on + # the `turn_prelude` line so the two shapes stay distinguishable in the + # logs once the flag has been on for a while. + deferred_build = False + # Refuse a turn against a session id another user already owns. # # Session ids travel in shareable URLs (`/s/{sessionId}`). Opening someone @@ -1304,12 +1684,29 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # stops the id from being forked at all. 404 rather than 403 so the # response says nothing about whether the session exists, matching what # `GET /sessions/{id}/metadata` already returns for the same case. - if await session_owned_by_other_user(input_data.session_id, user_id): + # + # ONE read of the session's META row, shared by everything in the preamble + # that used to fetch it again (PR-2, docs/specs/turn-latency-preamble.md). + # Measured on dev: eight separate reads of this item cost ~445ms of a + # ~455ms stage, because a GSI query from an AgentCore Runtime container is + # ~53ms rather than the ~12ms an in-region figure would suggest. + # + # Deliberately explicit rather than a per-request memo inside + # `_get_session_by_gsi`: CLAUDE.md's "never cache session state" rule has + # been paid for twice (#741, #751), and a snapshot callers opt into cannot + # leak into one that needs a fresh read. + session_meta = await load_session_meta(input_data.session_id, user_id) + if session_meta.owned_by_other: logger.warning( "Rejected invocation for session %s — owned by a different user", _sanitize_log(input_data.session_id), ) raise HTTPException(status_code=404, detail="Session not found") + # First of the preamble's five sub-stages (docs/specs/turn-latency-preamble.md). + # The coarse `preamble` number survives as `groups.preamble` in the emitted + # line, so the four-turn baseline in the agent-state-feedback spec stays + # comparable across this split. + prelude.mark("preamble.ownership") # Resume requests reuse the cached agent and its paused interrupt state; # they bypass quota, file resolution, and RAG augmentation because those # already ran on the original turn that got paused. @@ -1344,6 +1741,10 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g await _resolve_accessible_skill_ids(current_user), input_data.enabled_skills, ) + # Near-zero on a turn that selects no skills — the opt-in default (D6) + # short-circuits before touching RBAC or the skill table. A non-trivial + # number here means the RBAC cache missed or the owner-index query is slow. + prelude.mark("preamble.skills") # A "Continue" after a max_tokens truncation. Like resume, it bypasses # quota / RAG / file resolution and does NOT clear the turn state; unlike # resume there is no interrupt to validate — the agent is rebuilt from the @@ -1381,7 +1782,11 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g session_id=input_data.session_id, user_id=user_id, auth_token=auth_token, - enabled_tools=input_data.enabled_tools, + # Same auto-enable seam as the main turn, so a spreadsheet + # session's dispatch reads the slot the real turns fill. + enabled_tools=await _apply_attachment_tool_autoenable( + input_data.enabled_tools, current_user, input_data.session_id, user_id + ), model_id=input_data.model_id, system_prompt=input_data.system_prompt, caching_enabled=caching_enabled, @@ -1395,6 +1800,7 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # This path builds no injected tools, but shares a cache slot # with the real turns that do. Read the slot; never seed it. cache_write=False, + assistant_id=input_data.rag_assistant_id, ) payload = await dispatch_app_tool_call( agent, @@ -1437,7 +1843,11 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g session_id=input_data.session_id, user_id=user_id, auth_token=auth_token, - enabled_tools=input_data.enabled_tools, + # Same auto-enable seam as the main turn, so a spreadsheet + # session's dispatch reads the slot the real turns fill. + enabled_tools=await _apply_attachment_tool_autoenable( + input_data.enabled_tools, current_user, input_data.session_id, user_id + ), model_id=input_data.model_id, system_prompt=input_data.system_prompt, caching_enabled=caching_enabled, @@ -1450,6 +1860,7 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g accessible_skill_ids=effective_skill_ids, # Same partial-toolset hazard as app_tool_call above. cache_write=False, + assistant_id=input_data.rag_assistant_id, ) payload = dispatch_app_context_update( agent, @@ -1491,7 +1902,9 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g if not is_resume and not is_continuation: try: from apis.shared.sessions.metadata import pop_pending_attachments - recovered_upload_ids = await pop_pending_attachments(input_data.session_id, user_id) + recovered_upload_ids = await pop_pending_attachments( + input_data.session_id, user_id, snapshot=session_meta + ) except Exception as e: logger.error("Failed to recover pending attachments: %s", e, exc_info=True) @@ -1532,14 +1945,39 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # budget; we skip them inline and surface a note instead of # letting Bedrock reject the turn. all_files = list(input_data.files) if input_data.files else [] + upload_ids_to_resolve = list(input_data.file_upload_ids or []) + + # Per-message file count (spec §4E / PR-6). Applied here, before the + # S3 fetch, so a sixth file is reported to the user instead of silently + # truncated by the resolver — and so a client cannot fan out unbounded + # S3 reads. With the guard off, the resolver's own backstop (5) applies + # exactly as it did before. + turn_guard_on = attachment_turn_guard_enabled() + dropped_over_count_names: list[str] = [] + dropped_over_count_total = 0 + if turn_guard_on: + ( + all_files, + upload_ids_to_resolve, + dropped_over_count_names, + dropped_over_count_total, + ) = _apply_message_file_cap(all_files, upload_ids_to_resolve, MAX_FILES_PER_MESSAGE) + if dropped_over_count_total: + logger.warning( + "Dropped %d attachment(s) over the %d-per-message cap", + dropped_over_count_total, + MAX_FILES_PER_MESSAGE, + ) - if input_data.file_upload_ids: + if upload_ids_to_resolve: try: file_resolver = get_file_resolver() resolved_files = await file_resolver.resolve_files( user_id=user_id, - upload_ids=input_data.file_upload_ids, - max_files=5, # Bedrock document limit + upload_ids=upload_ids_to_resolve, + # Already capped above when the guard is on; the resolver's + # own backstop is the pre-guard behaviour. + max_files=None if turn_guard_on else 5, ) for rf in resolved_files: all_files.append( @@ -1602,7 +2040,42 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g f"{[(f.filename, _estimate_decoded_size(f)) for f in oversized_inline]}" ) - attachment_marker_names = _attachment_marker_names(all_files, oversized_inline) + # Aggregate budget for the turn (spec §4E / PR-6): the inline set is one + # persisted message, and a message over ~7.5 MB raw fails the AgentCore + # Memory write with SessionException. Trim first-fit in attachment order; + # the trimmed files join the oversized note path, never the exception. + over_budget_inline: list = [] + if turn_guard_on and files_to_send: + files_to_send, over_budget_inline, requested_inline_bytes = _apply_inline_byte_budget( + files_to_send, INLINE_ATTACHMENTS_MAX_TOTAL_BYTES + ) + if over_budget_inline: + logger.warning( + "Attachment turn over quota: requested_bytes=%d cap_bytes=%d " + "inline_files=%d dropped_files=%d", + requested_inline_bytes, + INLINE_ATTACHMENTS_MAX_TOTAL_BYTES, + len(files_to_send) + len(over_budget_inline), + len(over_budget_inline), + ) + _emit_attachment_over_quota_metric( + requested_bytes=requested_inline_bytes, + cap_bytes=INLINE_ATTACHMENTS_MAX_TOTAL_BYTES, + inline_count=len(files_to_send) + len(over_budget_inline), + dropped_count=len(over_budget_inline), + ) + + # Both classes were dropped from the turn entirely; the marker must not + # promise a card for either. + attachment_marker_names = _attachment_marker_names( + all_files, oversized_inline + over_budget_inline + ) + + # Covers the unconsumed-attachment recovery read, the S3 fetch behind + # `resolve_files`, and the inline/tabular/oversized partitioning. Expected + # to be ~0 on a turn with no attachments; if it is not, the hypothesis in + # docs/specs/turn-latency-preamble.md is wrong about where the time is. + prelude.mark("preamble.files") # Pre-create session metadata so OAuth interrupts and other state can # attach to the session row from turn one. Best-effort; on failure the @@ -1615,13 +2088,15 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # already moved past. is_new_session = False if not is_resume and not is_continuation: - is_new_session = await ensure_session_metadata_exists(input_data.session_id, user_id) + is_new_session = await ensure_session_metadata_exists( + input_data.session_id, user_id, snapshot=session_meta + ) try: from apis.shared.sessions.metadata import ( clear_paused_turn, clear_pending_interrupts, ) - await clear_paused_turn(input_data.session_id, user_id) + await clear_paused_turn(input_data.session_id, user_id, snapshot=session_meta) # The snapshot's breadcrumbs go with it. They are the other half of # the same record, and a breadcrumb that outlives the snapshot # re-renders a prompt the user can no longer answer: the resume @@ -1629,7 +2104,9 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # here specifically because this runs at the *head* of a non-resume # turn — any breadcrumb this turn goes on to write lands later, on # its own `done` event. - await clear_pending_interrupts(input_data.session_id, user_id) + await clear_pending_interrupts( + input_data.session_id, user_id, snapshot=session_meta + ) except Exception as e: logger.error("Failed to clear stale paused_turn on new turn: %s", e, exc_info=True) @@ -1641,7 +2118,7 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g if not is_resume: try: from apis.shared.sessions.metadata import clear_truncated_turn - await clear_truncated_turn(input_data.session_id, user_id) + await clear_truncated_turn(input_data.session_id, user_id, snapshot=session_meta) except Exception as e: logger.error("Failed to clear stale truncated_turn on new turn: %s", e, exc_info=True) @@ -1654,7 +2131,9 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # in the stream generator below. try: from apis.shared.sessions.metadata import clear_interrupted_turn - interrupted_turn_reason = await clear_interrupted_turn(input_data.session_id, user_id) + interrupted_turn_reason = await clear_interrupted_turn( + input_data.session_id, user_id, snapshot=session_meta + ) except Exception as e: logger.error("Failed to clear stale interrupted_turn on new turn: %s", e, exc_info=True) @@ -1691,6 +2170,14 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g ) ) + # The prime suspect: five of the preamble's eight reads of the session's + # META row live in this stage (pre-create plus the four stale-marker + # clears), each on its own round trip, and four of them short-circuit + # without writing anything. The title task is inside the boundary because + # spawning it is first-turn session state; it is an `asyncio.create_task`, + # so it contributes nothing to the number. + prelude.mark("preamble.session_state") + # Check quota if enforcement is enabled quota_warning_event = None quota_session_notice_event = None @@ -1698,7 +2185,23 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g if is_quota_enforcement_enabled() and not is_resume and not is_continuation: try: quota_checker = get_quota_checker() - quota_result = await quota_checker.check_quota(user=current_user, session_id=input_data.session_id) + # Hand the quota checker the session cost we already read (PR-2b). + # ONLY when the row actually carries `totalCost`: absent means a + # legacy row that `get_session_metadata` still needs to backfill, + # and `None` routes the checker back to that read. Passing 0.0 for + # a missing attribute would silence the notice on exactly the + # long-lived conversations it exists to catch. + session_total_cost = None + if session_meta.row is not None and "totalCost" in session_meta.row: + try: + session_total_cost = float(session_meta.row["totalCost"]) + except (TypeError, ValueError): + session_total_cost = None + quota_result = await quota_checker.check_quota( + user=current_user, + session_id=input_data.session_id, + session_total_cost=session_total_cost, + ) if not quota_result.allowed: # Quota blocked - stream as SSE instead of 429 for better UX @@ -1727,6 +2230,11 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g # Log error but don't block request - fail open for quota errors logger.error("Error checking quota for user", exc_info=True) + # The quota round trip: a cached tier resolve, a cached O(1) cost-summary + # GetItem, and — the uncached one — the per-session notice, which reads the + # META row for the eighth time this turn. + prelude.mark("preamble.quota") + # If quota exceeded, stream the quota exceeded message instead of agent response if quota_exceeded_event: return StreamingResponse( @@ -2268,6 +2776,10 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g % is_agent_mention ) + # Assistant resolution, the knowledge-base search and its metadata writes. + # Zero on a plain chat turn, which is what makes it worth separating. + prelude.mark("rag") + # Append active custom system prompt (if any). Gating rules + lookup live # in `system_prompt_resolver.py` so they can be unit-tested independently # of the route. @@ -2402,6 +2914,18 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g mantle_region=snapshot.mantle_region, agent_type=snapshot.agent_type, is_resume=True, + # The original turn's key carried whether the session had a + # readable document; the gate is monotonic, so re-asking it + # rebuilds the same key (an orphaned paused agent otherwise). + has_document_tools=await _document_tools_gate( + input_data.session_id, user_id + ), + # The assistant the original turn ran against is a key element + # (spreadsheet tools close over it). Replay the snapshot's + # value, not the request's: a snapshot written before the + # field existed carries None, which misses the slot and + # rebuilds — the pre-existing eviction path, never a wrong hit. + assistant_id=snapshot.assistant_id, # Resume must rebuild the SAME cache key the original turn used, # or the paused agent is orphaned. New snapshots carry the # original turn's exact effective set in enabled_skills, so @@ -2521,6 +3045,20 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g if agent_tools_override is not None else input_data.enabled_tools ) + # A session holding a spreadsheet gets the Spreadsheet Analysis + # tools whether or not the picker has them on, gated on the + # caller's RBAC grant. Applied to the *effective* list so it + # flows into the cache key, every builder below, the attachment + # guidance and the paused-turn snapshot as one value. Sticky + # across the session (see `_session_has_tabular`), so the key + # does not flip between the attach turn and the follow-up. + effective_enabled_tools = await _apply_attachment_tool_autoenable( + effective_enabled_tools, + current_user, + input_data.session_id, + user_id, + turn_has_tabular=bool(diverted_tabular), + ) # An Agent's skill bindings replace the request's skills for this turn so # ChatAgent's AgentSkills plugin discloses exactly the bound set (D5, @@ -2566,6 +3104,18 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g ) extra_tools = extra_tools + memory_tools + # document_read for any session that carries a readable attachment + # (this turn's uploads count). Gated on session state, not the + # picker; its presence goes into the cache key below rather than + # vetoing the cache, so an attachment session that could keep a + # warm agent still does. + document_tools = await _build_document_tools( + session_id=input_data.session_id, + user_id=user_id, + turn_upload_ids=input_data.file_upload_ids, + ) + extra_tools = extra_tools + document_tools + # Can this turn's agent be cached despite carrying injected tools? # Only when every builder that fired closes over values the cache # key already carries (session, user, enabled_tools). Derived from @@ -2577,24 +3127,66 @@ async def invocations(request: InvocationRequest, current_user: User = Depends(g has_memory_binding=bool(memory_tools), ) - agent = await get_agent( - session_id=input_data.session_id, - user_id=user_id, - auth_token=auth_token, - enabled_tools=effective_enabled_tools, - model_id=effective_model_id, - system_prompt=system_prompt, # Use assistant's instructions if available - caching_enabled=caching_enabled, - provider=effective_provider, - inference_params=inference_params, - mantle_api_mode=mantle_api_mode, - mantle_region=mantle_region, - agent_type=effective_agent_type, - extra_tools=extra_tools, - is_resume=False, - accessible_skill_ids=effective_skill_ids, - extra_tools_key_described=extra_tools_key_described, - ) + # System-prompt assembly, the single-flight lease, skill + # resolution and every tool builder (documents, attachments, + # memory, agent binding). + prelude.mark("tools") + + def _mark_build_stage(stage: str) -> None: + """Namespace a build sub-stage under `agent_build.` so the + emitted line groups them the way the preamble's are.""" + prelude.mark(f"agent_build.{stage}") + + async def _build_main_agent(): + """The turn's agent. Called eagerly here, or from the stream. + + A closure rather than an inline call because it now has two + call sites — see `defer_agent_build` below — and eighteen + keyword arguments that must not drift between them. + """ + return await get_agent( + session_id=input_data.session_id, + user_id=user_id, + auth_token=auth_token, + enabled_tools=effective_enabled_tools, + model_id=effective_model_id, + system_prompt=system_prompt, # Use assistant's instructions if available + caching_enabled=caching_enabled, + provider=effective_provider, + inference_params=inference_params, + mantle_api_mode=mantle_api_mode, + mantle_region=mantle_region, + agent_type=effective_agent_type, + extra_tools=extra_tools, + is_resume=False, + accessible_skill_ids=effective_skill_ids, + extra_tools_key_described=extra_tools_key_described, + has_document_tools=bool(document_tools), + assistant_id=input_data.rag_assistant_id, + build_stage_recorder=_mark_build_stage, + ) + + # Defer the build into the stream so it can be narrated. + # + # Measured on dev: a cold agent-cache miss spends 1478ms here + # against a 2542ms pre-stream window, and every millisecond of it + # is dead air — FastAPI flushes headers when this handler returns, + # so until then there is no channel to say anything on. Deferring + # opens the response first and emits a `preparing` frame, turning + # the longest silence in the product into a sentence. + # + # A warm turn spends 0-38ms here, so this changes nothing for the + # common case; it exists for the cold one. + # See docs/specs/agent-state-feedback.md PR-3. + if agent_preparing_phase_enabled(): + agent = None + deferred_build = True + else: + agent = await _build_main_agent() + # The remainder after the sub-stages the build itself recorded; + # `groups.agent_build` sums them, keeping the pre-split + # number comparable exactly as it did for the preamble. + prelude.mark("agent_build.rest") # Resume requests must target interrupts that the cached agent # actually has paused. Cache eviction, a process restart, or a @@ -2697,6 +3289,10 @@ def _session_title_sse() -> Optional[str]: diverted_presentations, oversized_inline, effective_enabled_tools, + over_budget=over_budget_inline, + dropped_over_count_names=dropped_over_count_names, + dropped_over_count_total=dropped_over_count_total, + max_files=MAX_FILES_PER_MESSAGE, ) # When multiple spreadsheets are visible, ship the full inventory # up front so the agent can disambiguate intentionally instead of @@ -2793,6 +3389,12 @@ def _session_title_sse() -> Optional[str]: original_message=input_data.message if message_will_be_modified else None, interrupt_responses=interrupt_responses_payload, continue_truncated=is_continuation, + # The turn's true start, so the end-of-turn recap spans the + # WHOLE turn. The coordinator's own clock starts when its + # generator is iterated, which is AFTER the deferred agent + # build — measured on dev, that under-reported a 7.8s turn as + # 2.1s. + turn_started_at=prelude.started_at, # Which Agent ran this turn (#756). Recorded on the cost row so a # deliberate `@`-mention prefix swap is distinguishable from the # nondeterministic-ordering regression the fingerprints exist to catch. @@ -2867,17 +3469,135 @@ def _session_title_sse() -> Optional[str]: # generator's finally is the release site for the happy path (the two # except handlers below cover pre-stream failures). async def _guarded_stream() -> AsyncGenerator[str, None]: - heartbeat_task = ( - asyncio.create_task(_lease_heartbeat_loop(session_lease, agent)) - if session_lease is not None - else None - ) + nonlocal agent + + heartbeat_task = None try: + # The deferred agent build (PR-3). Narrated, because this is + # where a cold turn spends over a second with nothing on the + # wire. The frame goes out FIRST so the client hears something + # the moment the response opens; the build follows. + if agent is None: + # Announce the build unconditionally; the SPA decides + # whether it is worth SHOWING. + # + # This used to race the build against a 250ms timer here + # and emit only if it was still running, so a warm build + # (0-38ms) never flashed a phase nobody can read. That + # cannot work: `create_agent` is synchronous + # (`agent_factory.py`), so a cold build occupies the event + # loop for its whole duration and `asyncio.wait` cannot + # fire its timeout — it returned only once the build was + # already done, `finished` was non-empty, and the frame was + # never sent. Verified on dev: a 1548ms build, six times + # the threshold, emitted nothing. + # + # A timer only works where the clock actually runs, which + # is the client. The SPA holds this phase for 250ms before + # rendering it, so a warm build still never shows — see + # `message-list.component.ts`. + yield ( + "event: agent_status\ndata: " + + json.dumps( + { + "type": "agent_status", + "sessionId": input_data.session_id, + "phase": "preparing", + } + ) + + "\n\n" + ) + try: + agent = await _build_main_agent() + except Exception as build_error: + # The handler has already returned, so the two `except` + # arms below cannot see this — a build that fails here + # would otherwise be a silent, hung stream. Surface it + # the way every other mid-stream failure is surfaced + # (CLAUDE.md: errors stream as assistant messages), and + # let the `finally` release the lease. + logger.error( + "Deferred agent build failed", exc_info=True + ) + error_event = build_conversational_error_event( + code=ErrorCode.AGENT_ERROR, + error=build_error, + session_id=input_data.session_id, + recoverable=True, + ) + async for frame in stream_conversational_message( + message=error_event.message, + stop_reason="error", + metadata_event=error_event, + session_id=input_data.session_id, + user_id=user_id, + user_input=input_data.message, + ): + yield frame + return + prelude.mark("agent_build.rest") + + # Tell the client the build is OVER. + # + # Without this the SPA can only infer it from the next + # status, which is `thinking` — and that does not arrive + # until the head-of-turn context work and the event loop's + # startup have also run, well over the 250ms the SPA waits + # before rendering "Getting ready…". So a 1ms cache-hit + # build still showed the label: `preparing` was not a state + # with an end, it was just the latest event. Measured on + # dev, builds of 0ms, 1ms and 40ms all rendered it. + # + # `durationMs` is the build's own measured time, which is + # what makes this frame worth more than a bare marker: it + # says how long the thing the user was told about took. + yield ( + "event: agent_status\ndata: " + + json.dumps( + { + "type": "agent_status", + "sessionId": input_data.session_id, + "phase": "prepared", + "durationMs": prelude.last_stage_ms, + } + ) + + "\n\n" + ) + + # Emitted here rather than before the return: with the build + # deferred, "the window before the client can hear anything" + # ends at the agent, not at the response. + prelude.emit( + session_id=input_data.session_id, + stream_kind="agent", + extra={ + "isResume": is_resume, + "hasAssistant": bool(input_data.rag_assistant_id), + "deferredBuild": deferred_build, + }, + ) + + # Started only once the agent exists — it is the heartbeat's + # first argument. + heartbeat_task = ( + asyncio.create_task(_lease_heartbeat_loop(session_lease, agent)) + if session_lease is not None + else None + ) + async for chunk in stream_with_quota_warning(): yield chunk finally: await _release_turn_lease(heartbeat_task, session_lease) + # Everything after the agent build: citation assembly, the title task, + # the lease acquire and the generator wiring. The `turn_prelude` line + # itself is emitted from inside `_guarded_stream`, once the agent is + # actually ready — with the build deferred, that is the true end of + # the window this measures. The early-return paths above (quota + # exceeded, app tool calls) are not agent turns and never emit. + prelude.mark("stream_setup") + # Stream response from agent as SSE (with optional files) # Note: Compression is handled by GZipMiddleware if configured in main.py return StreamingResponse( diff --git a/backend/src/apis/inference_api/chat/service.py b/backend/src/apis/inference_api/chat/service.py index bcd5b9719..4e70d79a3 100644 --- a/backend/src/apis/inference_api/chat/service.py +++ b/backend/src/apis/inference_api/chat/service.py @@ -8,7 +8,7 @@ import logging import hashlib import os -from typing import Any, Dict, Optional, List, Tuple +from typing import Any, Callable, Dict, Optional, List, Tuple import boto3 @@ -61,10 +61,27 @@ def _create_cache_key( freshness_hash: str, agent_type: Optional[str], skills_hash: str = "", + document_tools: bool = False, + assistant_id: Optional[str] = None, ) -> Tuple: """ Create a cache key for agent instances. + `assistant_id` is the assistant (RAG corpus) the turn ran against. The + spreadsheet-analysis builders close over it, so without it in the key a + cached agent could answer a later turn against the wrong corpus — which + is why that family bypassed the cache until the key carried it. Empty + string when no assistant is attached, so keys for assistant-less turns are + byte-identical to the pre-field ones. + + `document_tools` is whether the turn built the session-state-gated + ``document_read`` tool (the session has a readable attachment). It is not + in `enabled_tools`, so without this element an agent cached before the + first upload would be served — without the tool — to every turn after it. + The gate is monotonic in practice (files stay once uploaded), so the key + flips at most once per session, on the attach turn, when restored history + carries no document yet to lose. + `freshness_hash` is a short digest of the enabled tools' current `updated_at` values (see `freshness.get_freshness_hash`). When an admin edits a tool's config, the hash changes and the cache misses, @@ -95,6 +112,8 @@ def _create_cache_key( provider or "bedrock", freshness_hash, agent_type or "chat", + bool(document_tools), + assistant_id or "", skills_hash, ) @@ -153,7 +172,9 @@ def _adopt_session_conversation(agent: BaseAgent, session_id: str) -> None: ``agent.messages`` lives inside ``TurnBasedSessionManager.initialize()`` (document stripping, content-block sanitizing, compaction slicing, pairing repair) and so runs before we get here; after construction the list is only - appended to. A future compaction that rebinds mid-life would silently break + appended to — or, for the pending-cut apply at the head of a turn, sliced + **in place** by slice assignment (``messages[:] = ...``), which keeps the + alias. A future compaction that rebinds mid-life would silently break the alias — ``test_second_cache_key_for_a_session_shares_the_conversation`` is what catches that. @@ -174,12 +195,14 @@ def _adopt_session_conversation(agent: BaseAgent, session_id: str) -> None: return live = None + live_wrapper = None for key, cached in _agent_cache.items(): if key[0] != session_id: continue cached_inner = getattr(cached, "agent", None) if isinstance(getattr(cached_inner, "messages", None), list): live = cached_inner # newest wins — dict preserves insertion order + live_wrapper = cached if live is None or live.messages is inner.messages: return @@ -203,6 +226,18 @@ def _adopt_session_conversation(agent: BaseAgent, session_id: str) -> None: ) inner.messages = live.messages + # The list's coordinate system travels with it. Compaction expresses its + # checkpoint as ``_live_offset + index into this list`` and the pending-cut + # apply slices the list in place at that offset, so an instance that adopts + # the list must adopt the offset too or it would slice at the wrong place. + try: + src_sm = getattr(live_wrapper, "session_manager", None) + dst_sm = getattr(agent, "session_manager", None) + if src_sm is not None and dst_sm is not None and hasattr(src_sm, "_live_offset"): + dst_sm._live_offset = src_sm._live_offset + except Exception: # noqa: BLE001 - never let bookkeeping break a turn + logger.debug("Session %s: could not sync compaction live offset", scrub_log(session_id), exc_info=True) + async def get_agent( session_id: str, @@ -224,6 +259,9 @@ async def get_agent( accessible_skill_ids: Optional[List[str]] = None, extra_tools_key_described: bool = False, cache_write: bool = True, + has_document_tools: bool = False, + assistant_id: Optional[str] = None, + build_stage_recorder: Optional[Callable[[str], None]] = None, ) -> BaseAgent: """ Get or create agent instance with current configuration for session @@ -298,6 +336,8 @@ async def get_agent( freshness_hash=freshness_hash, agent_type=agent_type, skills_hash=skills_hash, + document_tools=has_document_tools, + assistant_id=assistant_id, ) # Whether this turn's injected tools (if any) let it use the cache at all. @@ -371,7 +411,23 @@ async def get_agent( # kwarg, so keep it off that path. if resolved_agent_type != "voice": create_kwargs["accessible_skill_ids"] = accessible_skill_ids - agent = create_agent(**create_kwargs) + # Decompose the build into sub-stages, same move that opened the preamble + # (docs/specs/turn-latency-preamble.md). A contextvar rather than a kwarg: + # the explicit alternative threads a parameter through a type registry and + # three agent classes that do not share constructor signatures, and a + # mis-set timing mark costs a wrong number, not wrong behaviour. See + # `apis/shared/observability/build_stages.py` for why that asymmetry with + # PR-2's explicit snapshot is deliberate. + from apis.shared.observability.build_stages import ( + reset_stage_recorder, + set_stage_recorder, + ) + + _stage_token = set_stage_recorder(build_stage_recorder) + try: + agent = create_agent(**create_kwargs) + finally: + reset_stage_recorder(_stage_token) # One session is one conversation, even when a turn runs under a different # configuration (an `@`-mention, a different toolset). Runs before the @@ -392,6 +448,9 @@ async def get_agent( agent._construction_snapshot["agent_type"] = resolved_agent_type if resolved_agent_type != "voice" and accessible_skill_ids is not None: agent._construction_snapshot["enabled_skills"] = list(accessible_skill_ids) + # The assistant is a key element (spreadsheet tools close over it), so + # resume must replay it verbatim or the paused agent is orphaned. + agent._construction_snapshot["assistant_id"] = assistant_id # Don't cache agents whose context-bound extra_tools captured anything the # key doesn't describe — a cached agent holds the *old* closures, so reuse diff --git a/backend/src/apis/inference_api/chat/turn_timing.py b/backend/src/apis/inference_api/chat/turn_timing.py new file mode 100644 index 000000000..bd3295df1 --- /dev/null +++ b/backend/src/apis/inference_api/chat/turn_timing.py @@ -0,0 +1,292 @@ +"""Wall-clock marks for the work that happens BEFORE the SSE stream opens. + +WHY THIS EXISTS +--------------- +Measured on dev (docs/specs/agent-state-feedback.md), a warm artifact turn spent +**3.75 seconds between the user's click and the first SSE byte**, against ~3.8s +for the entire streamed response. A cold turn spent 6.7s before the first +``agent_status`` frame. + +That window is invisible from both ends. The client has nothing to show but a +generic "Thinking"; the server cannot narrate it, because FastAPI flushes +response headers when the handler returns its ``StreamingResponse`` and +``invocations`` awaits model resolution, the quota check, RAG retrieval, tool +building and ``get_agent`` *before* that return. There is no channel open. + +Before restructuring anything to open that channel — the three options in the +spec all carry real risk — we need to know which stage owns the time. On a warm +turn the agent cache hits, so ``get_agent`` is probably NOT the bulk of it, and +narrating the wrong stage would be exactly the waste the CLAUDE.md +cost-effectiveness tenet exists to catch. + +WHAT IT COSTS +------------- +Nothing against the model: this never touches the prompt, the conversation, or +the cacheable prefix. At runtime it is a ``perf_counter()`` call per mark and +one log line per turn. + +HOW TO READ IT +-------------- +One structured line per turn, on the inference-api runtime log group +(``-DEFAULT``). The guard on the prod account makes Logs Insights +unusable, so read it with ``filter-log-events``:: + + aws logs filter-log-events --log-group-name \\ + --filter-pattern turn_prelude --profile \\ + --query 'events[].message' --output text > /tmp/prelude.txt + +``stages`` holds the delta in ms for each stage in the order it ran, so the +slowest one is the answer. ``totalMs`` is click-adjacent: it starts when the +handler is entered, so it excludes the app-api hop and any Runtime cold start — +compare it against the client-side gap to size what is left. + +A stage name may carry a dotted prefix (``preamble.quota``) to record a +sub-stage. ``groups`` then sums every stage sharing a prefix, so the coarse +number a stage used to report survives its own decomposition — see +``emit``. + +WHY THE LOG LINE IS NOT ENOUGH +------------------------------ +The line above answers "where did THIS turn's time go". It cannot answer "did +the change make it faster", because a turn's prelude varies by more than any +fix will move it: a cold container against a warm one is 6.7s vs 3.75s, and an +agent-cache miss against a hit is 1478ms vs 0. The dev baseline this +instrumentation was built against is **four turns**, which is noise. + +So ``emit`` also writes one EMF record per turn into +``AgentCoreStack/TurnLatency``, giving CloudWatch percentiles over thousands of +turns instead of a handful of grepped lines. The stage marks stay ungated +(they cost two attribute reads); the metrics ride the +``TURN_LATENCY_METRICS_ENABLED`` kill switch, because unlike a ``perf_counter`` +call they cost log ingestion and custom-metric charges — the same reasoning +that gave ``PROMPT_CACHE_OBSERVABILITY_ENABLED`` its switch. + +**What the metrics cannot tell you.** Everything here starts at handler entry, +so no metric in this namespace sees the app-api hop, auth, or Runtime routing +— ~478ms on a warm path and ~1.5s on a cold one. Only the client can measure +that; ``tests/load`` does. See docs/specs/turn-latency-preamble.md § Validating +a change, and do not read a flat ``PreambleMs`` as proof a fix failed to reach +the user, or a falling one as proof it did. +""" + +from __future__ import annotations + +import json +import logging +import os +import time +from typing import Any, Dict, List, Optional, Tuple + +logger = logging.getLogger(__name__) + +# Its own namespace, not the prompt-cache one: these are milliseconds, they +# want percentiles rather than sums, and a latency regression and a cost +# regression are read by different people at different times. +TURN_LATENCY_EMF_NAMESPACE = os.environ.get( + "TURN_LATENCY_EMF_NAMESPACE", "AgentCoreStack/TurnLatency" +) + +# Default ON; only the literal "false" disables. Empty/unset stays enabled, +# because a workflow env var can materialize as "" (house rule). +TURN_LATENCY_METRICS_ENABLED_ENV = "TURN_LATENCY_METRICS_ENABLED" + + +def turn_latency_metrics_enabled() -> bool: + """Whether the per-turn EMF record is emitted (env kill switch).""" + return os.environ.get(TURN_LATENCY_METRICS_ENABLED_ENV, "").strip().lower() != "false" + + +def _metric_name(stage: str) -> str: + """``preamble.session_state`` -> ``PreambleSessionStateMs``. + + Derived rather than looked up in a table, so a new mark cannot silently + go unmeasured — the failure mode of a hand-maintained list is a stage that + exists in the log and not on the dashboard, which is exactly the gap this + module was written to close. The cost of the automatic direction is that a + new mark quietly creates a new metric stream (~$0.30/month); that is the + cheaper mistake. + """ + parts = [p for p in stage.replace(".", "_").split("_") if p] + return "".join(p[:1].upper() + p[1:] for p in parts) + "Ms" + + +class TurnPrelude: + """Records named marks across the pre-stream stages of one turn. + + Best-effort in every direction, like the status hook it exists to inform: + a mark is two attribute reads and an append, and ``emit`` swallows + everything. A measurement must never be able to break the turn it measures. + """ + + __slots__ = ("_t0", "_wall_t0", "_last", "_marks") + + def __init__(self) -> None: + now = time.perf_counter() + self._t0 = now + self._last = now + # TWO clocks, deliberately. `perf_counter` is monotonic and is what the + # stage deltas must use — a clock step mid-turn would otherwise produce + # a negative stage. But the stream coordinator measures with + # `time.time()`, and a duration is only meaningful within one clock + # domain, so the value handed to it is captured here in ITS domain. + self._wall_t0 = time.time() + self._marks: List[Tuple[str, float]] = [] + + def mark(self, stage: str) -> None: + """Close the stage that just finished and open the next one. + + ``stage`` may be dotted (``preamble.quota``) to record a sub-stage; + ``emit`` sums the prefix back into ``groups``. + """ + try: + now = time.perf_counter() + self._marks.append((stage, (now - self._last) * 1000.0)) + self._last = now + except Exception: # noqa: BLE001 - never break a turn to measure it + logger.debug("Turn prelude mark skipped", exc_info=True) + + @property + def started_at(self) -> float: + """``time.time()`` for the moment the handler was entered. + + Wall clock, not ``perf_counter``, because the only consumer is the + stream coordinator's turn duration and it subtracts this from a + ``time.time()`` reading — mixing the two domains yields a meaningless + number, not a slightly wrong one. + + It exists because the coordinator's own ``stream_start_time`` no longer + marks the start of the turn: the agent build was deferred into the + stream generator (PR-3) and runs before that generator is iterated, so + measuring from it excludes the build. On dev, a turn the user waited + 7.8s for reported 2.1s. + """ + return self._wall_t0 + + @property + def last_stage_ms(self) -> Optional[int]: + """Duration of the stage most recently closed by ``mark``, in ms. + + Lets a caller report the stage it just finished without timing it a + second time in a different clock. + """ + if not self._marks: + return None + return int(self._marks[-1][1]) + + @property + def total_ms(self) -> int: + return max(0, int((time.perf_counter() - self._t0) * 1000)) + + def _groups(self) -> Dict[str, int]: + """Sum the dotted sub-stages back into the stage they decompose. + + A stage that gets split into sub-stages would otherwise take its own + history with it: the four-turn dev baseline in + docs/specs/agent-state-feedback.md is stated in terms of a single + ``preamble`` number, and nothing else in the log reproduces it. Summing + the prefix keeps the coarse series comparable across the split while + ``stages`` answers the new question. + + Undotted stages are deliberately absent — a group of one is noise, and + the reader already has that number in ``stages``. + """ + totals: Dict[str, int] = {} + for name, ms in self._marks: + prefix, dot, _ = name.partition(".") + if not dot: + continue + totals[prefix] = totals.get(prefix, 0) + int(ms) + return totals + + def emit( + self, + *, + session_id: str, + stream_kind: str, + extra: Optional[Dict[str, Any]] = None, + ) -> None: + """Log one structured line describing where the prelude spent its time. + + No user id and no message content: this is a latency record, and the + content-free rule applies to it the same as to the cost rows. + """ + # Read ONCE and shared with the metrics below. Two reads would make + # `totalMs` in the log line and `PreludeTotalMs` on the dashboard + # disagree — by microseconds, which is worse than by a lot: nobody + # chases a big difference for long, and everybody chases a small one. + total_ms = self.total_ms + try: + payload: Dict[str, Any] = { + "sessionId": session_id, + "streamKind": stream_kind, + "totalMs": total_ms, + "stages": {name: int(ms) for name, ms in self._marks}, + } + groups = self._groups() + if groups: + payload["groups"] = groups + if extra: + payload.update(extra) + logger.info("turn_prelude %s", json.dumps(payload, default=str)) + except Exception: # noqa: BLE001 + logger.debug("Turn prelude emit skipped", exc_info=True) + + # Separate try: a metrics failure must not cost the log line, and the + # log line failing must not cost the metrics. They are two independent + # readers of the same turn. + try: + self._emit_metrics( + stream_kind=stream_kind, + session_id=session_id, + total_ms=total_ms, + extra=extra, + ) + except Exception: # noqa: BLE001 + logger.debug("Turn prelude metrics skipped", exc_info=True) + + def _emit_metrics( + self, + *, + stream_kind: str, + session_id: str, + total_ms: int, + extra: Optional[Dict[str, Any]] = None, + ) -> None: + """One EMF record per turn, for percentiles over the fleet. + + Dimension-less, matching every other EMF caller in the repo: the + discriminators a reader actually wants (``isResume``, ``deferredBuild``, + cold vs warm) ride as queryable log properties instead. Making them + dimensions would multiply metric streams and, worse, invite reading a + percentile off a slice too thin to have one. + + ``sessionId`` is a property for the same reason it is one on the + prompt-cache records — unbounded cardinality. No message content: a + latency record is held to the same content-free rule as the cost rows. + """ + if not turn_latency_metrics_enabled(): + return + + from apis.shared.observability.emf import emit_emf_metrics + + metrics: Dict[str, float] = {"PreludeTotalMs": total_ms} + for name, ms in self._marks: + metrics[_metric_name(name)] = int(ms) + # Group totals last: on a decomposed stage they are the only place the + # coarse number exists, and no stage name can collide with one (a group + # key is only ever produced by a dotted stage, which never renders to + # the same metric name as its own prefix). + for prefix, total in self._groups().items(): + metrics.setdefault(_metric_name(prefix), total) + + properties: Dict[str, Any] = {"streamKind": stream_kind, "sessionId": session_id} + for key in ("isResume", "deferredBuild", "hasAssistant"): + if extra and key in extra: + properties[key] = extra[key] + + emit_emf_metrics( + namespace=TURN_LATENCY_EMF_NAMESPACE, + metrics=metrics, + properties=properties, + units={name: "Milliseconds" for name in metrics}, + ) diff --git a/backend/src/apis/inference_api/main.py b/backend/src/apis/inference_api/main.py index aad7824c9..dba6ee162 100644 --- a/backend/src/apis/inference_api/main.py +++ b/backend/src/apis/inference_api/main.py @@ -27,9 +27,7 @@ _startup_logger.warning(".env file not found at %s", env_path) from fastapi import FastAPI -from fastapi.staticfiles import StaticFiles from fastapi.middleware.cors import CORSMiddleware -from fastapi.middleware.gzip import GZipMiddleware from contextlib import asynccontextmanager import logging @@ -70,12 +68,6 @@ async def lifespan(app: FastAPI): if code_interpreter_id: logger.info(f"AgentCore Code Interpreter ID: {code_interpreter_id}") - # Log storage directories - upload_dir = os.getenv('UPLOAD_DIR', 'uploads') - output_dir_name = os.getenv('OUTPUT_DIR', 'output') - generated_images_dir_name = os.getenv('GENERATED_IMAGES_DIR', 'generated_images') - logger.info(f"Storage directories - Upload: {upload_dir}, Output: {output_dir_name}, Images: {generated_images_dir_name}") - # Log API URLs (if configured) frontend_url = os.getenv('FRONTEND_URL') if frontend_url: @@ -86,16 +78,13 @@ async def lifespan(app: FastAPI): if cors_origins: logger.info(f"CORS Origins: {cors_origins}") - # Create output directories if they don't exist - base_dir = Path(__file__).parent.parent - output_dir = os.path.join(base_dir, "output") - uploads_dir = os.path.join(base_dir, "uploads") - generated_images_dir = os.path.join(base_dir, "generated_images") + # Pull the first turn's lazy imports and boto service-model loads forward + # to container start, off the request path. Daemon thread: /ping answers + # immediately and a request that arrives mid-warm-up waits on the import + # lock rather than redoing the work. See apis/inference_api/warmup.py. + from apis.inference_api.warmup import start_warmup_in_background - os.makedirs(output_dir, exist_ok=True) - os.makedirs(uploads_dir, exist_ok=True) - os.makedirs(generated_images_dir, exist_ok=True) - logger.info("Output directories ready") + start_warmup_in_background() yield # Application is running @@ -118,14 +107,31 @@ async def lifespan(app: FastAPI): register_aws_client_error_handler(app) logger.info("Registered AWS ClientError handler") -# Add GZip compression middleware for SSE streams -# Compresses responses over 1KB, reducing bandwidth by 50-70% +# Compress responses over 1KB. Despite what this block used to claim, it is +# emphatically *not* "for SSE streams": Starlette excludes `text/event-stream` +# from compression by content type, so on the one route this service exists to +# serve — the `/invocations` SSE turn — it compresses nothing at all. +# +# What stock `GZipMiddleware` did do on that route is withhold +# `http.response.start` until the first body chunk, because until then it can't +# know whether it will need to set `Content-Encoding`. On an SSE turn the first +# chunk is the model's first token, so the response headers were arriving behind +# the agent's entire thinking time — measured locally at 2.0s of pure delay on a +# turn that stalls 2.0s before its first event, with or without `Accept-Encoding: +# gzip`. app-api's chat proxy reads this response's `content-type` before it can +# open its own stream to the SPA, so that delay was propagating all the way to +# the browser. +# +# `StreamSafeGZipMiddleware` keeps the compression and forwards an excluded +# response's headers immediately; see its module docstring. +from apis.shared.middleware.compression import StreamSafeGZipMiddleware + app.add_middleware( - GZipMiddleware, + StreamSafeGZipMiddleware, minimum_size=1000, # Only compress responses > 1KB compresslevel=6 # Balance between speed and compression ratio (1-9) ) -logger.info("Added GZip middleware for response compression") +logger.info("Added gzip compression middleware (SSE and pre-encoded bodies excluded)") # Bridge AgentCore Runtime headers (WorkloadAccessToken, OAuth2CallbackUrl, # session ID) into BedrockAgentCoreContext so downstream code can look up @@ -163,26 +169,6 @@ async def lifespan(app: FastAPI): # only proxies /invocations and /ping, so user-facing /connectors/* paths can't # be reached through this service. See apis/app_api/connectors/routes.py. -# Mount static file directories for serving generated content -# These are created by tools (visualization, code interpreter, etc.) -# Use parent directory (src/) as base -base_dir = Path(__file__).parent.parent -output_dir = os.path.join(base_dir, "output") -uploads_dir = os.path.join(base_dir, "uploads") -generated_images_dir = os.path.join(base_dir, "generated_images") - -if os.path.exists(output_dir): - app.mount("/output", StaticFiles(directory=output_dir), name="output") - logger.info(f"Mounted static files: /output -> {output_dir}") - -if os.path.exists(uploads_dir): - app.mount("/uploads", StaticFiles(directory=uploads_dir), name="uploads") - logger.info(f"Mounted static files: /uploads -> {uploads_dir}") - -if os.path.exists(generated_images_dir): - app.mount("/generated_images", StaticFiles(directory=generated_images_dir), name="generated_images") - logger.info(f"Mounted static files: /generated_images -> {generated_images_dir}") - if __name__ == "__main__": import uvicorn # Watch the full backend/src tree so edits to shared modules (agents/, diff --git a/backend/src/apis/inference_api/warmup.py b/backend/src/apis/inference_api/warmup.py new file mode 100644 index 000000000..7e530a387 --- /dev/null +++ b/backend/src/apis/inference_api/warmup.py @@ -0,0 +1,107 @@ +"""Startup warm-up for the inference-api container. + +Every new conversation lands on a fresh AgentCore micro-VM. Whatever the first +message has to do before its first token — import the symbolic-math library +behind the calculator tool, load botocore's service models for the clients the +turn will build — is paid on that user's first-word latency. Splitting cold +first-turns into phases on a 40k-user load run put that residual at 3.6 s +(docs/specs/load-test-assessment-2026-09.md §1 fix 2). Precompiled bytecode +(the Dockerfile) removes the compile; this module moves the rest to container +start, where it overlaps the Runtime's own readiness wait instead of the +user's request. + +Runs on a daemon thread so ``/ping`` answers immediately. Python's import lock +makes a first request that arrives mid-warm-up wait for the in-progress import +rather than redo it, so there is no double work. Best-effort throughout: a +failure here is logged and the first turn simply pays what it always paid. + +Gated by ``INFERENCE_WARMUP_ENABLED`` (default on; ``=false`` is the kill +switch, per the flags convention in CLAUDE.md). +""" + +import importlib +import logging +import os +import threading +import time +from typing import Callable, Iterable + +logger = logging.getLogger(__name__) + +WARMUP_ENABLED_ENV = "INFERENCE_WARMUP_ENABLED" + +# Modules the first turn imports lazily. Ordered heaviest-first so a request +# that arrives mid-warm-up finds the expensive ones already in progress. +WARM_MODULES: tuple[str, ...] = ( + # sympy, via the calculator tool (default-on in the registry) — the single + # largest lazy import on the first-turn path. + "strands_tools.calculator", + # Agent construction path: model factory, session manager, hooks. + "agents.main_agent.core.agent_factory", + "agents.main_agent.base_agent", + "agents.main_agent.session.session_factory", + "apis.inference_api.chat.service", +) + +# boto3 clients the turn builds. Creating one loads and parses the service's +# JSON model into the default session's loader cache, which every later client +# for the same service reuses — that parse, not the socket, is the cost. +WARM_BOTO_SERVICES: tuple[str, ...] = ( + "bedrock-runtime", + "bedrock-agentcore", + "dynamodb", + "s3", +) + + +def warmup_enabled() -> bool: + """Whether startup warm-up runs. Empty/unset means on.""" + return os.environ.get(WARMUP_ENABLED_ENV, "").strip().lower() != "false" + + +def _timed(label: str, fn: Callable[[], None]) -> None: + started = time.perf_counter() + try: + fn() + except Exception as e: # noqa: BLE001 - warm-up must never fail startup + logger.info("warmup step=%s outcome=skipped error=%s", label, e) + return + logger.info("warmup step=%s outcome=ok ms=%d", label, int((time.perf_counter() - started) * 1000)) + + +def warm_modules(modules: Iterable[str] = WARM_MODULES) -> None: + """Import each module once; a missing optional dependency is logged, not raised.""" + for name in modules: + _timed(f"import:{name}", lambda name=name: importlib.import_module(name)) + + +def warm_boto_clients(services: Iterable[str] = WARM_BOTO_SERVICES) -> None: + """Build one client per service so the service model is parsed and cached.""" + region = os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") + if not region: + logger.info("warmup step=boto outcome=skipped error=no region configured") + return + import boto3 + + for service in services: + _timed(f"boto:{service}", lambda service=service: boto3.client(service, region_name=region)) + + +def run_warmup() -> None: + """The whole warm-up, synchronously. Exposed for tests and for callers that + want it inline.""" + started = time.perf_counter() + warm_modules() + warm_boto_clients() + logger.info("warmup complete ms=%d", int((time.perf_counter() - started) * 1000)) + + +def start_warmup_in_background() -> threading.Thread | None: + """Kick off :func:`run_warmup` on a daemon thread. Returns the thread, or + ``None`` when the kill switch is set.""" + if not warmup_enabled(): + logger.info("warmup disabled via %s", WARMUP_ENABLED_ENV) + return None + thread = threading.Thread(target=run_warmup, name="inference-warmup", daemon=True) + thread.start() + return thread diff --git a/backend/src/apis/shared/agent_templates/__init__.py b/backend/src/apis/shared/agent_templates/__init__.py new file mode 100644 index 000000000..3b321bdd6 --- /dev/null +++ b/backend/src/apis/shared/agent_templates/__init__.py @@ -0,0 +1,11 @@ +"""Admin-managed catalog of agent templates. + +A template is a curated, Save-able starting point for a new agent — a subset of +the ``Agent`` shape (name, emoji, description, instructions, starters, model and +bindings) plus catalog-management metadata (enabled, sort order, pitch). Users +pick one on the create-agent page and the form opens pre-filled. + +Mirrors the ``system_prompts`` admin-catalog layout: models / repository / +service here, admin CRUD under ``app_api/admin/agent_templates``, and a public +read endpoint under ``app_api/agent_templates`` that the picker consumes. +""" diff --git a/backend/src/apis/shared/agent_templates/models.py b/backend/src/apis/shared/agent_templates/models.py new file mode 100644 index 000000000..50da68b27 --- /dev/null +++ b/backend/src/apis/shared/agent_templates/models.py @@ -0,0 +1,366 @@ +"""Models for admin-managed agent templates. + +Admins curate a small catalog of templates; any signed-in user reads the +enabled ones through the public ``GET /templates`` endpoint and starts a new +agent from one. The catalog is data, not code — the hardcoded client +``agent-templates.ts`` set is the seed source, not the source of truth. + +Storage uses a single table with PK ``TEMPLATE#``, SK +``METADATA``. No GSI: the catalog is small (tens of items), so a full Scan is +appropriate — this matches the ``system_prompts`` catalog it is modelled on. + +Two wire shapes are deliberately different: + +* **Admin** responses are snake_case and carry the whole record (including + ``enabled``, ``sort_order`` and audit fields), matching the ``system_prompts`` + admin convention. +* **Public** responses are camelCase and project only the ``TemplateDraft`` + subset the client already consumes (``{ draft, pitch }[]``), so the existing + picker / reconcile / form-population path is unchanged when it switches its + data source from the hardcoded array to this endpoint. +""" + +from dataclasses import dataclass, field +from typing import Any, Dict, List, Literal, Optional + +from pydantic import BaseModel, ConfigDict, Field +from pydantic.alias_generators import to_camel + +from apis.shared.timestamps import utc_now_iso + +# Bounds mirror the create-agent form's own field limits closely enough to keep +# a template Save-able without surprises. +MAX_NAME_LENGTH = 128 +MAX_DESCRIPTION_LENGTH = 1_024 +MAX_INSTRUCTIONS_LENGTH = 100_000 +MAX_PITCH_LENGTH = 200 +MAX_STARTERS = 12 +MAX_TAGS = 16 +MAX_BINDINGS = 64 + +TemplateStatus = Literal["enabled", "disabled"] + +# BindingKind mirrors the frontend union +# ('knowledge_base' | 'tool' | 'skill' | 'memory_space'). Stored as a free +# string so a fork can add binding kinds without a backend change; unknown +# kinds pass through and the client reconciles/ignores what it cannot bind. + + +def _utc_now() -> str: + return utc_now_iso() + + +@dataclass +class TemplateModelConfig: + """A model selection on a template. ``model_id=None`` ⇒ platform default.""" + + model_id: Optional[str] = None + params: Dict[str, Any] = field(default_factory=dict) + + +@dataclass +class TemplateBinding: + """One binding on a template — same shape as an ``AgentBinding``.""" + + kind: str + ref: str + config: Dict[str, Any] = field(default_factory=dict) + + +@dataclass +class AgentTemplate: + """Admin-managed agent template stored in DynamoDB.""" + + template_id: str + name: str + description: str + emoji: str + instructions: str + tags: List[str] + starters: List[str] + model_config_: TemplateModelConfig + bindings: List[TemplateBinding] + # ── catalog management (never part of the public client draft) ── + pitch: str + status: TemplateStatus + sort_order: int + created_at: str + updated_at: str + created_by: Optional[str] = None + updated_by: Optional[str] = None + + def to_dynamo_item(self) -> Dict[str, Any]: + item: Dict[str, Any] = { + "PK": f"TEMPLATE#{self.template_id}", + "SK": "METADATA", + "templateId": self.template_id, + "name": self.name, + "description": self.description, + "emoji": self.emoji, + "instructions": self.instructions, + "tags": list(self.tags), + "starters": list(self.starters), + "modelConfig": { + "modelId": self.model_config_.model_id, + "params": self.model_config_.params or {}, + }, + "bindings": [ + {"kind": b.kind, "ref": b.ref, "config": b.config or {}} + for b in self.bindings + ], + "pitch": self.pitch, + "status": self.status, + "sortOrder": self.sort_order, + "createdAt": self.created_at, + "updatedAt": self.updated_at, + } + if self.created_by: + item["createdBy"] = self.created_by + if self.updated_by: + item["updatedBy"] = self.updated_by + return item + + @classmethod + def from_dynamo_item(cls, item: Dict[str, Any]) -> "AgentTemplate": + try: + created_at = item["createdAt"] + updated_at = item["updatedAt"] + except KeyError as e: + raise ValueError( + f"Agent template item {item.get('PK', '?')} is missing required " + f"timestamp field: {e.args[0]}" + ) from e + + status = item.get("status", "enabled") + if status not in ("enabled", "disabled"): + # Defensive: an unknown status is hidden from users, never shown. + status = "disabled" + + mc = item.get("modelConfig") or {} + model_config = TemplateModelConfig( + model_id=mc.get("modelId"), + params=dict(mc.get("params") or {}), + ) + bindings = [ + TemplateBinding( + kind=b.get("kind", ""), + ref=b.get("ref", ""), + config=dict(b.get("config") or {}), + ) + for b in (item.get("bindings") or []) + ] + + # sortOrder may arrive as a Decimal from DynamoDB. + raw_sort = item.get("sortOrder", 0) + try: + sort_order = int(raw_sort) + except (TypeError, ValueError): + sort_order = 0 + + return cls( + template_id=item["templateId"], + name=item["name"], + description=item.get("description", ""), + emoji=item.get("emoji", ""), + instructions=item.get("instructions", ""), + tags=list(item.get("tags") or []), + starters=list(item.get("starters") or []), + model_config_=model_config, + bindings=bindings, + pitch=item.get("pitch", ""), + status=status, + sort_order=sort_order, + created_at=created_at, + updated_at=updated_at, + created_by=item.get("createdBy"), + updated_by=item.get("updatedBy"), + ) + + +# ============================================================================= +# Shared request sub-models +# ============================================================================= + + +class ModelConfigPayload(BaseModel): + """Wire model config. ``model_id=None`` ⇒ platform default. + + Serializes camelCase (``modelId``) to match the client ``TemplateDraft``. + ``protected_namespaces=()`` silences Pydantic's ``model_``-prefix warning. + """ + + model_config = ConfigDict( + populate_by_name=True, + alias_generator=to_camel, + protected_namespaces=(), + ) + + model_id: Optional[str] = None + params: Dict[str, Any] = Field(default_factory=dict) + + +class BindingPayload(BaseModel): + model_config = ConfigDict(populate_by_name=True) + + kind: str = Field(..., min_length=1, max_length=64) + ref: str = Field(..., min_length=1, max_length=256) + config: Dict[str, Any] = Field(default_factory=dict) + + +# ============================================================================= +# Admin request/response models (snake_case wire) +# ============================================================================= + + +class AgentTemplateCreate(BaseModel): + """Create payload. ``template_id`` optional — slugified from name if absent.""" + + template_id: Optional[str] = Field(None, min_length=1, max_length=128) + name: str = Field(..., min_length=1, max_length=MAX_NAME_LENGTH) + description: str = Field("", max_length=MAX_DESCRIPTION_LENGTH) + emoji: str = Field("", max_length=16) + instructions: str = Field("", max_length=MAX_INSTRUCTIONS_LENGTH) + tags: List[str] = Field(default_factory=list, max_length=MAX_TAGS) + starters: List[str] = Field(default_factory=list, max_length=MAX_STARTERS) + model_cfg: ModelConfigPayload = Field( + default_factory=ModelConfigPayload, alias="modelConfig" + ) + bindings: List[BindingPayload] = Field(default_factory=list, max_length=MAX_BINDINGS) + pitch: str = Field("", max_length=MAX_PITCH_LENGTH) + status: TemplateStatus = "enabled" + sort_order: int = 0 + + # Accept both the alias (``modelConfig``) and the field name on input; + # ``protected_namespaces=()`` avoids Pydantic's ``model_``-prefix warning. + model_config = ConfigDict(populate_by_name=True, protected_namespaces=()) + + +class AgentTemplateUpdate(BaseModel): + """Partial update — every field optional.""" + + name: Optional[str] = Field(None, min_length=1, max_length=MAX_NAME_LENGTH) + description: Optional[str] = Field(None, max_length=MAX_DESCRIPTION_LENGTH) + emoji: Optional[str] = Field(None, max_length=16) + instructions: Optional[str] = Field(None, max_length=MAX_INSTRUCTIONS_LENGTH) + tags: Optional[List[str]] = Field(None, max_length=MAX_TAGS) + starters: Optional[List[str]] = Field(None, max_length=MAX_STARTERS) + model_cfg: Optional[ModelConfigPayload] = Field(None, alias="modelConfig") + bindings: Optional[List[BindingPayload]] = Field(None, max_length=MAX_BINDINGS) + pitch: Optional[str] = Field(None, max_length=MAX_PITCH_LENGTH) + status: Optional[TemplateStatus] = None + sort_order: Optional[int] = None + + model_config = ConfigDict(populate_by_name=True, protected_namespaces=()) + + +class AgentTemplateAdminResponse(BaseModel): + """Full admin response (snake_case).""" + + template_id: str + name: str + description: str + emoji: str + instructions: str + tags: List[str] + starters: List[str] + model_cfg: ModelConfigPayload = Field(serialization_alias="modelConfig") + bindings: List[BindingPayload] + pitch: str + status: TemplateStatus + sort_order: int + created_at: str + updated_at: str + created_by: Optional[str] = None + updated_by: Optional[str] = None + + model_config = ConfigDict(protected_namespaces=()) + + @classmethod + def from_template(cls, t: AgentTemplate) -> "AgentTemplateAdminResponse": + return cls( + template_id=t.template_id, + name=t.name, + description=t.description, + emoji=t.emoji, + instructions=t.instructions, + tags=t.tags, + starters=t.starters, + model_cfg=ModelConfigPayload( + model_id=t.model_config_.model_id, params=t.model_config_.params + ), + bindings=[ + BindingPayload(kind=b.kind, ref=b.ref, config=b.config) + for b in t.bindings + ], + pitch=t.pitch, + status=t.status, + sort_order=t.sort_order, + created_at=t.created_at, + updated_at=t.updated_at, + created_by=t.created_by, + updated_by=t.updated_by, + ) + + +class AgentTemplateAdminListResponse(BaseModel): + templates: List[AgentTemplateAdminResponse] + total: int + + +# ============================================================================= +# Public response models (camelCase — matches the client TemplateDraft) +# ============================================================================= + + +class TemplateDraftResponse(BaseModel): + """The client ``TemplateDraft`` shape, camelCase, consumed by the picker. + + Only the subset the create-agent form reads — no catalog/audit fields. + """ + + model_config = ConfigDict(populate_by_name=True, alias_generator=to_camel) + + template_id: str + name: str + description: str + emoji: str + instructions: str + tags: List[str] + starters: List[str] + # Serialized as "modelConfig" via the camel alias generator. + model_cfg: ModelConfigPayload = Field(serialization_alias="modelConfig") + bindings: List[BindingPayload] + + +class TemplateCatalogEntryResponse(BaseModel): + """One picker entry: ``{ draft, pitch }`` — matches ``TemplateCatalogEntry``.""" + + draft: TemplateDraftResponse + pitch: str + + @classmethod + def from_template(cls, t: AgentTemplate) -> "TemplateCatalogEntryResponse": + return cls( + draft=TemplateDraftResponse( + template_id=t.template_id, + name=t.name, + description=t.description, + emoji=t.emoji, + instructions=t.instructions, + tags=t.tags, + starters=t.starters, + model_cfg=ModelConfigPayload( + model_id=t.model_config_.model_id, params=t.model_config_.params + ), + bindings=[ + BindingPayload(kind=b.kind, ref=b.ref, config=b.config) + for b in t.bindings + ], + ), + pitch=t.pitch, + ) + + +class PublicTemplateListResponse(BaseModel): + templates: List[TemplateCatalogEntryResponse] + total: int diff --git a/backend/src/apis/shared/agent_templates/repository.py b/backend/src/apis/shared/agent_templates/repository.py new file mode 100644 index 000000000..9d68bac8a --- /dev/null +++ b/backend/src/apis/shared/agent_templates/repository.py @@ -0,0 +1,265 @@ +"""DynamoDB repository for admin-managed agent templates. + +PK ``TEMPLATE#``, SK ``METADATA``. Listing is a full Scan — the +catalog is small (tens of items at most) so a GSI is unnecessary, matching the +``system_prompts`` repository this is modelled on. +""" + +import asyncio +import logging +import os +import re +import uuid +from typing import List, Optional + +import boto3 +from botocore.exceptions import ClientError + +from apis.shared.caching import config_cache +from apis.shared.timestamps import utc_now_iso + +from .models import ( + AgentTemplate, + AgentTemplateCreate, + AgentTemplateUpdate, + TemplateBinding, + TemplateModelConfig, +) + +logger = logging.getLogger(__name__) + +_SLUG_RE = re.compile(r"[^a-z0-9]+") + + +def slugify(value: str) -> str: + """Derive a URL/DynamoDB-safe template id from a name. + + Lowercased, non-alphanumeric runs collapsed to a single hyphen, trimmed. + Falls back to a uuid when the name has no usable characters. + """ + slug = _SLUG_RE.sub("-", value.strip().lower()).strip("-") + return slug or f"template-{uuid.uuid4().hex[:8]}" + + +class AgentTemplatesRepository: + """CRUD for agent templates in DynamoDB. + + Full Scan is used for listing — the catalog is expected to be small, so + Scan is appropriate and avoids a GSI. + """ + + def __init__(self, table_name: Optional[str] = None, region: Optional[str] = None): + self._table_name = table_name or os.getenv("DYNAMODB_AGENT_TEMPLATES_TABLE_NAME") + self._region = region or os.getenv("AWS_REGION", "us-west-2") + self._enabled = bool(self._table_name) + + if not self._enabled: + logger.warning( + "DYNAMODB_AGENT_TEMPLATES_TABLE_NAME not set. " + "Agent templates repository is disabled." + ) + return + + profile = os.getenv("AWS_PROFILE") + if profile: + session = boto3.Session(profile_name=profile) + self._dynamodb = session.resource("dynamodb", region_name=self._region) + else: + self._dynamodb = boto3.resource("dynamodb", region_name=self._region) + self._table = self._dynamodb.Table(self._table_name) + logger.info(f"Initialized agent templates repository: table={self._table_name}") + + @property + def enabled(self) -> bool: + return self._enabled + + async def list_templates(self, enabled_only: bool = False) -> List[AgentTemplate]: + """Return all templates, sorted by (sort_order, name). + + When ``enabled_only`` is set, disabled templates are filtered out — this + is what the public picker endpoint uses. + """ + if not self._enabled: + return [] + + try: + items = await config_cache.get_or_load( + config_cache.AGENT_TEMPLATES, + lambda: asyncio.to_thread(self._scan_template_items), + ) + except ClientError: + logger.error("Error listing agent templates", exc_info=True) + raise + + templates = [AgentTemplate.from_dynamo_item(item) for item in items] + if enabled_only: + templates = [t for t in templates if t.status == "enabled"] + templates.sort(key=lambda t: (t.sort_order, t.name.lower())) + return templates + + def _scan_template_items(self) -> List[dict]: + """Scan the raw TEMPLATE# items. Blocking; call via ``asyncio.to_thread``.""" + response = self._table.scan( + FilterExpression="SK = :sk", + ExpressionAttributeValues={":sk": "METADATA"}, + ) + items = response.get("Items", []) + while "LastEvaluatedKey" in response: + response = self._table.scan( + FilterExpression="SK = :sk", + ExpressionAttributeValues={":sk": "METADATA"}, + ExclusiveStartKey=response["LastEvaluatedKey"], + ) + items.extend(response.get("Items", [])) + return items + + async def get_template(self, template_id: str) -> Optional[AgentTemplate]: + """Return a single template by id, or None if not found.""" + if not self._enabled: + return None + try: + response = self._table.get_item( + Key={"PK": f"TEMPLATE#{template_id}", "SK": "METADATA"} + ) + item = response.get("Item") + if not item: + return None + return AgentTemplate.from_dynamo_item(item) + except ClientError: + logger.error("Error getting agent template", exc_info=True) + raise + + async def create_template( + self, data: AgentTemplateCreate, created_by: Optional[str] = None + ) -> AgentTemplate: + """Create a template and return it. + + Raises ``ValueError`` if the resolved id already exists (surfaced as a + 409 by the route). + """ + if not self._enabled: + raise RuntimeError("Agent templates repository is not enabled") + + template_id = (data.template_id or slugify(data.name)).strip() + now = utc_now_iso() + template = AgentTemplate( + template_id=template_id, + name=data.name, + description=data.description, + emoji=data.emoji, + instructions=data.instructions, + tags=list(data.tags), + starters=list(data.starters), + model_config_=TemplateModelConfig( + model_id=data.model_cfg.model_id, + params=dict(data.model_cfg.params or {}), + ), + bindings=[ + TemplateBinding(kind=b.kind, ref=b.ref, config=dict(b.config or {})) + for b in data.bindings + ], + pitch=data.pitch, + status=data.status, + sort_order=data.sort_order, + created_at=now, + updated_at=now, + created_by=created_by, + updated_by=created_by, + ) + + try: + self._table.put_item( + Item=template.to_dynamo_item(), + ConditionExpression="attribute_not_exists(PK)", + ) + except ClientError as e: + if e.response.get("Error", {}).get("Code") == "ConditionalCheckFailedException": + raise ValueError(f"Template '{template_id}' already exists") + logger.error("Error creating agent template", exc_info=True) + raise + + config_cache.invalidate(config_cache.AGENT_TEMPLATES) + logger.info(f"Created agent template: {template_id} name={template.name!r}") + return template + + async def update_template( + self, template_id: str, updates: AgentTemplateUpdate + ) -> Optional[AgentTemplate]: + """Apply a partial update. Returns None if not found. + + A conditional put guards against TOCTOU resurrection — if another admin + deletes the row between our read and write, the put fails rather than + recreating it. + """ + if not self._enabled: + return None + + existing = await self.get_template(template_id) + if not existing: + return None + + data = updates.model_dump(exclude_none=True, by_alias=False) + # model config is nested — apply it onto the dataclass explicitly. + mc = data.pop("model_cfg", None) + if mc is not None: + existing.model_config_ = TemplateModelConfig( + model_id=mc.get("model_id"), + params=dict(mc.get("params") or {}), + ) + bindings = data.pop("bindings", None) + if bindings is not None: + existing.bindings = [ + TemplateBinding( + kind=b.get("kind", ""), + ref=b.get("ref", ""), + config=dict(b.get("config") or {}), + ) + for b in bindings + ] + for field_name, value in data.items(): + setattr(existing, field_name, value) + existing.updated_at = utc_now_iso() + + try: + self._table.put_item( + Item=existing.to_dynamo_item(), + ConditionExpression="attribute_exists(PK)", + ) + except ClientError as e: + if e.response.get("Error", {}).get("Code") == "ConditionalCheckFailedException": + logger.warning(f"Agent template {template_id} disappeared during update") + return None + logger.error("Error updating agent template", exc_info=True) + raise + + config_cache.invalidate(config_cache.AGENT_TEMPLATES) + logger.info(f"Updated agent template: {template_id}") + return existing + + async def delete_template(self, template_id: str) -> bool: + """Delete a template. Returns True if deleted, False if not found.""" + if not self._enabled: + return False + existing = await self.get_template(template_id) + if not existing: + return False + try: + self._table.delete_item( + Key={"PK": f"TEMPLATE#{template_id}", "SK": "METADATA"} + ) + except ClientError: + logger.error("Error deleting agent template", exc_info=True) + raise + config_cache.invalidate(config_cache.AGENT_TEMPLATES) + logger.info(f"Deleted agent template: {template_id}") + return True + + +_repository: Optional[AgentTemplatesRepository] = None + + +def get_agent_templates_repository() -> AgentTemplatesRepository: + global _repository + if _repository is None: + _repository = AgentTemplatesRepository() + return _repository diff --git a/backend/src/apis/shared/agent_templates/seed.py b/backend/src/apis/shared/agent_templates/seed.py new file mode 100644 index 000000000..261be054c --- /dev/null +++ b/backend/src/apis/shared/agent_templates/seed.py @@ -0,0 +1,177 @@ +"""Neutral default agent templates (idempotent seed). + +These are org-agnostic **examples** any fork keeps, edits, or deletes — NOT +fixtures the product depends on. They carry: + +* no organization-specific tool refs (``bindings`` is empty), so a fork with a + different tool catalog shows no dropped-tool notices out of the box; +* ``model_id = None`` (platform default), so a fork's own default model is used; +* domain-neutral guardrail *shapes* in ``instructions`` — patterns to adapt, not + policy to obey. + +The hardcoded client ``agent-templates.ts`` set (Course Helper, Q&A, Roleplay, +Demo) is intentionally NOT copied verbatim: Course Helper is higher-ed-specific, +and the Demo entry is a dev/QA fixture, never seeded. + +Seeding is idempotent: an id that already exists is skipped, so re-running never +clobbers an admin's edits. +""" + +import logging +from typing import List + +from .models import ( + AgentTemplateCreate, + ModelConfigPayload, +) +from .service import AgentTemplatesService, get_agent_templates_service + +logger = logging.getLogger(__name__) + + +def default_seed_templates() -> List[AgentTemplateCreate]: + """The neutral default set, in display order.""" + return [ + AgentTemplateCreate( + template_id="document-qa", + name="Document Q&A Assistant", + description=( + "Answers questions from a body of material you upload — a handbook, " + "policy set, product docs or wiki. It answers strictly from those " + "documents, points back to where each answer came from, and says when " + "something isn't covered instead of guessing." + ), + emoji="📚", + instructions=( + "You are a Q&A assistant for a specific set of documents. Answer " + "questions accurately from the material in your knowledge base and " + "nothing else.\n\n" + "## What you do\n" + "- Answer using only the documents provided to you.\n" + "- Point the reader to where the answer comes from (document name, " + "section or heading) so they can verify it.\n" + "- Summarize, compare and explain what the material says in plain " + "language.\n\n" + "## Ground every answer in the source material\n" + "- If the documents do not contain the answer, say so plainly — " + "\"I don't know based on the material I have\" — rather than guessing.\n" + "- Do not speculate or invent facts, figures, dates, names, contact " + "details or links.\n" + "- If two documents disagree, surface the conflict rather than " + "silently picking one.\n\n" + "## Tone\n" + "Be clear, direct and neutral. A good answer is one the reader can " + "trust and check." + ), + tags=[], + starters=[ + "What does the material say about this?", + "Summarize the key points on this topic.", + "Where in the documents is this covered?", + "Is this addressed anywhere in the material?", + ], + model_cfg=ModelConfigPayload(model_id=None, params={}), + bindings=[], + pitch="Answers strictly from the documents you give it, and admits what it can't find.", + status="enabled", + sort_order=10, + ), + AgentTemplateCreate( + template_id="roleplay-partner", + name="Roleplay Practice Partner", + description=( + "A practice partner that plays a role so you can rehearse a real " + "conversation — a language exchange, an interview, a customer call or " + "a difficult discussion. It stays in character to make practice feel " + "real, then steps out to give feedback when you ask." + ), + emoji="🎭", + instructions=( + "You are a roleplay practice partner. You take on a role the user " + "chooses so they can practice a real-world conversation in a safe, " + "low-stakes way.\n\n" + "## How you play the role\n" + "- Ask briefly what scenario, role and difficulty the user wants if " + "they have not said, then stay in character and make it feel real.\n" + "- Keep your turns natural and appropriately short — this is the " + "user's practice, not a monologue.\n" + "- Adapt to the user's level and gently stretch it.\n\n" + "## Break character when it matters\n" + "- Step out the moment the user asks for help, feedback, a hint or a " + "translation; a short \"(stepping out of character)\" is enough, then " + "offer to resume.\n" + "- Break character immediately if the user seems genuinely distressed " + "or needs real help rather than practice, and point them to an " + "appropriate real resource.\n\n" + "## Stay safe and respectful\n" + "- Keep everything respectful and non-harmful whatever role you play. " + "Play challenging characters as challenging, never as cruel.\n" + "- Do not present invented specifics as real facts.\n\n" + "## After a scene\n" + "Give brief, kind, concrete feedback: what worked, one or two things " + "to try next time, and an offer to run it again." + ), + tags=[], + starters=[ + "Let's practice a job interview for a role I'm applying to.", + "Be my conversation partner in a language I'm learning, at a beginner level.", + "Roleplay a customer with a complaint so I can practice responding.", + "Help me rehearse a difficult conversation with a coworker.", + ], + model_cfg=ModelConfigPayload(model_id=None, params={}), + bindings=[], + pitch="Plays a role so you can rehearse interviews, languages and tough conversations.", + status="enabled", + sort_order=20, + ), + AgentTemplateCreate( + template_id="general-assistant", + name="General Assistant", + description=( + "A plain, helpful assistant with no special configuration — a blank " + "starting point you can shape into whatever you need." + ), + emoji="💬", + instructions=( + "You are a helpful, honest assistant. Answer the user's questions " + "clearly and concisely, ask a clarifying question when the request is " + "ambiguous, and say plainly when you are not sure or do not know rather " + "than guessing. Be respectful and constructive." + ), + tags=[], + starters=[ + "Help me draft something.", + "Explain this topic simply.", + "Brainstorm some ideas with me.", + ], + model_cfg=ModelConfigPayload(model_id=None, params={}), + bindings=[], + pitch="A blank, helpful assistant you can shape into anything.", + status="enabled", + sort_order=30, + ), + ] + + +async def seed_default_templates( + service: AgentTemplatesService | None = None, +) -> int: + """Idempotently write the neutral default templates. + + Returns the number of templates actually created (0 if all already exist). + Safe to run repeatedly — existing ids are skipped, never overwritten. + """ + svc = service or get_agent_templates_service() + created = 0 + for payload in default_seed_templates(): + existing = await svc.get_template(payload.template_id) + if existing is not None: + continue + try: + await svc.create_template(payload, created_by="seed") + created += 1 + except ValueError: + # Lost a race with a concurrent seed/create — already exists. + continue + logger.info("Seeded %d default agent templates", created) + return created diff --git a/backend/src/apis/shared/agent_templates/service.py b/backend/src/apis/shared/agent_templates/service.py new file mode 100644 index 000000000..fa9c5347f --- /dev/null +++ b/backend/src/apis/shared/agent_templates/service.py @@ -0,0 +1,45 @@ +"""Service layer for admin-managed agent templates. + +Routes call this rather than the repository directly so business rules (public +enabled-only listing, seeding) have a single home. Mirrors +``system_prompts.service``. +""" + +from typing import List, Optional + +from .models import AgentTemplate, AgentTemplateCreate, AgentTemplateUpdate +from .repository import AgentTemplatesRepository, get_agent_templates_repository + + +class AgentTemplatesService: + def __init__(self, repository: AgentTemplatesRepository): + self._repo = repository + + async def list_templates(self, enabled_only: bool = False) -> List[AgentTemplate]: + return await self._repo.list_templates(enabled_only=enabled_only) + + async def get_template(self, template_id: str) -> Optional[AgentTemplate]: + return await self._repo.get_template(template_id) + + async def create_template( + self, data: AgentTemplateCreate, created_by: Optional[str] = None + ) -> AgentTemplate: + return await self._repo.create_template(data, created_by=created_by) + + async def update_template( + self, template_id: str, updates: AgentTemplateUpdate + ) -> Optional[AgentTemplate]: + return await self._repo.update_template(template_id, updates) + + async def delete_template(self, template_id: str) -> bool: + return await self._repo.delete_template(template_id) + + +_service: Optional[AgentTemplatesService] = None + + +def get_agent_templates_service() -> AgentTemplatesService: + global _service + if _service is None: + _service = AgentTemplatesService(get_agent_templates_repository()) + return _service diff --git a/backend/src/apis/shared/aws_clients.py b/backend/src/apis/shared/aws_clients.py new file mode 100644 index 000000000..e77bd2870 --- /dev/null +++ b/backend/src/apis/shared/aws_clients.py @@ -0,0 +1,124 @@ +"""Process-cached boto3 clients and resources. + +WHY THIS EXISTS +--------------- +`sessions/metadata.py` alone constructs `boto3.resource("dynamodb")` in 28 +separate function bodies, and the repo has 89 such sites across `apis/` and +`agents/`. Each construction resolves an endpoint, builds a credential +resolver and registers event hooks. + +Measured: **~327ms for the first construction in a process, then ~1.5ms each.** +That sounds negligible until a single request makes a dozen of them. After +`docs/specs/turn-latency-preamble.md` PR-2 collapsed the preamble's eight +DynamoDB reads into one, `preamble.session_state` still measured 17-21ms on +dev while doing *no IO at all* — six helpers each building a client before +reaching their snapshot short-circuit. That residual is what this module +removes. + +A boto3 resource is safe to share: botocore clients are thread-safe for API +calls, and credential refresh is handled inside the shared session. What is +NOT safe is sharing one across a *moto* boundary — see below. + +THE MOTO TRAP, AND WHY `reset_cached_clients` EXISTS +---------------------------------------------------- +`moto.mock_aws()` is entered per test (`tests/*/conftest.py`). A client built +inside test A's mock context keeps pointing at test A's backend, which is torn +down when that test ends. Cache it, and test B silently talks to a dead +backend — or worse, to a live AWS endpoint. + +The failure would be **order-dependent and confusing**, which is exactly the +shape of bug this repo has already paid for elsewhere (a static memo leaking +across SPA spec files). So the cache is explicitly resettable, and the `aws` +fixture resets it on both entry and exit. A test that constructs its own +client directly is unaffected either way. +""" + +from __future__ import annotations + +import logging +import threading +from typing import Any, Dict, Optional, Tuple + +logger = logging.getLogger(__name__) + +# Keyed by (service_name, region). `region=None` means "whatever the ambient +# configuration resolves to", which is what every existing call site relies on. +_resources: Dict[Tuple[str, Optional[str]], Any] = {} +_clients: Dict[Tuple[str, Optional[str]], Any] = {} + +# Construction is not atomic and two threads can race to build the same entry. +# The loser's client is simply discarded — harmless, but the lock keeps the +# dicts consistent and makes the "built once" claim in the docstring true. +_lock = threading.Lock() + + +def get_resource(service_name: str, region_name: Optional[str] = None) -> Any: + """A process-cached ``boto3.resource``. + + Falls back to an uncached resource if construction fails inside the lock, + so a transient credential problem cannot poison the cache for the life of + the process. + """ + key = (service_name, region_name) + cached = _resources.get(key) + if cached is not None: + return cached + + import boto3 + + with _lock: + cached = _resources.get(key) + if cached is not None: + return cached + resource = ( + boto3.resource(service_name, region_name=region_name) + if region_name + else boto3.resource(service_name) + ) + _resources[key] = resource + return resource + + +def get_client(service_name: str, region_name: Optional[str] = None) -> Any: + """A process-cached ``boto3.client``.""" + key = (service_name, region_name) + cached = _clients.get(key) + if cached is not None: + return cached + + import boto3 + + with _lock: + cached = _clients.get(key) + if cached is not None: + return cached + client = ( + boto3.client(service_name, region_name=region_name) + if region_name + else boto3.client(service_name) + ) + _clients[key] = client + return client + + +def get_dynamodb_table(table_name: str, region_name: Optional[str] = None) -> Any: + """A DynamoDB ``Table`` bound to the cached resource. + + The `Table` object itself is deliberately NOT cached. It is a thin handle + over the resource — the expensive part is the resource, which is shared — + and caching handles by name would keep a dict entry alive per table for no + measurable gain. + """ + return get_resource("dynamodb", region_name).Table(table_name) + + +def reset_cached_clients() -> None: + """Drop every cached client and resource. + + Exists for tests. `moto.mock_aws()` is entered per test, so a client built + under one test's mock must never be reused by the next — see the module + docstring. Production code has no reason to call this. + """ + with _lock: + _resources.clear() + _clients.clear() diff --git a/backend/src/apis/shared/browser_takeover/__init__.py b/backend/src/apis/shared/browser_takeover/__init__.py new file mode 100644 index 000000000..62295548b --- /dev/null +++ b/backend/src/apis/shared/browser_takeover/__init__.py @@ -0,0 +1,31 @@ +"""Wire contract for handing the browser to the user so they can sign in. + +Shared by the producer (``agents.builtin_tools.browser.takeover_tool``), the +transport (``agents.main_agent.streaming.stream_coordinator``) and the +persistence layer (``apis.shared.sessions``), which is why it lives here rather +than in any one of them — see the import-boundary rule in CLAUDE.md. +""" + +from .models import ( + BrowserLoginRequiredEvent, + BrowserSessionRef, + TakeoverOutcome, + Viewport, + assert_no_url, + decode_ref, + encode_ref, + format_outcome, + parse_outcome, +) + +__all__ = [ + "BrowserLoginRequiredEvent", + "BrowserSessionRef", + "TakeoverOutcome", + "Viewport", + "assert_no_url", + "decode_ref", + "encode_ref", + "format_outcome", + "parse_outcome", +] diff --git a/backend/src/apis/shared/browser_takeover/models.py b/backend/src/apis/shared/browser_takeover/models.py new file mode 100644 index 000000000..9d606fd29 --- /dev/null +++ b/backend/src/apis/shared/browser_takeover/models.py @@ -0,0 +1,314 @@ +"""Shapes for the ``request_user_login`` takeover interrupt. + +The agent hands the browser to the user, the turn pauses on a Strands +interrupt, the user signs in with their own hands in a live view of the real +browser, and the turn resumes with the session authenticated. Three components +share these shapes: + +- ``agents.builtin_tools.browser.takeover_tool`` — takes control and raises the + interrupt with a :class:`BrowserSessionRef` payload. +- ``agents.main_agent.streaming.stream_coordinator`` — turns the pending + interrupt into the ``browser_login_required`` SSE event. +- ``apis.shared.sessions`` — persists the same reference on the conversation's + session-metadata row (spec D4) so app-api can mint a live-view URL without + reading agent state. + +Design notes +------------ +* **No URL, anywhere.** ``generate_live_view_url`` signs with SigV4 *query* + auth and caps at 300 seconds. A URL in a tool result is re-emitted by the + model truncated at the ``?`` (PR #1101), and a URL on an SSE event is dead + before a human can react to it. So nothing here carries one: the SPA gets an + identifier and asks app-api for a fresh URL when it needs one (spec D2). + :func:`assert_no_url` exists so a test can hold that line. +* **Credentials are never a parameter.** The whole point of takeover is that + the user types their password into a real browser. Nothing in this contract + accepts a secret, and nothing should be added that does. +* **The resume payload is tolerant.** Like ``parse_answers`` for + ``ask_user_question``: an unrecognized shape degrades to "the user didn't + finish", never to an exception. A paused turn's only route out must not be + able to fail on a shape mismatch. +* **Cost.** The tool spec is a constant in ``toolConfig`` for granted users; + the prompt, the live view and the user's keystrokes never enter the + conversation. Only the one-line outcome below re-enters it. +""" + +from __future__ import annotations + +import json +import logging +import re +from typing import Any, Dict, Literal, Optional + +from pydantic import BaseModel, ConfigDict, Field + +logger = logging.getLogger(__name__) + +# The outcome the tool reports back to the model. +TakeoverOutcome = Literal["completed", "skipped", "expired", "failed"] + +# Free-text note the user may attach when finishing ("couldn't sign in", say). +# Bounded for the same reason every per-turn payload is (CLAUDE.md token tenet). +MAX_NOTE_CHARS = 500 + +# The reason payload must never carry a URL. Deliberately broad — it matches a +# bare host as well as a scheme — because what we are guarding against is a +# *presigned* URL slipping in, and those are long, opaque and unmistakable. +_URL_PATTERN = re.compile(r"https?://|wss?://", re.IGNORECASE) + + +class Viewport(BaseModel): + """Remote browser viewport, in CSS pixels. + + Carried on the event rather than duplicated as a constant in the frontend: + the DCV viewer's ``remoteWidth``/``remoteHeight`` must match the session's + viewport or the stream crops, and a second copy of ``1280x800`` in the SPA + is a copy that will drift (spec D3). + """ + + model_config = ConfigDict(populate_by_name=True) + + width: int = Field(..., description="Viewport width in CSS pixels") + height: int = Field(..., description="Viewport height in CSS pixels") + + +class BrowserSessionRef(BaseModel): + """Identity of the browser session a takeover is happening on. + + This is both the interrupt's payload and the projection written to the + conversation's session-metadata row (spec D4). It deliberately holds + identifiers only: app-api resolves them into a short-lived live-view URL + server-side, per call. + """ + + model_config = ConfigDict(populate_by_name=True) + + browser_session_id: str = Field( + ..., + alias="browserSessionId", + description="AgentCore browser session id", + ) + browser_id: str = Field( + ..., + alias="browserId", + description="Browser resource identifier the session runs on", + ) + viewport: Viewport = Field(..., description="Remote viewport the stream renders at") + control_state: Literal["agent", "user"] = Field( + default="user", + alias="controlState", + description=( + "Who holds the browser. While 'user' the automation stream is " + "DISABLED at the service and the idle reaper leaves the session " + "alone" + ), + ) + deadline_at: Optional[str] = Field( + default=None, + alias="deadlineAt", + description=( + "ISO 8601 instant after which an unfinished takeover is abandoned: " + "control is released, the session becomes reapable, and the turn " + "completes with an ordinary error rather than staying pinned" + ), + ) + target_url: Optional[str] = Field( + default=None, + alias="targetUrl", + description=( + "Page the browser is parked on, shown to the user so they know " + "what they are signing into. A plain navigational URL the agent " + "already browsed — never a presigned one" + ), + ) + + +class BrowserLoginRequiredEvent(BaseModel): + """SSE event telling the SPA to offer the live view for a paused turn. + + Mirrors ``UserQuestionRequiredEvent``: emitted after ``message_stop`` in the + same ``done`` block as ``oauth_required`` and ``user_question_required``, + and answered by POSTing an ``interrupt_responses`` entry keyed by + ``interruptId``. + + ``sessionId`` is the **conversation** id, as on every other SSE event in + this codebase; the browser session is ``browserSessionId``. The spec sketch + wrote a single ambiguous ``sessionId``, which would have collided with that + convention the moment app-api's live-view route (spec D2) needed the + conversation id to scope ownership. + """ + + model_config = ConfigDict(populate_by_name=True) + + type: str = "browser_login_required" + interrupt_id: str = Field(..., alias="interruptId") + tool_use_id: str = Field(..., alias="toolUseId") + session_id: str = Field(..., alias="sessionId", description="Conversation id") + browser_session_id: str = Field(..., alias="browserSessionId") + browser_id: str = Field(..., alias="browserId") + viewport: Viewport + deadline_at: Optional[str] = Field(default=None, alias="deadlineAt") + target_url: Optional[str] = Field(default=None, alias="targetUrl") + reason: Optional[str] = Field( + default=None, + description="The agent's one-line explanation of what it needs signed into", + ) + sandbox_origin: str = Field( + default="", + alias="sandboxOrigin", + description=( + "Origin the SPA frames the live-view page from — the same " + "mcp-sandbox origin MCP Apps use, whose CloudFront function locks " + "`frame-ancestors` to the SPA and composes `connect-src` from the " + "`?csp=` query. Empty when the sandbox origin is not deployed, in " + "which case the SPA shows the prompt without a viewer rather than " + "framing nothing" + ), + ) + + def to_sse_format(self) -> str: + payload = self.model_dump(by_alias=True, exclude_none=True) + return ( + f"event: browser_login_required\n" + f"data: {json.dumps(payload)}\n\n" + ) + + +# Fields allowed to hold a URL, and why each is not the thing this guard is for: +# +# targetUrl — the page the agent already browsed to, shown so the user +# knows what they are signing into. Plain and navigational. +# sandboxOrigin — a deployment constant (the mcp-sandbox origin). It carries +# no credential and is identical for every user of an +# environment. +# +# What the guard IS for is a *presigned* live-view URL: SigV4 query-signed, +# short-lived, and a credential in URL form. Add to this set only for a value +# with the same property — constant, public, and not a bearer of authority. +_URL_ALLOWED_KEYS = frozenset( + {"targetUrl", "target_url", "sandboxOrigin", "sandbox_origin"} +) + + +def assert_no_url(payload: Any) -> None: + """Raise if anything in ``payload`` looks like a URL, except the keys in + :data:`_URL_ALLOWED_KEYS`. + + The rule from PR #1101 is easy to state and easy to regress, because the + obvious way to make the frontend's job simpler is to put the live-view URL + on the event. This makes that a test failure rather than a support ticket. + """ + if isinstance(payload, dict): + for key, value in payload.items(): + if key in _URL_ALLOWED_KEYS: + continue + assert_no_url(value) + return + if isinstance(payload, (list, tuple)): + for item in payload: + assert_no_url(item) + return + if isinstance(payload, str) and _URL_PATTERN.search(payload): + raise ValueError( + "A URL reached the browser-takeover payload. Live-view URLs are " + "minted by app-api per request and must never travel on the " + "interrupt, the SSE event or the tool result." + ) + + +def _clean_note(value: Any) -> Optional[str]: + if value is None: + return None + text = str(value).strip() + return text[:MAX_NOTE_CHARS] if text else None + + +def parse_outcome(response: Any) -> tuple[TakeoverOutcome, Optional[str]]: + """Map a resume payload onto ``(outcome, note)``. + + Accepted shapes, in the order they are tried: + + * ``{"completed": true}`` — the user signed in and handed the browser back. + * ``{"skipped": true}`` / ``{"cancelled": true}`` — they declined. + * ``{"expired": true}`` — the client noticed the deadline lapsed first. + * ``{"outcome": "completed" | ...}`` — explicit. + * a bare string naming any of the above. + + Never raises: anything unrecognized reads as ``skipped``, so the turn + always resumes. ``note`` is the user's optional free-text remark. + """ + if isinstance(response, str): + folded = response.strip().casefold() + if folded in ("completed", "complete", "done", "ok"): + return "completed", None + if folded == "expired": + return "expired", None + return "skipped", None + + if not isinstance(response, dict): + return "skipped", None + + note = _clean_note(response.get("note") or response.get("text")) + + explicit = response.get("outcome") + if isinstance(explicit, str): + folded = explicit.strip().casefold() + if folded in ("completed", "skipped", "expired", "failed"): + return folded, note # type: ignore[return-value] + + if response.get("completed"): + return "completed", note + if response.get("expired"): + return "expired", note + return "skipped", note + + +def format_outcome(outcome: TakeoverOutcome, note: Optional[str] = None) -> str: + """Render the tool result the model reads. + + One short line. This text re-enters the conversation and is paid for on + every subsequent turn, so it says what happened and what to do next, and + nothing else — no page content, no identifiers, and above all no URL. + """ + lines = { + "completed": ( + "The user signed in and handed the browser back. The session is " + "authenticated — continue browsing from the page it is on." + ), + "skipped": ( + "The user declined to sign in. Do not ask again; report what you " + "could not reach and continue with what is publicly available." + ), + "expired": ( + "The sign-in window lapsed before the user finished, and the " + "browser was released. Say so plainly and continue without the " + "authenticated content; offer to try again if they ask." + ), + "failed": ( + "Handing the browser to the user failed. Continue without the " + "authenticated content and say that sign-in was unavailable." + ), + } + text = lines.get(outcome, lines["skipped"]) + return f"{text} (User note: {note})" if note else text + + +def encode_ref(ref: BrowserSessionRef) -> str: + """JSON-encode a session reference for DynamoDB persistence. + + Stored as a string for the same reason ``PendingInterrupt.tool_input`` is: + DynamoDB coerces floats to ``Decimal`` on the way in and the decoded + payload is rendered verbatim. + """ + return json.dumps(ref.model_dump(by_alias=True, exclude_none=True)) + + +def decode_ref(encoded: Optional[str]) -> Optional[Dict[str, Any]]: + """Best-effort inverse of :func:`encode_ref` for the reload path.""" + if not encoded: + return None + try: + decoded = json.loads(encoded) + except (TypeError, ValueError): + return None + return decoded if isinstance(decoded, dict) else None diff --git a/backend/src/apis/shared/caching/__init__.py b/backend/src/apis/shared/caching/__init__.py index 1aee3a99b..08ad88004 100644 --- a/backend/src/apis/shared/caching/__init__.py +++ b/backend/src/apis/shared/caching/__init__.py @@ -4,6 +4,7 @@ MANAGED_MODELS, OAUTH_PROVIDERS_ALL, OAUTH_PROVIDERS_ENABLED, + AGENT_TEMPLATES, SYSTEM_PROMPTS, TOOL_CATALOG, ConfigListCache, @@ -17,6 +18,7 @@ "MANAGED_MODELS", "OAUTH_PROVIDERS_ALL", "OAUTH_PROVIDERS_ENABLED", + "AGENT_TEMPLATES", "SYSTEM_PROMPTS", "TOOL_CATALOG", "ConfigListCache", diff --git a/backend/src/apis/shared/caching/config_cache.py b/backend/src/apis/shared/caching/config_cache.py index e53b4b6b5..086c8c383 100644 --- a/backend/src/apis/shared/caching/config_cache.py +++ b/backend/src/apis/shared/caching/config_cache.py @@ -67,6 +67,7 @@ MANAGED_MODELS = "managed_models" TOOL_CATALOG = "tool_catalog" SYSTEM_PROMPTS = "system_prompts" +AGENT_TEMPLATES = "agent_templates" # Providers are read two ways — the enabled-only GSI query that `/connectors/` # uses on first load, and the full scan the admin console uses. They are diff --git a/backend/src/apis/shared/costs/calculator.py b/backend/src/apis/shared/costs/calculator.py index 788e64e0e..0c26040fc 100644 --- a/backend/src/apis/shared/costs/calculator.py +++ b/backend/src/apis/shared/costs/calculator.py @@ -6,17 +6,25 @@ - Multi-provider cost support (Bedrock, OpenAI, Gemini) """ -from typing import Dict, Tuple +from typing import Dict, Optional, Tuple from .models import CostBreakdown class CostCalculator: """Calculate costs from token usage and pricing""" + # Bedrock's cache-write premium for a 1-hour TTL, as a multiple of the + # model's base input rate (5m is 1.25x — that is what the catalog's + # cacheWritePricePerMtok already carries). See the prompt-cache contract + # in CLAUDE.md. + LONG_TTL_CACHE_WRITE_MULTIPLIER = 2.0 + @staticmethod def calculate_message_cost( usage: Dict[str, int], - pricing: Dict[str, float] + pricing: Dict[str, float], + *, + long_ttl_static_prefix_tokens: Optional[int] = None, ) -> Tuple[float, CostBreakdown]: """ Calculate cost for a single message @@ -24,6 +32,15 @@ def calculate_message_cost( Args: usage: Token usage dict with inputTokens, outputTokens, etc. pricing: Pricing dict with inputPricePerMtok, etc. + long_ttl_static_prefix_tokens: When the turn ran with the 1h TTL on + the static (tools + system) cachePoints, the size of that static + segment. Cache writes that cover it are billed at the 1h premium + (2x base) instead of the 5m one. Which part of the write was + static is inferred from the read: a static segment that was + read (cacheRead >= static) was not written, so the write is all + history at 5m; otherwise the unread remainder of the static + segment was written at 1h. ``None`` (the default) prices every + cache write at the catalog's 5m rate — today's behavior. Returns: Tuple of (total_cost, cost_breakdown) @@ -79,7 +96,15 @@ def calculate_message_cost( input_cost = (input_tokens / 1_000_000) * input_price output_cost = (output_tokens / 1_000_000) * output_price cache_read_cost = (cache_read_tokens / 1_000_000) * cache_read_price - cache_write_cost = (cache_write_tokens / 1_000_000) * cache_write_price + + long_ttl_written = 0 + if long_ttl_static_prefix_tokens and long_ttl_static_prefix_tokens > 0 and cache_write_tokens > 0: + long_ttl_written = max(0, min(cache_write_tokens, int(long_ttl_static_prefix_tokens) - cache_read_tokens)) + long_ttl_write_price = input_price * CostCalculator.LONG_TTL_CACHE_WRITE_MULTIPLIER + cache_write_cost = ( + (long_ttl_written / 1_000_000) * long_ttl_write_price + + ((cache_write_tokens - long_ttl_written) / 1_000_000) * cache_write_price + ) total_cost = input_cost + output_cost + cache_read_cost + cache_write_cost diff --git a/backend/src/apis/shared/feature_flags.py b/backend/src/apis/shared/feature_flags.py index 31555cb95..b9108c686 100644 --- a/backend/src/apis/shared/feature_flags.py +++ b/backend/src/apis/shared/feature_flags.py @@ -92,6 +92,36 @@ def workspace_tools_enabled() -> bool: return os.environ.get("WORKSPACE_TOOLS_ENABLED", "").strip().lower() != "false" +def document_read_enabled() -> bool: + """Whether the ``document_read`` agent tool is injected for sessions that + carry a readable attachment (``docs/specs/document-context-offload.md`` + §4B). **Default ON with a kill switch** (house style): unset or empty + resolves to enabled; only the literal ``"false"`` disables. + + This is the *only* control on the tool. It is deliberately not gated on + RBAC or the tool picker: the governing capability is the user's own + attachment, and the ``workspace_files`` catalog key is granted to no prod + role, so an RBAC gate would ship the recovery path dark. + """ + return os.environ.get("DOCUMENT_READ_ENABLED", "").strip().lower() != "false" + + +def attachment_tool_autoenable_enabled() -> bool: + """Whether a turn that carries (or a session that holds) a spreadsheet + attachment gets the Spreadsheet Analysis tools injected for that session + even when the picker has them off — gated on the caller's RBAC grant, so + it enables, never grants. **Default ON with a kill switch** (house style): + unset or empty resolves to enabled; only the literal ``"false"`` disables. + + Why: CSV/XLSX never go inline (``_partition_attachments``), and the + analysis tools are opt-in in the picker, so on the default tool set an + attached spreadsheet was a dead end — the model told the user to go find a + sidebar toggle (docs/specs/load-test-assessment-2026-09.md §1 fix 5). The + user's own attachment is the governing intent. + """ + return os.environ.get("ATTACHMENT_TOOL_AUTOENABLE_ENABLED", "").strip().lower() != "false" + + def agents_enabled() -> bool: """Whether the Agent Designer surface is enabled for this environment. @@ -258,6 +288,71 @@ def agent_status_enabled() -> bool: return os.environ.get("AGENT_STATUS_ENABLED", "").strip().lower() != "false" +def agent_status_live_drain_enabled() -> bool: + """Whether ``agent_status`` transitions are drained DURING agent-stream silence. + + The hook records its transitions from inside Strands' event loop, which has + no route to the SSE stream — so the coordinator drains them. Draining + between yields of the agent stream means a transition can only leave the + container when the agent stream next produces an event, and during tool + execution the agent stream produces nothing. A ``tool_start`` therefore + queued for exactly the silence it existed to explain and arrived bundled + with its own ``tool_end`` (measured: a three-tool browse turn narrated + nothing for 4.5s). This flag turns on the concurrent drain that fixes it. + + **Default ON with a kill switch** (house style): unset or empty resolves to + enabled; only the literal ``"false"`` (case-insensitive) disables. + + Its own kill switch rather than riding ``AGENT_STATUS_ENABLED`` because the + two carry different risk. That flag gates what the hook *records*, all of + it in-process. This one changes how the turn's stream is consumed — the + merge races the agent stream against a short timer — so a regression here + would be a streaming bug, not a missing status line. Turning it off + restores the original between-yields drain exactly, leaving every other + part of the feature intact. + + Costs nothing against the model: it changes only when an already-recorded + transition is written to the SSE channel. Nothing reaches the prompt. + """ + return ( + os.environ.get("AGENT_STATUS_LIVE_DRAIN_ENABLED", "").strip().lower() + != "false" + ) + + +def agent_preparing_phase_enabled() -> bool: + """Whether the agent build runs INSIDE the response stream, narrated. + + Measured on dev (docs/specs/agent-state-feedback.md): a cold agent-cache + miss spends **1478ms** in ``get_agent`` against a 2542ms pre-stream window, + while a warm turn spends 0-38ms there. Because FastAPI flushes response + headers when the handler returns its ``StreamingResponse``, and + ``get_agent`` is awaited before that return, the whole of that 1478ms is + dead air: the client has no channel and the server has nothing to say on. + + With this on, the non-resume path defers the build into the stream + generator and emits one ``agent_status`` ``preparing`` frame before it, so + the response opens immediately and the wait is narrated instead of silent. + + **Default ON with a kill switch** (house style): unset or empty resolves to + enabled; only the literal ``"false"`` (case-insensitive) disables, which + restores the eager build exactly. + + Deliberately NOT applied to resume turns. Resume validates the submitted + interrupt ids against the rebuilt agent's paused state and **400s** on a + mismatch; that guard has to run before any byte is sent, and it needs the + agent to run at all. Resume also reuses a cached agent by construction, so + it is the case with the least to gain. + + Costs nothing against the model: one SSE frame, and the same build either + way. Nothing reaches the prompt. + """ + return ( + os.environ.get("AGENT_PREPARING_PHASE_ENABLED", "").strip().lower() + != "false" + ) + + def tool_summaries_enabled() -> bool: """Whether tool batches get a model-generated one-line summary. @@ -352,3 +447,88 @@ def ask_user_question_enabled() -> bool: every time it flipped. """ return os.environ.get("ASK_USER_QUESTION_ENABLED", "").strip().lower() != "false" + + +def browser_takeover_enabled() -> bool: + """Whether a turn can hand the browser to the user so they can sign in. + + Covers the ``request_user_login`` built-in tool, its Strands interrupt, the + ``browser_login_required`` SSE event and the ``browser_login`` + ``PendingInterrupt`` breadcrumb. **Default ON with a kill switch** (house + style, mirroring ``ask_user_question_enabled``): unset or empty resolves to + enabled; only the literal ``"false"`` (case-insensitive) disables. + + While off the tool is never registered, so it never reaches ``toolConfig`` + and the model cannot pause a turn behind a prompt no client is listening + for. The tool re-checks the flag at call time as well, so a registry built + before a flip cannot strand a turn. Note this flag is a second gate, not + the first: ``request_user_login`` is its own catalog entry, so a role that + was never granted it never sees the tool regardless of the flag + (``docs/specs/authenticated-web-assessment.md`` D1). + + Cost note (CLAUDE.md token-effectiveness tenet): flipping this changes + ``toolConfig`` for granted users and therefore re-writes the cacheable + prefix once per in-flight session — the ordinary cost of a deploy-time tool + change. Do not derive it from conversation state. + """ + return os.environ.get("BROWSER_TAKEOVER_ENABLED", "").strip().lower() != "false" + + +def response_feedback_enabled() -> bool: + """Whether users can thumb an assistant message up or down. + + Covers the ``PUT`` / ``DELETE /sessions/{id}/messages/{messageId}/feedback`` + routes, the ``feedback`` field merged into ``GET /sessions/{id}/messages``, + and the ``thumbsUp`` / ``thumbsDown`` session rollups. **Default ON with a + kill switch** (house style, mirroring ``cost_diagnostics_enabled``): unset + or empty resolves to enabled; only the literal ``"false"`` (case- + insensitive) disables. While off the write routes 404 and the read merge + is skipped; rows already written stay in the table. Name and default per + ``docs/specs/response-feedback.md`` §5. + + The signal is content-free by construction (a ±1, a timestamp and an + optional reason *code* from a fixed enum — never free text), which is + what lets it join the ``C#`` cost row's turn class on the admin session + profile without the profile ever reading the conversation. See + ``docs/specs/document-context-offload.md`` §5 row 7 / §6.1. + """ + return os.environ.get("RESPONSE_FEEDBACK_ENABLED", "").strip().lower() != "false" +def attachment_turn_guard_enabled() -> bool: + """Whether a turn's attachments are held to the per-message file count and + the aggregate inline-bytes budget before the message is built. + + Covers ``_apply_message_file_cap`` and ``_apply_inline_byte_budget`` in + the inference API chat route (docs/specs/document-context-offload.md §4E, + PR-6). **Default ON with a kill switch** (house style): unset or empty + resolves to enabled; only the literal ``"false"`` (case-insensitive) + disables. + + While off the route behaves as before this shipped: the ``file_upload_ids`` + resolver silently truncates at five, direct ``files`` are uncounted, and + a turn whose attachments sum past the AgentCore Memory event quota fails + at ``create_message`` with a ``SessionException``. The tuning knobs + (``INLINE_ATTACHMENTS_MAX_TOTAL_BYTES``, ``FILE_UPLOAD_MAX_FILES_PER_MESSAGE``) + live in ``apis.shared.files.models``. + """ + return os.environ.get("ATTACHMENT_TURN_GUARD_ENABLED", "").strip().lower() != "false" + + +def feedback_eval_sampling_enabled() -> bool: + """Whether down-thumbed turns may be sent to AgentCore Evaluations. + + Covers ``POST /admin/feedback/evaluations/run`` (the offline batch that + judges recent down-thumbs, response-feedback spec §11 PR-4). **Defaults + OFF** (the ``FINE_TUNING_ENABLED``-style opt-in): set + ``FEEDBACK_EVAL_SAMPLING_ENABLED=true`` to turn it on. + + Off by default on purpose, not by caution: the judge is an AWS-managed + evaluator that reads the conversation's spans — the full system prompt + and every user message of the sampled session. The evaluations spike + (``docs/specs/agentcore-evaluations-spike-findings.md`` §2) says to make + that decision explicitly per environment rather than let it happen as a + side effect, and the feedback spec's §8 puts conversation content behind + a scope. Flipping this flag is that decision. The read surfaces (the + queue list, the profile's judged aggregates) are not gated — they show + numbers only and tolerate the absence of any judged row. + """ + return os.environ.get("FEEDBACK_EVAL_SAMPLING_ENABLED", "false").strip().lower() == "true" diff --git a/backend/src/apis/shared/feedback_eval/__init__.py b/backend/src/apis/shared/feedback_eval/__init__.py new file mode 100644 index 000000000..f170aa11b --- /dev/null +++ b/backend/src/apis/shared/feedback_eval/__init__.py @@ -0,0 +1 @@ +"""Eval sampling: down-thumbed turns as the evaluation harness's input queue.""" diff --git a/backend/src/apis/shared/feedback_eval/sampler.py b/backend/src/apis/shared/feedback_eval/sampler.py new file mode 100644 index 000000000..29fde6115 --- /dev/null +++ b/backend/src/apis/shared/feedback_eval/sampler.py @@ -0,0 +1,230 @@ +"""Eval sampling — response-feedback spec §11 PR-4. + +A down-thumb is a *sampler*: it marks the small subset of turns worth +spending judge tokens on. This module turns the queue of recent down-thumbs +into AgentCore Evaluations calls, routed by the thumb's reason code (spec +§6), and writes a **content-free** verdict back onto the thumb row. + +Three rules, from the spec and the evaluations spike: + +* **Offline batch only.** ``run_sampling_batch`` is driven by an admin + request (or a future schedule), never by a turn. An inline per-turn judge + is exactly what the cost tenet exists to stop. +* **Numbers leave, prose does not.** The Evaluate API returns an + ``explanation`` that quotes the conversation. ``summarize`` drops it; the + storage write refuses anything that still carries one; the content-policy + denylist names it. +* **The judge is a seam.** :class:`Judge` is a one-method protocol so tests + and forks without AgentCore Evaluations inject their own; + :class:`AgentCoreJudge` is the SDK adapter, imported lazily. + +Routing (spec §6 → the 16 built-in evaluators the spike verified): + + wrong → Builtin.Correctness, Builtin.Faithfulness + instructions → Builtin.InstructionFollowing + length → Builtin.Conciseness (the style signal) + tool_failed → no judge — ops, corroborated against the call's tool census + outdated → no judge yet — KB-freshness join is a follow-up + other / none → Builtin.Helpfulness (the generic judge) +""" + +from __future__ import annotations + +import logging +import os +from dataclasses import dataclass, field +from datetime import timedelta +from typing import Any, Dict, List, Optional, Protocol, Sequence + +logger = logging.getLogger(__name__) + +RUNTIME_LOG_GROUP_ENV = "AGENTCORE_RUNTIME_LOG_GROUP" + +#: Reason code → built-in evaluator ids. An empty tuple means "no judge for +#: this bucket" (it is an ops signal, or not yet joinable), not "skip". +EVALUATORS_BY_REASON: Dict[Optional[str], tuple] = { + "wrong": ("Builtin.Correctness", "Builtin.Faithfulness"), + "instructions": ("Builtin.InstructionFollowing",), + "length": ("Builtin.Conciseness",), + "tool_failed": (), + "outdated": (), + "other": ("Builtin.Helpfulness",), + None: ("Builtin.Helpfulness",), +} + + +def evaluators_for(reason: Optional[str]) -> tuple: + """The evaluator ids a reason bucket routes to (unknown codes → generic).""" + return EVALUATORS_BY_REASON.get(reason, EVALUATORS_BY_REASON[None]) + + +class Judge(Protocol): + """Anything that can score one conversation with a set of evaluators.""" + + def judge(self, session_id: str, evaluator_ids: Sequence[str]) -> List[Dict[str, Any]]: + """Raw ``evaluationResults`` items for the session, or ``[]``.""" + + +class AgentCoreJudge: + """``bedrock_agentcore.evaluation.EvaluationClient`` over the runtime log + group. Lazy imports keep ``apis.shared`` importable in images without the + SDK. The session id sent is the *runtime* session id + (``sid-``), which is how the chat proxy pins spans.""" + + def __init__(self, log_group_name: Optional[str] = None, region_name: Optional[str] = None, + look_back: timedelta = timedelta(days=7)): + self.log_group_name = log_group_name or os.environ.get(RUNTIME_LOG_GROUP_ENV, "").strip() + self.region_name = region_name or os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") + self.look_back = look_back + self._client = None + + def judge(self, session_id: str, evaluator_ids: Sequence[str]) -> List[Dict[str, Any]]: + if not self.log_group_name: + raise RuntimeError(f"{RUNTIME_LOG_GROUP_ENV} is not configured") + if self._client is None: + from bedrock_agentcore.evaluation import EvaluationClient # lazy: heavy, optional + + self._client = EvaluationClient(region_name=self.region_name) + from apis.shared.harness.runner import runtime_session_id_for + + return self._client.run( + evaluator_ids=list(evaluator_ids), + session_id=runtime_session_id_for(session_id), + log_group_name=self.log_group_name, + look_back_time=self.look_back, + ) + + +def summarize(results: Sequence[Dict[str, Any]]) -> Dict[str, Dict[str, Any]]: + """Collapse raw ``evaluationResults`` into ``{evaluatorId: {value, rating, + n, tokens}}`` — numbers and the evaluator's own rating vocabulary (its + ``label``, stored as ``rating`` because ``label`` is a content-bearing + path elsewhere in the table and the denylist is by name). The + ``explanation`` (prose quoting the conversation) and the span context + are dropped here and nowhere later. TRACE-level evaluators return one + result per trace; ``value`` is their mean and ``n`` says how many.""" + out: Dict[str, Dict[str, Any]] = {} + for item in results: + evaluator = item.get("evaluatorId") or item.get("evaluatorName") + if not evaluator or item.get("errorCode"): + continue + value = item.get("value") + if not isinstance(value, (int, float)): + continue + slot = out.setdefault(evaluator, {"value": 0.0, "n": 0, "tokens": 0}) + n = slot["n"] + slot["value"] = round((slot["value"] * n + float(value)) / (n + 1), 4) + slot["n"] = n + 1 + slot["tokens"] += int((item.get("tokenUsage") or {}).get("totalTokens") or 0) + rating = item.get("label") + if isinstance(rating, str) and rating: + slot["rating"] = rating + return out + + +def corroborate_tool_failure(cost_row: Optional[Dict[str, Any]]) -> Optional[bool]: + """For a ``tool_failed`` thumb: did the call's tool census (``toolCalls`` + on the ``C#`` row) actually record an error? ``None`` when the row or the + census is missing — "could not check", not "no".""" + if not cost_row: + return None + census = cost_row.get("toolCalls") + if not isinstance(census, dict): + return None + for entry in census.values(): + errors = entry.get("errors") if isinstance(entry, dict) else None + try: + if int(errors or 0) > 0: + return True + except (TypeError, ValueError): + continue + return False + + +@dataclass +class SamplingReport: + """What one batch did — counts only, for the log line and tests.""" + candidates: int = 0 + judged: int = 0 + corroborated: int = 0 + skipped_no_judge: int = 0 + skipped_already: int = 0 + failed: int = 0 + tokens: int = 0 + errors: List[str] = field(default_factory=list) + + +def build_verdict( + reason: Optional[str], + results: Sequence[Dict[str, Any]], + cost_row: Optional[Dict[str, Any]], +) -> Dict[str, Any]: + """The content-free ``evaluation`` map for one thumb.""" + verdict: Dict[str, Any] = {"reason": reason or "none", "evaluators": list(evaluators_for(reason))} + if results: + verdict["scores"] = summarize(results) + if reason == "tool_failed": + corroborated = corroborate_tool_failure(cost_row) + if corroborated is not None: + verdict["toolFailureCorroborated"] = corroborated + return verdict + + +async def run_sampling_batch( + table, + judge: Judge, + *, + limit: int = 10, + cost_row_lookup=None, +) -> SamplingReport: + """Judge up to ``limit`` recent, not-yet-judged down-thumbs. + + ``cost_row_lookup(session_id, message_id) -> Optional[dict]`` supplies the + call's ``C#`` row for tool-failure corroboration; optional. Every thumb is + independent: one judge failure is recorded and the batch continues. + """ + from apis.shared.sessions.feedback import is_explicit, list_recent_down_thumbs, store_evaluation + + report = SamplingReport() + rows = [r for r in list_recent_down_thumbs(table, limit=limit * 3) if is_explicit(r)] + for row in rows: + if report.judged + report.skipped_no_judge + report.failed >= limit: + break + if row.get("evaluatedAt"): + report.skipped_already += 1 + continue + report.candidates += 1 + session_id = str(row.get("sessionId") or "") + user_id = str(row.get("userId") or "") + try: + message_id = int(row.get("messageId")) + except (TypeError, ValueError): + report.failed += 1 + continue + reason = row.get("reason") if isinstance(row.get("reason"), str) else None + evaluator_ids = evaluators_for(reason) + cost_row = cost_row_lookup(session_id, message_id) if cost_row_lookup else None + results: List[Dict[str, Any]] = [] + try: + if evaluator_ids: + results = judge.judge(session_id, evaluator_ids) + verdict = build_verdict(reason, results, cost_row) + store_evaluation(table, user_id, session_id, message_id, verdict) + except Exception as e: # noqa: BLE001 - one bad thumb must not stop the batch + report.failed += 1 + report.errors.append(type(e).__name__) + logger.warning("eval sampling failed for a thumb: %s", type(e).__name__) + continue + if evaluator_ids: + report.judged += 1 + report.tokens += sum(s.get("tokens", 0) for s in verdict.get("scores", {}).values()) + else: + report.skipped_no_judge += 1 + if verdict.get("toolFailureCorroborated"): + report.corroborated += 1 + logger.info( + "eval sampling batch: candidates=%d judged=%d no_judge=%d already=%d failed=%d tokens=%d", + report.candidates, report.judged, report.skipped_no_judge, report.skipped_already, + report.failed, report.tokens, + ) + return report diff --git a/backend/src/apis/shared/files/__init__.py b/backend/src/apis/shared/files/__init__.py index 9a12684b1..6fa4213eb 100644 --- a/backend/src/apis/shared/files/__init__.py +++ b/backend/src/apis/shared/files/__init__.py @@ -21,6 +21,9 @@ TABULAR_EXTENSIONS, PRESENTATION_MIME_TYPES, PRESENTATION_EXTENSIONS, + SHEET_PREVIEW_MIME_TYPES, + SheetPreview, + SheetPreviewResponse, INLINE_DOCUMENT_MAX_BYTES, get_file_format, is_allowed_mime_type, @@ -69,6 +72,9 @@ "TABULAR_EXTENSIONS", "PRESENTATION_MIME_TYPES", "PRESENTATION_EXTENSIONS", + "SHEET_PREVIEW_MIME_TYPES", + "SheetPreview", + "SheetPreviewResponse", "INLINE_DOCUMENT_MAX_BYTES", "get_file_format", "is_allowed_mime_type", diff --git a/backend/src/apis/shared/files/document_digest.py b/backend/src/apis/shared/files/document_digest.py new file mode 100644 index 000000000..c7b474679 --- /dev/null +++ b/backend/src/apis/shared/files/document_digest.py @@ -0,0 +1,471 @@ +"""DocumentDigest — a bounded, once-per-upload description of a document. + +The offload design (``docs/specs/document-context-offload.md`` §4A) keeps a +document inline on the turn that introduces it and hands later turns a +*digest* plus a ``document_read`` handle instead. This module builds that +digest, off the model path, when an upload completes (PR-2). Nothing in the +chat path reads it yet; PR-3 renders it in place of the restore placeholder. + +Two parts, deliberately split: + +* **Outline** — deterministic, no model: page / paragraph / line count, a + heading outline with the unit each heading starts on, table / figure + mentions, and a text sample. PDF text comes from ``pypdfium2`` (already a + dependency), DOCX from the stdlib extractor in ``document_read``, the text + family from the bytes. A document with no detectable headings gets an + outline sampled from page first-lines so the model still has anchors. +* **Abstract** — 3–5 sentences from a cheap text model over the sample plus + the outline (Nova Micro, the same model the tool-batch summaries and the + compaction summary use; ``DOCUMENT_DIGEST_MODEL_ID`` overrides). Fail-open: + a model error leaves the digest without an abstract, never without an + outline. + +The rendered form (``render_digest``) is what enters context later. It is +hard-capped at ``DOCUMENT_DIGEST_MAX_TOKENS`` (default 1,500, chars/4): the +outline is trimmed from the end first, then the abstract — escaped *before* it +is cut, so the cap is measured on what is actually rendered, and dropped +entirely rather than allowed to overshoot. Every digest +records its rendered token estimate so the cap is a stored fact per file. + +Content policy: the digest carries model prose about the user's document +(``abstract``) and heading text (``sections``). Both are denylisted in +``content_policy.CONTENT_BEARING``; the numeric fields (``status``, +``tokens``, ``count``, ``format``) are projectable. +""" + +from __future__ import annotations + +import asyncio +import logging +import os +import re +import time +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from pydantic import BaseModel, Field + +from .document_read import _open_pdf, _pdf_page_text, docx_paragraphs, document_format_for + +logger = logging.getLogger(__name__) + +DIGEST_VERSION = 1 + +#: Same heuristic the compaction estimator uses (``compaction_policy.CHARS_PER_TOKEN``); +#: duplicated here because ``apis.shared`` must not import the agent layer. +_CHARS_PER_TOKEN = 4 + +#: Rendered budget. The spec's ceiling: a digest must be strictly smaller than +#: any document worth offloading (the trigger's floor is 5,000 tokens). +DOCUMENT_DIGEST_MAX_TOKENS = int(os.environ.get("DOCUMENT_DIGEST_MAX_TOKENS", 1_500)) +#: Outline entries kept (evenly sampled when the document has more headings). +DOCUMENT_DIGEST_MAX_SECTIONS = int(os.environ.get("DOCUMENT_DIGEST_MAX_SECTIONS", 40)) +#: Characters of document text handed to the abstract model. +DOCUMENT_DIGEST_SAMPLE_CHARS = int(os.environ.get("DOCUMENT_DIGEST_SAMPLE_CHARS", 12_000)) +#: Cheap text model for the abstract. Nova Micro is what the tool-batch +#: summaries and the compaction summary already run on; it is text-only, +#: which is fine because the outline extractor hands it text. +DOCUMENT_DIGEST_MODEL_ID = os.environ.get("DOCUMENT_DIGEST_MODEL_ID", "").strip() or "us.amazon.nova-micro-v1:0" +_ABSTRACT_MAX_OUTPUT_TOKENS = 320 +_HEADING_MAX_CHARS = 90 +_SECTION_TITLE_CHARS = 80 +_SAMPLE_PER_PAGE_CHARS = 600 + + +def document_digest_enabled() -> bool: + """Default ON with a kill switch (house style): only the literal "false" disables.""" + return os.environ.get("DOCUMENT_DIGEST_ENABLED", "").strip().lower() != "false" + + +# --------------------------------------------------------------------------- +# Model +# --------------------------------------------------------------------------- + + +class DigestSection(BaseModel): + """One outline entry: the unit (page / paragraph / line, 1-indexed) a + heading starts on and the heading text, bounded.""" + + start: int + title: str + + +class DocumentDigest(BaseModel): + """What is persisted on ``FileMetadata.digest`` (as a plain dict).""" + + version: int = DIGEST_VERSION + status: str = "ready" # ready | failed + format: Optional[str] = None + unit: str = "page" # page | paragraph | line + count: int = 0 + sections: List[DigestSection] = Field(default_factory=list) + tables: int = 0 + figures: int = 0 + chars: int = 0 + abstract: Optional[str] = None + tokens: int = 0 # rendered-form estimate (chars/4) + model_id: Optional[str] = None + extractor_ms: int = 0 + error: Optional[str] = None # exception class name only, never a message + generated_at: str = Field(default_factory=lambda: datetime.now(timezone.utc).isoformat()) + + def to_item(self) -> Dict[str, Any]: + return self.model_dump(exclude_none=True) + + @classmethod + def from_item(cls, item: Any) -> Optional["DocumentDigest"]: + if not isinstance(item, dict): + return None + try: + return cls(**item) + except Exception: # noqa: BLE001 - a malformed row reads as "no digest" + return None + + +# --------------------------------------------------------------------------- +# Outline extraction (deterministic) +# --------------------------------------------------------------------------- + +_NUMBERED = re.compile( + r"^(?:\d+(?:\.\d+)*[.)]?\s+\S|[IVXLC]{1,6}[.)]\s+\S|(?:article|section|chapter|part|appendix|schedule|exhibit|title)\s+[\w-]+)", + re.IGNORECASE, +) +_TABLE = re.compile(r"\btable\s+\d+|\btable\b\s*[:\-]", re.IGNORECASE) +_FIGURE = re.compile(r"\b(?:figure|fig\.|chart|diagram)\s+\d+", re.IGNORECASE) +_HTML_TAG = re.compile(r"<[^>]+>") +_MD_HEADING = re.compile(r"^\s{0,3}#{1,6}\s+(.+?)\s*#*\s*$") + + +def looks_like_heading(line: str, *, standalone: bool = False) -> bool: + """Cheap, format-agnostic heading test: short, no sentence punctuation at + the end, and either numbered, ALL CAPS, or title-cased with few words. + + ``standalone`` is for units that are already a whole paragraph (DOCX, + text lines): a one-word capitalized paragraph ("Risks") is a heading + there, while a one-word line inside a PDF page is usually noise. + """ + text = line.strip() + if not (3 <= len(text) <= _HEADING_MAX_CHARS) or text.endswith((".", ",", ";", ":")): + return False + if _NUMBERED.match(text): + return True + letters = [c for c in text if c.isalpha()] + if len(letters) >= 2 and all(c.isupper() for c in letters): + return True + words = [w for w in re.split(r"\s+", text) if w] + if len(words) <= 10: + capitalized = sum(1 for w in words if w[0].isupper() or not w[0].isalpha()) + return capitalized / len(words) >= 0.7 and len(words) >= (1 if standalone else 2) + return False + + +def _strip_html(text: str) -> str: + return _HTML_TAG.sub(" ", text) + + +def _units_for(fmt: str, raw: bytes) -> Tuple[str, List[str]]: + """``(unit name, unit texts)`` — one string per page / paragraph / line.""" + if fmt == "pdf": + pdf = _open_pdf(raw) + try: + return "page", [_pdf_page_text(pdf, i) for i in range(len(pdf))] + finally: + pdf.close() + if fmt == "docx": + return "paragraph", docx_paragraphs(raw) + text = raw.decode("utf-8", errors="replace") + if fmt == "html": + text = _strip_html(text) + return "line", text.splitlines() + + +def _sample_sections(sections: List[DigestSection], limit: int) -> List[DigestSection]: + if len(sections) <= limit: + return sections + step = len(sections) / limit + return [sections[int(i * step)] for i in range(limit)] + + +def extract_outline(fmt: str, raw: bytes) -> DocumentDigest: + """The deterministic half of the digest (no abstract, no tokens yet).""" + unit, texts = _units_for(fmt, raw) + sections: List[DigestSection] = [] + tables = figures = chars = 0 + for index, text in enumerate(texts, start=1): + chars += len(text) + tables += len(_TABLE.findall(text)) + figures += len(_FIGURE.findall(text)) + lines = text.splitlines() if unit == "page" else [text] + for line in lines: + md = _MD_HEADING.match(line) if fmt == "md" else None + title = md.group(1) if md else ( + line.strip() if looks_like_heading(line, standalone=unit != "page") else None + ) + if title: + sections.append(DigestSection(start=index, title=title[:_SECTION_TITLE_CHARS])) + if unit != "page": + break + if not sections and unit == "page" and texts: + # No headings (a scan, a form): anchor the outline on page first-lines. + for index, text in enumerate(texts, start=1): + first = next((ln.strip() for ln in text.splitlines() if ln.strip()), "") + if first: + sections.append(DigestSection(start=index, title=first[:_SECTION_TITLE_CHARS])) + return DocumentDigest( + format=fmt, + unit=unit, + count=len(texts), + sections=_sample_sections(sections, DOCUMENT_DIGEST_MAX_SECTIONS), + tables=tables, + figures=figures, + chars=chars, + ) + + +def text_sample(fmt: str, raw: bytes, limit: int = DOCUMENT_DIGEST_SAMPLE_CHARS) -> str: + """Up to ``limit`` characters spread across the document (not just its + head), so the abstract sees the middle and the end of a long file.""" + _, texts = _units_for(fmt, raw) + if not texts: + return "" + if fmt == "pdf": + per_page = max(_SAMPLE_PER_PAGE_CHARS, limit // max(1, len(texts))) + pieces = [t.strip()[:per_page] for t in texts if t.strip()] + return "\n\n".join(pieces)[:limit] + return "\n".join(t for t in texts if t.strip())[:limit] + + +# --------------------------------------------------------------------------- +# Abstract (cheap model, fail-open) +# --------------------------------------------------------------------------- + +_ABSTRACT_SYSTEM_PROMPT = ( + "You write short, factual abstracts of documents for an assistant that will " + "later retrieve specific pages on demand. Write 3 to 5 plain sentences: what " + "the document is, who it is for or from, its main parts, and any notable " + "numbers, dates or decisions. No preamble, no bullet points, no quotes." +) + + +def _abstract_prompt(outline: DocumentDigest, sample: str) -> str: + lines = [f"Document type: {outline.format}, {outline.count} {outline.unit}s."] + if outline.sections: + lines.append("Outline:") + lines.extend(f"- ({outline.unit} {s.start}) {s.title}" for s in outline.sections[:25]) + lines.append("Text sample:") + lines.append(sample) + return "\n".join(lines) + + +async def generate_abstract(outline: DocumentDigest, sample: str, model_id: str = DOCUMENT_DIGEST_MODEL_ID) -> Optional[str]: + """3–5 sentences from the cheap model, or ``None`` (never raises).""" + if not sample.strip(): + return None + try: + import boto3 + except ImportError: # pragma: no cover + return None + try: + client = boto3.client("bedrock-runtime", region_name=os.environ.get("AWS_REGION", "us-west-2")) + response = await asyncio.to_thread( + client.converse, + modelId=model_id, + messages=[{"role": "user", "content": [{"text": _abstract_prompt(outline, sample)}]}], + system=[{"text": _ABSTRACT_SYSTEM_PROMPT}], + inferenceConfig={"temperature": 0.2, "maxTokens": _ABSTRACT_MAX_OUTPUT_TOKENS, "topP": 0.9}, + ) + if response.get("stopReason") == "max_tokens": + logger.debug("Document abstract hit the token ceiling; discarding") + return None + text = response["output"]["message"]["content"][0]["text"].strip() + return re.sub(r"\s+", " ", text) or None + except Exception: # noqa: BLE001 - an abstract is never worth a failed upload + logger.debug("Document abstract generation skipped", exc_info=True) + return None + + +# --------------------------------------------------------------------------- +# Rendering (what PR-3 puts in context) and the budget +# --------------------------------------------------------------------------- + + +def _truncate_escaped(text: str, limit: int) -> str: + """Cut an already-escaped string to ``limit`` characters without splitting + an entity — ``&`` must never be left as ``&am``. + + Escaping *after* slicing (what this replaces) silently broke the budget: + each ``&`` becomes five characters, so a slice measured on the raw text + could render up to 5x longer. Measured: an abstract of ``&`` rendered 388 + tokens against a 100-token budget. + """ + if limit <= 0: + return "" + cut = text[:limit] + amp = cut.rfind("&") + if amp != -1 and ";" not in cut[amp:]: + cut = cut[:amp] + return cut.rstrip() + + +def _xml_escape(text: str) -> str: + return text.replace("&", "&").replace("<", "<").replace(">", ">").replace('"', """) + + +def estimate_tokens(text: str) -> int: + return len(text) // _CHARS_PER_TOKEN + + +def render_digest( + digest: DocumentDigest, + *, + filename: str, + upload_id: str, + budget_tokens: int = DOCUMENT_DIGEST_MAX_TOKENS, + include_handle: bool = True, +) -> str: + """The ```` block, trimmed to ``budget_tokens``. + + Sections are dropped from the end first (the model can always ask + ``document_read`` for more), then the abstract is truncated. The opening + tag always fits: it is the ``document_read`` handle. + + ``include_handle=False`` omits ``upload_id``. Callers pass it when + ``DOCUMENT_READ_ENABLED=false`` has taken the tool away: the outline and + abstract are still strictly more than the pre-PR-3 placeholder, but + advertising a retrieval id for a tool the model does not have would invite + a call that cannot be made. See ``feature_flags.document_read_enabled``. + """ + handle = f'upload_id="{upload_id}" ' if include_handle else "" + header = ( + f'{_xml_escape(digest.abstract)}" if digest.abstract else "" + section_lines = [ + f'
{_xml_escape(s.title)}
' for s in digest.sections + ] + + def _build(abstract: str, sections: Sequence[str]) -> str: + parts = [header] + if abstract: + parts.append(abstract) + parts.extend(sections) + parts.append(footer) + return "\n".join(parts) + + budget_chars = max(0, budget_tokens) * _CHARS_PER_TOKEN + kept = list(section_lines) + text = _build(abstract_line, kept) + while len(text) > budget_chars and kept: + kept.pop() + text = _build(abstract_line, kept) + if len(text) > budget_chars and abstract_line: + room = budget_chars - len(_build("", kept)) - len(" ") - 1 + if room > 20 and digest.abstract: + # Escape first, then cut: the budget is measured on what is + # actually rendered, not on the raw text it came from. + trimmed = _truncate_escaped(_xml_escape(digest.abstract), room - 1) + "…" + text = _build(f" {trimmed}", kept) + else: + text = _build("", kept) + if len(text) > budget_chars: + # The cap is hard. Drop the abstract entirely rather than overshoot; + # only the opening tag — the ``document_read`` handle — is allowed to + # survive a budget this small. + text = _build("", kept) + return text + + +# --------------------------------------------------------------------------- +# Build +# --------------------------------------------------------------------------- + + +async def build_digest( + *, + raw: bytes, + mime_type: str, + filename: str, + upload_id: str, + model_id: str = DOCUMENT_DIGEST_MODEL_ID, + with_abstract: bool = True, +) -> DocumentDigest: + """Outline + abstract + rendered token estimate. Never raises: an + extraction failure yields ``status="failed"`` with the exception class + name; an abstract failure yields a digest without one.""" + started = time.monotonic() + fmt = document_format_for(mime_type, filename) + if fmt is None: + return DocumentDigest(status="failed", error="NotADocument", extractor_ms=0) + try: + outline = await asyncio.to_thread(extract_outline, fmt, raw) + sample = await asyncio.to_thread(text_sample, fmt, raw) if with_abstract else "" + except Exception as e: # noqa: BLE001 + logger.warning("Document digest extraction failed (%s)", type(e).__name__, exc_info=True) + return DocumentDigest( + status="failed", format=fmt, error=type(e).__name__, + extractor_ms=int((time.monotonic() - started) * 1000), + ) + if with_abstract: + outline.abstract = await generate_abstract(outline, sample, model_id=model_id) + outline.model_id = model_id if outline.abstract else None + outline.extractor_ms = int((time.monotonic() - started) * 1000) + outline.tokens = estimate_tokens(render_digest(outline, filename=filename, upload_id=upload_id)) + return outline + + +def record_digest(digest: DocumentDigest) -> None: + """One content-free EMF record per digest build. Never raises.""" + logger.info( + "document_digest: status=%s format=%s %s=%d sections=%d tokens=%d abstract=%s ms=%d", + digest.status, digest.format, digest.unit, digest.count, len(digest.sections), + digest.tokens, bool(digest.abstract), digest.extractor_ms, + ) + try: + from apis.shared.observability.prompt_cache import prompt_cache_observability_enabled + from apis.shared.observability.emf import emit_emf_metrics + + if not prompt_cache_observability_enabled(): + return + emit_emf_metrics( + "AgentCoreStack/Compaction", + metrics={ + "DocumentDigestGenerated": 1, + "DocumentDigestTokens": int(digest.tokens), + "DocumentDigestMs": int(digest.extractor_ms), + }, + properties={ + "format": digest.format, + "outcome": ( + "failed" if digest.status != "ready" + else ("ready" if digest.abstract else "no_abstract") + ), + }, + units={"DocumentDigestTokens": "Count", "DocumentDigestMs": "Milliseconds"}, + ) + except Exception as e: # noqa: BLE001 + logger.debug("document_digest EMF skipped: %s", e) + + +__all__ = [ + "DIGEST_VERSION", + "DOCUMENT_DIGEST_MAX_SECTIONS", + "DOCUMENT_DIGEST_MAX_TOKENS", + "DOCUMENT_DIGEST_MODEL_ID", + "DigestSection", + "DocumentDigest", + "build_digest", + "document_digest_enabled", + "extract_outline", + "generate_abstract", + "looks_like_heading", + "record_digest", + "render_digest", + "text_sample", +] diff --git a/backend/src/apis/shared/files/document_read.py b/backend/src/apis/shared/files/document_read.py new file mode 100644 index 000000000..9d09535e8 --- /dev/null +++ b/backend/src/apis/shared/files/document_read.py @@ -0,0 +1,735 @@ +"""Document read service — page-range and pattern retrieval over uploaded documents. + +Backs the ``document_read`` agent tool (``agents/builtin_tools/document_read_tool.py``; +design in ``docs/specs/document-context-offload.md`` §4B). The tool is the +recovery path for a document that is no longer inline in the model's context: +history restore strips inline document bytes (Bedrock rejects duplicate +document names across a conversation), and the offload trigger will later +replace a document with a digest on purpose. Either way the model needs a way +to pull *part* of a document back at native fidelity. + +Three retrieval modes, modelled on Strands' ``retrieve_offloaded_content``: + +* **pages** — a PDF page range, re-assembled server-side into a new PDF that + contains only those pages and returned as a native ``document`` block. The + model sees the pages as it would the original (text layer plus page + images), not as flattened text. Hard-capped by ``max_pages`` so one call can + never re-inject a whole document. +* **pattern** — a case-insensitive regex over the document's text layer. + Returns the matching lines with their page numbers (PDF) or line / + paragraph numbers (text, DOCX), so the model can then ask for the pages + that matter. Bounded match count. +* **text** — bounded text for the text-family formats (txt / md / html via the + workspace read path, DOCX via a stdlib extractor), with ``offset`` + continuation. + +Plus a **list** mode (no ``upload_id``) enumerating the session's readable +documents, and an **index** mode (a PDF with neither range nor pattern) that +returns the page count and a short per-page snippet index. + +Every read goes through the DynamoDB user-files table (ownership by key shape, +``PK = USER#{userId}``) and reuses the workspace service's S3 client. No +operation accepts a model-supplied S3 key. Nothing here emits metrics or +touches the agent layer — the tool does that. +""" + +from __future__ import annotations + +import asyncio +import io +import logging +import os +import re +import time +import uuid +import zipfile +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Sequence, Tuple +from xml.etree import ElementTree + +from .models import FileMetadata, FileStatus, get_file_format, is_presentation_file, is_tabular_file +from .repository import get_file_upload_repository +from .workspace import ( + WORKSPACE_READ_MAX_BYTES, + WorkspaceError, + WorkspaceFileNotFoundError, + WorkspaceValidationError, + _get_owned_ready_file, + _require_identity, + _s3, + is_text_mime, + read_workspace_file, +) + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Bounds (env-backed; every one caps a per-turn payload) +# --------------------------------------------------------------------------- + +#: Default page-range size when the model does not say. 8 pages of a dense PDF +#: is ~10–20k tokens — a bounded, deliberate spend, not a whole document. +DOCUMENT_READ_MAX_PAGES = int(os.environ.get("DOCUMENT_READ_MAX_PAGES", 8)) +#: Absolute ceiling on one call's page count, whatever ``max_pages`` the model +#: asks for. The tool's contract is "one call cannot re-inject the document". +DOCUMENT_READ_HARD_MAX_PAGES = int(os.environ.get("DOCUMENT_READ_HARD_MAX_PAGES", 20)) +#: Pattern mode returns at most this many matching lines. +DOCUMENT_READ_MAX_MATCHES = int(os.environ.get("DOCUMENT_READ_MAX_MATCHES", 40)) +#: Index mode lists a snippet for at most this many pages. +DOCUMENT_READ_INDEX_PAGES = int(os.environ.get("DOCUMENT_READ_INDEX_PAGES", 40)) +#: Wall-clock bound on one pattern scan. The backstop behind +#: ``catastrophic_pattern``: a single ``re.search`` cannot be cancelled, but the +#: walk across lines and pages can be stopped, so a merely-slow pattern over a +#: long document degrades to a partial answer instead of hanging the turn to the +#: 600 s SSE timeout. ``0`` disables the bound. +DOCUMENT_READ_PATTERN_BUDGET_SECONDS = float( + os.environ.get("DOCUMENT_READ_PATTERN_BUDGET_SECONDS", 2.0) +) +#: Characters kept per matching / snippet line. +_LINE_CHARS = 200 +_SNIPPET_CHARS = 90 +#: Lines of context on either side of a pattern match. +_CONTEXT_LINES = 1 +#: Documents larger than this are not searched or sliced (the upload cap is +#: 4 MB, so this only guards a misconfigured bucket). +_MAX_DOCUMENT_BYTES = int(os.environ.get("DOCUMENT_READ_MAX_SOURCE_BYTES", 8 * 1024 * 1024)) + +#: Bedrock document formats the tool can read. Tabular and presentation files +#: have their own tools; images have no page/text structure to retrieve. +DOCUMENT_CLASS_FORMATS = frozenset({"pdf", "docx", "txt", "html", "md"}) +_TEXT_FORMATS = frozenset({"txt", "html", "md"}) + +_DOCX_MIME = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" +_W_NS = "{http://schemas.openxmlformats.org/wordprocessingml/2006/main}" + + +class DocumentReadError(WorkspaceError): + """A document_read failure surfaced conversationally by the tool.""" + + +# --------------------------------------------------------------------------- +# Classification +# --------------------------------------------------------------------------- + + +def document_format_for(mime_type: str, filename: str = "") -> Optional[str]: + """The Bedrock document format the tool can read this file as, or ``None``. + + ``None`` for tabular files (spreadsheet tools), presentations (PowerPoint + tools), images, and anything the upload allowlist does not know. + """ + if is_tabular_file(filename or "", mime_type or "") or is_presentation_file(filename or "", mime_type or ""): + return None + fmt = get_file_format((mime_type or "").lower().split(";")[0].strip()) + if fmt is None and is_text_mime(mime_type or ""): + fmt = "txt" + return fmt if fmt in DOCUMENT_CLASS_FORMATS else None + + +def is_document_class(mime_type: str, filename: str = "") -> bool: + """True when ``document_read`` has something to retrieve from this file.""" + return document_format_for(mime_type, filename) is not None + + +# --------------------------------------------------------------------------- +# Result shape +# --------------------------------------------------------------------------- + + +@dataclass +class DocumentReadResult: + """What one read produced. ``payload`` is JSON for the tool result; the + optional ``document_block`` is a native Bedrock document block appended + after it. The counters are what the tool records (content-free).""" + + mode: str + payload: Dict[str, Any] + document_block: Optional[Dict[str, Any]] = None + pages_returned: int = 0 + bytes_returned: int = 0 + format: Optional[str] = None + extra_blocks: List[Dict[str, Any]] = field(default_factory=list) + + +# --------------------------------------------------------------------------- +# Listing +# --------------------------------------------------------------------------- + + +async def list_session_documents(user_id: str, session_id: str) -> Dict[str, Any]: + """The session's READY documents the tool can read, newest first.""" + _require_identity(user_id, session_id) + files = await get_file_upload_repository().list_session_files(session_id, status=FileStatus.READY) + documents: List[Dict[str, Any]] = [] + for meta in files: + if meta.user_id != user_id: + continue + fmt = document_format_for(meta.mime_type, meta.filename) + if fmt is None: + continue + documents.append( + { + "upload_id": meta.upload_id, + "filename": meta.filename, + "format": fmt, + "size_bytes": meta.size_bytes, + "modes": _modes_for(fmt), + } + ) + return {"documents": documents, "count": len(documents)} + + +def _modes_for(fmt: str) -> List[str]: + if fmt == "pdf": + return ["page_range", "pattern"] + return ["pattern", "text"] + + +async def session_has_documents(user_id: str, session_id: str) -> bool: + """Whether the session has at least one READY document the tool can read. + + The tool-injection gate. One ``SessionIndex`` query; callers memoize the + positive answer because it is monotonic for practical purposes (a file, + once uploaded, stays unless the user deletes it). + """ + _require_identity(user_id, session_id) + files = await get_file_upload_repository().list_session_files(session_id, status=FileStatus.READY) + return any(meta.user_id == user_id and is_document_class(meta.mime_type, meta.filename) for meta in files) + + +async def session_has_tabular_files(user_id: str, session_id: str) -> bool: + """Whether the session has at least one READY spreadsheet (CSV/XLSX). + + The Spreadsheet Analysis auto-enable gate — the tabular counterpart of + :func:`session_has_documents`, same one-query cost and the same + "memoize the positive answer" contract for callers. Spreadsheets never go + inline, so a session holding one is a session whose turns need the + analysis tools to read it at all. + """ + _require_identity(user_id, session_id) + files = await get_file_upload_repository().list_session_files(session_id, status=FileStatus.READY) + return any(meta.user_id == user_id and is_tabular_file(meta.filename, meta.mime_type) for meta in files) + + +# --------------------------------------------------------------------------- +# Reading +# --------------------------------------------------------------------------- + + +def parse_page_range(value: Any) -> Optional[Tuple[int, int]]: + """``"4-7"`` / ``"4"`` / ``"4..7"`` / ``"4:7"`` / ``{"start": 4, "end": 7}`` + → ``(4, 7)`` (1-indexed, inclusive). ``None`` for empty input. Raises + ``WorkspaceValidationError`` on anything else.""" + if value is None: + return None + if isinstance(value, dict): + start, end = value.get("start"), value.get("end", value.get("start")) + elif isinstance(value, (list, tuple)) and len(value) in (1, 2): + start, end = value[0], value[-1] + elif isinstance(value, int): + start = end = value + else: + text = str(value).strip() + if not text: + return None + match = re.fullmatch(r"\s*(\d+)\s*(?:(?:-|\.\.|:|–|to)\s*(\d+))?\s*", text) + if not match: + raise WorkspaceValidationError( + f"Unrecognized page_range '{value}'. Use 'start-end' with 1-indexed page numbers, e.g. '4-7'." + ) + start, end = match.group(1), match.group(2) or match.group(1) + try: + start_i, end_i = int(start), int(end) + except (TypeError, ValueError): + raise WorkspaceValidationError(f"Unrecognized page_range '{value}'.") from None + if start_i < 1 or end_i < start_i: + raise WorkspaceValidationError( + f"Invalid page_range '{value}': pages are 1-indexed and end must not precede start." + ) + return start_i, end_i + + +def clamp_max_pages(max_pages: Any) -> int: + try: + requested = int(max_pages) if max_pages is not None else DOCUMENT_READ_MAX_PAGES + except (TypeError, ValueError): + requested = DOCUMENT_READ_MAX_PAGES + return max(1, min(requested, DOCUMENT_READ_HARD_MAX_PAGES)) + + +async def read_document( + user_id: str, + session_id: str, + upload_id: str, + *, + page_range: Any = None, + pattern: Optional[str] = None, + max_pages: Any = None, + offset: int = 0, +) -> DocumentReadResult: + """Read part of one document. See the module docstring for the modes.""" + _require_identity(user_id, session_id) + if not upload_id: + raise WorkspaceValidationError("upload_id is required (call with no arguments to list documents)") + if offset < 0: + raise WorkspaceValidationError("offset must be >= 0") + + meta = await _get_owned_ready_file(user_id, upload_id) + fmt = document_format_for(meta.mime_type, meta.filename) + if fmt is None: + raise DocumentReadError( + f"'{meta.filename}' is not a readable document. Spreadsheets go through " + "analyze_spreadsheet, presentations through read_powerpoint_presentation, " + "and images are only available inline." + ) + if meta.size_bytes > _MAX_DOCUMENT_BYTES: + raise DocumentReadError(f"'{meta.filename}' is too large to read ({meta.size_bytes} bytes).") + + pages = parse_page_range(page_range) + pattern_text = (pattern or "").strip() or None + limit = clamp_max_pages(max_pages) + + base = { + "upload_id": meta.upload_id, + "filename": meta.filename, + "format": fmt, + "size_bytes": meta.size_bytes, + } + + if fmt == "pdf": + raw = await _fetch_bytes(meta) + if pattern_text: + return await asyncio.to_thread(_pdf_pattern, raw, pattern_text, base) + if pages: + return await asyncio.to_thread(_pdf_pages, raw, pages, limit, base, meta.filename) + return await asyncio.to_thread(_pdf_index, raw, base) + + if pages: + raise WorkspaceValidationError( + f"page_range applies to PDFs only; '{meta.filename}' is {fmt}. Use pattern or offset instead." + ) + + if fmt == "docx": + raw = await _fetch_bytes(meta) + return await asyncio.to_thread(_docx_read, raw, pattern_text, offset, base) + + # txt / md / html + if pattern_text: + raw = await _fetch_bytes(meta) + return await asyncio.to_thread(_text_pattern, raw.decode("utf-8", errors="replace"), pattern_text, base, "line") + result = await read_workspace_file(user_id, upload_id, offset=offset) + payload = { + **base, + "mode": "text", + "content": result.get("content", ""), + "offset": result.get("offset", offset), + "truncated": bool(result.get("truncated")), + "next_offset": result.get("next_offset"), + } + return DocumentReadResult( + mode="text", payload=payload, bytes_returned=len(payload["content"].encode("utf-8")), format=fmt + ) + + +# --------------------------------------------------------------------------- +# S3 +# --------------------------------------------------------------------------- + + +def _get_object_bytes(bucket: str, key: str) -> bytes: + return _s3().get_object(Bucket=bucket, Key=key)["Body"].read() + + +async def _fetch_bytes(meta: FileMetadata) -> bytes: + raw = await asyncio.to_thread(_get_object_bytes, meta.s3_bucket, meta.s3_key) + if len(raw) > _MAX_DOCUMENT_BYTES: + raise DocumentReadError(f"'{meta.filename}' is too large to read ({len(raw)} bytes).") + return raw + + +# --------------------------------------------------------------------------- +# PDF (pypdfium2 — already a dependency for attachment thumbnails) +# --------------------------------------------------------------------------- + + +def _open_pdf(raw: bytes): + import pypdfium2 as pdfium # lazy: native lib, not needed on non-PDF paths + + try: + return pdfium.PdfDocument(io.BytesIO(raw)) + except Exception as e: # noqa: BLE001 + raise DocumentReadError(f"Could not open the PDF: {e}") from e + + +def _pdf_page_text(pdf, index: int) -> str: + page = pdf[index] + try: + textpage = page.get_textpage() + try: + return textpage.get_text_bounded() or "" + finally: + textpage.close() + finally: + page.close() + + +def _pdf_pages(raw: bytes, pages: Tuple[int, int], limit: int, base: Dict[str, Any], filename: str) -> DocumentReadResult: + import pypdfium2 as pdfium + + start, end = pages + pdf = _open_pdf(raw) + try: + count = len(pdf) + if start > count: + raise WorkspaceValidationError(f"page_range starts at {start} but the document has {count} pages.") + end = min(end, count) + truncated = False + if end - start + 1 > limit: + end = start + limit - 1 + truncated = True + sub = pdfium.PdfDocument.new() + try: + sub.import_pages(pdf, pages=list(range(start - 1, end))) + buffer = io.BytesIO() + sub.save(buffer) + finally: + sub.close() + finally: + pdf.close() + + sliced = buffer.getvalue() + returned = end - start + 1 + name = _unique_document_name(filename, f"p{start}-{end}") + payload = { + **base, + "mode": "pages", + "page_count": count, + "pages": {"start": start, "end": end}, + "pages_returned": returned, + "truncated_to_max_pages": truncated, + "next_start": end + 1 if (truncated and end < count) else None, + "page_numbering": ( + f"The attached document contains original pages {start}-{end} only; " + f"its page k is original page {start} + k - 1." + ), + } + block = {"document": {"format": "pdf", "name": name, "source": {"bytes": sliced}}} + return DocumentReadResult( + mode="pages", payload=payload, document_block=block, + pages_returned=returned, bytes_returned=len(sliced), format="pdf", + ) + + +def _pdf_pattern(raw: bytes, pattern: str, base: Dict[str, Any]) -> DocumentReadResult: + regex, note = _compile(pattern) + pdf = _open_pdf(raw) + # The clock starts after the open: lazily importing the pypdfium2 native + # library and parsing the document are setup, not scanning, and charging + # them to the scan's budget can report a timeout on pages never examined. + clock = _Budget(DOCUMENT_READ_PATTERN_BUDGET_SECONDS) + pages_searched = 0 + try: + count = len(pdf) + matches: List[Dict[str, Any]] = [] + pages_matched: List[int] = [] + for index in range(count): + if len(matches) >= DOCUMENT_READ_MAX_MATCHES or clock.out_of_time(): + break + pages_searched = index + 1 + lines = _pdf_page_text(pdf, index).splitlines() + page_no = index + 1 + hit = False + for entry in _grep_lines(lines, regex, DOCUMENT_READ_MAX_MATCHES - len(matches), clock): + entry["page"] = page_no + matches.append(entry) + hit = True + if hit: + pages_matched.append(page_no) + finally: + pdf.close() + payload = { + **base, + "mode": "pattern", + "pattern": pattern, + "page_count": count, + "pages_matched": pages_matched, + "match_count": len(matches), + "truncated": len(matches) >= DOCUMENT_READ_MAX_MATCHES, + "matches": matches, + "hint": "Call again with page_range to read the matching pages at full fidelity.", + } + if note: + payload["pattern_note"] = note + if clock.expired: + payload["timed_out"] = True + payload["pages_searched"] = pages_searched + payload["hint"] = ( + f"The search ran out of time after {pages_searched} of {count} pages; these are " + "the matches found so far. Use a simpler pattern, or read a page_range directly." + ) + return DocumentReadResult(mode="pattern", payload=payload, pages_returned=0, format="pdf") + + +def _pdf_index(raw: bytes, base: Dict[str, Any]) -> DocumentReadResult: + pdf = _open_pdf(raw) + try: + count = len(pdf) + index: List[Dict[str, Any]] = [] + for i in range(min(count, DOCUMENT_READ_INDEX_PAGES)): + snippet = "" + for line in _pdf_page_text(pdf, i).splitlines(): + line = line.strip() + if line: + snippet = line[:_SNIPPET_CHARS] + break + index.append({"page": i + 1, "snippet": snippet}) + finally: + pdf.close() + payload = { + **base, + "mode": "index", + "page_count": count, + "page_index": index, + "index_truncated": count > DOCUMENT_READ_INDEX_PAGES, + "hint": ( + f"Read specific pages with page_range (max {DOCUMENT_READ_MAX_PAGES} per call, " + f"hard cap {DOCUMENT_READ_HARD_MAX_PAGES}) or locate them first with pattern." + ), + } + return DocumentReadResult(mode="index", payload=payload, format="pdf") + + +# --------------------------------------------------------------------------- +# DOCX (stdlib: zipfile + ElementTree over word/document.xml) +# --------------------------------------------------------------------------- + + +def docx_paragraphs(raw: bytes) -> List[str]: + """Paragraph text of a .docx, in document order. Tables contribute their + cell paragraphs; runs are joined; tabs and line breaks become whitespace. + Good enough for pattern search and bounded reading, not a renderer.""" + try: + with zipfile.ZipFile(io.BytesIO(raw)) as archive: + xml = archive.read("word/document.xml") + except (zipfile.BadZipFile, KeyError) as e: + raise DocumentReadError(f"Could not open the Word document: {e}") from e + try: + root = ElementTree.fromstring(xml) + except ElementTree.ParseError as e: + raise DocumentReadError(f"Could not parse the Word document: {e}") from e + paragraphs: List[str] = [] + for para in root.iter(f"{_W_NS}p"): + parts: List[str] = [] + for node in para.iter(): + if node.tag == f"{_W_NS}t": + parts.append(node.text or "") + elif node.tag in (f"{_W_NS}tab",): + parts.append("\t") + elif node.tag in (f"{_W_NS}br", f"{_W_NS}cr"): + parts.append(" ") + text = "".join(parts).strip() + if text: + paragraphs.append(text) + return paragraphs + + +def _docx_read(raw: bytes, pattern: Optional[str], offset: int, base: Dict[str, Any]) -> DocumentReadResult: + paragraphs = docx_paragraphs(raw) + if pattern: + result = _text_pattern("\n".join(paragraphs), pattern, base, "paragraph") + result.format = "docx" + return result + text = "\n\n".join(paragraphs) + encoded = text.encode("utf-8") + if offset and offset >= len(encoded): + raise WorkspaceValidationError(f"offset {offset} is beyond the end of the document text ({len(encoded)} bytes)") + chunk = encoded[offset: offset + WORKSPACE_READ_MAX_BYTES] + end = offset + len(chunk) + truncated = end < len(encoded) + payload = { + **base, + "mode": "text", + "paragraph_count": len(paragraphs), + "content": chunk.decode("utf-8", errors="replace"), + "offset": offset, + "truncated": truncated, + "next_offset": end if truncated else None, + } + return DocumentReadResult(mode="text", payload=payload, bytes_returned=len(chunk), format="docx") + + +# --------------------------------------------------------------------------- +# Text helpers +# --------------------------------------------------------------------------- + + +def _scan_flags(text: str): + """Yield ``(index, char)`` for the regex-significant characters of ``text`` + — skipping escaped characters and the contents of ``[...]`` classes, where + ``*``, ``+``, ``|`` and parentheses are literals.""" + i, in_class, n = 0, False, len(text) + while i < n: + ch = text[i] + if ch == "\\": + i += 2 + continue + if in_class: + if ch == "]": + in_class = False + i += 1 + continue + if ch == "[": + in_class = True + i += 1 + continue + yield i, ch + i += 1 + + +def _unbounded_quantifier_at(pattern: str, i: int) -> bool: + """Is there an unbounded quantifier (``*``, ``+`` or ``{n,}``) at ``i``?""" + if i >= len(pattern): + return False + if pattern[i] in "*+": + return True + if pattern[i] == "{": + close = pattern.find("}", i) + return close != -1 and pattern[i + 1: close].endswith(",") + return False + + +def _has_unbounded_quantifier(body: str) -> bool: + return any( + _unbounded_quantifier_at(body, i) for i, ch in _scan_flags(body) if ch in "*+{" + ) + + +def catastrophic_pattern(pattern: str) -> bool: + """Whether ``pattern`` has the nested-quantifier shape that makes Python's + backtracking engine run in exponential time. + + ``(a+)+``, ``(\\w+\\s*)*``, ``(a+|b){2,}`` — an unbounded quantifier applied to + a group that itself contains one. Measured on the real engine: ``(a+)+$`` + against ``"a"*24 + "!"`` takes 0.87 s, 26 takes 3.5 s, 28 takes 14 s. + Extracted PDF lines run 60–100 characters, so such a pattern never returns. + + Deliberately narrow. It flags only the unambiguous family, so ordinary + patterns — ``\\d+``, ``(invoice|receipt)``, ``(foo)+`` — are never refused. + Shapes it does not catch (overlapping literal alternations like ``(a|a)+``, + backreference blowups) are bounded by the wall-clock budget instead, not by + this test. + """ + if not pattern: + return False + starts: List[int] = [] + for i, ch in _scan_flags(pattern): + if ch == "(": + starts.append(i) + elif ch == ")" and starts: + body = pattern[starts.pop() + 1: i] + if _unbounded_quantifier_at(pattern, i + 1) and _has_unbounded_quantifier(body): + return True + return False + + +def _compile(pattern: str) -> Tuple["re.Pattern[str]", Optional[str]]: + """``(regex, note)``. The note is non-``None`` when the pattern was searched + literally instead of as a regex, and is carried to the model on the payload + so a silently different result is never presented as the requested one. + + Two reasons to fall back, both degrading rather than erroring: the pattern + does not compile, or it has the nested-quantifier shape that would hang the + turn (``catastrophic_pattern``).""" + if catastrophic_pattern(pattern): + logger.warning("document_read: refusing catastrophic pattern, searching literally") + return re.compile(re.escape(pattern), re.IGNORECASE), ( + "This pattern nests one unbounded quantifier inside another, which can take " + "exponential time to match, so it was searched as literal text instead. " + "Rewrite it without the nesting (for example '\\w+' rather than '(\\w+)+')." + ) + try: + return re.compile(pattern, re.IGNORECASE), None + except re.error: + return re.compile(re.escape(pattern), re.IGNORECASE), ( + "This pattern is not a valid regular expression, so it was searched as literal text." + ) + + +class _Budget: + """Wall-clock bound on one pattern scan. + + The backstop behind ``catastrophic_pattern``: it cannot interrupt a single + ``re.search`` — CPython exposes no way to cancel one, and the scan runs in a + worker thread where signals are unavailable — but it stops the *walk*, so a + merely-slow pattern over a 200-page document cannot run away. A scan that + runs out reports ``timed_out`` with the matches it already has. + """ + + __slots__ = ("_deadline", "expired") + + def __init__(self, seconds: float = 0.0) -> None: + self._deadline = (time.monotonic() + seconds) if seconds > 0 else None + self.expired = False + + def out_of_time(self) -> bool: + if self._deadline is not None and time.monotonic() > self._deadline: + self.expired = True + return self.expired + + +def _grep_lines( + lines: Sequence[str], + regex: "re.Pattern[str]", + budget: int, + clock: Optional[_Budget] = None, +) -> List[Dict[str, Any]]: + """Matching lines with ``_CONTEXT_LINES`` of context, 1-indexed.""" + out: List[Dict[str, Any]] = [] + for i, line in enumerate(lines): + if len(out) >= budget or (clock is not None and clock.out_of_time()): + break + if not regex.search(line): + continue + before = [ln.strip()[:_LINE_CHARS] for ln in lines[max(0, i - _CONTEXT_LINES): i]] + after = [ln.strip()[:_LINE_CHARS] for ln in lines[i + 1: i + 1 + _CONTEXT_LINES]] + out.append({"line": i + 1, "text": line.strip()[:_LINE_CHARS], "before": before, "after": after}) + return out + + +def _text_pattern(text: str, pattern: str, base: Dict[str, Any], unit: str) -> DocumentReadResult: + regex, note = _compile(pattern) + lines = text.splitlines() + clock = _Budget(DOCUMENT_READ_PATTERN_BUDGET_SECONDS) + matches = _grep_lines(lines, regex, DOCUMENT_READ_MAX_MATCHES, clock) + payload = { + **base, + "mode": "pattern", + "pattern": pattern, + "unit": unit, + f"{unit}_count": len(lines), + "match_count": len(matches), + "truncated": len(matches) >= DOCUMENT_READ_MAX_MATCHES, + "matches": matches, + } + if note: + payload["pattern_note"] = note + if clock.expired: + payload["timed_out"] = True + payload["hint"] = ( + "The search ran out of time; these are the matches found so far. " + "Use a simpler pattern, or read the document directly." + ) + return DocumentReadResult(mode="pattern", payload=payload, format=base.get("format")) + + +_NAME_UNSAFE = re.compile(r"[^a-zA-Z0-9\s\-\(\)\[\]]") + + +def _unique_document_name(filename: str, suffix: str) -> str: + """A Bedrock-safe document name that cannot collide with any other block in + the conversation. Bedrock allows alphanumerics, whitespace, hyphens, + parentheses and square brackets, and rejects duplicate names across the + whole message history — so every slice gets a fresh random tail.""" + stem = filename.rsplit(".", 1)[0] if "." in filename else filename + stem = re.sub(r"\s+", " ", _NAME_UNSAFE.sub(" ", stem)).strip()[:40] or "document" + return f"{stem} {suffix} {uuid.uuid4().hex[:6]}" diff --git a/backend/src/apis/shared/files/document_tokens.py b/backend/src/apis/shared/files/document_tokens.py new file mode 100644 index 000000000..ea5f12b48 --- /dev/null +++ b/backend/src/apis/shared/files/document_tokens.py @@ -0,0 +1,98 @@ +"""Token weight of an inline attachment — PDFs are dual-encoded, bytes/4 is not. + +``docs/specs/document-context-offload.md`` §6.1 reports a document's weight as +``documentTokens`` and §4C decides whether an eviction is worth its prefix +re-write from the same number. Both used the compaction estimator's ``bytes/4``, +which is **wrong for PDFs by an order of magnitude**. + +Why: Bedrock understands a PDF page as an image *and* an extracted text layer +(the dual encoding §3 of the spec relies on, and the reason the design returns +native blocks rather than flattened text). The image channel dominates, and it +has nothing to do with the file's byte size — a 27 KB 60-page PDF and a 27 MB +60-page scan cost about the same. ``bytes/4`` sees only the bytes. + +Measured on dev 2026-09-16 (session ``61de2256``): a 27,578-byte, 60-page PDF +produced a **109.1K-token** first write against a 14.6K static prefix, so the +document was ≈94K tokens. ``bytes/4`` reported **6,894** — low by ~14×. At that +error the spec's "document share of the prefix" reads ~6% where the truth was +~86%, which inverts the ship/abandon call in the evaluation spec §4.2. + +The model here: **pages x ``PDF_PAGE_TOKEN_ESTIMATE``**, floored by ``bytes/4`` +so a byte-heavy PDF is never scored below the old number. The default (1,500) is +``compaction_policy.IMAGE_TOKEN_ESTIMATE``'s own rationale — Anthropic image +tokens are ~(w*h)/750, and a letter page at ~1000x1100 lands there. Against the +measurement above it is 1,500 vs 1,573 actual per page: ~5% low, deliberately, +because this number gates an eviction and over-stating it would evict documents +whose re-write has not earned it. + +Still a heuristic — Bedrock reports no per-block usage — but one whose dominant +term is now the right term. Non-PDF formats keep ``bytes/4``: Bedrock extracts +them to text, so there is no image channel to miss. + +Never raises: an unparseable PDF falls back to ``bytes/4``, which is exactly +today's behaviour. +""" + +from __future__ import annotations + +import logging +import os +from typing import Any, Optional + +logger = logging.getLogger(__name__) + +#: Same heuristic as ``compaction_policy.CHARS_PER_TOKEN``; duplicated because +#: ``apis.shared`` must not import the agent layer. +CHARS_PER_TOKEN = 4 + +#: Estimated tokens for one rendered PDF page (the image channel). Env-tunable +#: so the number can be re-fit from measured rows without a deploy. +PDF_PAGE_TOKEN_ESTIMATE = int(os.environ.get("PDF_PAGE_TOKEN_ESTIMATE", 1_500)) + + +def pdf_page_count(raw: bytes) -> Optional[int]: + """Page count of ``raw``, or ``None`` if it will not open. + + Costs ~0.1 ms even for a 200-page file (measured), so callers on the + per-turn path do not need to memoize it. + """ + if not isinstance(raw, (bytes, bytearray)) or not raw: + return None + try: + import io + + import pypdfium2 as pdfium + + pdf = pdfium.PdfDocument(io.BytesIO(bytes(raw))) + try: + return len(pdf) + finally: + pdf.close() + except Exception: # noqa: BLE001 - estimator must never raise + logger.debug("pdf_page_count: unparseable PDF, falling back to bytes/4", exc_info=True) + return None + + +def estimate_document_tokens(fmt: Any, raw: Any) -> int: + """Estimated tokens an inline document block costs in the prefix. + + PDFs: ``max(pages * PDF_PAGE_TOKEN_ESTIMATE, bytes/4)``. Everything else: + ``bytes/4``. + """ + if not isinstance(raw, (bytes, bytearray)): + return 0 + byte_tokens = len(raw) // CHARS_PER_TOKEN + if str(fmt or "").lower() != "pdf": + return byte_tokens + pages = pdf_page_count(raw) + if not pages: + return byte_tokens + return max(pages * PDF_PAGE_TOKEN_ESTIMATE, byte_tokens) + + +__all__ = [ + "CHARS_PER_TOKEN", + "PDF_PAGE_TOKEN_ESTIMATE", + "estimate_document_tokens", + "pdf_page_count", +] diff --git a/backend/src/apis/shared/files/file_resolver.py b/backend/src/apis/shared/files/file_resolver.py index 6ce4a7b5f..7af143e95 100644 --- a/backend/src/apis/shared/files/file_resolver.py +++ b/backend/src/apis/shared/files/file_resolver.py @@ -53,7 +53,7 @@ async def resolve_files( self, user_id: str, upload_ids: List[str], - max_files: int = 5 + max_files: Optional[int] = 5 ) -> List[ResolvedFileContent]: """ Resolve upload IDs to file content objects. @@ -61,7 +61,11 @@ async def resolve_files( Args: user_id: Owner user ID (for authorization) upload_ids: List of upload IDs to resolve - max_files: Maximum files to process (Bedrock limit is 5) + max_files: Backstop on how many IDs are fetched. ``None`` fetches + every ID given. The chat route applies the per-message cap + *before* calling this so it can tell the user which files + were left out; this cap only exists so no caller can fan out + an unbounded number of S3 reads by accident. Returns: List of ResolvedFileContent objects with base64-encoded bytes @@ -71,7 +75,15 @@ async def resolve_files( """ resolved_files = [] - for upload_id in upload_ids[:max_files]: + if max_files is not None and len(upload_ids) > max_files: + logger.warning( + "resolve_files truncating %d upload ID(s) to %d", + len(upload_ids), + max_files, + ) + upload_ids = upload_ids[:max_files] + + for upload_id in upload_ids: try: file_content = await self._resolve_single_file(user_id, upload_id) if file_content: diff --git a/backend/src/apis/shared/files/models.py b/backend/src/apis/shared/files/models.py index ae0b7459c..81b8b491d 100644 --- a/backend/src/apis/shared/files/models.py +++ b/backend/src/apis/shared/files/models.py @@ -165,6 +165,41 @@ def is_presentation_file(filename: str, mime_type: str) -> bool: os.environ.get("INLINE_DOCUMENT_MAX_BYTES", 4 * 1024 * 1024) # 4MB ) +# A turn's inline attachments are persisted as ONE message, and the message — +# not the file — is what AgentCore Memory bounds. Anything over the SDK's +# ~72 KB conversational limit is written as a base64 ``blob`` payload, so raw +# attachment bytes inflate by 4/3 on the way in and are then held to the +# 10 MB event quota. 10 MB × 3/4 = 7.5 MB of raw bytes per turn is the break +# point. Above it ``create_message`` raises ``SessionException`` — a hole in +# history — which is strictly worse than the per-file oversized note, so the +# turn is trimmed to this budget *before* it is built. Prod measurement +# (docs/specs/document-context-offload-validation.md, Claim 7): ~1.3–1.4% of +# attachment turns exceed it, several with only 3–4 files, so the per-file +# cap above and the SPA's 5-file cap do not protect on their own. +# ``0`` (or any non-positive value) disables the aggregate budget. +# +# Why 7.0 MB and not the 7.5 MB the arithmetic above suggests: base64 of N raw +# bytes is ``4*ceil(N/3)``, so 7,500,000 encodes to **exactly 10,000,000** — the +# quota itself, with nothing left for the event's JSON envelope (role, content +# keys, the prompt text block, per-file metadata, the wrapper). A guard whose +# default sits precisely on the break point it exists to stay under does not +# prevent the failure it was written for. 7,000,000 encodes to 9,333,336 and +# leaves ~666 KB of headroom, which comfortably covers the envelope while still +# admitting every attachment turn measured in prod (p90 cluster 2.58 MB, largest +# legitimate 29.89 MB — already over either number and correctly trimmed). +INLINE_ATTACHMENTS_MAX_TOTAL_BYTES = int( + os.environ.get("INLINE_ATTACHMENTS_MAX_TOTAL_BYTES", 7_000_000) # 7.0MB +) + +# Files per message. The SPA enforces the same number client-side +# (``MAX_FILES_PER_MESSAGE`` in file-upload.service.ts); this is the server +# side of it, shared by the ``file_upload_ids`` resolver and the direct +# ``files`` path so a sixth file is reported to the user instead of silently +# truncated. ``0`` (or any non-positive value) disables the count cap. +MAX_FILES_PER_MESSAGE = int( + os.environ.get("FILE_UPLOAD_MAX_FILES_PER_MESSAGE", 5) +) + # ============================================================================= # Database Models (stored in DynamoDB) @@ -204,6 +239,14 @@ class FileMetadata(BaseModel): # never part of an access decision. source: str = Field(default="upload", description="Origin of the file") + # DocumentDigest (docs/specs/document-context-offload.md §4A), built once + # when a document upload completes and stored as a plain map — see + # ``apis.shared.files.document_digest``. ``None`` = never generated (files + # uploaded before PR-2, non-documents, or the flag off). Carries model + # prose (``abstract``) and heading text (``sections``): content-bearing, + # denylisted in the admin projections. + digest: Optional[dict] = Field(None, description="DocumentDigest map, when generated") + # Timestamps created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc)) updated_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc)) @@ -230,7 +273,7 @@ def to_dynamo_item(self) -> dict: if ttl_value is None: ttl_value = int(self.created_at.timestamp()) + (365 * 24 * 60 * 60) - return { + item = { "PK": f"USER#{self.user_id}", "SK": f"FILE#{self.upload_id}", "GSI1PK": f"CONV#{self.session_id}", @@ -250,6 +293,9 @@ def to_dynamo_item(self) -> dict: "updatedAt": to_iso(self.updated_at), "ttl": ttl_value, } + if self.digest: + item["digest"] = self.digest + return item @classmethod def from_dynamo_item(cls, item: dict) -> "FileMetadata": @@ -271,6 +317,7 @@ def from_dynamo_item(cls, item: dict) -> "FileMetadata": created_at=from_iso(created_at) if created_at else datetime.now(timezone.utc), updated_at=from_iso(updated_at) if updated_at else datetime.now(timezone.utc), ttl=item.get("ttl"), + digest=item.get("digest") if isinstance(item.get("digest"), dict) else None, ) @@ -377,6 +424,58 @@ class TextSnippetResponse(BaseModel): model_config = ConfigDict(populate_by_name=True) +class SheetPreview(BaseModel): + """One worksheet, read into display-ready strings.""" + + name: str = Field(..., description="Worksheet name as it appears on the tab") + headers: List[str] = Field( + ..., description="First row of the sheet, used as column labels" + ) + rows: List[List[str]] = Field( + ..., description="Body rows, each padded to len(headers)" + ) + total_rows: int = Field( + ..., + alias="totalRows", + description="Body rows the sheet claims to have, which may exceed len(rows)", + ) + truncated: bool = Field( + ..., description="True when a cap stopped the read short of the sheet's end" + ) + truncated_by: Optional[str] = Field( + None, + alias="truncatedBy", + description="Which cap fired: 'rows' or 'columns'", + ) + + model_config = ConfigDict(populate_by_name=True) + + +class SheetPreviewResponse(BaseModel): + """Response for GET /api/files/{uploadId}/sheet-preview.""" + + upload_id: str = Field(..., alias="uploadId") + filename: str + sheets: List[SheetPreview] = Field( + ..., description="Visible worksheets, in workbook order" + ) + truncated: bool = Field( + ..., + description="True when any sheet was cut short or sheets were dropped", + ) + + model_config = ConfigDict(populate_by_name=True) + + +# MIME types the spreadsheet reader can turn into a grid. Deliberately +# only OOXML: the pre-2007 .xls binary format needs a different library +# (xlrd), and it is not worth one for a format nothing in the product +# generates. +SHEET_PREVIEW_MIME_TYPES = frozenset({ + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", +}) + + # MIME types the thumbnail renderer can currently produce a preview image for. # Callers should consult this set before invoking the thumbnail endpoint to # avoid hammering the service for unsupported types. diff --git a/backend/src/apis/shared/files/repository.py b/backend/src/apis/shared/files/repository.py index cce1a1dc5..3d3ba3f16 100644 --- a/backend/src/apis/shared/files/repository.py +++ b/backend/src/apis/shared/files/repository.py @@ -139,6 +139,40 @@ async def update_file_status( logger.error(f"Error updating file status {upload_id}: {e}") raise + async def update_file_digest( + self, user_id: str, upload_id: str, digest: dict + ) -> Optional[FileMetadata]: + """Store a ``DocumentDigest`` map on the file row (``SET digest``). + + Idempotent last-write-wins; ``None`` when the row no longer exists + (the file was deleted while the digest was being built). + """ + return self.update_file_digest_sync(user_id, upload_id, digest) + + def update_file_digest_sync( + self, user_id: str, upload_id: str, digest: dict + ) -> Optional[FileMetadata]: + """Synchronous body of :meth:`update_file_digest` — the restore path + persists a lazily built digest from synchronous code (see + :meth:`list_session_files_sync`).""" + try: + response = self._table.update_item( + Key={"PK": f"USER#{user_id}", "SK": f"FILE#{upload_id}"}, + UpdateExpression="SET digest = :digest, updatedAt = :now", + ExpressionAttributeValues={ + ":digest": self._convert_floats_to_decimals(dict(digest)), + ":now": utc_now_iso(), + }, + ConditionExpression="attribute_exists(PK)", + ReturnValues="ALL_NEW", + ) + return FileMetadata.from_dynamo_item(response["Attributes"]) + except ClientError as e: + if e.response["Error"]["Code"] == "ConditionalCheckFailedException": + return None + logger.error(f"Error updating file digest {upload_id}: {e}") + raise + async def delete_file(self, user_id: str, upload_id: str) -> Optional[FileMetadata]: """ Delete a file metadata record. @@ -257,6 +291,19 @@ async def list_session_files( Returns: List of FileMetadata """ + return self.list_session_files_sync(session_id, status) + + def list_session_files_sync( + self, session_id: str, status: Optional[FileStatus] = None + ) -> List[FileMetadata]: + """Synchronous body of :meth:`list_session_files`. + + The session manager's restore path (``TurnBasedSessionManager.initialize``) + runs synchronously inside the Strands agent constructor, under a + running event loop it cannot re-enter, and needs the session's upload + rows to rehydrate stripped documents. boto3 is synchronous anyway; the + async method is a thin wrapper over this. + """ try: query_params = { "IndexName": "SessionIndex", diff --git a/backend/src/apis/shared/middleware/compression.py b/backend/src/apis/shared/middleware/compression.py new file mode 100644 index 000000000..46911ff55 --- /dev/null +++ b/backend/src/apis/shared/middleware/compression.py @@ -0,0 +1,164 @@ +"""StreamSafeGZipMiddleware — gzip JSON responses without touching SSE. + +Nothing compressed app-api responses before this. CloudFront can't do it for +us: the `/api/*` behaviour is deliberately `compress: false` +(`spa-distribution-construct.ts`) precisely so CloudFront never buffers a +`text/event-stream`, and that decision stands — compression belongs at the +origin, where the response's own content type is known, not at an edge that +has to guess from a path pattern. + +The payoff is on the JSON surface, which is most of what app-api serves; the +risk is entirely on the SSE surface, which is the product's main path (the +chat proxy in `app_api/chat/proxy_routes.py`, plus assistants and +`chat/converse_routes.py`). So this middleware is defined by what it refuses +to touch. + +**Two things are passed straight through, untouched:** + +1. Responses whose `Content-Type` is in :data:`EXCLUDED_CONTENT_TYPES` — + `text/event-stream` first and foremost, plus payload types that are + already compressed on the wire (zip archives, raster images, media, + web fonts), where gzip costs CPU and returns nothing. +2. Responses that already carry a `Content-Encoding`, whatever it is. That + header is the origin saying "this body is already encoded"; re-encoding + it would produce a body no client can read. + +Starlette's own `GZipMiddleware` skips both of those as of its +`DEFAULT_EXCLUDED_CONTENT_TYPES`/`content_encoding_set` checks, and this +class reuses that machinery rather than reimplementing it. What it adds is +the part Starlette does not do: **it sends `http.response.start` eagerly.** + +Starlette's responders withhold the response-start message until the first +`http.response.body` arrives, because until then they can't know whether +they'll need to set `Content-Encoding` and drop `Content-Length`. That is +correct for a body they might compress, and wrong for one they've already +decided not to. On an SSE turn the first body chunk is the model's first +token — seconds away, or up to `_SSE_KEEPALIVE_SECONDS` if the turn opens +with a silent tool call — and today those response headers go out +immediately. Buffering them until the first event would delay every client's +"stream is open" transition behind the agent's thinking time, on the one +path in this service least worth adding a layer to (the same reasoning that +made `ProxiedRedirectMiddleware` raw ASGI rather than `BaseHTTPMiddleware`). + +So: as soon as the response-start message identifies an excluded response, +this forwards it and flips into pure passthrough for the rest of the +exchange. Everything else — threshold, `Vary`, streaming compression, +`http.response.pathsend` — is Starlette's, unmodified. + +Non-HTTP scopes (the voice WebSocket proxy) never reach a responder at all. +""" + +from __future__ import annotations + +from starlette.datastructures import Headers +from starlette.middleware.gzip import ( + DEFAULT_EXCLUDED_CONTENT_TYPES, + GZipMiddleware, + GZipResponder, + IdentityResponder, +) +from starlette.types import ASGIApp, Message, Receive, Scope, Send + +#: Content-type prefixes this middleware never compresses. +#: +#: `text/event-stream` is the correctness entry — gzipping an SSE stream +#: delays or breaks event flushes. It comes from Starlette's own default so +#: the two can't drift; the rest are efficiency entries, payload types that +#: arrive already compressed and would only burn CPU for ~0% gain. Matched +#: with `str.startswith`, so a charset parameter (`text/event-stream; +#: charset=utf-8`, which Starlette appends to every `text/*` media type) +#: still matches. +#: +#: Deliberately *not* excluded: `application/pdf` and `image/svg+xml`, both +#: of which compress usefully and both of which CloudFront itself lists as +#: compressible. +EXCLUDED_CONTENT_TYPES: tuple[str, ...] = ( + *DEFAULT_EXCLUDED_CONTENT_TYPES, + "application/gzip", + "application/x-7z-compressed", + "application/x-bzip2", + "application/x-gzip", + "application/x-zip-compressed", + "application/zip", + "audio/", + "font/woff", + "image/avif", + "image/gif", + "image/jpeg", + "image/png", + "image/webp", + "video/", +) + + +def is_excluded(headers: Headers) -> bool: + """Whether this response must be passed through uncompressed. + + True for an already-encoded body and for any content type in + :data:`EXCLUDED_CONTENT_TYPES`. A response with no `Content-Type` at all + is not excluded — Starlette's size threshold already covers the empty + bodies that case usually means. + """ + if "content-encoding" in headers: + return True + return headers.get("content-type", "").startswith(EXCLUDED_CONTENT_TYPES) + + +class _EagerPassthroughMixin: + """Forward an excluded response's start message without buffering it. + + Mixed in ahead of Starlette's responders so `super()` still owns every + response this does *not* claim. + """ + + #: Class-level default; flipped on the instance by the first + #: `http.response.start` that turns out to be excluded. + _passthrough = False + + async def send_with_compression(self, message: Message) -> None: + if not self._passthrough: + if message["type"] != "http.response.start": + await super().send_with_compression(message) + return + if not is_excluded(Headers(raw=message["headers"])): + await super().send_with_compression(message) + return + # Excluded: nothing downstream will alter the headers, so the + # client can have them now rather than when the first chunk + # lands. `started` keeps the base class consistent in case + # anything else consults it. + self._passthrough = True + self.started = True + await self.send(message) + + +class _IdentityPassthroughResponder(_EagerPassthroughMixin, IdentityResponder): + """Non-gzip clients: adds `Vary`, compresses nothing.""" + + +class _GZipPassthroughResponder(_EagerPassthroughMixin, GZipResponder): + """gzip-capable clients: compresses everything not excluded.""" + + +class StreamSafeGZipMiddleware(GZipMiddleware): + """`GZipMiddleware` that never buffers an excluded response. + + Same constructor and same behaviour for compressible responses; see the + module docstring for what changes and why. + """ + + async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: + if scope["type"] != "http": + # WebSockets (voice mode) and lifespan: not ours. + await self.app(scope, receive, send) + return + + responder: ASGIApp + if "gzip" in Headers(scope=scope).get("Accept-Encoding", ""): + responder = _GZipPassthroughResponder( + self.app, self.minimum_size, compresslevel=self.compresslevel + ) + else: + responder = _IdentityPassthroughResponder(self.app, self.minimum_size) + + await responder(scope, receive, send) diff --git a/backend/src/apis/shared/observability/build_stages.py b/backend/src/apis/shared/observability/build_stages.py new file mode 100644 index 000000000..eedb04b80 --- /dev/null +++ b/backend/src/apis/shared/observability/build_stages.py @@ -0,0 +1,83 @@ +"""A per-turn recorder for stages inside the agent build. + +WHY THIS EXISTS +--------------- +`docs/specs/turn-latency-preamble.md` decomposed the `preamble` stage into five +sub-stages and the measurement immediately named its owner — five reads of one +DynamoDB row — which three PRs then removed, taking the warm preamble from +455ms to 95ms. + +`agent_build` is now the largest number left in the same log line: **~2950ms on +a cold turn**, against 1-47ms warm. Nothing says which part of it that is. The +spec's standing hypothesis was the serial MCP `tools/list` pre-flight, and that +hypothesis is *unverified* — dev loads exactly one external MCP server, and +`AgentFactory.create_agent` receives `tools` and `session_manager` already +built, so the expensive work is upstream of it in the agent's own constructor. + +This module is the same move that worked for the preamble: measure first. + +WHY A CONTEXTVAR, WHEN PR-2 ARGUED AGAINST IMPLICIT STATE +--------------------------------------------------------- +PR-2 deliberately threaded its session snapshot as an explicit parameter and +rejected an implicit per-request memo. This does the opposite, and the +asymmetry is the point rather than an inconsistency: + +- there, implicit state that went wrong meant **stale session data in + production** — the bug CLAUDE.md names and this repo has shipped twice; +- here, implicit state that goes wrong means **a timing number is missing or + attributed to the wrong stage**. Nothing the user sees, nothing persisted, + nothing that reaches the model. + +Against that, the explicit alternative costs a kwarg threaded through +`get_agent` -> a type registry -> three agent classes, two of which do not +accept the same constructor arguments (`VoiceAgent` already needs a special +case for `accessible_skill_ids`). That is a lot of production signature for a +measurement, and every one of those signatures is a place to get it wrong. + +NOT propagated across threads, deliberately. `contextvars` do not cross into a +`ThreadPoolExecutor`, and the MCP load path explicitly crosses one. Every mark +here is taken on the calling thread, so the boundary is invisible — but a +future caller that tries to mark from inside that executor will silently record +nothing, which is why this note exists. +""" + +from __future__ import annotations + +import contextvars +import logging +from typing import Callable, Optional + +logger = logging.getLogger(__name__) + +_recorder: contextvars.ContextVar[Optional[Callable[[str], None]]] = contextvars.ContextVar( + "agent_build_stage_recorder", default=None +) + + +def set_stage_recorder(recorder: Optional[Callable[[str], None]]) -> contextvars.Token: + """Install the recorder for the current context. Returns a reset token.""" + return _recorder.set(recorder) + + +def reset_stage_recorder(token: contextvars.Token) -> None: + """Restore whatever recorder was installed before ``set_stage_recorder``.""" + try: + _recorder.reset(token) + except Exception: # noqa: BLE001 - a measurement must not break a turn + logger.debug("Stage recorder reset skipped", exc_info=True) + + +def mark_stage(stage: str) -> None: + """Close the build stage that just finished. + + A no-op when nothing is installed, which is every caller outside the + inference-api turn path — tests, the scheduled-runs Lambda, app-api. Never + raises: the whole module is a measurement, and a measurement that can break + the thing it measures is worse than no measurement. + """ + try: + recorder = _recorder.get() + if recorder is not None: + recorder(stage) + except Exception: # noqa: BLE001 + logger.debug("Build stage mark skipped", exc_info=True) diff --git a/backend/src/apis/shared/observability/content_policy.py b/backend/src/apis/shared/observability/content_policy.py index 795485d28..5e8dad291 100644 --- a/backend/src/apis/shared/observability/content_policy.py +++ b/backend/src/apis/shared/observability/content_policy.py @@ -60,10 +60,14 @@ "stateReason", "lastError", "errorDetail", + # F# feedback rows, judged: the evaluator's prose quotes the conversation + "explanation", # FILE# upload rows "filename", # user-chosen "s3Key", # embeds the filename "s3Uri", + "digest.abstract", # model-generated abstract of the document + "digest.sections", # heading text lifted from the document }) #: The one path a content-free reader may *request* but must never *return*. @@ -127,6 +131,21 @@ def is_content_bearing(path: str) -> bool: "toolCallCount", "toolErrorCount", "compactionCount", + # Compaction decisions by kind (per-call ledger rolled up; see + # `TurnBasedSessionManager.record_compaction_event`) + "compactionAppliedCount", + "compactionForcedCount", + "compactionFloorUnreachableCount", + # Message-feedback rollups (live F# rows written while diagnostics were + # on; see `apis.shared.sessions.feedback`) + "thumbsUp", + "thumbsDown", + # Document lifecycle rollups (per-call document fields summed; see + # `apis.shared.sessions.metadata.DOCUMENT_ROLLUP_ATTRS`) + "fullDocumentCalls", + "digestOnlyCalls", + "documentReadCalls", + "documentReadPages", ) #: C# rows for the cost anatomy and the session profile's trajectory. @@ -146,6 +165,26 @@ def is_content_bearing(path: str) -> bool: "prefixFingerprints", "contextWindow", "toolCalls", # per-call census, optional (PR-3) + # Context ledger, optional: the agent's stable prefix split + # ({system, tools} tokens), the conversation window's cumulative trim + # count, and the compaction decisions taken before this call — all + # numbers, never text. + "prefixTokens", + "windowRemovedMessages", + "compactionEvents", + # Document context, optional: the attachment footprint of the live + # context at this call (counts, estimated tokens, a format→count map keyed + # by Bedrock's format enum) and the document_read retrievals the call + # requested. Numbers and enum keys only — never a filename or a byte. + "hasDocuments", + "documentCount", + "documentTokens", + "documentDigests", + "documentsAttached", + "documentSlices", + "documentSliceTokens", + "documentMime", + "documentReads", ) #: FILE# rows for the session profile's attachment summary. @@ -157,12 +196,37 @@ def is_content_bearing(path: str) -> bool: "source", "status", "createdAt", + # DocumentDigest coverage (numbers and the format enum only; the + # abstract and section titles are denylisted above). + "digest.status", + "digest.format", + "digest.count", + "digest.tokens", +) + +#: F# rows for the session profile's feedback join. A thumb is a ±1, an +#: optional reason *code*, a `signal` discriminator (explicit / implicit, +#: response-feedback spec §10) and a timestamp — never text, by the request +#: model's closed enum (`apis.shared.sessions.models.FEEDBACK_REASONS`). +FEEDBACK_ROW_PROJECTION: Tuple[str, ...] = ( + "sessionId", + "messageId", + "value", + "reason", + "signal", + "kind", # implicit rows: copy / continue (closed enum) + "count", # implicit rows: how many times it fired + "retryMessageId", # a message index, the retry-with-correction link + "evaluation", # judged verdict: per-evaluator value/label/n/tokens, never the explanation + "evaluatedAt", + "updatedAt", ) ALL_PROJECTIONS: Dict[str, Tuple[str, ...]] = { "SESSION_ROW_PROJECTION": SESSION_ROW_PROJECTION, "CALL_ROW_PROJECTION": CALL_ROW_PROJECTION, "FILE_ROW_PROJECTION": FILE_ROW_PROJECTION, + "FEEDBACK_ROW_PROJECTION": FEEDBACK_ROW_PROJECTION, } diff --git a/backend/src/apis/shared/rbac/admin_scopes.py b/backend/src/apis/shared/rbac/admin_scopes.py index d2ba36356..a052a2d0d 100644 --- a/backend/src/apis/shared/rbac/admin_scopes.py +++ b/backend/src/apis/shared/rbac/admin_scopes.py @@ -94,6 +94,12 @@ class AdminScope: group=GROUP_AI_CONFIG, description="Manage skill bundles, their reference files, and skill role grants.", ), + AdminScope( + id="admin.agent_templates", + label="Agent Templates", + group=GROUP_AI_CONFIG, + description="Manage the catalog of agent templates users start new agents from.", + ), AdminScope( id="admin.connectors", label="Connectors", diff --git a/backend/src/apis/shared/sessions/feedback.py b/backend/src/apis/shared/sessions/feedback.py new file mode 100644 index 000000000..4dd7bfc3e --- /dev/null +++ b/backend/src/apis/shared/sessions/feedback.py @@ -0,0 +1,419 @@ +"""Message feedback (thumbs up / down) storage — the ``F#`` row family. + +The outcome signal the cost work has been missing (document-context-offload +spec §5 row 7, §6.1 "the outcome signal"; compaction thresholds spec §7.2). +One row per ``(user, session, message)``, content-free by construction: a +``value`` of +1 / -1, a timestamp and an optional reason *code* from +:data:`FEEDBACK_REASONS`. No free text can be stored here — the request +model's ``reason`` is a closed ``Literal`` and this module never accepts a +string outside the tuple. + +Schema (``sessions-metadata`` table, beside the ``C#`` / ``D#`` rows — see the +row-family summary in ``apis.shared.sessions.metadata``):: + + PK: USER#{user_id} + SK: F#{session_id}#{message_id} + GSI_PK: SESSION#{session_id} (SessionLookupIndex) + GSI_SK: F#{message_id} + sessionId, messageId, userId, value, reason?, signal, retryMessageId?, updatedAt, ttl + +``signal`` is ``"explicit"`` for a thumb. Implicit signals +(``docs/specs/response-feedback.md`` §10 — today ``copy`` and ``continue``) +live in the same family under their own key so they never collide with the +thumb:: + + SK: F#{session_id}#{message_id}#{kind} + GSI_SK: F#{message_id}#{kind} + sessionId, messageId, userId, signal="implicit", kind, count (ADD), updatedAt, ttl + +Every thumb reader filters to explicit rows, so the two are never summed +(§10's rule); the admin profile counts implicit rows per kind separately. + +Thumb rows also carry ``GSI1PK = FEEDBACK#down`` / ``FEEDBACK#up`` and +``GSI1SK = updatedAt`` on the existing ``UserTimestampIndex``, so "recent +down-thumbs across the fleet" is one query with no new index — the eval +sampler's input queue (spec §11 PR-4). A judged row gains ``evaluation`` +(content-free: per-evaluator value / label / tokens, never the judge's +explanation) and ``evaluatedAt``. + +``retryMessageId`` is the index of the user message the SPA sent as a +*retry with correction* after a down-thumb (response-feedback spec §7 "the +retry loop", §11 PR-1's consequence). It is a link, never the correction's +text: the profile counts retries and prices the rework from the cost rows +the link points at. A later re-thumb without the field keeps the link. + +``messageId`` is the same 0-based index the ``C#`` cost row carries for the +assistant message, so an admin read joins feedback to the call's turn class +(``hasDocuments`` / ``documentDigests`` / ``documentReads``, #1137) on +``(sessionId, messageId)`` with no second lookup. The SK is deterministic, so +a second thumb on the same message *replaces* the first (``put_item``), and a +``DELETE`` removes it — one thumb per (user, message) by key design. + +Session-row rollups: ``thumbsUp`` / ``thumbsDown`` are ``ADD``ed on the ``S#`` +row like ``toolCallCount``, only while ``COST_DIAGNOSTICS_ENABLED`` (an absent +attribute reads "not tracked", never 0). A replace adjusts both counters so +the rollup always equals the count of live ``F#`` rows written while the +diagnostics were on. + +Everything is gated by :func:`apis.shared.feature_flags.response_feedback_enabled` +at the route; this module is deliberately flag-free so a backfill or a test +can drive it directly. +""" + +from __future__ import annotations + +import logging +import os +from datetime import datetime, timedelta, timezone +from typing import Any, Dict, List, Optional + +from .models import FEEDBACK_REASONS, IMPLICIT_SIGNAL_KINDS, MessageFeedback +from .preview import is_preview_session + +logger = logging.getLogger(__name__) + +#: Retention, matching the ``C#`` cost rows the feedback joins to. +FEEDBACK_TTL_DAYS = 365 + + +class SessionNotOwned(Exception): + """The session has no metadata row for this user (missing or another user's).""" + + +def _table_name() -> str: + name = os.environ.get("DYNAMODB_SESSIONS_METADATA_TABLE_NAME") + if not name: + raise RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME environment variable is required") + return name + + +def _table(): + import boto3 + + return boto3.resource("dynamodb").Table(_table_name()) + + +def feedback_sk(session_id: str, message_id: int) -> str: + return f"F#{session_id}#{message_id}" + + +def _keys(user_id: str, session_id: str, message_id: int) -> Dict[str, str]: + return {"PK": f"USER#{user_id}", "SK": feedback_sk(session_id, message_id)} + + +def _to_model(item: Dict[str, Any]) -> MessageFeedback: + retry = item.get("retryMessageId") + return MessageFeedback( + value=int(item.get("value", 0)), + reason=item.get("reason") if item.get("reason") in FEEDBACK_REASONS else None, + retry_message_id=int(retry) if retry is not None else None, + updated_at=str(item.get("updatedAt", "")), + ) + + +async def _owned_session_sk(session_id: str, user_id: str, table) -> str: + """The session row's SK, or raise :class:`SessionNotOwned`.""" + from .metadata import _get_session_by_gsi + + existing = await _get_session_by_gsi(session_id, user_id, table) + if not existing or not existing.get("SK"): + raise SessionNotOwned(session_id) + return existing["SK"] + + +def _bump_rollups(table, user_id: str, session_sk: str, *, up: int, down: int) -> None: + """``ADD thumbsUp :up, thumbsDown :down`` on the session row, only while + the content-free diagnostics are on. Best-effort: a failed bump never + fails the user's click — the rows stay authoritative and the profile + prefers them when present.""" + from apis.shared.feature_flags import cost_diagnostics_enabled + + if not cost_diagnostics_enabled(): + return + try: + table.update_item( + Key={"PK": f"USER#{user_id}", "SK": session_sk}, + UpdateExpression="ADD thumbsUp :up, thumbsDown :down", + ExpressionAttributeValues={":up": int(up), ":down": int(down)}, + ) + except Exception as e: # noqa: BLE001 - rollup drift is tolerable, a lost click is not + logger.debug("feedback rollup bump failed for %s: %s", session_sk, e) + + +async def put_message_feedback( + session_id: str, + user_id: str, + message_id: int, + value: int, + reason: Optional[str] = None, + retry_message_id: Optional[int] = None, +) -> MessageFeedback: + """Write (or replace) this user's thumb on one message. + + An upsert (``update_item``): ``value`` / ``reason`` / ``updatedAt`` are + replaced on every call, ``retryMessageId`` is set when given and kept + otherwise, so a user who thumbs again after retrying does not lose the + link. ``ReturnValues=ALL_OLD`` tells us what a replace replaced, so the + session rollups move by the delta rather than double-counting. + + Raises :class:`SessionNotOwned` when the session is not this user's, and + ``ValueError`` on a value outside ``{1, -1}`` or a reason outside + :data:`FEEDBACK_REASONS` — the route's request model already rejects + both, this is the storage layer refusing to become a text field. + """ + if value not in (1, -1): + raise ValueError("feedback value must be 1 or -1") + if reason is not None and reason not in FEEDBACK_REASONS: + raise ValueError("feedback reason must be one of the fixed codes") + if retry_message_id is not None and (not isinstance(retry_message_id, int) or retry_message_id < 0): + raise ValueError("retryMessageId must be a non-negative message index") + if is_preview_session(session_id): + # Preview sessions persist nothing; echo the thumb so the UI is consistent. + return MessageFeedback(value=value, reason=reason, retry_message_id=retry_message_id, updated_at=_now()) + + table = _table() + session_sk = await _owned_session_sk(session_id, user_id, table) + + now = _now() + ttl = int((datetime.now(timezone.utc) + timedelta(days=FEEDBACK_TTL_DAYS)).timestamp()) + sets = { + "GSI_PK": f"SESSION#{session_id}", + "GSI_SK": f"F#{message_id}", + # Fleet queue on UserTimestampIndex: recent down-thumbs in one query. + "GSI1PK": f"FEEDBACK#{'down' if value == -1 else 'up'}", + "GSI1SK": now, + "sessionId": session_id, + "messageId": int(message_id), + "userId": user_id, + "#value": int(value), + "#signal": "explicit", + "updatedAt": now, + "#ttl": ttl, + } + if reason: + sets["reason"] = reason + if retry_message_id is not None: + sets["retryMessageId"] = int(retry_message_id) + names = {"#value": "value", "#signal": "signal", "#ttl": "ttl"} + values: Dict[str, Any] = {} + set_parts = [] + for i, (attr, val) in enumerate(sets.items()): + placeholder = f":v{i}" + values[placeholder] = val + set_parts.append(f"{attr} = {placeholder}") + expression = "SET " + ", ".join(set_parts) + if not reason: + expression += " REMOVE reason" + + response = table.update_item( + Key=_keys(user_id, session_id, message_id), + UpdateExpression=expression, + ExpressionAttributeNames=names, + ExpressionAttributeValues=values, + ReturnValues="ALL_OLD", + ) + previous = response.get("Attributes") or {} + previous_value = int(previous.get("value", 0)) if previous else 0 + item: Dict[str, Any] = { + "value": int(value), + "reason": reason, + "retryMessageId": ( + int(retry_message_id) if retry_message_id is not None else previous.get("retryMessageId") + ), + "updatedAt": now, + } + + up = (1 if value == 1 else 0) - (1 if previous_value == 1 else 0) + down = (1 if value == -1 else 0) - (1 if previous_value == -1 else 0) + if up or down: + _bump_rollups(table, user_id, session_sk, up=up, down=down) + + logger.info("👍 feedback stored for message %s in session %s", message_id, session_id) + return _to_model(item) + + +async def delete_message_feedback(session_id: str, user_id: str, message_id: int) -> bool: + """Remove this user's thumb on one message. Returns whether a row existed.""" + if is_preview_session(session_id): + return False + table = _table() + session_sk = await _owned_session_sk(session_id, user_id, table) + + response = table.delete_item(Key=_keys(user_id, session_id, message_id), ReturnValues="ALL_OLD") + previous = response.get("Attributes") or {} + if not previous: + return False + previous_value = int(previous.get("value", 0)) + _bump_rollups( + table, user_id, session_sk, + up=-1 if previous_value == 1 else 0, + down=-1 if previous_value == -1 else 0, + ) + return True + + +async def record_implicit_signal(session_id: str, user_id: str, message_id: int, kind: str) -> None: + """Count one implicit signal on one message (``ADD count :one``). Same + ownership rule as a thumb; a kind outside :data:`IMPLICIT_SIGNAL_KINDS` + is refused so the family stays a closed vocabulary. No session rollup: + the rows are the read model and the profile counts them directly.""" + if kind not in IMPLICIT_SIGNAL_KINDS: + raise ValueError("implicit signal kind must be one of the fixed codes") + if is_preview_session(session_id): + return + table = _table() + await _owned_session_sk(session_id, user_id, table) + ttl = int((datetime.now(timezone.utc) + timedelta(days=FEEDBACK_TTL_DAYS)).timestamp()) + table.update_item( + Key={"PK": f"USER#{user_id}", "SK": f"{feedback_sk(session_id, message_id)}#{kind}"}, + UpdateExpression=( + "SET GSI_PK = :gpk, GSI_SK = :gsk, sessionId = :sid, messageId = :mid, userId = :uid, " + "#signal = :sig, kind = :kind, updatedAt = :now, #ttl = :ttl ADD #count :one" + ), + ExpressionAttributeNames={"#signal": "signal", "#ttl": "ttl", "#count": "count"}, + ExpressionAttributeValues={ + ":gpk": f"SESSION#{session_id}", + ":gsk": f"F#{message_id}#{kind}", + ":sid": session_id, + ":mid": int(message_id), + ":uid": user_id, + ":sig": "implicit", + ":kind": kind, + ":now": _now(), + ":ttl": ttl, + ":one": 1, + }, + ) + + +def list_recent_down_thumbs(table, limit: int = 20) -> List[Dict[str, Any]]: + """Newest-first down-thumb rows across the fleet (``UserTimestampIndex``, + partition ``FEEDBACK#down``). Raw rows — the caller is the sampler, which + needs ``userId`` to write the verdict back; the admin list goes through + the projected storage reader instead.""" + from boto3.dynamodb.conditions import Key + + response = table.query( + IndexName="UserTimestampIndex", + KeyConditionExpression=Key("GSI1PK").eq("FEEDBACK#down"), + ScanIndexForward=False, + Limit=max(1, int(limit)), + ) + return list(response.get("Items", [])) + + +def list_feedback_in_window( + table, + start: str, + end: str, + value: str = "down", + limit: int = 2000, +) -> List[Dict[str, Any]]: + """Explicit thumbs of one polarity in an ISO time window, oldest first. + + The ``FEEDBACK#down`` / ``FEEDBACK#up`` partitions on + ``UserTimestampIndex`` are sorted by ``updatedAt``, so a window is a + ``between`` on the sort key — no scan, no new index. Raw rows; the admin + surface reads through the projected storage reader instead. + + Note implicit signals (spec §10) never appear here: they are written + without the ``GSI1*`` keys, so the fleet view is explicit-only by + construction as well as by filter. + """ + from boto3.dynamodb.conditions import Key + + partition = "FEEDBACK#down" if value == "down" else "FEEDBACK#up" + items: List[Dict[str, Any]] = [] + last_key = None + while len(items) < limit: + kwargs: Dict[str, Any] = { + "IndexName": "UserTimestampIndex", + "KeyConditionExpression": ( + Key("GSI1PK").eq(partition) & Key("GSI1SK").between(start, end) + ), + "ScanIndexForward": True, + "Limit": min(500, limit - len(items)), + } + if last_key: + kwargs["ExclusiveStartKey"] = last_key + response = table.query(**kwargs) + items.extend(response.get("Items", [])) + last_key = response.get("LastEvaluatedKey") + if not last_key: + break + return items[:limit] + + +def store_evaluation( + table, + user_id: str, + session_id: str, + message_id: int, + evaluation: Dict[str, Any], +) -> None: + """Attach a content-free judged result to a thumb row. ``evaluation`` is + the sampler's summary (``apis.shared.feedback_eval.sampler.build_verdict``); + this function refuses anything carrying an ``explanation`` so the judge's + prose about the conversation can never land beside the cost row.""" + if _carries_explanation(evaluation): + raise ValueError("evaluation summaries must not carry the judge's explanation") + from .metadata import _convert_floats_to_decimal + + table.update_item( + Key=_keys(user_id, session_id, message_id), + UpdateExpression="SET evaluation = :e, evaluatedAt = :t", + ExpressionAttributeValues={":e": _convert_floats_to_decimal(evaluation), ":t": _now()}, + ConditionExpression="attribute_exists(PK)", + ) + + +def _carries_explanation(obj: Any) -> bool: + if isinstance(obj, dict): + return any(k == "explanation" or _carries_explanation(v) for k, v in obj.items()) + if isinstance(obj, list): + return any(_carries_explanation(v) for v in obj) + return False + + +def query_session_feedback(table, session_id: str, user_id: Optional[str] = None) -> Dict[str, MessageFeedback]: + """All ``F#`` rows for a session via ``SessionLookupIndex``, keyed by + message id (as ``str``, matching the metadata index). With ``user_id`` + set, rows belonging to another user are dropped (the SPA read); the + admin reader passes ``None`` and gets every user's thumb.""" + from boto3.dynamodb.conditions import Key + + out: Dict[str, MessageFeedback] = {} + last_key = None + while True: + kwargs: Dict[str, Any] = { + "IndexName": "SessionLookupIndex", + "KeyConditionExpression": Key("GSI_PK").eq(f"SESSION#{session_id}") & Key("GSI_SK").begins_with("F#"), + } + if last_key: + kwargs["ExclusiveStartKey"] = last_key + response = table.query(**kwargs) + for item in response.get("Items", []): + if user_id is not None and item.get("userId") != user_id: + continue + if not is_explicit(item): + continue + raw_id = item.get("messageId") + try: + message_id = str(int(raw_id)) + except (TypeError, ValueError): + continue + out[message_id] = _to_model(item) + last_key = response.get("LastEvaluatedKey") + if not last_key: + break + return out + + +def is_explicit(row: Dict[str, Any]) -> bool: + """A thumb, as opposed to a §10 implicit signal. Rows written before the + discriminator existed carry none and are explicit by construction.""" + return row.get("signal") in (None, "explicit") + + +def _now() -> str: + return datetime.now(timezone.utc).isoformat() diff --git a/backend/src/apis/shared/sessions/metadata.py b/backend/src/apis/shared/sessions/metadata.py index 8d336ae80..5d6fe11dd 100644 --- a/backend/src/apis/shared/sessions/metadata.py +++ b/backend/src/apis/shared/sessions/metadata.py @@ -5,6 +5,22 @@ Architecture: - Cloud: Stores metadata in DynamoDB table specified by DYNAMODB_SESSIONS_METADATA_TABLE_NAME + +Row families on the ``sessions-metadata`` table (PK = ``USER#{user_id}``; all +carry ``GSI_PK = SESSION#{session_id}`` so ``SessionLookupIndex`` lists one +session's rows by prefix): + + S#{session_id} session row (rollups, preferences, compaction state) + C#{timestamp}#{uuid} one model call's cost/usage record; ``messageId`` = the + assistant message's 0-based index (``_store_message_metadata_cloud``) + D#{session_id}#{message_id} the user's original prompt text for display (``store_user_display_text``) + F#{session_id}#{message_id} the user's thumb on an assistant message — value ±1, optional + reason code, timestamp; content-free (``apis.shared.sessions.feedback``). + Same ``messageId`` as the ``C#`` row, so feedback joins the call's + turn class on ``(sessionId, messageId)`` in one lookup. + +``GSI_SK`` is ``META`` / ``C#{timestamp}`` / ``D#{message_id}`` / ``F#{message_id}`` +respectively. Only ``C#`` / ``D#`` / ``F#`` rows carry a ``ttl``. """ import logging @@ -12,6 +28,9 @@ import math import os import base64 +from dataclasses import dataclass + +from apis.shared.aws_clients import get_dynamodb_table from typing import Iterable, List, Optional, Tuple, Any, Dict from decimal import Decimal @@ -157,11 +176,9 @@ async def store_user_display_text( return try: - import boto3 from datetime import datetime, timezone, timedelta - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) timestamp = datetime.now(timezone.utc).isoformat() ttl = int((datetime.now(timezone.utc) + timedelta(days=365)).timestamp()) @@ -230,12 +247,10 @@ async def _store_message_metadata_cloud( - TTL only affects cost records (sessions don't have ttl) """ try: - import boto3 import uuid as uuid_lib from datetime import datetime, timezone, timedelta - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(table_name) + table = get_dynamodb_table(table_name) # Prepare item for DynamoDB metadata_dict = message_metadata.model_dump(by_alias=True, exclude_none=True) @@ -941,12 +956,10 @@ async def _store_session_metadata_cloud( - Direct session lookup via GSI """ try: - import boto3 from botocore.exceptions import ClientError from datetime import datetime, timezone - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(table_name) + table = get_dynamodb_table(table_name) # First, check if session exists via GSI to get current SK existing_session = await _get_session_by_gsi(session_id, user_id, table) @@ -1091,7 +1104,9 @@ def _recency_gsi_keys( return {} -async def ensure_session_metadata_exists(session_id: str, user_id: str) -> bool: +async def ensure_session_metadata_exists( + session_id: str, user_id: str, snapshot: Optional["SessionMetaSnapshot"] = None +) -> bool: """Idempotently create a session metadata row if it doesn't exist yet. Returns ``True`` when a new row was created (caller can use this as the @@ -1122,16 +1137,23 @@ async def ensure_session_metadata_exists(session_id: str, user_id: str) -> bool: raise RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME environment variable is required") try: - import boto3 from botocore.exceptions import ClientError from datetime import datetime, timezone - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) # Catch a pre-existing row (legacy S#ACTIVE#… or already-migrated S#{id}) so # we don't create a second row for the same session. - existing = await _get_session_by_gsi(session_id, user_id, table) + # + # Both this and the ownership check below come out of ONE query when + # the caller passes a snapshot (PR-2) — they always could, since a + # single `SessionLookupIndex` response contains every META row for the + # session; `_get_session_by_gsi` just discarded the half the ownership + # check needed. `snapshot=None` keeps both reads exactly as they were. + existing = ( + snapshot.row if snapshot is not None + else await _get_session_by_gsi(session_id, user_id, table) + ) if existing is not None: return False @@ -1140,7 +1162,11 @@ async def ensure_session_metadata_exists(session_id: str, user_id: str) -> bool: # can't see the other row) and fork the session id across two users. # The invocations route rejects these turns outright; this is the # backstop for every other path that pre-creates metadata. - if await session_owned_by_other_user(session_id, user_id): + owned_by_other = ( + snapshot.owned_by_other if snapshot is not None + else await session_owned_by_other_user(session_id, user_id) + ) + if owned_by_other: logger.warning( "Refusing to create metadata for session %s — already owned by another user", session_id, @@ -1205,10 +1231,8 @@ async def update_session_title(session_id: str, user_id: str, title: str) -> Non raise RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME environment variable is required") try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -1252,10 +1276,8 @@ async def set_session_unread(session_id: str, user_id: str, unread: bool) -> Non raise RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME environment variable is required") try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -1330,11 +1352,9 @@ async def update_session_activity( raise RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME environment variable is required") try: - import boto3 from datetime import datetime, timezone - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -1467,10 +1487,8 @@ async def set_selected_prompt_id( return False try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -1517,6 +1535,101 @@ async def set_selected_prompt_id( return False +@dataclass(frozen=True) +class SessionMetaSnapshot: + """One read of a session's ``META`` rows, shared by the whole preamble. + + WHY THIS EXISTS + --------------- + Measured on dev (docs/specs/turn-latency-preamble.md), a warm turn read + this one item **eight times** before the first model call — the ownership + guard, the attachment pop, the metadata pre-create, four stale-marker + clears, and the quota session-notice — each on its own round trip. A GSI + query from inside an AgentCore Runtime container costs **~53ms** (measured, + not assumed: ``preamble.ownership`` is exactly one query and nothing else), + so those reads were ~445ms of a ~455ms stage. + + This is the single read they now share. + + WHY IT IS PASSED EXPLICITLY, NOT CACHED + --------------------------------------- + An implicit per-request memo inside ``_get_session_by_gsi`` would be a + smaller diff and the wrong shape. CLAUDE.md's rule — *never cache session + state; per-session state must be re-read per turn and must never move + backwards* — exists because this repo has shipped that bug twice (#741 + conversation history, #751 compaction state). A snapshot that callers opt + into by passing it cannot leak into a caller that needs a fresh read; a + memo keyed on the request can, and the failure is silent. + + So every consumer keeps working exactly as before when ``snapshot`` is + ``None``, which is the default and what every non-preamble caller gets. + + ``row is None`` is a real answer ("this user has no META row"), not + "unknown" — which is why consumers take the snapshot object rather than the + row dict. Passing ``row`` alone would make "no row yet" indistinguishable + from "nothing was prefetched", and a brand-new session would silently fall + back to re-reading. + """ + + row: Optional[dict] + """This user's ``META`` row, decimal-converted, or ``None`` if absent.""" + + owned_by_other: bool + """``META`` rows exist for this session and none of them are this user's.""" + + +async def load_session_meta(session_id: str, user_id: str) -> SessionMetaSnapshot: + """Read a session's ``META`` rows once, answering ownership and content. + + Replaces an ownership probe and a row lookup that were separate queries of + the same index for the same key. Both answers come out of one response + because they were always in it — ``_get_session_by_gsi`` simply discarded + the information the ownership check needed (it returns ``None`` both for + "no such session" and for "someone else's", which is the ambiguity + ``session_owned_by_other_user`` exists to resolve). + + Best-effort in the same direction as the helpers it feeds: any failure + yields ``row=None, owned_by_other=False``, i.e. "nothing known, nothing + blocked". That matches what a failed ownership probe already did (fail + open) and what a failed row read already did (treat as absent), so a + DynamoDB outage degrades the preamble exactly as it did before. + """ + empty = SessionMetaSnapshot(row=None, owned_by_other=False) + + sessions_metadata_table = os.environ.get("DYNAMODB_SESSIONS_METADATA_TABLE_NAME") + if not sessions_metadata_table or is_preview_session(session_id): + return empty + + try: + import boto3 + from boto3.dynamodb.conditions import Key + + table = get_dynamodb_table(sessions_metadata_table) + + response = table.query( + IndexName="SessionLookupIndex", + KeyConditionExpression=Key("GSI_PK").eq(f"SESSION#{session_id}") + & Key("GSI_SK").eq("META"), + ) + items = response.get("Items", []) or [] + if not items: + return empty + + # Scan ALL rows for this user's rather than trusting items[0] — a + # cross-user fork gives two rows sharing GSI_PK/GSI_SK, returned in an + # unspecified order. Same reasoning as `_get_session_by_gsi`, which + # this consolidates rather than replaces. + mine = next((i for i in items if i.get("userId") == user_id), None) + if mine is not None: + return SessionMetaSnapshot(row=_convert_decimal_to_float(mine), owned_by_other=False) + + logger.warning("Session %s belongs to a different user", session_id) + return SessionMetaSnapshot(row=None, owned_by_other=True) + except Exception as e: + logger.debug("Session meta load failed, treating as absent: %s", e) + return empty + + async def session_owned_by_other_user(session_id: str, user_id: str) -> bool: """Whether this session id already has a metadata row owned by someone else. @@ -1553,8 +1666,7 @@ async def session_owned_by_other_user(session_id: str, user_id: str) -> bool: import boto3 from boto3.dynamodb.conditions import Key - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) response = table.query( IndexName="SessionLookupIndex", @@ -1742,6 +1854,21 @@ async def _bump_session_aggregates( update_parts_add.append("toolErrorCount :toolErrors") values[":toolCalls"] = tool_calls_total values[":toolErrors"] = tool_errors_total + # Compaction decisions, counted per kind from the call's + # `compactionEvents` ledger. `checkpoint` is deliberately not + # here — `_save_compaction_state(record_event=True)` already + # bumps `compactionCount` for it, and two counters for one event + # would disagree under concurrency. + for kind, attr in _COMPACTION_EVENT_COUNTERS.items(): + update_parts_add.append(f"{attr} :{attr}") + values[f":{attr}"] = _compaction_event_count(message_metadata, kind) + # Document lifecycle rollups (docs/specs/document-context-offload.md + # §6.1): how many calls ran with the full document inline vs. a + # digest only, and how much document_read pulled back. Written + # as 0 while the diagnostics are on, like the counters above. + for attr, value in _document_rollups(message_metadata).items(): + update_parts_add.append(f"{attr} :{attr}") + values[f":{attr}"] = value update_expression = ( "ADD " + ", ".join(update_parts_add) + " SET " + ", ".join(update_parts_set) @@ -1764,6 +1891,63 @@ async def _bump_session_aggregates( logger.debug("bump_session_aggregates failed (will be backfilled on read): %s", e) +#: Session-row counter per compaction event kind (see +#: ``TurnBasedSessionManager.record_compaction_event``). Written as 0 while +#: the diagnostics are on so the attribute exists from the first call. +_COMPACTION_EVENT_COUNTERS = { + "applied": "compactionAppliedCount", + "forced": "compactionForcedCount", + "floor_unreachable": "compactionFloorUnreachableCount", +} + + +#: Session-row counters derived from a call's document fields. ``fullDocumentCalls`` +#: and ``digestOnlyCalls`` are the digest-vs-full turn shares; the two +#: ``documentRead*`` counters sum the call's ``documentReads`` ledger entry. +DOCUMENT_ROLLUP_ATTRS = ("fullDocumentCalls", "digestOnlyCalls", "documentReadCalls", "documentReadPages") + + +def _document_rollups(message_metadata: Any) -> Dict[str, int]: + """``{attr: delta}`` for every ``DOCUMENT_ROLLUP_ATTRS`` entry, from the + call's ``hasDocuments`` / ``documentDigests`` / ``documentReads`` extras. + Absent or malformed fields count as zero — the bump must never fail.""" + extra = getattr(message_metadata, "model_extra", None) + extra = extra if isinstance(extra, dict) else {} + has_documents = bool(extra.get("hasDocuments")) + try: + digests = int(extra.get("documentDigests") or 0) + except (TypeError, ValueError): + digests = 0 + reads = extra.get("documentReads") + reads = reads if isinstance(reads, dict) else {} + + def _int(value: Any) -> int: + try: + return int(value or 0) + except (TypeError, ValueError): + return 0 + + return { + "fullDocumentCalls": 1 if has_documents else 0, + "digestOnlyCalls": 1 if (digests > 0 and not has_documents) else 0, + "documentReadCalls": _int(reads.get("calls")), + "documentReadPages": _int(reads.get("pages")), + } + + +def _compaction_event_count(message_metadata: Any, kind: str) -> int: + """How many events of ``kind`` the call's ``compactionEvents`` extra carries. + + Malformed entries count as zero rather than raising — the aggregate bump + must never fail on them. + """ + extra = getattr(message_metadata, "model_extra", None) + events = extra.get("compactionEvents") if isinstance(extra, dict) else None + if not isinstance(events, list): + return 0 + return sum(1 for e in events if isinstance(e, dict) and e.get("kind") == kind) + + def _tool_census_totals(message_metadata: Any) -> tuple[int, int]: """``(calls, errors)`` summed over the call's ``toolCalls`` extra field. @@ -1962,8 +2146,7 @@ async def session_exists_for_other_user(session_id: str, current_user_id: str) - import boto3 from boto3.dynamodb.conditions import Key - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) response = table.query( IndexName='SessionLookupIndex', @@ -2029,8 +2212,7 @@ async def _get_all_message_metadata_cloud(session_id: str, user_id: str, table_n import boto3 from boto3.dynamodb.conditions import Key - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(table_name) + table = get_dynamodb_table(table_name) logger.info(f"🔍 Querying cost records via GSI for session {session_id}") @@ -2089,6 +2271,20 @@ async def _get_all_message_metadata_cloud(session_id: str, user_id: str, table_n metadata_index[message_id] = {"displayText": display_text} logger.debug(f"🔗 Merged displayText for user message {message_id}") + # Merge this user's thumbs (F# rows) so a reload restores the SPA's + # pressed state. Skipped while the feature is off — the rows stay. + from apis.shared.feature_flags import response_feedback_enabled + + if response_feedback_enabled(): + from .feedback import query_session_feedback + + try: + for message_id, feedback in query_session_feedback(table, session_id, user_id).items(): + entry = metadata_index.setdefault(message_id, {}) + entry["feedback"] = feedback.model_dump(by_alias=True, exclude_none=True) + except Exception as e: # noqa: BLE001 - feedback is a UI enhancement, never block history + logger.warning(f"Failed to merge message feedback: {e}") + logger.info(f"📋 Metadata keys: {sorted(metadata_index.keys())}") return metadata_index @@ -2128,8 +2324,7 @@ async def _get_session_metadata_cloud( import boto3 from boto3.dynamodb.conditions import Key - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(table_name) + table = get_dynamodb_table(table_name) # Use GSI for session lookup by ID response = table.query( @@ -2423,8 +2618,7 @@ async def _list_user_sessions_cloud( from boto3.dynamodb.conditions import Key from botocore.exceptions import ClientError - dynamodb = boto3.resource('dynamodb') - table = dynamodb.Table(table_name) + table = get_dynamodb_table(table_name) cursor = _decode_list_cursor(next_token) want = (limit + 1) if limit else None @@ -2638,10 +2832,8 @@ async def add_pending_interrupt( return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -2696,10 +2888,8 @@ async def add_export_receipt( return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -2747,10 +2937,8 @@ async def remove_pending_interrupts( return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -2780,7 +2968,9 @@ async def remove_pending_interrupts( logger.error("Failed to remove pending_interrupts: %s", e, exc_info=True) -async def clear_pending_interrupts(session_id: str, user_id: str) -> None: +async def clear_pending_interrupts( + session_id: str, user_id: str, snapshot: Optional["SessionMetaSnapshot"] = None +) -> None: """Drop every pending-interrupt breadcrumb for a session. Distinct from :func:`remove_pending_interrupts`, which drops specific ids @@ -2811,12 +3001,16 @@ async def clear_pending_interrupts(session_id: str, user_id: str) -> None: return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) - existing = await _get_session_by_gsi(session_id, user_id, table) + # The preamble reads this row once and shares it (PR-2); `None` keeps + # the original per-call read for every other caller. + existing = ( + snapshot.row + if snapshot is not None + else await _get_session_by_gsi(session_id, user_id, table) + ) if not existing: return @@ -2873,10 +3067,8 @@ async def set_paused_turn( return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -2913,7 +3105,9 @@ async def get_paused_turn(session_id: str, user_id: str) -> Optional[PausedTurnS return metadata.paused_turn -async def clear_paused_turn(session_id: str, user_id: str) -> None: +async def clear_paused_turn( + session_id: str, user_id: str, snapshot: Optional["SessionMetaSnapshot"] = None +) -> None: """Drop the paused-turn snapshot for a session. Called on successful resume completion, on explicit dismiss, and at the @@ -2925,12 +3119,16 @@ async def clear_paused_turn(session_id: str, user_id: str) -> None: return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) - existing = await _get_session_by_gsi(session_id, user_id, table) + # The preamble reads this row once and shares it (PR-2); `None` keeps + # the original per-call read for every other caller. + existing = ( + snapshot.row + if snapshot is not None + else await _get_session_by_gsi(session_id, user_id, table) + ) if not existing: return @@ -2951,6 +3149,120 @@ async def clear_paused_turn(session_id: str, user_id: str) -> None: logger.error("Failed to clear paused_turn: %s", e, exc_info=True) +async def set_browser_session( + session_id: str, + user_id: str, + ref: Dict[str, Any], +) -> None: + """Project the conversation's browser session onto its metadata row. + + Spec D4. The agent-side source of truth is ``agent.state``, which the + Strands session manager restores from AgentCore Memory — a store app-api + cannot read. app-api is where the live-view route has to live (the + AgentCore Runtime data plane proxies only ``/invocations`` and ``/ping``, + so a route on inference-api would 404 in cloud), and it needs the browser + session's identity to mint a URL. Hence this projection. + + Idempotent overwrite. Per the "one session can be served by more than one + agent" rule, readers must re-read this row rather than caching it on an + agent instance. + + **Identifiers only.** Anything URL-shaped is rejected before the write: a + live-view URL is SigV4 query-signed with a 300-second cap, so a persisted + one is stale by the time anything reads it back, and persisting one at all + is the mistake PR #1101 already paid for once. + + No-op when the session metadata record is missing or the table env var is + unset (preview/anonymous flows). + """ + sessions_metadata_table = os.environ.get("DYNAMODB_SESSIONS_METADATA_TABLE_NAME") + if not sessions_metadata_table: + logger.warning( + "DYNAMODB_SESSIONS_METADATA_TABLE_NAME not set; skipping browser_session persistence" + ) + return + + try: + from apis.shared.browser_takeover import assert_no_url + + assert_no_url(ref) + except ValueError as e: + logger.error("Refusing to persist browser_session: %s", e) + return + + try: + + table = get_dynamodb_table(sessions_metadata_table) + + existing = await _get_session_by_gsi(session_id, user_id, table) + if not existing: + logger.info( + "Skipping browser_session write — session %s not found", session_id + ) + return + + sk = existing.get("SK") + if not sk: + logger.warning( + "Session %s has no SK; cannot update browser_session", session_id + ) + return + + table.update_item( + Key={"PK": f"USER#{user_id}", "SK": sk}, + UpdateExpression="SET #bs = :bs", + ExpressionAttributeNames={"#bs": "browserSession"}, + ExpressionAttributeValues={":bs": _convert_floats_to_decimal(ref)}, + ) + logger.info("Persisted browser_session for session %s", session_id) + except Exception as e: + # Best-effort: a write failure must not break the live SSE flow. The + # cost is that app-api cannot mint a live view for this takeover, so + # the user sees the prompt without a working viewer — degraded, not + # broken, and the turn still resumes on skip. + logger.error("Failed to persist browser_session: %s", e, exc_info=True) + + +async def clear_browser_session(session_id: str, user_id: str) -> None: + """Drop the browser-session projection for a conversation. + + Called when the browser session ends. Leaving a stale row behind would have + app-api mint live-view URLs for a session that no longer exists. + """ + sessions_metadata_table = os.environ.get("DYNAMODB_SESSIONS_METADATA_TABLE_NAME") + if not sessions_metadata_table: + return + + try: + + table = get_dynamodb_table(sessions_metadata_table) + + existing = await _get_session_by_gsi(session_id, user_id, table) + if not existing or not existing.get("SK"): + return + if "browserSession" not in existing: + return # Already clear + + table.update_item( + Key={"PK": f"USER#{user_id}", "SK": existing["SK"]}, + UpdateExpression="REMOVE #bs", + ExpressionAttributeNames={"#bs": "browserSession"}, + ) + logger.info("Cleared browser_session for session %s", session_id) + except Exception as e: + logger.error("Failed to clear browser_session: %s", e, exc_info=True) + + +async def get_browser_session( + session_id: str, user_id: str +) -> Optional[Dict[str, Any]]: + """Return the persisted browser-session projection, if any.""" + metadata = await get_session_metadata(session_id, user_id) + if not metadata: + return None + return metadata.browser_session + + async def set_truncated_turn(session_id: str, user_id: str) -> None: """Mark that the last turn ended in a recoverable max_tokens truncation. @@ -2966,10 +3278,8 @@ async def set_truncated_turn(session_id: str, user_id: str) -> None: return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -2992,7 +3302,9 @@ async def set_truncated_turn(session_id: str, user_id: str) -> None: logger.error("Failed to persist truncated_turn: %s", e, exc_info=True) -async def clear_truncated_turn(session_id: str, user_id: str) -> None: +async def clear_truncated_turn( + session_id: str, user_id: str, snapshot: Optional["SessionMetaSnapshot"] = None +) -> None: """Drop the truncated-turn marker. Called at the start of any new turn that isn't an interrupt-resume @@ -3005,12 +3317,16 @@ async def clear_truncated_turn(session_id: str, user_id: str) -> None: return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) - existing = await _get_session_by_gsi(session_id, user_id, table) + # The preamble reads this row once and shares it (PR-2); `None` keeps + # the original per-call read for every other caller. + existing = ( + snapshot.row + if snapshot is not None + else await _get_session_by_gsi(session_id, user_id, table) + ) if not existing: return @@ -3085,12 +3401,10 @@ async def set_interrupted_turn( reason = "unknown" try: - import boto3 from datetime import datetime, timezone from botocore.exceptions import ClientError - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -3152,7 +3466,9 @@ async def set_interrupted_turn( logger.error("Failed to persist interrupted_turn: %s", e, exc_info=True) -async def clear_interrupted_turn(session_id: str, user_id: str) -> Optional[str]: +async def clear_interrupted_turn( + session_id: str, user_id: str, snapshot: Optional["SessionMetaSnapshot"] = None +) -> Optional[str]: """Pop the interrupted-turn marker, returning the reason it recorded. Called at the start of any new turn that isn't an interrupt-resume, so a @@ -3172,12 +3488,16 @@ async def clear_interrupted_turn(session_id: str, user_id: str) -> Optional[str] return None try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) - existing = await _get_session_by_gsi(session_id, user_id, table) + # The preamble reads this row once and shares it (PR-2); `None` keeps + # the original per-call read for every other caller. + existing = ( + snapshot.row + if snapshot is not None + else await _get_session_by_gsi(session_id, user_id, table) + ) if not existing: return None @@ -3260,11 +3580,9 @@ async def set_pending_attachments( return try: - import boto3 from datetime import datetime, timezone - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -3312,10 +3630,8 @@ async def clear_pending_attachments(session_id: str, user_id: str) -> None: return try: - import boto3 - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) existing = await _get_session_by_gsi(session_id, user_id, table) if not existing: @@ -3341,7 +3657,9 @@ async def clear_pending_attachments(session_id: str, user_id: str) -> None: logger.error("Failed to clear pending_attachments: %s", e, exc_info=True) -async def pop_pending_attachments(session_id: str, user_id: str) -> List[str]: +async def pop_pending_attachments( + session_id: str, user_id: str, snapshot: Optional["SessionMetaSnapshot"] = None +) -> List[str]: """Atomically take the pending-attachment upload IDs, clearing the marker. Returns the IDs only when the marker is younger than @@ -3358,13 +3676,17 @@ async def pop_pending_attachments(session_id: str, user_id: str) -> List[str]: return [] try: - import boto3 from datetime import datetime, timezone - dynamodb = boto3.resource("dynamodb") - table = dynamodb.Table(sessions_metadata_table) + table = get_dynamodb_table(sessions_metadata_table) - existing = await _get_session_by_gsi(session_id, user_id, table) + # The preamble reads this row once and shares it (PR-2); `None` keeps + # the original per-call read for every other caller. + existing = ( + snapshot.row + if snapshot is not None + else await _get_session_by_gsi(session_id, user_id, table) + ) if not existing: return [] diff --git a/backend/src/apis/shared/sessions/models.py b/backend/src/apis/shared/sessions/models.py index 82dc611b0..ce88d44ad 100644 --- a/backend/src/apis/shared/sessions/models.py +++ b/backend/src/apis/shared/sessions/models.py @@ -27,7 +27,7 @@ class PendingInterrupt(BaseModel): reload — without it, a browser refresh leaves the prompt stuck and the tool call orphaned in ``pending`` forever. - Three variants share this shape (discriminated by ``kind``): + Four variants share this shape (discriminated by ``kind``): - ``oauth`` — written by ``OAuthConsentHook``. Carries ``provider_id``; the frontend re-fetches a fresh consent URL via ``initiate-consent`` @@ -44,6 +44,13 @@ class PendingInterrupt(BaseModel): already looking at. Unlike the other two this interrupt is raised by the tool itself via ``ToolContext``, not by a hook — the persisted shape and the resume path are identical either way. + - ``browser_login`` — written for ``request_user_login``. Carries + ``browser_session`` (JSON-encoded :class:`BrowserSessionRef`) so the live + view can be re-offered after a refresh. Deliberately holds **no URL**: + live-view URLs are SigV4 query-signed and expire within 300 seconds, so + a stored one is always stale by the time it is read — app-api mints a + fresh one per request instead (``docs/specs/authenticated-web- + assessment.md`` D2). Default ``kind`` is ``oauth`` for backward compatibility with rows written before per-tool approval shipped. @@ -51,7 +58,7 @@ class PendingInterrupt(BaseModel): model_config = ConfigDict(populate_by_name=True) interrupt_id: str = Field(..., alias="interruptId", description="Strands interrupt id used to resume the paused turn") - kind: Literal["oauth", "tool_approval", "user_question"] = Field( + kind: Literal["oauth", "tool_approval", "user_question", "browser_login"] = Field( default="oauth", description="Discriminator: which variant this interrupt represents", ) @@ -96,6 +103,20 @@ class PendingInterrupt(BaseModel): description="(user_question) JSON-encoded list of questions to re-render", ) + # browser_login-only fields + browser_session: Optional[str] = Field( + default=None, + alias="browserSession", + description=( + "(browser_login) JSON-encoded BrowserSessionRef — identifiers and " + "viewport only, never a live-view URL" + ), + ) + reason: Optional[str] = Field( + default=None, + description="(browser_login) The agent's one-line explanation of what needs signing into", + ) + class PausedTurnSnapshot(BaseModel): """Frozen agent-construction context for a turn that paused on OAuth consent. @@ -150,6 +171,15 @@ class PausedTurnSnapshot(BaseModel): "Mantle turn targets the same region. None for non-Mantle turns and " "snapshots written before the field existed.", ) + assistant_id: Optional[str] = Field( + default=None, + alias="assistantId", + description="Assistant (RAG corpus) the paused turn ran against. It is an " + "agent-cache key element because the spreadsheet-analysis tools " + "close over it, so resume replays it verbatim to land on the " + "paused agent's slot. None for assistant-less turns and snapshots " + "written before the field existed (those miss and rebuild).", + ) captured_at: str = Field(..., alias="capturedAt", description="ISO 8601 timestamp when the turn paused") expires_at: str = Field(..., alias="expiresAt", description="ISO 8601 timestamp after which the snapshot is no longer valid for resume") @@ -287,6 +317,21 @@ class SessionMetadata(BaseModel): alias="pendingAttachmentsAt", description="ISO 8601 timestamp the pending-attachment marker was written; recovery is TTL-bounded against it", ) + browser_session: Optional[Dict[str, Any]] = Field( + default=None, + alias="browserSession", + description=( + "Identity of the AgentCore browser session this conversation is " + "driving, projected here so app-api can mint a live-view URL for a " + "takeover (docs/specs/authenticated-web-assessment.md D4). The " + "agent-side source of truth stays on `agent.state`; this is the " + "copy app-api can read, and per the 'one session, more than one " + "agent' rule it is re-read per turn rather than cached on an agent " + "instance. Identifiers, viewport and control state only — never a " + "live-view URL, which is SigV4 query-signed and dead within 300 " + "seconds of being minted" + ), + ) # Denormalized cost + context aggregates for the session-cost badge. # Maintained by _bump_session_aggregates after each turn (write-time @@ -427,6 +472,46 @@ class SessionSteerResponse(BaseModel): model_config = ConfigDict(populate_by_name=True) +class BrowserLiveViewResponse(BaseModel): + """A short-lived Live View URL for the conversation's browser session. + + The URL is SigV4 *query*-signed and lives at most 300 seconds, so this is + minted per request rather than stored or streamed + (``docs/specs/authenticated-web-assessment.md`` D2). The viewer re-requests + against ``expiresAt``, which is why a twenty-minute sign-in works. + + ``viewport`` rides the response because DCV's ``remoteWidth``/ + ``remoteHeight`` must match the browser session's real viewport or the + stream crops — the viewer must not carry its own copy of 1280x800. + + ⚠️ This is the one place a presigned browser URL is allowed to travel, and + only because it goes to an authenticated browser over the SPA's own + session. It must never reach a tool result, an SSE event, a persisted row + or a log line. + """ + + model_config = ConfigDict(populate_by_name=True) + + url: str = Field(..., description="Presigned Live View URL, valid until expiresAt") + expires_at: str = Field( + ..., + alias="expiresAt", + description="ISO 8601 instant the signature expires; the viewer refreshes before this", + ) + viewport: Dict[str, int] = Field( + ..., + description="The browser session's real viewport, for DCV remoteWidth/remoteHeight", + ) + control_state: str = Field( + default="agent", + alias="controlState", + description=( + "'user' while the automation stream is DISABLED and the human can " + "drive; 'agent' when the view is read-only because the agent holds it" + ), + ) + + class SessionMetadataResponse(BaseModel): """Response containing session metadata""" @@ -642,6 +727,64 @@ class Citation(BaseModel): text: str = Field(..., description="Relevant text excerpt from the document") +#: Reason codes a down-thumb may carry — the six buckets of +#: ``docs/specs/response-feedback.md`` §6, each of which routes to an +#: evaluator or an ops signal. A closed enum, never free text: the row is +#: content-free by construction so it can sit beside the ``C#`` cost row and +#: be read by the admin profile without reading the conversation. The spec's +#: "something else → free text" is deliberately not here; that hand-off is +#: the existing Agent report dialog (spec §3), which already has moderation. +FEEDBACK_REASONS = ("wrong", "instructions", "length", "tool_failed", "outdated", "other") +FeedbackReason = Literal["wrong", "instructions", "length", "tool_failed", "outdated", "other"] + + +#: Implicit signals (response-feedback spec §10): denser than thumbs, no UI +#: cost, written to the same ``F#`` family under ``signal: "implicit"`` and +#: never summed with them. ``copy`` = the response was copied out; +#: ``continue`` = a truncated / interrupted response was resumed. Edit-and- +#: resend has no affordance in the SPA yet; abandonment is deferred (its +#: base rate is indistinguishable from a satisfied user going quiet). +IMPLICIT_SIGNAL_KINDS = ("copy", "continue") +ImplicitSignalKind = Literal["copy", "continue"] + + +class ImplicitSignalRequest(BaseModel): + """Body of ``POST /sessions/{id}/messages/{messageId}/signals``.""" + + model_config = ConfigDict(populate_by_name=True) + + kind: ImplicitSignalKind = Field(..., description="Which implicit signal fired (closed enum)") + + +class MessageFeedback(BaseModel): + """One user's thumb on one assistant message (``F#`` row, see + ``apis.shared.sessions.metadata``). ``value`` is +1 (up) or -1 (down); + ``reason`` is an optional code from ``FEEDBACK_REASONS``.""" + + model_config = ConfigDict(populate_by_name=True) + + value: Literal[1, -1] = Field(..., description="+1 for thumbs up, -1 for thumbs down") + reason: Optional[FeedbackReason] = Field(None, description="Optional reason code (never free text)") + retry_message_id: Optional[int] = Field( + None, alias="retryMessageId", ge=0, + description="Index of the user message sent as a retry-with-correction after this thumb (content-free link)", + ) + updated_at: str = Field(..., alias="updatedAt", description="ISO timestamp of the latest thumb") + + +class MessageFeedbackRequest(BaseModel): + """Body of ``PUT /sessions/{id}/messages/{messageId}/feedback``.""" + + model_config = ConfigDict(populate_by_name=True) + + value: Literal[1, -1] = Field(..., description="+1 for thumbs up, -1 for thumbs down") + reason: Optional[FeedbackReason] = Field(None, description="Optional reason code (never free text)") + retry_message_id: Optional[int] = Field( + None, alias="retryMessageId", ge=0, + description="Set when the user sent a retry-with-correction: that user message's index", + ) + + class MessageMetadata(BaseModel): """Metadata associated with a single message""" @@ -654,8 +797,23 @@ class MessageMetadata(BaseModel): cost: Optional[Union[float, Dict[str, float]]] = Field(None, description="Cost for this message — either a total float (legacy) or a breakdown dict with total, inputCost, outputCost, cacheReadCost, cacheWriteCost") citations: Optional[List[Dict[str, str]]] = Field(None, description="RAG citations for this message (stored as dicts for flexible JSON storage)") display_text: Optional[str] = Field(None, alias="displayText", description="Original user message text before RAG augmentation (for clean UI display)") - # Note: Feedback will be added in future implementation - # feedback: Optional[Feedback] = None + # One user's thumb on this message, merged from the ``F#`` row on read + # (see ``apis.shared.sessions.metadata``). Content-free: a ±1, a timestamp + # and an optional reason code. + feedback: Optional[MessageFeedback] = Field(None, description="User thumbs up/down on this assistant message") + # Set on the LAST assistant message of a turn only — it describes the turn, + # not the message. See `turn_duration_ms` on the write path. + turn_duration_ms: Optional[int] = Field( + None, + alias="turnDurationMs", + description=( + "How long the whole turn took, in ms, measured server-side from " + "the invocation arriving to the stream ending. Deliberately NOT " + "derivable from `latency.endToEndLatency`, which prefers the " + "provider's own API-call time and so excludes tool execution and " + "the pre-stream agent build." + ), + ) class Message(BaseModel): diff --git a/backend/src/apis/shared/skills/models.py b/backend/src/apis/shared/skills/models.py index 49e0d1fe0..81cd57310 100644 --- a/backend/src/apis/shared/skills/models.py +++ b/backend/src/apis/shared/skills/models.py @@ -26,6 +26,13 @@ # Regex for a skill_id — identical shape to tool_id (see ToolCreateRequest). SKILL_ID_PATTERN = r"^[a-z][a-z0-9_]{2,49}$" +# Max length of a skill ``description``. This is the Agent Skills spec limit for +# the SKILL.md frontmatter ``description`` field, so a skill authored for Claude +# imports here unchanged. It is a real bound, not a formality: the description is +# the Level-1 catalog line injected into the cacheable system prompt for every +# enabled skill, on every turn. +SKILL_DESCRIPTION_MAX_LENGTH = 1024 + # ``owner_id`` of an admin-catalog skill. Anything else is a user-authored # skill (Skills v2 PR-3): governed by ownership, not by RBAC ``granted_skills``. SYSTEM_OWNER_ID = "system" @@ -291,7 +298,7 @@ class SkillCreateRequest(BaseModel): display_name: str = Field( ..., min_length=1, max_length=100, alias="displayName" ) - description: str = Field(..., max_length=500) + description: str = Field(..., max_length=SKILL_DESCRIPTION_MAX_LENGTH) # SKILL.md body — uncapped (instructions can be long); empty is allowed so # an admin can save a draft and fill it in later. instructions: str = Field(default="") @@ -308,7 +315,9 @@ class SkillUpdateRequest(BaseModel): display_name: Optional[str] = Field( None, min_length=1, max_length=100, alias="displayName" ) - description: Optional[str] = Field(None, max_length=500) + description: Optional[str] = Field( + None, max_length=SKILL_DESCRIPTION_MAX_LENGTH + ) instructions: Optional[str] = None compose: Optional[List[str]] = None status: Optional[SkillStatus] = None diff --git a/backend/src/apis/shared/storage/dynamodb_storage.py b/backend/src/apis/shared/storage/dynamodb_storage.py index 8faaf08af..fd8ca5cf8 100644 --- a/backend/src/apis/shared/storage/dynamodb_storage.py +++ b/backend/src/apis/shared/storage/dynamodb_storage.py @@ -337,6 +337,122 @@ async def get_session_cost_records( except ClientError as e: raise Exception(f"Failed to get session cost records: {e}") + async def get_session_feedback_rows( + self, + session_id: str, + ) -> List[Dict[str, Any]]: + """All ``F#`` message-feedback rows for a session, any user — admin + scope, content-free by projection (``FEEDBACK_ROW_PROJECTION``). + Each row: ``messageId`` (the assistant message's index, the same key + the ``C#`` row carries), ``value`` ±1, optional ``reason`` code, + ``updatedAt``. Empty when the session has no thumbs. + """ + from boto3.dynamodb.conditions import Key + from apis.shared.observability.content_policy import ( + FEEDBACK_ROW_PROJECTION, + build_projection, + strip_content, + ) + + projection, names = build_projection(FEEDBACK_ROW_PROJECTION) + try: + items: List[Dict[str, Any]] = [] + last_evaluated_key = None + while True: + query_kwargs = { + "IndexName": "SessionLookupIndex", + "KeyConditionExpression": ( + Key("GSI_PK").eq(f"SESSION#{session_id}") + & Key("GSI_SK").begins_with("F#") + ), + "ProjectionExpression": projection, + "ExpressionAttributeNames": names, + } + if last_evaluated_key: + query_kwargs["ExclusiveStartKey"] = last_evaluated_key + response = self.sessions_metadata_table.query(**query_kwargs) + items.extend(response.get("Items", [])) + last_evaluated_key = response.get("LastEvaluatedKey") + if not last_evaluated_key: + break + except ClientError as e: + raise Exception(f"Failed to get session feedback rows: {e}") + + return [strip_content(self._convert_decimal_to_float(item)) for item in items] + + async def get_recent_down_thumbs(self, limit: int = 50) -> List[Dict[str, Any]]: + """Newest-first down-thumb rows across the fleet — the eval sampler's + queue as an admin sees it: content-free by projection, no user id.""" + from boto3.dynamodb.conditions import Key + from apis.shared.observability.content_policy import ( + FEEDBACK_ROW_PROJECTION, + build_projection, + strip_content, + ) + + projection, names = build_projection(FEEDBACK_ROW_PROJECTION) + try: + response = self.sessions_metadata_table.query( + IndexName="UserTimestampIndex", + KeyConditionExpression=Key("GSI1PK").eq("FEEDBACK#down"), + ScanIndexForward=False, + Limit=max(1, min(int(limit), 200)), + ProjectionExpression=projection, + ExpressionAttributeNames=names, + ) + except ClientError as e: + raise Exception(f"Failed to list recent down-thumbs: {e}") + return [strip_content(self._convert_decimal_to_float(item)) for item in response.get("Items", [])] + + async def get_feedback_in_window( + self, + start: str, + end: str, + limit: int = 2000, + ) -> List[Dict[str, Any]]: + """Every explicit thumb, both polarities, in an ISO time window — + content-free by projection (no user id: the fleet view is aggregates + only, per response-feedback spec §8 rule 1). + + Returns at most ``limit`` rows per polarity; the caller reports + truncation rather than silently under-counting. + """ + from boto3.dynamodb.conditions import Key + from apis.shared.observability.content_policy import ( + FEEDBACK_ROW_PROJECTION, + build_projection, + strip_content, + ) + + projection, names = build_projection(FEEDBACK_ROW_PROJECTION) + rows: List[Dict[str, Any]] = [] + for partition in ("FEEDBACK#down", "FEEDBACK#up"): + collected: List[Dict[str, Any]] = [] + last_key = None + try: + while len(collected) < limit: + kwargs: Dict[str, Any] = { + "IndexName": "UserTimestampIndex", + "KeyConditionExpression": ( + Key("GSI1PK").eq(partition) & Key("GSI1SK").between(start, end) + ), + "ScanIndexForward": True, + "Limit": min(500, limit - len(collected)), + "ProjectionExpression": projection, + "ExpressionAttributeNames": names, + } + if last_key: + kwargs["ExclusiveStartKey"] = last_key + response = self.sessions_metadata_table.query(**kwargs) + collected.extend(response.get("Items", [])) + last_key = response.get("LastEvaluatedKey") + if not last_key: + break + except ClientError as e: + raise Exception(f"Failed to query feedback window: {e}") + rows.extend(collected[:limit]) + return [strip_content(self._convert_decimal_to_float(item)) for item in rows] + async def get_session_diagnostic_row( self, session_id: str, @@ -825,11 +941,20 @@ async def get_user_session_diagnostics( self, user_id: str, active_since: Optional[str] = None, + include_deleted: bool = False, ) -> List[Dict[str, Any]]: """One user's session rows, content-free, with everything a diagnostic list needs: cost/cache rollups, context, model, enabled tool ids, agent binding, compaction coordinates and the behavioral counters. + ``include_deleted=True`` keeps soft-deleted rows (``deleted`` / + ``status="deleted"``). A delete is a tombstone, not a refund: the + session's ``C#`` rows and its share of the user's period total survive + it, so an audit that hides these rows cannot account for the user's + spend — one prod user showed a single $3.77 conversation against + $20.32 of period cost. The default stays exclusive for callers that + list what the user can still open. + Same bounded base-table query as :meth:`get_user_session_costs`; the difference is the projection (``SESSION_ROW_PROJECTION``) and the post-processing: ``compaction.summary`` is measured into @@ -848,6 +973,7 @@ async def get_user_session_diagnostics( active_since=active_since, projection=projection, names=names, + include_deleted=include_deleted, ) return [self._content_free_session_row(item) for item in items] @@ -857,13 +983,15 @@ async def _query_user_session_rows( active_since: Optional[str], projection: str, names: Optional[Dict[str, str]], + include_deleted: bool = False, ) -> List[Dict[str, Any]]: """Shared body of the two per-user session readers. ``PK = USER#``, ``SK begins_with S#`` — matches both the static (``S#``) and legacy (``S#ACTIVE#…``) schemes and no other row - family. Paginates, converts Decimals, drops soft-deleted rows, and - applies ``active_since`` client-side (``lastMessageAt`` is not a key). + family. Paginates, converts Decimals, drops soft-deleted rows unless + ``include_deleted``, and applies ``active_since`` client-side + (``lastMessageAt`` is not a key). """ from boto3.dynamodb.conditions import Key @@ -890,7 +1018,7 @@ async def _query_user_session_rows( results = [] for item in items: item_float = self._convert_decimal_to_float(item) - if item_float.get("deleted"): + if item_float.get("deleted") and not include_deleted: continue if active_since: last_message_at = item_float.get("lastMessageAt") or "" diff --git a/backend/src/apis/shared/tools/injected.py b/backend/src/apis/shared/tools/injected.py index 2d7b00671..6efbf886e 100644 --- a/backend/src/apis/shared/tools/injected.py +++ b/backend/src/apis/shared/tools/injected.py @@ -42,9 +42,19 @@ # Session workspace files. A single toggle that provisions list/read/write. WORKSPACE_TOOL_IDS = frozenset({"workspace_files"}) -# Every id owned by a per-request factory. Memory-Space tools are deliberately -# absent: they are gated on an Agent's memory binding rather than on -# ``enabled_tools``, so they never reach the filter. +# Document retrieval (bound to session/user). Listed for the record, and +# deliberately NOT folded into ``INJECTED_TOOL_IDS`` below: ``document_read`` +# is gated on the session having a readable attachment, not on +# ``enabled_tools`` — there is no catalog entry, no RBAC grant and no picker +# toggle (docs/specs/document-context-offload.md §4B; the ``workspace_files`` +# key it would otherwise hang off is granted to no prod role). Like the +# Memory-Space tools it never reaches ``ToolFilter``. +DOCUMENT_TOOL_IDS = frozenset({"document_read"}) + +# Every id owned by a per-request factory. Memory-Space and document tools are +# deliberately absent: they are gated on an Agent's memory binding / the +# session's attachments rather than on ``enabled_tools``, so they never reach +# the filter. INJECTED_TOOL_IDS = frozenset( SPREADSHEET_TOOL_IDS | ARTIFACT_TOOL_IDS @@ -72,24 +82,41 @@ # `initialize()` + AgentCore Memory restore (see # docs/specs/agent-cache-extra-tools-bypass.md §1–§2). # -# Deliberately starting at ARTIFACT only. That spec's §6 asks for a -# single-builder experiment rather than more observational data, because the -# bypass→cache-write correlation is confounded by workload; artifacts are the -# clean arm (no `assistant_id`, no memory binding, ~957 sessions). The rest are -# excluded for now, each for a stated reason: +# The set started at ARTIFACT only, as the single-variable arm of that spec's +# §6 experiment. The arm read clean (§8, 2026-08-05: hit/hit/hit after turn 1 +# once #841 pinned sessions to a microVM, ~60% per-turn latency win, no +# prompt-cache regression), so the families that capture exactly what +# artifacts capture are promoted on the same reasoning. Each builder in +# `inference_api/chat/routes.py` closes over `(session_id, user_id)` and +# nothing else — both are key elements: +# +# - WORD (`_build_word_document_tools`), EXCEL +# (`_build_excel_spreadsheet_tools`), POWERPOINT +# (`_build_powerpoint_presentation_tools`), WORKSPACE +# (`_build_workspace_tools`). +# +# - SPREADSHEET (`_build_spreadsheet_tools`) closes over `assistant_id` as +# well. That was NOT a key element until `_create_cache_key` gained it, +# alongside `PausedTurnSnapshot.assistant_id` and the resume path replaying +# it — all three have to agree or a paused agent is orphaned. With the key +# carrying it, the closure is described and the family is eligible. This +# is the dominant cohort: the 2026-08-03 prod read put +# `analyze_spreadsheet` on ~2,669 of 3,565 sessions. +# +# Still excluded: # -# - SPREADSHEET: `make_*_tool(assistant_id, …)` closes over `assistant_id`, -# which is NOT a key element. Needs the key (and `PausedTurnSnapshot`) -# extended first, per that spec's §6. -# - WORD / EXCEL / POWERPOINT / WORKSPACE: capture only session+user, so they -# are *eligible on the same reasoning as artifacts* — held back only so the -# experiment measures one variable. Promote them once the artifact arm reads -# clean. # - Memory-Space tools: capture the resolved binding (space id + access) and # are not gated on `enabled_tools` at all, so they are not in any set here. # `get_agent`'s caller must treat a live memory binding as an independent # veto — see `injected_tools_are_key_described`. -KEY_DESCRIBED_INJECTED_TOOL_IDS = frozenset(ARTIFACT_TOOL_IDS) +KEY_DESCRIBED_INJECTED_TOOL_IDS = frozenset( + ARTIFACT_TOOL_IDS + | WORD_DOCUMENT_TOOL_IDS + | EXCEL_SPREADSHEET_TOOL_IDS + | POWERPOINT_PRESENTATION_TOOL_IDS + | WORKSPACE_TOOL_IDS + | SPREADSHEET_TOOL_IDS +) def injected_tools_are_key_described( diff --git a/backend/tests/agents/builtin_tools/browser/test_takeover_tool.py b/backend/tests/agents/builtin_tools/browser/test_takeover_tool.py new file mode 100644 index 000000000..51db31d4a --- /dev/null +++ b/backend/tests/agents/builtin_tools/browser/test_takeover_tool.py @@ -0,0 +1,707 @@ +"""Tests for `request_user_login` and the pool's control primitives. + +No AWS: the pool is driven against a fake `BrowserClient` that records +`UpdateBrowserStream` calls, and the interrupt against a fake `ToolContext` +that behaves the way Strands does — raise on the first pass, return the stored +response on the second, with the tool re-executed from the top in between. + +The traps these exist to hold down: + +* **Re-entry.** Strands re-runs an interrupted tool from the start, so a naive + implementation takes control twice — the second time against a browser the + user has already handed back. +* **Release on every path.** A turn that leaves the automation stream DISABLED + bricks every later `browse_web` call in the conversation. +* **Reaper pinning.** Idleness is counted in agent tool calls, of which a + takeover makes none, so the plain rule reaps the browser mid-login — but an + unbounded exemption leaves an abandoned session billing to its TTL. +""" + +from __future__ import annotations + +import asyncio +import json +import logging +import time +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from typing import Any, Dict, List, Optional + +import pytest + +from agents.builtin_tools.browser import session_pool, takeover_tool +from agents.builtin_tools.browser.takeover_tool import request_user_login +from strands.interrupt import InterruptException + + +def _call(tool, **kwargs): + """Reach the underlying callable inside the Strands tool wrapper.""" + inner = getattr(tool, "_tool_func", None) or getattr(tool, "func", None) or tool + if hasattr(inner, "__wrapped__"): + inner = inner.__wrapped__ + return inner(**kwargs) + + +# --------------------------------------------------------------------------- +# Fakes +# --------------------------------------------------------------------------- + + +class FakeState: + """`AgentState` deep-copies on read, and so does this.""" + + def __init__(self) -> None: + self._data: Dict[str, Any] = {} + + def get(self, key: str) -> Any: + return json.loads(json.dumps(self._data.get(key))) if key in self._data else None + + def set(self, key: str, value: Any) -> None: + self._data[key] = json.loads(json.dumps(value)) + + +class FakeAgent: + def __init__(self) -> None: + self.state = FakeState() + + +class FakeClient: + """Records the stream flips a real `BrowserClient` would make.""" + + def __init__(self) -> None: + self.identifier = "browser-abc" + self.session_id = "bs-1" + self.stream: List[str] = [] + self.take_fails = False + + def take_control(self) -> None: + if self.take_fails: + raise RuntimeError("UpdateBrowserStream refused") + self.stream.append("DISABLED") + + def release_control(self) -> None: + self.stream.append("ENABLED") + + def stop(self) -> bool: + return True + + +@dataclass +class FakeCdp: + url: str = "https://www.jstor.org/action/showLogin" + closed: bool = False + + async def evaluate(self, expression: str, **_: Any) -> Any: + return self.url if "location.href" in expression else None + + async def close(self) -> None: + self.closed = True + + +class FakeContext: + """A `ToolContext` that reproduces Strands' interrupt semantics. + + `interrupt()` raises the first time and returns the stored response + afterwards — which is exactly why the tool body runs twice. + """ + + def __init__(self, agent: FakeAgent, response: Any = None) -> None: + self.agent = agent + self.tool_use = {"toolUseId": "tu-1"} + self.response = response + self.reasons: List[Any] = [] + + def interrupt(self, name: str, reason: Any = None, response: Any = None) -> Any: + self.reasons.append(reason) + if self.response is None: + raise InterruptException(_FakeInterrupt(name, reason)) + return self.response + + +@dataclass +class _FakeInterrupt: + name: str + reason: Any + id: str = "v1:tool_call:tu-1:abc" + + +@pytest.fixture +def pooled(monkeypatch): + """A live session in the pool, wired to fakes instead of AWS.""" + session_pool._live.clear() + client = FakeClient() + live = session_pool._LiveSession( + session_id="bs-1", + identifier="browser-abc", + client=client, + cdp=FakeCdp(), # type: ignore[arg-type] + ) + session_pool._live["bs-1"] = live + + async def _acquire(agent): + session_pool._write_state( + agent, {"sessionId": "bs-1", "identifier": "browser-abc"} + ) + return live + + monkeypatch.setattr(session_pool, "acquire", _acquire) + # The tool refuses to hand over a browser that has never navigated, so most + # cases here need a conversation that has already browsed. `FakeAgent` + # instances are made per test, so this patches the check rather than the + # state; the one case that exercises the refusal overrides it. + monkeypatch.setattr(session_pool, "has_session", lambda _agent: True) + yield live + session_pool._live.clear() + + +# --------------------------------------------------------------------------- +# The pool's control primitives +# --------------------------------------------------------------------------- + + +class TestTakeAndReleaseControl: + @pytest.mark.asyncio + async def test_take_control_disables_the_automation_stream(self, pooled) -> None: + agent = FakeAgent() + + ref = await session_pool.take_control(agent) + + assert pooled.client.stream == ["DISABLED"] + assert ref["browserSessionId"] == "bs-1" + assert ref["viewport"] == session_pool.DEFAULT_VIEWPORT + assert ref["deadlineAt"] is not None + + @pytest.mark.asyncio + async def test_take_control_is_idempotent_and_does_not_extend_the_deadline( + self, pooled + ) -> None: + agent = FakeAgent() + + first = await session_pool.take_control(agent) + deadline = pooled.control_deadline + second = await session_pool.take_control(agent) + + # One stream flip, not two — the resume pass must not re-take a browser + # the user already holds. + assert pooled.client.stream == ["DISABLED"] + assert pooled.control_deadline == deadline + assert first["browserSessionId"] == second["browserSessionId"] + + @pytest.mark.asyncio + async def test_release_control_re_enables_and_is_idempotent(self, pooled) -> None: + agent = FakeAgent() + await session_pool.take_control(agent) + + assert await session_pool.release_control(agent) is True + assert await session_pool.release_control(agent) is False + assert pooled.client.stream == ["DISABLED", "ENABLED"] + + @pytest.mark.asyncio + async def test_release_works_when_the_resume_lands_in_another_container( + self, monkeypatch, pooled + ) -> None: + # The takeover happened elsewhere: this process has the state but not + # the socket. Returning early here would leave the stream DISABLED + # forever and brick every later browse_web call. + agent = FakeAgent() + session_pool._write_state( + agent, {"sessionId": "bs-1", "identifier": "browser-abc"} + ) + session_pool._live.clear() + + remote = FakeClient() + monkeypatch.setattr( + "bedrock_agentcore.tools.browser_client.BrowserClient", + lambda **_: remote, + ) + + assert await session_pool.release_control(agent) is True + assert remote.stream == ["ENABLED"] + + @pytest.mark.asyncio + async def test_a_failed_release_still_drops_the_local_pin(self, pooled) -> None: + agent = FakeAgent() + await session_pool.take_control(agent) + + def _boom() -> None: + raise RuntimeError("service refused") + + pooled.client.release_control = _boom + + assert await session_pool.release_control(agent) is False + # Pin dropped anyway: keeping it on the strength of a failed API call + # is how an abandoned browser bills to its TTL. + assert pooled.user_controlled() is False + + +class TestReaperPinning: + @pytest.mark.asyncio + async def test_a_session_the_user_is_driving_is_not_reaped(self, pooled) -> None: + agent = FakeAgent() + await session_pool.take_control(agent) + # Idle by the agent's reckoning — a takeover makes no tool calls. + pooled.last_used = time.monotonic() - session_pool.IDLE_REAP_SECONDS - 60 + + await session_pool._reap_idle() + + assert "bs-1" in session_pool._live + assert pooled.cdp.closed is False + + @pytest.mark.asyncio + async def test_an_abandoned_takeover_becomes_reapable_past_the_deadline( + self, pooled + ) -> None: + agent = FakeAgent() + await session_pool.take_control(agent) + pooled.last_used = time.monotonic() - session_pool.IDLE_REAP_SECONDS - 60 + # The user opened the live view and walked away. + pooled.control_deadline = time.monotonic() - 1 + + await session_pool._reap_idle() + + assert "bs-1" not in session_pool._live + + @pytest.mark.asyncio + async def test_an_ordinary_idle_session_is_still_reaped(self, pooled) -> None: + pooled.last_used = time.monotonic() - session_pool.IDLE_REAP_SECONDS - 60 + + await session_pool._reap_idle() + + assert "bs-1" not in session_pool._live + + +# --------------------------------------------------------------------------- +# The tool +# --------------------------------------------------------------------------- + + +class TestRequestUserLogin: + @pytest.mark.asyncio + async def test_first_pass_takes_control_and_pauses_the_turn(self, pooled) -> None: + agent = FakeAgent() + context = FakeContext(agent) + + with pytest.raises(InterruptException): + await _call( + request_user_login, + tool_context=context, + reason="Sign in to JSTOR so I can check the results page.", + ) + + assert pooled.client.stream == ["DISABLED"] + payload = context.reasons[0] + assert payload["type"] == "browser_login_required" + assert payload["browserSessionId"] == "bs-1" + assert payload["reason"].startswith("Sign in to JSTOR") + + @pytest.mark.asyncio + async def test_the_interrupt_payload_carries_no_live_view_url(self, pooled) -> None: + from apis.shared.browser_takeover import assert_no_url + + agent = FakeAgent() + context = FakeContext(agent) + + with pytest.raises(InterruptException): + await _call(request_user_login, tool_context=context, reason="Sign in.") + + assert_no_url(context.reasons[0]) # must not raise + assert "liveViewUrl" not in context.reasons[0] + + @pytest.mark.asyncio + async def test_the_page_is_named_so_the_user_knows_what_they_sign_into( + self, pooled + ) -> None: + agent = FakeAgent() + context = FakeContext(agent) + + with pytest.raises(InterruptException): + await _call(request_user_login, tool_context=context, reason="Sign in.") + + assert context.reasons[0]["targetUrl"] == "https://www.jstor.org/action/showLogin" + + @pytest.mark.asyncio + async def test_resume_releases_control_and_reports_success(self, pooled) -> None: + agent = FakeAgent() + + # Pass 1: pauses. + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + # Pass 2: Strands re-runs the tool from the top with the response. + result = await _call( + request_user_login, + tool_context=FakeContext(agent, response={"completed": True}), + reason="Sign in.", + ) + + assert result["status"] == "success" + assert "authenticated" in result["content"][0]["text"] + # Exactly one DISABLED: the re-run must not take control a second time. + assert pooled.client.stream == ["DISABLED", "ENABLED"] + + @pytest.mark.asyncio + async def test_resume_without_the_session_does_not_start_a_second_browser( + self, monkeypatch, pooled + ) -> None: + # The resume landed in a container that never saw the takeover. Without + # the state marker the tool would re-derive its descriptor through + # `acquire`, which starts a *fresh* browser — unauthenticated, billed, + # and reported to the model as a successful sign-in. + agent = FakeAgent() + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + + async def _must_not_acquire(_agent): + raise AssertionError("resume must not start or re-acquire a session") + + monkeypatch.setattr(session_pool, "acquire", _must_not_acquire) + + result = await _call( + request_user_login, + tool_context=FakeContext(agent, response={"completed": True}), + reason="Sign in.", + ) + + assert result["status"] == "success" + + @pytest.mark.asyncio + async def test_the_takeover_marker_is_cleared_so_a_later_call_can_pause_again( + self, pooled + ) -> None: + agent = FakeAgent() + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + + assert session_pool.read_takeover(agent) is not None + + await _call( + request_user_login, + tool_context=FakeContext(agent, response={"completed": True}), + reason="Sign in.", + ) + + assert session_pool.read_takeover(agent) is None + + @pytest.mark.asyncio + async def test_a_declined_sign_in_still_releases_the_browser(self, pooled) -> None: + agent = FakeAgent() + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + + result = await _call( + request_user_login, + tool_context=FakeContext(agent, response={"skipped": True}), + reason="Sign in.", + ) + + assert result["status"] == "success" + assert "not ask again" in result["content"][0]["text"] + # Release is unconditional: leaving the stream DISABLED after a decline + # would brick every later browse_web call in the conversation. + assert pooled.client.stream[-1] == "ENABLED" + + @pytest.mark.asyncio + async def test_a_completion_far_past_the_deadline_reads_as_expired( + self, pooled + ) -> None: + agent = FakeAgent() + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + + # The session was released and made reapable while the user was away, + # so we cannot honestly report an authenticated browser. + marker = session_pool.read_takeover(agent) + marker["ref"]["deadlineAt"] = ( + datetime.now(timezone.utc) - timedelta(hours=1) + ).isoformat() + session_pool.write_takeover(agent, marker) + + result = await _call( + request_user_login, + tool_context=FakeContext(agent, response={"completed": True}), + reason="Sign in.", + ) + + assert "lapsed" in result["content"][0]["text"] + + @pytest.mark.asyncio + async def test_a_completion_inside_the_grace_window_is_honoured( + self, pooled + ) -> None: + agent = FakeAgent() + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + + # Finished at 7:59, POST landed at 8:01. Throwing the sign-in away here + # would waste a login the user actually completed. + marker = session_pool.read_takeover(agent) + marker["ref"]["deadlineAt"] = ( + datetime.now(timezone.utc) - timedelta(seconds=5) + ).isoformat() + session_pool.write_takeover(agent, marker) + + result = await _call( + request_user_login, + tool_context=FakeContext(agent, response={"completed": True}), + reason="Sign in.", + ) + + assert "authenticated" in result["content"][0]["text"] + + @pytest.mark.asyncio + async def test_an_unparseable_deadline_does_not_discard_a_real_sign_in( + self, pooled + ) -> None: + agent = FakeAgent() + with pytest.raises(InterruptException): + await _call( + request_user_login, tool_context=FakeContext(agent), reason="Sign in." + ) + + marker = session_pool.read_takeover(agent) + marker["ref"]["deadlineAt"] = "not a timestamp" + session_pool.write_takeover(agent, marker) + + result = await _call( + request_user_login, + tool_context=FakeContext(agent, response={"completed": True}), + reason="Sign in.", + ) + + assert "authenticated" in result["content"][0]["text"] + + @pytest.mark.asyncio + async def test_a_failed_handover_errors_instead_of_pausing(self, pooled) -> None: + agent = FakeAgent() + context = FakeContext(agent) + pooled.client.take_fails = True + + result = await _call(request_user_login, tool_context=context, reason="Sign in.") + + assert result["status"] == "error" + assert context.reasons == [] # the turn was never paused + + @pytest.mark.asyncio + async def test_no_browser_session_errors_instead_of_handing_over_a_blank_tab( + self, monkeypatch, pooled + ) -> None: + # The agent called this before browsing anywhere. Starting a session to + # hand the user a blank tab bills an AgentCore browser and pauses the + # turn to achieve nothing. + context = FakeContext(FakeAgent()) + + async def _must_not_acquire(_agent): + raise AssertionError("must not start a session just to hand it over") + + monkeypatch.setattr(session_pool, "has_session", lambda _agent: False) + monkeypatch.setattr(session_pool, "acquire", _must_not_acquire) + + result = await _call(request_user_login, tool_context=context, reason="Sign in.") + + assert result["status"] == "error" + assert "browse_web" in result["content"][0]["text"] + assert context.reasons == [] + + @pytest.mark.asyncio + async def test_a_missing_reason_is_rejected_before_anything_is_touched( + self, pooled + ) -> None: + result = await _call( + request_user_login, tool_context=FakeContext(FakeAgent()), reason=" " + ) + + assert result["status"] == "error" + assert pooled.client.stream == [] + + @pytest.mark.asyncio + async def test_kill_switch_short_circuits_without_touching_the_browser( + self, monkeypatch, pooled + ) -> None: + monkeypatch.setenv("BROWSER_TAKEOVER_ENABLED", "false") + context = FakeContext(FakeAgent()) + + result = await _call(request_user_login, tool_context=context, reason="Sign in.") + + assert result["status"] == "error" + assert pooled.client.stream == [] + assert context.reasons == [] + + @pytest.mark.asyncio + async def test_the_tool_takes_no_credential_argument(self) -> None: + # Structural, and deliberately so: a credential as a tool argument + # would land in the prompt, in Memory and in the cacheable prefix, and + # be re-read every turn for the life of the conversation. + import inspect + + inner = getattr(request_user_login, "_tool_func", None) or request_user_login + params = set(inspect.signature(inner).parameters) + + assert params == {"tool_context", "reason"} + assert not params & {"username", "password", "credentials", "secret", "token"} + + +class TestRegistrationAndCatalog: + """The two gates: a kill switch, and a catalog entry granted per role.""" + + def test_registered_by_default(self, monkeypatch) -> None: + monkeypatch.delenv("BROWSER_TAKEOVER_ENABLED", raising=False) + from agents.main_agent.tools.tool_registry import create_default_registry + + assert create_default_registry().has_tool("request_user_login") + + def test_absent_when_disabled(self, monkeypatch) -> None: + monkeypatch.setenv("BROWSER_TAKEOVER_ENABLED", "false") + from agents.main_agent.tools.tool_registry import create_default_registry + + assert not create_default_registry().has_tool("request_user_login") + + def test_it_is_its_own_catalog_entry_not_a_browse_web_action(self) -> None: + # RBAC granularity is exactly one tool_id: an action on `browse_web` + # would ship an interactive browser in our AWS account to everyone who + # can browse. Separate entry, granted separately, zero prefix tokens + # for anyone without the grant. + from agents.main_agent.tools.tool_catalog import TOOL_CATALOG + + assert "request_user_login" in TOOL_CATALOG + assert TOOL_CATALOG["request_user_login"].tool_id == "request_user_login" + assert "browse_web" in TOOL_CATALOG + assert TOOL_CATALOG["request_user_login"] is not TOOL_CATALOG["browse_web"] + + +class TestViewportFidelity: + """DCV crops unless remoteWidth/remoteHeight match the real session.""" + + @pytest.mark.asyncio + async def test_the_reported_viewport_is_the_one_the_session_started_with( + self, monkeypatch, pooled + ) -> None: + # A session can outlive a config change, or be reconnected to from a + # container whose DEFAULT_VIEWPORT differs. Reporting today's default + # rather than the session's own would crop the stream. + pooled.viewport = {"width": 1600, "height": 900} + monkeypatch.setattr( + session_pool, "DEFAULT_VIEWPORT", {"width": 1280, "height": 800} + ) + + ref = await session_pool.take_control(FakeAgent()) + + assert ref["viewport"] == {"width": 1600, "height": 900} + + +class TestSessionUrlPolicy: + """The Chromium URL policy every session is started with (spec D6). + + ⚠️ Session-level policies are RECOMMENDED-only — the service rejects + MANAGED — so this constrains the *agent*, not a *human* holding the browser + during a takeover. The real control needs `CreateBrowser`. These assert the + policy is well-formed, is never MANAGED (which breaks every session), and + that a misconfiguration is loud rather than silently permissive. + """ + + def test_a_policy_is_built_from_the_s3_uri(self, monkeypatch) -> None: + monkeypatch.setenv("BROWSER_POLICY_S3", "s3://my-bucket/policies/managed.json") + + policies = session_pool._enterprise_policies() + + assert policies == [ + { + "type": "RECOMMENDED", + "location": { + "s3": {"bucket": "my-bucket", "prefix": "policies/managed.json"} + }, + } + ] + + def test_the_session_policy_is_never_managed(self, monkeypatch) -> None: + # Measured on dev 2026-09-19: the API's enum accepts MANAGED but the + # service rejects it — + # "Invalid value for parameter 'type'. MANAGED is not supported for + # session-level policies." + # It does not degrade: StartBrowserSession fails outright, so EVERY + # browser session dies and browse_web stops working. This test exists + # so nobody "tries MANAGED" here again. + monkeypatch.setenv("BROWSER_POLICY_S3", "s3://b/k.json") + + assert session_pool._enterprise_policies()[0]["type"] == "RECOMMENDED" + + def test_a_key_with_slashes_survives_intact(self, monkeypatch) -> None: + monkeypatch.setenv("BROWSER_POLICY_S3", "s3://b/a/b/c/managed.json") + + prefix = session_pool._enterprise_policies()[0]["location"]["s3"]["prefix"] + assert prefix == "a/b/c/managed.json" + + @pytest.mark.parametrize( + "value", ["", " ", "my-bucket/key.json", "https://example.com/p.json", "s3://", "s3://bucket-only"] + ) + def test_an_unusable_setting_yields_no_policy(self, monkeypatch, value) -> None: + monkeypatch.setenv("BROWSER_POLICY_S3", value) + + assert session_pool._enterprise_policies() is None + + def test_a_malformed_setting_is_logged_as_an_error(self, monkeypatch, caplog) -> None: + # Silently starting an unrestricted browser is the failure mode that + # matters here, so it must be loud in the logs. + monkeypatch.setenv("BROWSER_POLICY_S3", "not-a-uri") + + with caplog.at_level(logging.ERROR): + session_pool._enterprise_policies() + + assert "NO url policy" in caplog.text + + @pytest.mark.asyncio + async def test_start_passes_the_policy_through(self, monkeypatch) -> None: + monkeypatch.setenv("BROWSER_POLICY_S3", "s3://my-bucket/policies/managed.json") + captured: Dict[str, Any] = {} + + class RecordingClient: + def __init__(self, **_: Any) -> None: + pass + + def start(self, **kwargs: Any) -> str: + captured.update(kwargs) + return "bs-new" + + monkeypatch.setattr( + "bedrock_agentcore.tools.browser_client.BrowserClient", RecordingClient + ) + + await session_pool._start_remote_session() + + assert captured["enterprise_policies"][0]["type"] == "RECOMMENDED" + assert captured["viewport"] == session_pool.DEFAULT_VIEWPORT + + @pytest.mark.asyncio + async def test_start_omits_the_kwarg_entirely_when_unconfigured( + self, monkeypatch + ) -> None: + # An older SDK or a local dev box must not get `enterprise_policies=None` + # forwarded into the start call. + monkeypatch.delenv("BROWSER_POLICY_S3", raising=False) + captured: Dict[str, Any] = {} + + class RecordingClient: + def __init__(self, **_: Any) -> None: + pass + + def start(self, **kwargs: Any) -> str: + captured.update(kwargs) + return "bs-new" + + monkeypatch.setattr( + "bedrock_agentcore.tools.browser_client.BrowserClient", RecordingClient + ) + + await session_pool._start_remote_session() + + assert "enterprise_policies" not in captured diff --git a/backend/tests/agents/builtin_tools/test_document_read_tool.py b/backend/tests/agents/builtin_tools/test_document_read_tool.py new file mode 100644 index 000000000..9cdd1e4c0 --- /dev/null +++ b/backend/tests/agents/builtin_tools/test_document_read_tool.py @@ -0,0 +1,241 @@ +"""``document_read`` tool factory, its session-state gate, and the two places +its presence has to be accounted for: the agent cache key and the tool-result +offloader's exemption list. + +The gate is the load-bearing decision of docs/specs/document-context-offload.md +§4B: the tool exists for any session with a readable attachment, whatever the +user's RBAC grants or picker state, and its id never enters +``INJECTED_TOOL_IDS``. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from agents.builtin_tools.document_read_tool import ( + DOCUMENT_READ_TOOL_NAME, + make_document_read_tool, + record_document_read, +) +from apis.shared.files.document_read import DocumentReadResult +from apis.shared.files.workspace import WorkspaceFileNotFoundError, WorkspaceStorageNotConfiguredError +from apis.shared.tools.injected import DOCUMENT_TOOL_IDS, INJECTED_TOOL_IDS + +TOOL_MODULE = "agents.builtin_tools.document_read_tool" +ROUTES = "apis.inference_api.chat.routes" + + +async def _call(tool, *args, **kwargs): + fn = getattr(tool, "__wrapped__", None) or tool + return await fn(*args, **kwargs) + + +# --------------------------------------------------------------------------- +# Tool factory +# --------------------------------------------------------------------------- + + +class TestTool: + def test_identity_is_mandatory_and_the_name_is_stable(self): + tool = make_document_read_tool("s1", "u1") + assert tool.tool_name == DOCUMENT_READ_TOOL_NAME == "document_read" + with pytest.raises(ValueError): + make_document_read_tool("", "u1") + with pytest.raises(ValueError): + make_document_read_tool("s1", "") + + def test_id_is_recorded_but_never_an_injected_catalog_id(self): + assert DOCUMENT_TOOL_IDS == {"document_read"} + assert not (DOCUMENT_TOOL_IDS & INJECTED_TOOL_IDS) + + @pytest.mark.asyncio + async def test_no_upload_id_lists_the_sessions_documents(self, monkeypatch): + listing = AsyncMock(return_value={"documents": [{"upload_id": "up-1"}], "count": 1}) + monkeypatch.setattr(f"{TOOL_MODULE}.list_session_documents", listing) + result = await _call(make_document_read_tool("s1", "u1")) + listing.assert_awaited_once_with("u1", "s1") + assert result["status"] == "success" + payload = result["content"][0]["json"] + assert payload["count"] == 1 and "hint" in payload + + @pytest.mark.asyncio + async def test_page_read_appends_the_native_block_after_the_metadata(self, monkeypatch): + block = {"document": {"format": "pdf", "name": "policy p4-7 abc123", "source": {"bytes": b"%PDF"}}} + read = AsyncMock(return_value=DocumentReadResult( + mode="pages", payload={"pages_returned": 4}, document_block=block, pages_returned=4, bytes_returned=4, format="pdf", + )) + monkeypatch.setattr(f"{TOOL_MODULE}.read_document", read) + result = await _call(make_document_read_tool("s1", "u1"), upload_id="up-1", page_range="4-7", max_pages=6) + read.assert_awaited_once_with("u1", "s1", "up-1", page_range="4-7", pattern=None, max_pages=6, offset=0) + assert result["content"] == [{"json": {"pages_returned": 4}}, block] + + @pytest.mark.asyncio + async def test_empty_strings_mean_absent_arguments(self, monkeypatch): + read = AsyncMock(return_value=DocumentReadResult(mode="index", payload={})) + monkeypatch.setattr(f"{TOOL_MODULE}.read_document", read) + await _call(make_document_read_tool("s1", "u1"), upload_id="up-1", page_range="", pattern="") + assert read.await_args.kwargs["page_range"] is None and read.await_args.kwargs["pattern"] is None + + @pytest.mark.asyncio + @pytest.mark.parametrize( + "exc,fragment", + [ + (WorkspaceStorageNotConfiguredError("no bucket"), "not configured"), + (WorkspaceFileNotFoundError("No file with id 'up-9'"), "up-9"), + (RuntimeError("s3 down"), "s3 down"), + ], + ) + async def test_failures_surface_as_error_results_not_exceptions(self, monkeypatch, exc, fragment): + monkeypatch.setattr(f"{TOOL_MODULE}.read_document", AsyncMock(side_effect=exc)) + result = await _call(make_document_read_tool("s1", "u1"), upload_id="up-9") + assert result["status"] == "error" + assert fragment in result["content"][0]["text"] + + def test_metric_is_content_free_and_in_the_compaction_namespace(self, monkeypatch): + calls = [] + monkeypatch.setattr( + "apis.shared.observability.emf.emit_emf_metrics", + lambda ns, metrics, properties=None, units=None: calls.append((ns, metrics, properties)), + ) + monkeypatch.setattr("apis.shared.observability.prompt_cache.prompt_cache_observability_enabled", lambda: True) + record_document_read(DocumentReadResult( + mode="pages", payload={"filename": "secret.pdf"}, pages_returned=4, bytes_returned=900, format="pdf", + )) + assert calls == [( + "AgentCoreStack/Compaction", + {"DocumentRead": 1, "DocumentReadPages": 4, "DocumentReadBytes": 900}, + {"mode": "pages", "format": "pdf"}, + )] + assert "secret" not in repr(calls) + + +# --------------------------------------------------------------------------- +# Routes gate +# --------------------------------------------------------------------------- + + +@pytest.fixture +def clear_memo(): + from apis.inference_api.chat import routes + + routes._DOCUMENT_SESSIONS.clear() + yield + routes._DOCUMENT_SESSIONS.clear() + + +class TestRoutesGate: + @pytest.mark.asyncio + async def test_this_turns_uploads_build_the_tool_without_a_query(self, monkeypatch, clear_memo): + from apis.inference_api.chat.routes import _build_document_tools + + lookup = AsyncMock(return_value=False) + monkeypatch.setattr("apis.shared.files.document_read.session_has_documents", lookup) + tools = await _build_document_tools("s1", "u1", turn_upload_ids=["up-1"]) + assert [t.tool_name for t in tools] == ["document_read"] + lookup.assert_not_awaited() + + @pytest.mark.asyncio + async def test_a_session_with_a_document_gets_the_tool_regardless_of_enabled_tools(self, monkeypatch, clear_memo): + from apis.inference_api.chat.routes import _build_document_tools + + lookup = AsyncMock(return_value=True) + monkeypatch.setattr("apis.shared.files.document_read.session_has_documents", lookup) + assert len(await _build_document_tools("s1", "u1")) == 1 + lookup.assert_awaited_once_with("u1", "s1") + # Memoized: the second turn does not query again. + assert len(await _build_document_tools("s1", "u1")) == 1 + assert lookup.await_count == 1 + + @pytest.mark.asyncio + async def test_no_document_means_no_tool_and_no_memo(self, monkeypatch, clear_memo): + from apis.inference_api.chat.routes import _DOCUMENT_SESSIONS, _build_document_tools + + lookup = AsyncMock(return_value=False) + monkeypatch.setattr("apis.shared.files.document_read.session_has_documents", lookup) + assert await _build_document_tools("s1", "u1") == [] + assert await _build_document_tools("s1", "u1") == [] + assert lookup.await_count == 2 and "s1" not in _DOCUMENT_SESSIONS + + @pytest.mark.asyncio + async def test_lookup_failure_fails_closed(self, monkeypatch, clear_memo): + from apis.inference_api.chat.routes import _build_document_tools + + monkeypatch.setattr("apis.shared.files.document_read.session_has_documents", AsyncMock(side_effect=RuntimeError("ddb"))) + assert await _build_document_tools("s1", "u1") == [] + + @pytest.mark.asyncio + async def test_kill_switch_and_missing_identity(self, monkeypatch, clear_memo): + from apis.inference_api.chat.routes import _build_document_tools, _document_tools_gate + + monkeypatch.setenv("DOCUMENT_READ_ENABLED", "false") + assert await _build_document_tools("s1", "u1", turn_upload_ids=["up-1"]) == [] + assert await _document_tools_gate("s1", "u1", turn_upload_ids=["up-1"]) is False + monkeypatch.setenv("DOCUMENT_READ_ENABLED", "") + assert await _document_tools_gate("s1", "u1", turn_upload_ids=["up-1"]) is True + assert await _document_tools_gate("", "u1", turn_upload_ids=["up-1"]) is False + + def test_memo_is_bounded(self, monkeypatch, clear_memo): + from apis.inference_api.chat import routes + + monkeypatch.setattr(routes, "_DOCUMENT_SESSIONS_MAX", 3) + for i in range(5): + routes._remember_document_session(f"s{i}") + assert list(routes._DOCUMENT_SESSIONS) == ["s2", "s3", "s4"] + + +# --------------------------------------------------------------------------- +# Agent cache key +# --------------------------------------------------------------------------- + + +def test_cache_key_carries_the_document_tool_bit_without_moving_the_skills_hash(): + from apis.inference_api.chat import service + + base = dict( + session_id="s", user_id="u", enabled_tools=["t"], model_id="m", inference_params={}, + system_prompt=None, caching_enabled=False, provider="bedrock", freshness_hash="f", agent_type="chat", + ) + without = service._create_cache_key(**base, skills_hash="k") + with_docs = service._create_cache_key(**base, skills_hash="k", document_tools=True) + assert without != with_docs + assert without[-1] == with_docs[-1] == "k" + # Trailing layout: (..., document_tools, assistant_id, skills_hash). + assert without[-2] == with_docs[-2] == "" + assert without[-3] is False and with_docs[-3] is True + + +# --------------------------------------------------------------------------- +# Offloader exemption +# --------------------------------------------------------------------------- + + +class TestOffloaderExemption: + def test_document_read_results_are_never_offloaded(self): + from agents.main_agent.core.tool_result_offload import OFFLOAD_EXEMPT_TOOLS, _exempt_tool, _should_offload + + assert "document_read" in OFFLOAD_EXEMPT_TOOLS + assert _exempt_tool(SimpleNamespace(tool_use={"name": "document_read"})) is True + assert _exempt_tool(SimpleNamespace(tool_use={"name": "gmail_search"})) is False + assert _exempt_tool(SimpleNamespace(tool_use=None)) is False + assert _should_offload("document_read", 90_000) is False + assert _should_offload("gmail_search", 90_000) is True + + @pytest.mark.asyncio + async def test_mixin_returns_before_counting_tokens_for_an_exempt_tool(self): + from agents.main_agent.core.tool_result_offload import _OffloaderMixin + + class _Base: + async def _handle_tool_result(self, event): # pragma: no cover - must not run + raise AssertionError("base offloader reached for an exempt tool") + + class _Sut(_OffloaderMixin, _Base): + _max_result_tokens = 10 + + big = "x" * 4_000 + result = {"toolUseId": "t1", "status": "success", "content": [{"text": big}]} + event = SimpleNamespace(result=result, tool_use={"toolUseId": "t1", "name": "document_read"}) + await _Sut()._handle_tool_result(event) + assert event.result is result diff --git a/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py b/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py index 16d2caeee..39986fbf5 100644 --- a/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py +++ b/backend/tests/agents/main_agent/core/test_bedrock_cache_points.py @@ -358,3 +358,64 @@ def test_the_static_points_are_untouched(self, model): assert steered["system"] == plain["system"] # Everything before the mixed message is identical too. assert steered["messages"][:-1] == plain["messages"][:-1] + + +# --------------------------------------------------------------------------- +# PR-5: selective 1h TTL on the STATIC prefix (thresholds spec §3.6) +# --------------------------------------------------------------------------- + +class TestStaticPrefixLongTtl: + """AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h puts a 1h TTL on the tools + and system points only; the message point stays at the 5m default, and + the flag off emits exactly today's bytes.""" + + def _model(self, monkeypatch, value): + monkeypatch.setenv("AWS_REGION", "us-west-2") + if value is None: + monkeypatch.delenv("AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL", raising=False) + else: + monkeypatch.setenv("AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL", value) + from agents.main_agent.core.bedrock_count_tokens import CountTokensBedrockModel + + config = ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=True) + return config, CountTokensBedrockModel(**config.to_bedrock_config()) + + def _request(self, model): + return model.format_request( + [{"role": "user", "content": [{"text": "hi"}]}], + [{"name": "t", "description": "d", "inputSchema": {"json": {"type": "object", "properties": {}}}}], + system_prompt_content=[{"text": "sys"}, {"cachePoint": {"type": "default"}}], + ) + + def test_flag_on_sets_1h_on_tools_and_system_only(self, monkeypatch): + config, model = self._model(monkeypatch, "1h") + cc = config.to_bedrock_config()["cache_config"] + assert cc.tools_ttl == "1h" and cc.system_prompt_ttl == "1h" and cc.ttl is None + assert config.long_ttl_static_prefix() is True + req = self._request(model) + assert req["toolConfig"]["tools"][-1] == {"cachePoint": {"type": "default", "ttl": "1h"}} + assert req["system"][-1] == {"cachePoint": {"type": "default", "ttl": "1h"}} + assert req["messages"][-1]["content"][-1] == {"cachePoint": {"type": "default"}} + assert _count_cache_points(req) == 3 + + @pytest.mark.parametrize("value", [None, "", "5m", "2h", "true"]) + def test_anything_but_1h_is_todays_bytes(self, monkeypatch, value): + config, model = self._model(monkeypatch, value) + cc = config.to_bedrock_config()["cache_config"] + assert cc.tools_ttl is True and cc.system_prompt_ttl is True + assert config.long_ttl_static_prefix() is False + req = self._request(model) + assert req["toolConfig"]["tools"][-1] == {"cachePoint": {"type": "default"}} + assert req["system"][-1] == {"cachePoint": {"type": "default"}} + assert _count_cache_points(req) == 3 + + def test_non_anthropic_model_is_unaffected(self, monkeypatch): + monkeypatch.setenv("AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL", "1h") + config = ModelConfig(model_id="us.amazon.nova-micro-v1:0", caching_enabled=True) + cc = config.to_bedrock_config()["cache_config"] + assert cc.tools_ttl is False + assert config.long_ttl_static_prefix() is False + + def test_caching_disabled_means_no_long_ttl(self, monkeypatch): + monkeypatch.setenv("AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL", "1h") + assert ModelConfig(model_id=CLAUDE_MODEL_ID, caching_enabled=False).long_ttl_static_prefix() is False diff --git a/backend/tests/agents/main_agent/core/test_bedrock_count_tokens.py b/backend/tests/agents/main_agent/core/test_bedrock_count_tokens.py index c21241da5..5d23a677a 100644 --- a/backend/tests/agents/main_agent/core/test_bedrock_count_tokens.py +++ b/backend/tests/agents/main_agent/core/test_bedrock_count_tokens.py @@ -8,11 +8,18 @@ from unittest.mock import patch import pytest -from strands.models import BedrockModel +from botocore.exceptions import ClientError +from strands.models.bedrock import _SKIP_COUNT_TOKENS_MODELS, _clear_skip_count_tokens_cache +from strands.models.model import Model from agents.main_agent.core.bedrock_count_tokens import ( + COUNT_TOKENS_MAX_ATTEMPTS_ENV, + COUNT_TOKENS_TIMEOUT_ENV, CountTokensBedrockModel, base_foundation_model_id, + build_count_tokens_client_config, + count_tokens_max_attempts, + count_tokens_timeout_seconds, ) @@ -59,63 +66,201 @@ def _aws_region(monkeypatch): monkeypatch.setenv("AWS_DEFAULT_REGION", "us-east-1") -class TestCountTokensModelIdSwap: - """count_tokens must count against the base id, then restore the profile id - so invocation keeps using the inference profile.""" +PROFILE_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0" +BASE_ID = "anthropic.claude-haiku-4-5-20251001-v1:0" + + +class FakeCountClient: + """Stands in for the dedicated CountTokens boto client.""" + + def __init__(self, result=4242, raise_with=None): + self.result = result + self.raise_with = raise_with + self.calls = [] + + def count_tokens(self, **kwargs): + self.calls.append(kwargs) + if self.raise_with is not None: + raise self.raise_with + return {"inputTokens": self.result} + + +def _client_error(code, message="nope"): + return ClientError({"Error": {"Code": code, "Message": message}}, "CountTokens") + + +@pytest.fixture(autouse=True) +def _clean_skip_cache(): + _clear_skip_count_tokens_cache() + yield + _clear_skip_count_tokens_cache() + + +def _model(model_id=PROFILE_ID, native=True, client=None): + model = CountTokensBedrockModel(model_id=model_id, use_native_token_count=native) + if client is not None: + model._count_client = client + return model + + +class TestCountTokensModelId: + """The count goes to the API with the base id; the profile id in config is + never touched, before, during or after.""" @pytest.mark.asyncio - async def test_counts_against_base_id_and_restores_profile_id(self, _aws_region): - model = CountTokensBedrockModel( - model_id="us.anthropic.claude-haiku-4-5-20251001-v1:0", - use_native_token_count=True, - ) + async def test_counts_against_base_id_without_mutating_config(self, _aws_region): + client = FakeCountClient(result=4242) + model = _model(client=client) + + result = await model.count_tokens([], system_prompt="hi") + assert result == 4242 + assert client.calls[0]["modelId"] == BASE_ID + assert client.calls[0]["input"]["converse"]["system"] == [{"text": "hi"}] + assert model.config["model_id"] == PROFILE_ID + + @pytest.mark.asyncio + async def test_config_is_not_mutated_even_mid_call(self, _aws_region): + """A concurrent stream() must never observe the base id — that was the + hazard of the old swap-and-restore approach.""" + model = _model() seen = {} - async def fake_super_count(self, messages, tool_specs=None, system_prompt=None, system_prompt_content=None): - seen["model_id_during_count"] = self.config["model_id"] - return 4242 + class SpyClient(FakeCountClient): + def count_tokens(self, **kwargs): + seen["model_id_during_count"] = model.config["model_id"] + return super().count_tokens(**kwargs) - with patch.object(BedrockModel, "count_tokens", fake_super_count): - result = await model.count_tokens([], system_prompt="hi") + model._count_client = SpyClient(result=1) + await model.count_tokens([]) - assert result == 4242 - # The base id was used for the CountTokens call... - assert seen["model_id_during_count"] == "anthropic.claude-haiku-4-5-20251001-v1:0" - # ...and the profile id is restored for invocation afterward. - assert model.config["model_id"] == "us.anthropic.claude-haiku-4-5-20251001-v1:0" + assert seen["model_id_during_count"] == PROFILE_ID @pytest.mark.asyncio - async def test_no_swap_when_already_base_id(self, _aws_region): - model = CountTokensBedrockModel( - model_id="anthropic.claude-3-5-sonnet-20241022-v2:0", - use_native_token_count=True, - ) + async def test_base_id_passes_through_unchanged(self, _aws_region): + client = FakeCountClient(result=7) + model = _model(model_id="anthropic.claude-3-5-sonnet-20241022-v2:0", client=client) - seen = {} + await model.count_tokens([]) - async def fake_super_count(self, messages, tool_specs=None, system_prompt=None, system_prompt_content=None): - seen["model_id_during_count"] = self.config["model_id"] - return 7 + assert client.calls[0]["modelId"] == "anthropic.claude-3-5-sonnet-20241022-v2:0" - with patch.object(BedrockModel, "count_tokens", fake_super_count): - await model.count_tokens([]) + @pytest.mark.asyncio + async def test_native_flag_off_never_builds_a_client(self, _aws_region): + model = _model(native=False) - assert seen["model_id_during_count"] == "anthropic.claude-3-5-sonnet-20241022-v2:0" - assert model.config["model_id"] == "anthropic.claude-3-5-sonnet-20241022-v2:0" + with patch.object(CountTokensBedrockModel, "_get_count_client") as get_client: + result = await model.count_tokens([{"role": "user", "content": [{"text": "hello world"}]}]) + + get_client.assert_not_called() + assert isinstance(result, int) and result > 0 + + +class TestCountTokensBounded: + """A throttle costs one failed attempt and falls back to the heuristic; + it is never retried and never raised into the turn.""" @pytest.mark.asyncio - async def test_profile_id_restored_even_when_count_raises(self, _aws_region): - model = CountTokensBedrockModel( - model_id="us.anthropic.claude-haiku-4-5-20251001-v1:0", - use_native_token_count=True, + async def test_throttle_falls_back_to_heuristic_after_one_attempt(self, _aws_region): + client = FakeCountClient(raise_with=_client_error("ThrottlingException")) + model = _model(client=client) + messages = [{"role": "user", "content": [{"text": "x" * 400}]}] + + result = await model.count_tokens(messages) + + assert len(client.calls) == 1 + # chars/4 heuristic, not the native answer and not an exception. + assert result == await Model.count_tokens(model, messages) + # A throttle is transient: the model is NOT added to the skip list. + assert BASE_ID not in _SKIP_COUNT_TOKENS_MODELS + + @pytest.mark.asyncio + async def test_access_denied_is_remembered_and_skips_the_client(self, _aws_region): + client = FakeCountClient(raise_with=_client_error("AccessDeniedException")) + model = _model(client=client) + + await model.count_tokens([]) + await model.count_tokens([]) + + assert len(client.calls) == 1 + assert BASE_ID in _SKIP_COUNT_TOKENS_MODELS + + @pytest.mark.asyncio + async def test_unsupported_model_is_remembered(self, _aws_region): + client = FakeCountClient( + raise_with=_client_error("ValidationException", "The provided model doesn't support counting tokens.") ) + model = _model(client=client) + + await model.count_tokens([]) + + assert BASE_ID in _SKIP_COUNT_TOKENS_MODELS + + @pytest.mark.asyncio + async def test_unexpected_error_falls_back_without_raising(self, _aws_region): + client = FakeCountClient(raise_with=RuntimeError("boom")) + model = _model(client=client) + + result = await model.count_tokens([{"role": "user", "content": [{"text": "hi"}]}]) + + assert isinstance(result, int) + assert model.config["model_id"] == PROFILE_ID + + @pytest.mark.asyncio + async def test_missing_input_tokens_falls_back(self, _aws_region): + class NoneClient(FakeCountClient): + def count_tokens(self, **kwargs): + self.calls.append(kwargs) + return {} + + model = _model(client=NoneClient()) + + result = await model.count_tokens([{"role": "user", "content": [{"text": "hi"}]}]) + + assert isinstance(result, int) + + +class TestCountTokensClientConfig: + """The dedicated client is what bounds the cost of a throttle.""" + + def test_default_is_one_attempt_and_a_two_second_timeout(self, monkeypatch): + monkeypatch.delenv(COUNT_TOKENS_TIMEOUT_ENV, raising=False) + monkeypatch.delenv(COUNT_TOKENS_MAX_ATTEMPTS_ENV, raising=False) + + cfg = build_count_tokens_client_config() + + assert cfg.retries == {"total_max_attempts": 1, "mode": "standard"} + assert cfg.read_timeout == 2.0 + assert cfg.connect_timeout == 2.0 + + def test_env_overrides_are_honoured(self, monkeypatch): + monkeypatch.setenv(COUNT_TOKENS_TIMEOUT_ENV, "0.5") + monkeypatch.setenv(COUNT_TOKENS_MAX_ATTEMPTS_ENV, "3") + + cfg = build_count_tokens_client_config() + + assert cfg.retries["total_max_attempts"] == 3 + assert cfg.read_timeout == 0.5 + + def test_garbage_env_falls_back_to_defaults(self, monkeypatch): + monkeypatch.setenv(COUNT_TOKENS_TIMEOUT_ENV, "soon") + monkeypatch.setenv(COUNT_TOKENS_MAX_ATTEMPTS_ENV, "0") + + assert count_tokens_timeout_seconds() == 2.0 + assert count_tokens_max_attempts() == 1 - async def fake_super_count(self, *args, **kwargs): - raise RuntimeError("boom") + def test_client_is_built_lazily_in_the_invocation_region(self, _aws_region): + model = _model() + assert model._count_client is None - with patch.object(BedrockModel, "count_tokens", fake_super_count): - with pytest.raises(RuntimeError, match="boom"): - await model.count_tokens([]) + with patch("agents.main_agent.core.bedrock_count_tokens.boto3.client") as boto_client: + boto_client.return_value = object() + first = model._get_count_client() + second = model._get_count_client() - assert model.config["model_id"] == "us.anthropic.claude-haiku-4-5-20251001-v1:0" + assert first is second + boto_client.assert_called_once() + kwargs = boto_client.call_args.kwargs + assert boto_client.call_args.args == ("bedrock-runtime",) + assert kwargs["region_name"] == "us-east-1" + assert kwargs["config"].retries["total_max_attempts"] == 1 diff --git a/backend/tests/agents/main_agent/core/test_conversation_window.py b/backend/tests/agents/main_agent/core/test_conversation_window.py new file mode 100644 index 000000000..246402a7d --- /dev/null +++ b/backend/tests/agents/main_agent/core/test_conversation_window.py @@ -0,0 +1,32 @@ +"""The chat agent must not run under Strands' default 40-message window. + +docs/specs/compaction-model-relative-thresholds.md §3.0: compaction owns +history size; the SDK window would slide the front of the list every turn +past 40 messages (a prefix re-write per turn) and move the coordinates the +compaction checkpoint is expressed in. +""" + +from strands.agent.conversation_manager import SlidingWindowConversationManager + +from agents.main_agent.config.constants import Defaults +from agents.main_agent.core.agent_factory import AgentFactory + + +def test_default_window_is_large_not_forty(monkeypatch): + monkeypatch.delenv("AGENTCORE_CONVERSATION_WINDOW_MESSAGES", raising=False) + cm = AgentFactory.build_conversation_manager() + assert isinstance(cm, SlidingWindowConversationManager) + assert cm.window_size == Defaults.CONVERSATION_WINDOW_MESSAGES + assert cm.window_size >= 1000 + # Overflow recovery keeps tool-result truncation on. + assert cm.should_truncate_results is True + + +def test_env_override_restores_sdk_default(monkeypatch): + monkeypatch.setenv("AGENTCORE_CONVERSATION_WINDOW_MESSAGES", "40") + assert AgentFactory.build_conversation_manager().window_size == 40 + + +def test_garbage_env_falls_back(monkeypatch): + monkeypatch.setenv("AGENTCORE_CONVERSATION_WINDOW_MESSAGES", "lots") + assert AgentFactory.build_conversation_manager().window_size == Defaults.CONVERSATION_WINDOW_MESSAGES diff --git a/backend/tests/agents/main_agent/core/test_system_prompt_builder.py b/backend/tests/agents/main_agent/core/test_system_prompt_builder.py index c08dafd9e..f56e9859c 100644 --- a/backend/tests/agents/main_agent/core/test_system_prompt_builder.py +++ b/backend/tests/agents/main_agent/core/test_system_prompt_builder.py @@ -32,17 +32,17 @@ class TestBuildWithDateTrue: @patch( "agents.main_agent.core.system_prompt_builder.get_current_date_pacific", - return_value="2024-06-15 (Saturday) 10:00 PDT", + return_value="2024-06-15 (Saturday) PDT", ) def test_appends_current_date_line(self, mock_date): builder = SystemPromptBuilder() result = builder.build(include_date=True) - assert result.endswith("Current date: 2024-06-15 (Saturday) 10:00 PDT") + assert result.endswith("Current date: 2024-06-15 (Saturday) PDT") @patch( "agents.main_agent.core.system_prompt_builder.get_current_date_pacific", - return_value="2024-06-15 (Saturday) 10:00 PDT", + return_value="2024-06-15 (Saturday) PDT", ) def test_includes_base_prompt(self, mock_date): builder = SystemPromptBuilder() @@ -52,13 +52,13 @@ def test_includes_base_prompt(self, mock_date): @patch( "agents.main_agent.core.system_prompt_builder.get_current_date_pacific", - return_value="2024-01-01 (Monday) 08:00 PST", + return_value="2024-01-01 (Monday) PST", ) def test_date_separated_by_blank_line(self, mock_date): builder = SystemPromptBuilder() result = builder.build(include_date=True) - expected = f"{DEFAULT_SYSTEM_PROMPT}\n\nCurrent date: 2024-01-01 (Monday) 08:00 PST" + expected = f"{DEFAULT_SYSTEM_PROMPT}\n\nCurrent date: 2024-01-01 (Monday) PST" assert result == expected @@ -102,14 +102,14 @@ def test_uses_custom_prompt(self): @patch( "agents.main_agent.core.system_prompt_builder.get_current_date_pacific", - return_value="2024-03-20 (Wednesday) 15:00 PDT", + return_value="2024-03-20 (Wednesday) PDT", ) def test_build_with_date_uses_custom_prompt(self, mock_date): custom = "Custom prompt." builder = SystemPromptBuilder(base_prompt=custom) result = builder.build(include_date=True) - assert result == "Custom prompt.\n\nCurrent date: 2024-03-20 (Wednesday) 15:00 PDT" + assert result == "Custom prompt.\n\nCurrent date: 2024-03-20 (Wednesday) PDT" def test_none_base_prompt_falls_back_to_default(self): builder = SystemPromptBuilder(base_prompt=None) @@ -159,3 +159,28 @@ def test_build_without_date_includes_floor_and_user_prompt(self): assert result.startswith(PLATFORM_SAFETY_FLOOR) assert user_prompt in result + + +# --------------------------------------------------------------------------- +# KaTeX guidance: the SPA does not treat a bare "$" as a math delimiter +# --------------------------------------------------------------------------- +class TestKatexGuidance: + """The prompt must not resurrect the HTML-entity workaround for "$". + + The prompt once told the model to write other uses of "$" as "$". + That never worked: marked emits the entity into innerHTML, the browser + decodes it to a literal "$" in the text node, and KaTeX walks the DOM + afterwards -- so the entity form broke identically. It did, however, leak + the 9-character string "$100K" into generated .pptx/.xlsx cells. The + real fix is in the SPA (see katex-delimiters.ts), which drops the bare + "$...$" delimiter, so the model should write currency as a plain "$". + """ + + def test_does_not_tell_the_model_to_escape_dollar_signs(self): + assert "$" not in DEFAULT_SYSTEM_PROMPT + + def test_names_the_supported_inline_math_delimiters(self): + assert r"$...$ or \(...\) for inline math" in DEFAULT_SYSTEM_PROMPT + + def test_still_offers_katex_for_equations(self): + assert "KaTeX" in DEFAULT_SYSTEM_PROMPT diff --git a/backend/tests/agents/main_agent/core/test_tool_result_offload.py b/backend/tests/agents/main_agent/core/test_tool_result_offload.py new file mode 100644 index 000000000..4426bdff5 --- /dev/null +++ b/backend/tests/agents/main_agent/core/test_tool_result_offload.py @@ -0,0 +1,245 @@ +"""Tool-result offload at intake — thresholds spec §3.6 (PR-4).""" + +import sys +import types +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from agents.main_agent.core import tool_result_offload as tro +from agents.main_agent.core.tool_result_offload import ( + PREFILTER_RATIO, + build_tool_result_offloader, + offload_prefix, + tool_result_offload_enabled, +) + + +class _FakeStorage: + """Unified-Storage-shaped in-memory backend (write/read), like strands.storage.""" + + def __init__(self): + self.objects = {} + + async def write(self, key, data): + self.objects[key] = data + + async def read(self, key): + return self.objects.get(key) + + async def delete(self, key): + self.objects.pop(key, None) + + async def list(self, query=""): + return [k for k in self.objects if k.startswith(query)] + + +class _Agent: + """Weak-referenceable stand-in (the plugin keys WeakKeyDictionaries on the agent).""" + + def __init__(self, model=None): + self.model = model + self.event_loop_metrics = SimpleNamespace(cycle_count=1) + self.storage = None + self.sandbox = None + + +def _event(text, tool_name="gmail_search", count_tokens=None): + result = {"toolUseId": "t1", "status": "success", "content": [{"text": text}]} + model = SimpleNamespace(count_tokens=count_tokens or AsyncMock(return_value=len(text) // 4)) + agent = _Agent(model) + return SimpleNamespace( + result=result, + tool_use={"toolUseId": "t1", "name": tool_name}, + selected_tool=None, + cancel_message=None, + agent=agent, + ) + + +@pytest.fixture +def offloader(monkeypatch): + monkeypatch.setenv("AGENTCORE_TOOL_RESULT_OFFLOAD_MAX_TOKENS", "1000") + monkeypatch.setenv("AGENTCORE_TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS", "100") + storage = _FakeStorage() + plugin = build_tool_result_offloader(session_id="s1", user_id="u1", storage=storage) + assert plugin is not None + # Bind storage the way init_agent would. + plugin.init_agent(_Agent()) + return plugin, storage + + +class TestBuilder: + def test_flag_default_on_only_literal_false_off(self, monkeypatch): + monkeypatch.delenv("AGENTCORE_TOOL_RESULT_OFFLOAD_ENABLED", raising=False) + assert tool_result_offload_enabled() is True + monkeypatch.setenv("AGENTCORE_TOOL_RESULT_OFFLOAD_ENABLED", "false") + assert tool_result_offload_enabled() is False + assert build_tool_result_offloader(session_id="s", user_id="u", storage=_FakeStorage()) is None + + def test_no_bucket_means_no_plugin(self, monkeypatch): + monkeypatch.delenv("S3_USER_FILES_BUCKET_NAME", raising=False) + assert build_tool_result_offloader(session_id="s", user_id="u") is None + + def test_s3_backend_is_scoped_per_user_and_session(self, monkeypatch): + monkeypatch.setenv("S3_USER_FILES_BUCKET_NAME", "files-bucket") + captured = {} + + class FakeS3Storage: + def __init__(self, bucket, *, prefix="", region_name=None, **kw): + captured.update(bucket=bucket, prefix=prefix, region=region_name) + + async def write(self, k, d): ... + async def read(self, k): ... + async def delete(self, k): ... + async def list(self, q=""): return [] + + monkeypatch.setitem(sys.modules, "strands.storage", types.SimpleNamespace(S3Storage=FakeS3Storage)) + plugin = build_tool_result_offloader(session_id="sess", user_id="usr", region="us-west-2") + assert plugin is not None + assert captured == {"bucket": "files-bucket", "prefix": "compaction-offload/usr/sess", "region": "us-west-2"} + assert offload_prefix("usr", "sess") == "compaction-offload/usr/sess" + + def test_configuration(self, offloader): + plugin, _ = offloader + assert plugin._max_result_tokens == 1000 + assert plugin._preview_tokens == 100 + assert plugin._evict_after_cycles is None # no eviction from the model path + assert plugin._include_retrieval_tool is True + assert type(plugin).__name__ == "BoundedToolResultOffloader" + + def test_preview_is_clamped_below_gate(self, monkeypatch): + monkeypatch.setenv("AGENTCORE_TOOL_RESULT_OFFLOAD_MAX_TOKENS", "500") + monkeypatch.setenv("AGENTCORE_TOOL_RESULT_OFFLOAD_PREVIEW_TOKENS", "9000") + plugin = build_tool_result_offloader(session_id="s", user_id="u", storage=_FakeStorage()) + assert plugin._preview_tokens == 499 + + +class TestPrefilterAndOffload: + @pytest.mark.asyncio + async def test_small_result_skips_count_tokens_and_is_untouched(self, offloader): + plugin, storage = offloader + counter = AsyncMock(return_value=10) + ev = _event("short result", count_tokens=counter) + before = ev.result + await plugin._handle_tool_result(ev) + counter.assert_not_called() + assert ev.result is before and storage.objects == {} + + @pytest.mark.asyncio + async def test_borderline_result_is_measured_by_count_tokens(self, offloader): + plugin, _ = offloader + # ~600 estimated tokens: over the pre-filter (500) but under the gate (1000). + counter = AsyncMock(return_value=600) + ev = _event("x" * 2400, count_tokens=counter) + before = ev.result + await plugin._handle_tool_result(ev) + counter.assert_called_once() + assert ev.result is before + + @pytest.mark.asyncio + async def test_oversized_result_is_offloaded_with_preview_and_reference(self, offloader, monkeypatch): + plugin, storage = offloader + import apis.shared.observability.emf as emf + emitted = [] + monkeypatch.setattr(emf, "emit_emf_metrics", lambda ns, metrics, properties=None, units=None: emitted.append((ns, metrics, properties))) + monkeypatch.delenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", raising=False) + + body = "header line\n" + ("row data " * 2000) + ev = _event(body, count_tokens=AsyncMock(return_value=5000)) + await plugin._handle_tool_result(ev) + + text = ev.result["content"][0]["text"] + assert text.startswith("[Offloaded:") + assert "retrieve_offloaded_content" in text + assert "header line" in text # preview keeps the head + assert len(text) < len(body) // 4 + assert len(storage.objects) == 1 # the full block landed in storage + assert ev.result["toolUseId"] == "t1" and ev.result["status"] == "success" + assert emitted and emitted[0][0] == "AgentCoreStack/Compaction" + assert emitted[0][1]["ToolResultOffloaded"] == 1 + assert emitted[0][2]["toolName"] == "gmail_search" + assert "row data" not in str(emitted[0][2]) # content-free + + @pytest.mark.asyncio + async def test_storage_failure_keeps_original_result(self, offloader): + plugin, storage = offloader + + async def boom(key, data): + raise RuntimeError("s3 down") + + storage.write = boom + ev = _event("y" * 20000, count_tokens=AsyncMock(return_value=5000)) + before = ev.result + await plugin._handle_tool_result(ev) + assert ev.result is before + + @pytest.mark.asyncio + async def test_count_tokens_failure_keeps_original_result(self, offloader): + plugin, _ = offloader + ev = _event("z" * 20000, count_tokens=AsyncMock(side_effect=RuntimeError("AccessDenied"))) + before = ev.result + await plugin._handle_tool_result(ev) + assert ev.result is before + + def test_prefilter_ratio_is_half_the_gate(self): + assert PREFILTER_RATIO == 0.5 + + +class TestChatAgentWiring: + def test_chat_agent_adds_the_offloader_plugin(self, monkeypatch): + from unittest.mock import MagicMock + from agents.main_agent.chat_agent import ChatAgent + from agents.main_agent.core import AgentFactory + + monkeypatch.setenv("S3_USER_FILES_BUCKET_NAME", "files-bucket") + + class FakeS3Storage: + def __init__(self, bucket, *, prefix="", region_name=None, **kw): + self.prefix = prefix + + async def write(self, k, d): ... + async def read(self, k): ... + async def delete(self, k): ... + async def list(self, q=""): return [] + + monkeypatch.setitem(sys.modules, "strands.storage", types.SimpleNamespace(S3Storage=FakeS3Storage)) + captured = {} + monkeypatch.setattr(AgentFactory, "create_agent", staticmethod(lambda **kw: captured.update(kw) or MagicMock())) + + agent = ChatAgent.__new__(ChatAgent) + agent.system_prompt = "BASE" + agent.model_config = MagicMock() + agent.session_manager = MagicMock() + agent.session_id = "sess" + agent.user_id = "usr" + agent._accessible_skill_ids = None + monkeypatch.setattr(ChatAgent, "_build_filtered_tools", lambda self: [], raising=False) + monkeypatch.setattr(ChatAgent, "_create_hooks", lambda self: [], raising=False) + + agent._create_agent() + + plugins = captured["plugins"] + assert plugins and type(plugins[0]).__name__ == "BoundedToolResultOffloader" + assert plugins[0]._storage._prefix if hasattr(plugins[0]._storage, "_prefix") else True + + def test_chat_agent_without_bucket_passes_no_plugins(self, monkeypatch): + from unittest.mock import MagicMock + from agents.main_agent.chat_agent import ChatAgent + from agents.main_agent.core import AgentFactory + + monkeypatch.delenv("S3_USER_FILES_BUCKET_NAME", raising=False) + captured = {} + monkeypatch.setattr(AgentFactory, "create_agent", staticmethod(lambda **kw: captured.update(kw) or MagicMock())) + agent = ChatAgent.__new__(ChatAgent) + agent.system_prompt = "BASE" + agent.model_config = MagicMock() + agent.session_manager = MagicMock() + agent.session_id = "sess" + agent.user_id = "usr" + agent._accessible_skill_ids = None + monkeypatch.setattr(ChatAgent, "_build_filtered_tools", lambda self: [], raising=False) + monkeypatch.setattr(ChatAgent, "_create_hooks", lambda self: [], raising=False) + agent._create_agent() + assert captured["plugins"] is None diff --git a/backend/tests/agents/main_agent/property/conftest.py b/backend/tests/agents/main_agent/property/conftest.py index d7db8b78c..cdc3469ec 100644 --- a/backend/tests/agents/main_agent/property/conftest.py +++ b/backend/tests/agents/main_agent/property/conftest.py @@ -143,32 +143,3 @@ def st_compaction_state(draw): min_size=0, max_size=20, ) - - -# --------------------------------------------------------------------------- -# st_sse_event_dict — dicts with string keys and JSON-serializable values -# --------------------------------------------------------------------------- -_json_primitives = st.one_of( - st.none(), - st.booleans(), - st.integers(min_value=-(2**31), max_value=2**31), - st.floats(allow_nan=False, allow_infinity=False), - st.text(max_size=50), -) - -# Recursive strategy: primitives, lists of primitives, or nested dicts -_json_values = st.recursive( - _json_primitives, - lambda children: st.one_of( - st.lists(children, max_size=5), - st.dictionaries(st.text(max_size=10), children, max_size=5), - ), - max_leaves=15, -) - -st_sse_event_dict = st.dictionaries( - keys=st.text(min_size=1, max_size=20), - values=_json_values, - min_size=1, - max_size=8, -) diff --git a/backend/tests/agents/main_agent/property/test_pbt_agent_core.py b/backend/tests/agents/main_agent/property/test_pbt_agent_core.py index bbb7fbefd..4bf35fdbf 100644 --- a/backend/tests/agents/main_agent/property/test_pbt_agent_core.py +++ b/backend/tests/agents/main_agent/property/test_pbt_agent_core.py @@ -193,51 +193,6 @@ def test_output_matches_allowed_pattern_and_is_idempotent(self, filename: str): ) -import json - -from agents.main_agent.streaming.event_formatter import StreamEventFormatter - -from .conftest import st_sse_event_dict - - -class TestSSEFormatInvariantAndJSONRoundTrip: - """ - Feature: agent-core-tests, Property 6: SSE format invariant and JSON round-trip - - Validates: Requirements 12.9, 12.10 - """ - - @given(event_dict=st_sse_event_dict) - @settings(max_examples=100) - def test_sse_format_and_json_round_trip(self, event_dict: dict): - """ - Feature: agent-core-tests, Property 6: SSE format invariant and JSON round-trip - - Validates: Requirements 12.9, 12.10 - - For any dictionary with string keys and JSON-serializable values, - format_sse_event(d) should (a) start with "data: " and end with - "\\n\\n", and (b) the JSON payload extracted from between the prefix - and suffix should parse back to the original dictionary. - """ - result = StreamEventFormatter.format_sse_event(event_dict) - - # (a) SSE format invariant - assert result.startswith("data: "), ( - f"SSE event does not start with 'data: ': {result!r}" - ) - assert result.endswith("\n\n"), ( - f"SSE event does not end with '\\n\\n': {result!r}" - ) - - # (b) JSON round-trip — extract payload between prefix and suffix - payload_str = result[len("data: "):-len("\n\n")] - parsed = json.loads(payload_str) - assert parsed == event_dict, ( - f"JSON round-trip failed: expected {event_dict!r}, got {parsed!r}" - ) - - import hypothesis.strategies as st from hypothesis import given, settings diff --git a/backend/tests/agents/main_agent/quota/test_checker.py b/backend/tests/agents/main_agent/quota/test_checker.py index a12de05d8..7763d6892 100644 --- a/backend/tests/agents/main_agent/quota/test_checker.py +++ b/backend/tests/agents/main_agent/quota/test_checker.py @@ -541,3 +541,146 @@ async def test_blocked_turn_skips_the_session_read( assert result.allowed is False read.assert_not_awaited() + + +# =================================================================== +# PR-2b — the caller may supply the session cost it already read +# (docs/specs/turn-latency-preamble.md). The read it replaces was a +# measured 62ms GSI query in the inference-api preamble. +# =================================================================== + + +def _resolved(sample_tier, sample_assignment): + return ResolvedQuota( + user_id="test123", + tier=sample_tier, + matched_by="direct_user", + assignment=sample_assignment, + ) + + +def _user_summary(total=60.0): + return UserCostSummary( + userId="test123", + periodStart="2025-01-01T00:00:00Z", + periodEnd="2025-01-31T23:59:59Z", + totalCost=total, + models=[], + totalRequests=10, + totalInputTokens=1000, + totalOutputTokens=500, + totalCacheSavings=0.0, + ) + + +@pytest.mark.asyncio +async def test_supplied_session_cost_skips_the_metadata_read( + checker, mock_resolver, mock_cost_aggregator, mock_event_recorder, + sample_user, sample_tier, sample_assignment +): + """The whole point: a caller that already has the row does not pay again.""" + mock_resolver.resolve_user_quota.return_value = _resolved(sample_tier, sample_assignment) + mock_cost_aggregator.get_user_cost_summary.return_value = _user_summary() + + reader = AsyncMock() + with patch("apis.shared.sessions.metadata.get_session_metadata", reader): + result = await checker.check_quota( + sample_user, session_id="sess-1", session_total_cost=130.0 + ) + + reader.assert_not_awaited() + assert float(result.session_cost) == 130.0 + assert float(result.session_percentage_of_limit) == pytest.approx(26.0) + mock_event_recorder.record_session_notice_if_needed.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_supplied_cost_below_the_share_still_produces_no_notice( + checker, mock_resolver, mock_cost_aggregator, mock_event_recorder, + sample_user, sample_tier, sample_assignment +): + """The threshold decision must not change just because the number arrived + by a different route.""" + mock_resolver.resolve_user_quota.return_value = _resolved(sample_tier, sample_assignment) + mock_cost_aggregator.get_user_cost_summary.return_value = _user_summary() + + reader = AsyncMock() + with patch("apis.shared.sessions.metadata.get_session_metadata", reader): + result = await checker.check_quota( + sample_user, session_id="sess-1", session_total_cost=10.0 + ) + + reader.assert_not_awaited() + assert result.session_cost is None + mock_event_recorder.record_session_notice_if_needed.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_none_falls_back_to_the_read_and_its_lazy_backfill( + checker, mock_resolver, mock_cost_aggregator, mock_event_recorder, + sample_user, sample_tier, sample_assignment +): + """`None` means "not known", NEVER "zero". + + A legacy row written before write-time aggregation has no `totalCost`, and + `get_session_metadata` backfills it on read. If the caller's absent value + were read as 0.0 the backfill would be skipped and the notice would go + quiet on exactly the long-lived conversations it exists to catch — a + latency fix turning into a silent feature regression. + """ + mock_resolver.resolve_user_quota.return_value = _resolved(sample_tier, sample_assignment) + mock_cost_aggregator.get_user_cost_summary.return_value = _user_summary() + + backfilled = Mock() + backfilled.total_cost = 130.0 + reader = AsyncMock(return_value=backfilled) + + with patch("apis.shared.sessions.metadata.get_session_metadata", reader): + result = await checker.check_quota( + sample_user, session_id="sess-1", session_total_cost=None + ) + + reader.assert_awaited_once() + assert float(result.session_cost) == 130.0 + mock_event_recorder.record_session_notice_if_needed.assert_awaited_once() + + +@pytest.mark.asyncio +async def test_a_supplied_zero_is_honoured_as_zero_not_as_unknown( + checker, mock_resolver, mock_cost_aggregator, mock_event_recorder, + sample_user, sample_tier, sample_assignment +): + """A genuinely free session reports 0.0, which is falsy — a truthiness + check here would send it back to the read it was meant to avoid.""" + mock_resolver.resolve_user_quota.return_value = _resolved(sample_tier, sample_assignment) + mock_cost_aggregator.get_user_cost_summary.return_value = _user_summary() + + reader = AsyncMock() + with patch("apis.shared.sessions.metadata.get_session_metadata", reader): + result = await checker.check_quota( + sample_user, session_id="sess-1", session_total_cost=0.0 + ) + + reader.assert_not_awaited() + assert result.session_cost is None + + +@pytest.mark.asyncio +async def test_default_call_is_unchanged_for_callers_that_pass_nothing( + checker, mock_resolver, mock_cost_aggregator, mock_event_recorder, + sample_user, sample_tier, sample_assignment +): + """app-api's converse route calls this without a snapshot and must keep + reading exactly as before.""" + mock_resolver.resolve_user_quota.return_value = _resolved(sample_tier, sample_assignment) + mock_cost_aggregator.get_user_cost_summary.return_value = _user_summary() + + meta = Mock() + meta.total_cost = 130.0 + reader = AsyncMock(return_value=meta) + + with patch("apis.shared.sessions.metadata.get_session_metadata", reader): + result = await checker.check_quota(sample_user, session_id="sess-1") + + reader.assert_awaited_once() + assert float(result.session_cost) == 130.0 diff --git a/backend/tests/agents/main_agent/quota/test_resolver.py b/backend/tests/agents/main_agent/quota/test_resolver.py index e5ec2274c..dda776587 100644 --- a/backend/tests/agents/main_agent/quota/test_resolver.py +++ b/backend/tests/agents/main_agent/quota/test_resolver.py @@ -14,6 +14,25 @@ from apis.shared.auth.models import User +@pytest.fixture(autouse=True) +def _no_live_app_role_service(): + """Skip the AppRole quota branch, which builds a real RBAC service. + + ``QuotaResolver.resolve`` step 3 calls ``_get_app_role_service()`` and then + ``resolve_user_permissions`` -> DynamoDB. No test in this file exercises + that branch, and its ``except`` swallowed the failure — so these passed + while issuing a live AWS call. See the off-box socket guard in + ``tests/conftest.py``. + """ + from unittest.mock import patch as _patch + + with _patch( + "agents.main_agent.quota.resolver._get_app_role_service", + return_value=None, + ): + yield + + @pytest.fixture def mock_repository(): """Create a mock quota repository""" diff --git a/backend/tests/agents/main_agent/session/conftest.py b/backend/tests/agents/main_agent/session/conftest.py index 57aa832ee..d232ba130 100644 --- a/backend/tests/agents/main_agent/session/conftest.py +++ b/backend/tests/agents/main_agent/session/conftest.py @@ -126,6 +126,10 @@ def compaction_config() -> CompactionConfig: token_threshold=1000, protected_turns=3, max_tool_content_length=50, + # These suites pin the immediate-apply path (checkpoint moves at turn + # end). Paid-when-free deferral is the default in prod and has its own + # suite: test_compaction_deferred_apply.py. + deferred_apply_enabled=False, ) diff --git a/backend/tests/agents/main_agent/session/test_compaction_deferred_apply.py b/backend/tests/agents/main_agent/session/test_compaction_deferred_apply.py new file mode 100644 index 000000000..b977dc526 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_compaction_deferred_apply.py @@ -0,0 +1,253 @@ +"""Paid-when-free compaction — docs/specs/compaction-model-relative-thresholds.md §3.5. + +Post-turn parks the cut as PENDING; the head of the next turn applies it to +the live list in place only when the prefix re-write is free (cache expired, +model/agent switched) or unavoidable (hard ceiling). +""" + +import copy +import json +from datetime import datetime, timedelta, timezone +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.compaction_models import CompactionConfig, CompactionState +from agents.main_agent.session.compaction_summary import approx_tokens + +from .conftest import make_conversation + + +CONFIG = dict(enabled=True, token_threshold=1000, protected_turns=3, summary_token_budget=200) + + +def _now(): + return datetime.now(timezone.utc) + + +def _dump(messages): + return json.dumps(messages, sort_keys=True) + + +def _manager(make_session_manager, store, **overrides): + """Manager wired to an in-memory state store (deferred mode ON).""" + cfg = CompactionConfig(**{**CONFIG, **overrides}) + mgr = make_session_manager(compaction_config=cfg) + mgr._load_compaction_state = lambda: CompactionState.from_dict(store.get("compaction")) + + def _save(state, record_event=False): + state.updated_at = _now().isoformat() + store["compaction"] = state.to_dict() + + mgr._save_compaction_state = _save + mgr._retrieve_session_summaries = lambda: ["LTM summary"] + mgr.compaction_state = CompactionState.from_dict(store.get("compaction")) + return mgr + + +def _agent(messages): + agent = MagicMock() + agent.messages = messages + return agent + + +def _age(store, seconds): + store["compaction"]["updatedAt"] = (_now() - timedelta(seconds=seconds)).isoformat() + + +class TestPostTurnParksTheCut: + @pytest.mark.asyncio + async def test_over_ceiling_parks_pending_and_leaves_checkpoint(self, make_session_manager): + store = {} + mgr = _manager(make_session_manager, store) + live = make_conversation(5) + result = await mgr.update_after_turn(2000, current_messages=live) + assert result is not None and result.deferred is True + state = mgr.compaction_state + assert state.pending_checkpoint == 4 and state.checkpoint == 0 + assert state.pending_summary == "LTM summary" + assert state.pending_hard_ceiling == 1500 + assert state.armed is False + assert store["compaction"]["pendingCheckpoint"] == 4 + assert state.policy["deferred"] is True + # Nothing about the live list changed at turn end. + assert len(live) == 10 and "conversation_summary" not in _dump(live) + + @pytest.mark.asyncio + async def test_second_over_ceiling_turn_does_not_cut_deeper(self, make_session_manager): + store = {} + mgr = _manager(make_session_manager, store) + await mgr.update_after_turn(2000, current_messages=make_conversation(5)) + second = await mgr.update_after_turn(2500, current_messages=make_conversation(7)) # even at hard ceiling + assert second is None + assert mgr.compaction_state.pending_checkpoint == 4 + + @pytest.mark.asyncio + async def test_kill_switch_applies_immediately(self, make_session_manager): + store = {} + mgr = _manager(make_session_manager, store, deferred_apply_enabled=False) + result = await mgr.update_after_turn(2000, current_messages=make_conversation(5)) + assert result.deferred is False + assert mgr.compaction_state.checkpoint == 4 and mgr.compaction_state.pending_checkpoint is None + + @pytest.mark.asyncio + async def test_legacy_mode_applies_immediately(self, make_session_manager): + store = {} + mgr = _manager(make_session_manager, store, model_relative_enabled=False) + result = await mgr.update_after_turn(2000, current_messages=make_conversation(5)) + assert result.deferred is False and mgr.compaction_state.checkpoint == 4 + + +class TestHeadOfTurnApply: + async def _parked(self, make_session_manager, store): + """Park a cut at 1200 tokens: over the ceiling (1000), under hard (1500).""" + mgr = _manager(make_session_manager, store) + live = make_conversation(5) + await mgr.update_after_turn(1200, current_messages=live) + assert mgr.compaction_state.pending_checkpoint == 4 + return mgr, live + + @pytest.mark.asyncio + async def test_warm_cache_same_prefix_below_hard_waits(self, make_session_manager): + store = {} + mgr, live = await self._parked(make_session_manager, store) + before = _dump(live) + assert mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") is None + assert _dump(live) == before + assert mgr.compaction_state.pending_checkpoint == 4 and mgr.compaction_state.checkpoint == 0 + + @pytest.mark.asyncio + async def test_cache_expired_applies_in_place(self, make_session_manager): + store = {} + mgr, live = await self._parked(make_session_manager, store) + _age(store, 600) + list_id = id(live) + reason = mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") + assert reason == "cache_expired" + assert id(live) == list_id # slice assignment, never rebound + assert len(live) == 6 # 10 - 4 + assert live[0]["role"] == "user" and "conversation_summary" in live[0]["content"][0]["text"] + assert "LTM summary" in live[0]["content"][0]["text"] + state = mgr.compaction_state + assert state.checkpoint == 4 and state.truncation_anchor == 4 and state.summary == "LTM summary" + assert state.pending_checkpoint is None and state.pending_summary is None + assert mgr._live_offset == 4 + assert state.policy["applied"] == "cache_expired" + assert state.policy["cacheGapSeconds"] >= 599 + assert store["compaction"]["checkpoint"] == 4 and store["compaction"]["pendingCheckpoint"] is None + + @pytest.mark.asyncio + async def test_prefix_change_applies(self, make_session_manager): + store = {} + mgr = _manager(make_session_manager, store) + live = make_conversation(5) + # Turn 1 stamps the key; turn end persists it. + mgr.apply_pending_compaction(_agent(live), prefix_key="sonnet|default") + await mgr.update_after_turn(1200, current_messages=live) + assert store["compaction"]["lastPrefixKey"] == "sonnet|default" + # Same key, warm: waits. Different key (model switch): applies. + assert mgr.apply_pending_compaction(_agent(live), prefix_key="sonnet|default") is None + assert mgr.apply_pending_compaction(_agent(live), prefix_key="opus|default") == "prefix_changed" + assert len(live) == 6 + + @pytest.mark.asyncio + async def test_first_ever_prefix_key_is_not_a_change(self, make_session_manager): + store = {} + mgr, live = await self._parked(make_session_manager, store) # no key was ever stamped + assert mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") is None + + @pytest.mark.asyncio + async def test_hard_ceiling_forces_apply(self, make_session_manager): + store = {} + mgr, live = await self._parked(make_session_manager, store) + await mgr.update_after_turn(1600, current_messages=live) # pending waits post-turn... + assert mgr.compaction_state.pending_checkpoint == 4 + assert mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") == "hard_ceiling" # ...and lands pre-call + assert len(live) == 6 + + @pytest.mark.asyncio + async def test_live_apply_matches_a_cold_restore_of_the_same_state(self, make_session_manager): + """The in-place slice must produce the bytes _apply_compaction derives + from stored history under the promoted state (prefix cache parity).""" + store = {} + mgr, live = await self._parked(make_session_manager, store) + stored = copy.deepcopy(live) + _age(store, 600) + assert mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") == "cache_expired" + + cold = _manager(make_session_manager, store) + agent = _agent(copy.deepcopy(stored)) + cold._apply_compaction(agent) + assert _dump(agent.messages) == _dump(live) + assert cold._live_offset == mgr._live_offset == 4 + + @pytest.mark.asyncio + async def test_pending_beyond_live_list_is_dropped_not_applied(self, make_session_manager): + store = {} + mgr, live = await self._parked(make_session_manager, store) + _age(store, 600) + short = live[:2] # a list that does not reach the cut + assert mgr.apply_pending_compaction(_agent(short), prefix_key="m|default") is None + assert mgr.compaction_state.pending_checkpoint is None + assert mgr.compaction_state.checkpoint == 0 and len(short) == 2 + + @pytest.mark.asyncio + async def test_after_apply_next_turn_rearms_and_can_cut_again(self, make_session_manager): + store = {} + mgr, live = await self._parked(make_session_manager, store) + _age(store, 600) + mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") + assert await mgr.update_after_turn(800, current_messages=live) is None + assert mgr.compaction_state.armed is True + live.extend(make_conversation(4)) + result = await mgr.update_after_turn(2000, current_messages=live) + assert result is not None and result.deferred is True + assert mgr.compaction_state.pending_checkpoint > 4 # absolute: offset 4 + relative cut + + @pytest.mark.asyncio + async def test_apply_emits_metric_with_reason(self, make_session_manager, monkeypatch): + import apis.shared.observability.emf as emf + emitted = [] + monkeypatch.setattr(emf, "emit_emf_metrics", lambda ns, metrics, properties=None, units=None: emitted.append((metrics, properties))) + monkeypatch.delenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", raising=False) + store = {} + mgr, live = await self._parked(make_session_manager, store) + cut_metrics = [m for m, p in emitted if "CompactionCut" in m] + assert cut_metrics and cut_metrics[0]["CompactionDeferred"] == 1 + _age(store, 600) + mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") + applied = [(m, p) for m, p in emitted if "CompactionApplied" in m] + assert applied and applied[0][1]["applyReason"] == "cache_expired" + assert applied[0][0]["CompactionAppliedForced"] == 0 + + +class TestLedger: + @pytest.mark.asyncio + async def test_apply_records_an_applied_ledger_event(self, make_session_manager): + store = {} + mgr = _manager(make_session_manager, store) + live = make_conversation(5) + await mgr.update_after_turn(1200, current_messages=live) + mgr.record_compaction_event = MagicMock() + _age(store, 600) + assert mgr.apply_pending_compaction(_agent(live), prefix_key="m|default") == "cache_expired" + kinds = [c.args[0] for c in mgr.record_compaction_event.call_args_list] + assert kinds == ["applied"] + fields = mgr.record_compaction_event.call_args.kwargs + assert fields["checkpoint"] == 4 and fields["retainedMessages"] == 6 + assert fields["cacheGapSeconds"] >= 599 and fields["summaryTokens"] >= 0 + assert fields["promoted"] == 1 + + +class TestStateRoundTrip: + def test_pending_fields_round_trip_and_default_none(self): + assert CompactionState.from_dict({"checkpoint": 1}).pending_checkpoint is None + s = CompactionState(pending_checkpoint=7, pending_summary="s", pending_hard_ceiling=150, pending_since="t", last_prefix_key="m|a") + again = CompactionState.from_dict(s.to_dict()) + assert (again.pending_checkpoint, again.pending_summary, again.pending_hard_ceiling, again.pending_since, again.last_prefix_key) == (7, "s", 150, "t", "m|a") + + def test_from_env_flag(self, monkeypatch): + monkeypatch.delenv("AGENTCORE_MEMORY_COMPACTION_DEFERRED_APPLY_ENABLED", raising=False) + assert CompactionConfig.from_env().deferred_apply_enabled is True + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_DEFERRED_APPLY_ENABLED", "false") + assert CompactionConfig.from_env().deferred_apply_enabled is False diff --git a/backend/tests/agents/main_agent/session/test_compaction_event_ledger.py b/backend/tests/agents/main_agent/session/test_compaction_event_ledger.py new file mode 100644 index 000000000..1a240b7e4 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_compaction_event_ledger.py @@ -0,0 +1,45 @@ +"""`TurnBasedSessionManager.record_compaction_event` / `drain_compaction_events` +— the queue the `ContextLedgerHook` drains onto the next model call's cost row.""" + +import pytest + +from agents.main_agent.session.turn_based_session_manager import ( + _MAX_PENDING_COMPACTION_EVENTS, + COMPACTION_EVENT_KINDS, +) + + +@pytest.fixture(autouse=True) +def diagnostics_enabled(monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + + +def test_events_carry_numbers_only_and_drain_once(make_session_manager): + manager = make_session_manager() + manager.record_compaction_event( + "applied", checkpoint=12, summaryTokens=900.7, retainedMessages=30, + summary="never persisted", flag=True, + ) + drained = manager.drain_compaction_events() + assert drained == [{"kind": "applied", "checkpoint": 12, "summaryTokens": 900, "retainedMessages": 30}] + assert manager.drain_compaction_events() == [] + + +def test_unknown_kinds_are_ignored_and_the_queue_is_bounded(make_session_manager): + manager = make_session_manager() + manager.record_compaction_event("mystery", checkpoint=1) + assert manager.drain_compaction_events() == [] + for _ in range(_MAX_PENDING_COMPACTION_EVENTS + 3): + manager.record_compaction_event("forced") + assert len(manager.drain_compaction_events()) == _MAX_PENDING_COMPACTION_EVENTS + + +def test_reserved_kinds_exist_for_the_scheduling_policy(): + assert {"applied", "checkpoint", "forced", "floor_unreachable"} <= COMPACTION_EVENT_KINDS + + +def test_kill_switch_records_nothing(make_session_manager, monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + manager = make_session_manager() + manager.record_compaction_event("applied", checkpoint=3) + assert manager.drain_compaction_events() == [] diff --git a/backend/tests/agents/main_agent/session/test_compaction_models.py b/backend/tests/agents/main_agent/session/test_compaction_models.py index 898233625..25ce5075b 100644 --- a/backend/tests/agents/main_agent/session/test_compaction_models.py +++ b/backend/tests/agents/main_agent/session/test_compaction_models.py @@ -50,6 +50,13 @@ def test_to_dict_has_camel_case_keys(self): "updatedAt", "totalSummarizedTurns", "truncationAnchor", + "armed", + "policy", + "pendingCheckpoint", + "pendingSummary", + "pendingHardCeiling", + "pendingSince", + "lastPrefixKey", } def test_to_dict_values_match(self): @@ -174,3 +181,24 @@ def test_from_env_defaults_when_no_vars(self, monkeypatch): assert config.token_threshold == 100_000 assert config.protected_turns == 3 assert config.max_tool_content_length == 500 + + +# --------------------------------------------------------------------------- +# Hysteresis fields (docs/specs/compaction-model-relative-thresholds.md §3.3) +# --------------------------------------------------------------------------- + +class TestCompactionStateArmed: + def test_default_is_armed_with_no_policy(self): + state = CompactionState() + assert state.armed is True + assert state.policy is None + + def test_legacy_record_without_armed_defaults_to_armed(self): + state = CompactionState.from_dict({"checkpoint": 3}) + assert state.armed is True + + def test_roundtrips_armed_and_policy(self): + state = CompactionState(armed=False, policy={"ceiling": 100, "floor": 25}) + again = CompactionState.from_dict(state.to_dict()) + assert again.armed is False + assert again.policy == {"ceiling": 100, "floor": 25} diff --git a/backend/tests/agents/main_agent/session/test_compaction_policy.py b/backend/tests/agents/main_agent/session/test_compaction_policy.py new file mode 100644 index 000000000..11b8d73bd --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_compaction_policy.py @@ -0,0 +1,320 @@ +"""Model-relative compaction policy — docs/specs/compaction-model-relative-thresholds.md. + +Pins the §3.1 table, the kill switch, the floor-seeking cut (§3.2) and the +hysteresis rule (§3.3) as exercised through ``update_after_turn``. +""" + +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.compaction_models import CompactionConfig, CompactionState +from agents.main_agent.session.compaction_policy import ( + IMAGE_TOKEN_ESTIMATE, + CompactionPolicy, + choose_checkpoint, + estimate_message_tokens, +) + +from .conftest import make_assistant_message, make_conversation, make_user_message + + +# --------------------------------------------------------------------------- +# §3.1 — the table +# --------------------------------------------------------------------------- + +class TestPolicyResolution: + @pytest.mark.parametrize( + "window, ceiling, floor, hard", + [ + (128_000, 64_000, 16_000, 89_600), + (200_000, 100_000, 25_000, 140_000), + (256_000, 100_000, 25_000, 150_000), + (272_000, 100_000, 25_000, 150_000), + (1_000_000, 100_000, 25_000, 150_000), + ], + ) + def test_table_rows(self, window, ceiling, floor, hard): + policy = CompactionPolicy.resolve(CompactionConfig(), window) + assert (policy.ceiling, policy.floor, policy.hard_ceiling) == (ceiling, floor, hard) + assert policy.source == "model_relative" + assert policy.context_window == window + + def test_unknown_window_uses_fixed_threshold(self): + policy = CompactionPolicy.resolve(CompactionConfig(), None) + assert policy.source == "fixed" + assert policy.ceiling == 100_000 + assert policy.floor == 25_000 + assert policy.hard_ceiling == 150_000 + assert policy.hysteresis_enabled + + @pytest.mark.parametrize("bad", [0, -5, "abc"]) + def test_invalid_window_treated_as_unknown(self, bad): + assert CompactionPolicy.resolve(CompactionConfig(), bad).source == "fixed" + + def test_kill_switch_is_legacy(self): + policy = CompactionPolicy.resolve( + CompactionConfig(model_relative_enabled=False, token_threshold=1_000), 1_000_000 + ) + assert policy.source == "legacy" + assert policy.ceiling == 1_000 + assert policy.floor is None + assert policy.hard_ceiling is None + assert not policy.hysteresis_enabled + + def test_hard_never_below_ceiling_and_floor_always_below(self): + cfg = CompactionConfig(hard_ceiling_ratio=0.1, floor_ratio=2.0) + policy = CompactionPolicy.resolve(cfg, 200_000) + assert policy.hard_ceiling >= policy.ceiling + assert policy.floor < policy.ceiling + + def test_from_env_reads_policy_fields(self, monkeypatch): + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED", "false") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_CEILING_RATIO", "0.4") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_CEILING_CAP_TOKENS", "150000") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_FLOOR_RATIO", "0.2") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_HARD_CEILING_RATIO", "0.6") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_HARD_CEILING_MULTIPLIER", "1.2") + cfg = CompactionConfig.from_env() + assert cfg.model_relative_enabled is False + assert cfg.ceiling_ratio == 0.4 + assert cfg.ceiling_cap_tokens == 150_000 + assert cfg.floor_ratio == 0.2 + assert cfg.hard_ceiling_ratio == 0.6 + assert cfg.hard_ceiling_multiplier == 1.2 + + def test_kill_switch_default_on_and_only_literal_false_disables(self, monkeypatch): + monkeypatch.delenv("AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED", raising=False) + assert CompactionConfig.from_env().model_relative_enabled is True + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED", "") + assert CompactionConfig.from_env().model_relative_enabled is True + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED", "FALSE") + assert CompactionConfig.from_env().model_relative_enabled is False + + +# --------------------------------------------------------------------------- +# Estimator +# --------------------------------------------------------------------------- + +class TestEstimator: + def test_text_scales_with_length(self): + short = estimate_message_tokens(make_user_message("a" * 40)) + long = estimate_message_tokens(make_user_message("a" * 4000)) + assert long > short * 10 + + def test_image_is_flat(self): + msg = {"role": "user", "content": [{"image": {"format": "png", "source": {"bytes": b"x" * 10}}}]} + assert estimate_message_tokens(msg) >= IMAGE_TOKEN_ESTIMATE + + def test_tool_result_counts_inner_text(self): + msg = { + "role": "user", + "content": [{"toolResult": {"toolUseId": "t1", "content": [{"text": "r" * 4000}]}}], + } + assert estimate_message_tokens(msg) >= 1000 + + @pytest.mark.parametrize("weird", [None, "plain", {"role": "user"}, {"role": "user", "content": "str"}, {"content": [None, 3]}]) + def test_never_raises(self, weird): + assert estimate_message_tokens(weird) >= 0 + + +# --------------------------------------------------------------------------- +# §3.2 — floor-seeking cut +# --------------------------------------------------------------------------- + +def _conversation_with_sizes(sizes): + """user/assistant pairs where each user message carries `size` chars.""" + messages = [] + for i, size in enumerate(sizes): + messages.append(make_user_message("u" * size)) + messages.append(make_assistant_message(f"a{i}")) + return messages + + +class TestChooseCheckpoint: + def test_keeps_minimum_protected_turns(self): + msgs = make_conversation(3) + assert choose_checkpoint(msgs, [0, 2, 4], protected_turns=3, floor_tokens=10, history_tokens=None) == (0, None) + + def test_picks_oldest_cut_that_fits_under_floor(self): + # 6 turns; the last three are tiny, turns 2-3 are big. Floor admits the + # tail plus turn 3 but not turn 2 — oldest fitting cut is turn 3 (index 4). + msgs = _conversation_with_sizes([4000, 4000, 4000, 40, 40, 40]) + cutoffs = [0, 2, 4, 6, 8, 10] + # raw: big user ~1000 tok + overhead, small ~10+overhead; calibrate to 3500 total + cut, retained = choose_checkpoint(msgs, cutoffs, 3, floor_tokens=1400, history_tokens=3500) + assert cut == 4 + assert retained is not None and retained <= 1400 + + def test_falls_back_to_minimum_protection_when_tail_too_big(self): + msgs = _conversation_with_sizes([40, 40, 40, 8000, 8000, 8000]) + cutoffs = [0, 2, 4, 6, 8, 10] + cut, retained = choose_checkpoint(msgs, cutoffs, 3, floor_tokens=100, history_tokens=6000) + assert cut == 6 # cutoffs[-3] + assert retained > 100 + + def test_no_cut_when_everything_fits(self): + msgs = _conversation_with_sizes([40] * 6) + cut, _ = choose_checkpoint(msgs, [0, 2, 4, 6, 8, 10], 3, floor_tokens=10_000, history_tokens=200) + assert cut == 0 + + def test_calibration_scales_estimates_to_history_tokens(self): + msgs = _conversation_with_sizes([400] * 6) # equal turns + cutoffs = [0, 2, 4, 6, 8, 10] + # History measured at 6000 tokens → each turn ~1000. Floor 2500 admits + # only the two newest turns... but protection keeps three → min-protection. + cut, retained = choose_checkpoint(msgs, cutoffs, 3, floor_tokens=2500, history_tokens=6000) + assert cut == 6 + # Floor 3500 admits exactly three turns → cut at 6 again (oldest fitting) + cut2, _ = choose_checkpoint(msgs, cutoffs, 3, floor_tokens=3500, history_tokens=6000) + assert cut2 == 6 + # Floor 4500 admits four turns → cut at 4 + cut3, _ = choose_checkpoint(msgs, cutoffs, 3, floor_tokens=4500, history_tokens=6000) + assert cut3 == 4 + + def test_empty_messages_uses_minimum_protection(self): + assert choose_checkpoint([], [0, 2, 4, 6], 3, 10, 100)[0] == 2 + + +# --------------------------------------------------------------------------- +# §3.3 / §3.4 — hysteresis and coordinates through update_after_turn +# --------------------------------------------------------------------------- + +def _armed_manager(make_session_manager, compaction_config, checkpoint=0, armed=True): + mgr = make_session_manager(compaction_config=compaction_config) + mgr.compaction_state = CompactionState(checkpoint=checkpoint, armed=armed) + mgr._save_compaction_state = MagicMock() + mgr._retrieve_session_summaries = MagicMock(return_value=[]) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8] + mgr._all_messages_for_summary = make_conversation(5) + return mgr + + +class TestHysteresis: + """compaction_config fixture: threshold 1000 → fixed policy ceiling 1000, + floor 250, hard 1500 (window unknown).""" + + @pytest.mark.asyncio + async def test_cut_disarms(self, make_session_manager, compaction_config): + mgr = _armed_manager(make_session_manager, compaction_config) + result = await mgr.update_after_turn(1200) + assert result is not None + assert mgr.compaction_state.armed is False + assert result.forced is False + assert result.ceiling == 1000 and result.floor == 250 and result.hard_ceiling == 1500 + assert mgr.compaction_state.policy["source"] == "fixed" + + @pytest.mark.asyncio + async def test_over_ceiling_while_disarmed_is_a_noop(self, make_session_manager, compaction_config): + mgr = _armed_manager(make_session_manager, compaction_config, checkpoint=4, armed=False) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8, 10] + mgr._all_messages_for_summary = make_conversation(6) + result = await mgr.update_after_turn(1200) + assert result is None + assert mgr.compaction_state.checkpoint == 4 + assert mgr.compaction_state.armed is False + mgr._save_compaction_state.assert_called_once() + + @pytest.mark.asyncio + async def test_hard_ceiling_forces_a_cut_while_disarmed(self, make_session_manager, compaction_config): + mgr = _armed_manager(make_session_manager, compaction_config, checkpoint=4, armed=False) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8, 10] + mgr._all_messages_for_summary = make_conversation(6) + result = await mgr.update_after_turn(1500) + assert result is not None and result.forced is True + assert mgr.compaction_state.checkpoint == 6 + assert mgr.compaction_state.policy["forced"] is True + + @pytest.mark.asyncio + async def test_armed_cut_above_hard_ceiling_is_not_forced(self, make_session_manager, compaction_config): + """Forced means "ran while disarmed" — the spiral signal — not "was large".""" + mgr = _armed_manager(make_session_manager, compaction_config) + result = await mgr.update_after_turn(2500) # above hard=1500, but armed + assert result is not None and result.forced is False + assert mgr.compaction_state.policy["forced"] is False + + @pytest.mark.asyncio + async def test_under_ceiling_rearms(self, make_session_manager, compaction_config): + mgr = _armed_manager(make_session_manager, compaction_config, checkpoint=4, armed=False) + assert await mgr.update_after_turn(900) is None + assert mgr.compaction_state.armed is True + + @pytest.mark.asyncio + async def test_spiral_shape_cuts_exactly_once(self, make_session_manager, compaction_config): + """Input pinned above the ceiling for 10 turns: one cut, then no-ops.""" + mgr = _armed_manager(make_session_manager, compaction_config) + advances = 0 + for turn in range(10): + n_turns = 5 + turn + mgr._valid_cutoff_indices = list(range(0, 2 * n_turns, 2)) + mgr._all_messages_for_summary = make_conversation(n_turns) + if await mgr.update_after_turn(1200) is not None: + advances += 1 + assert advances == 1 + + @pytest.mark.asyncio + async def test_legacy_mode_never_disarms_and_uses_turn_count(self, make_session_manager): + cfg = CompactionConfig(enabled=True, deferred_apply_enabled=False, token_threshold=1000, protected_turns=3, model_relative_enabled=False) + mgr = _armed_manager(make_session_manager, cfg) + first = await mgr.update_after_turn(1200) + assert first is not None and first.floor is None + assert mgr.compaction_state.checkpoint == 4 # cutoffs[-3] + assert mgr.compaction_state.armed is True + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8, 10] + mgr._all_messages_for_summary = make_conversation(6) + second = await mgr.update_after_turn(1200) + assert second is not None and mgr.compaction_state.checkpoint == 6 + + +class TestCoordinates: + @pytest.mark.asyncio + async def test_checkpoint_is_live_offset_plus_relative_cut(self, make_session_manager, compaction_config): + mgr = _armed_manager(make_session_manager, compaction_config, checkpoint=10) + mgr._live_offset = 10 # restore sliced at absolute 10 + result = await mgr.update_after_turn(1200) + assert result is not None + assert result.previous_checkpoint == 10 + assert result.new_checkpoint == 14 # 10 + cutoffs[-3]=4 + assert mgr.compaction_state.truncation_anchor == 14 + + @pytest.mark.asyncio + async def test_context_window_flows_into_policy(self, make_session_manager): + cfg = CompactionConfig(enabled=True, deferred_apply_enabled=False, protected_turns=3) + mgr = _armed_manager(make_session_manager, cfg) + # 128k window → ceiling 64k; 60k is under it → no cut. + assert await mgr.update_after_turn(60_000, context_window=128_000) is None + assert mgr.compaction_state.checkpoint == 0 + # 1M window → ceiling capped at 100k (not 500k) → 150k cuts. + result = await mgr.update_after_turn(150_000, context_window=1_000_000) + assert result is not None and result.context_window == 1_000_000 and result.ceiling == 100_000 + assert result.hard_ceiling == 150_000 + + +class TestCompactionLedgerEvents: + """Decisions are handed to the per-call compaction ledger when it exists + (cost-diagnostics ``record_compaction_event``), and are a no-op otherwise.""" + + @pytest.mark.asyncio + async def test_forced_and_floor_unreachable_are_recorded(self, make_session_manager, compaction_config): + mgr = _armed_manager(make_session_manager, compaction_config, checkpoint=4, armed=False) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8, 10] + mgr._all_messages_for_summary = make_conversation(6) + mgr.record_compaction_event = MagicMock() + result = await mgr.update_after_turn(1500) # disarmed + hard ceiling → forced + assert result is not None and result.forced + kinds = [c.args[0] for c in mgr.record_compaction_event.call_args_list] + assert "forced" in kinds and "floor_unreachable" in kinds + forced_call = next(c for c in mgr.record_compaction_event.call_args_list if c.args[0] == "forced") + assert forced_call.kwargs == {"checkpoint": 6, "inputTokens": 1500} + floor_call = next(c for c in mgr.record_compaction_event.call_args_list if c.args[0] == "floor_unreachable") + assert floor_call.kwargs["retainedTokens"] > 250 + + @pytest.mark.asyncio + async def test_no_ledger_is_a_noop(self, make_session_manager, compaction_config, monkeypatch): + # The cost-diagnostics ledger now ships on this class, so absence has to + # be simulated: strip the recorder and prove the cut still lands rather + # than raising through ``_record_ledger_event``'s getattr seam. + mgr = _armed_manager(make_session_manager, compaction_config) + monkeypatch.delattr(type(mgr), "record_compaction_event") + assert not hasattr(mgr, "record_compaction_event") + assert await mgr.update_after_turn(1200) is not None # no AttributeError diff --git a/backend/tests/agents/main_agent/session/test_compaction_stability.py b/backend/tests/agents/main_agent/session/test_compaction_stability.py index ef386e602..9c883f85c 100644 --- a/backend/tests/agents/main_agent/session/test_compaction_stability.py +++ b/backend/tests/agents/main_agent/session/test_compaction_stability.py @@ -282,6 +282,76 @@ def test_missing_updated_at_treated_as_warm(self, restore_session): assert store["compaction"]["truncationAnchor"] == 0 +class TestTruncationAnchorLedger: + """Offload spec PR-5: the guard above holds by construction; these pin + the rows that prove it in prod — a ``truncation_anchor`` event carrying + the gap the decision was made on, and the gap on the restore-time + ``applied`` event.""" + + @pytest.fixture(autouse=True) + def diagnostics_enabled(self, monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + + def _restore_with_events(self, make_session_manager, compaction_config, state_store, stored): + """Same as ``restore_session`` but hands back the manager so the + ledger queue can be drained.""" + mgr = make_session_manager(compaction_config=compaction_config) + mgr._load_compaction_state = lambda: CompactionState.from_dict(state_store.get("compaction")) + + def _save(state, record_event=False): + state.updated_at = _iso(_now()) + state_store["compaction"] = state.to_dict() + + mgr._save_compaction_state = _save + mgr._retrieve_session_summaries = lambda: [] + session_agent = MagicMock() + session_agent.state = {} + session_agent.conversation_manager_state = {} + session_messages = [] + for msg in copy.deepcopy(stored): + sm = MagicMock() + sm.to_message.return_value = msg + session_messages.append(sm) + mgr.read_agent = MagicMock(return_value=session_agent) + mgr.list_messages = MagicMock(return_value=session_messages) + mgr._is_new_session = False + agent = MagicMock() + agent.agent_id = "default" + agent.messages = [] + agent.conversation_manager.restore_from_session.return_value = [] + agent.conversation_manager.removed_message_count = 0 + mgr.initialize(agent) + return mgr, agent + + def test_cold_advance_records_the_gap_it_was_decided_on(self, make_session_manager, compaction_config): + store = _stale_state(age_seconds=600) + mgr, _ = self._restore_with_events(make_session_manager, compaction_config, store, make_tool_conversation(8)) + events = [e for e in mgr.drain_compaction_events() if e["kind"] == "truncation_anchor"] + assert len(events) == 1 + event = events[0] + assert (event["anchorFrom"], event["anchorTo"]) == (0, 20) + # Measured before the save re-stamped updated_at — the real gap, not ~0. + assert event["cacheGapSeconds"] >= 599 + + def test_warm_cache_records_no_anchor_event(self, make_session_manager, compaction_config): + store = _fresh_state() + mgr, _ = self._restore_with_events(make_session_manager, compaction_config, store, make_tool_conversation(8)) + assert [e for e in mgr.drain_compaction_events() if e["kind"] == "truncation_anchor"] == [] + + def test_restore_applied_event_carries_the_gap(self, make_session_manager, compaction_config): + stored = make_tool_conversation(8) + store = _stale_state(age_seconds=900, checkpoint=8, truncation_anchor=8, summary="earlier turns") + mgr, _ = self._restore_with_events(make_session_manager, compaction_config, store, stored) + applied = [e for e in mgr.drain_compaction_events() if e["kind"] == "applied"] + assert len(applied) == 1 and applied[0]["checkpoint"] == 8 + assert applied[0]["cacheGapSeconds"] >= 899 + + def test_kind_is_registered(self): + from agents.main_agent.session.turn_based_session_manager import COMPACTION_EVENT_KINDS + + assert "truncation_anchor" in COMPACTION_EVENT_KINDS + + class TestCacheWindowExpired: def test_none_and_garbage_are_not_expired(self): assert TurnBasedSessionManager._cache_window_expired(None, 300) is False diff --git a/backend/tests/agents/main_agent/session/test_compaction_summary.py b/backend/tests/agents/main_agent/session/test_compaction_summary.py new file mode 100644 index 000000000..08681ad85 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_compaction_summary.py @@ -0,0 +1,213 @@ +"""Bounded compaction summary — spiral spec PR-2 / thresholds spec §3.6.""" + +import sys +import types +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.compaction_models import CompactionConfig, CompactionState +from agents.main_agent.session.compaction_summary import ( + approx_tokens, + bound_summary, + truncate_records_newest_first, +) + +from .conftest import make_conversation + + +BUDGET = 100 # tokens → 400 chars + + +@pytest.fixture +def bedrock(monkeypatch): + """Patch boto3 so no test reaches Bedrock; returns the converse mock.""" + converse = MagicMock() + client = MagicMock() + client.converse = converse + module = types.SimpleNamespace(client=MagicMock(return_value=client)) + monkeypatch.setitem(sys.modules, "boto3", module) + return converse + + +def _model_reply(text, stop="end_turn"): + return {"stopReason": stop, "output": {"message": {"content": [{"text": text}]}}} + + +class TestTruncateNewestFirst: + def test_keeps_newest_records_that_fit(self): + records = ["old " * 50, "mid " * 50, "new " * 50] # 200 chars each + out = truncate_records_newest_first(records, budget_tokens=110) # 440 chars + assert out.startswith("mid") and out.endswith("new ") + assert "old" not in out + + def test_keeps_tail_of_newest_when_nothing_fits(self): + newest = "x" * 1000 + "TAIL" + out = truncate_records_newest_first(["ancient", newest], budget_tokens=10) # 40 chars + assert out.endswith("TAIL") and len(out) == 40 + + def test_empty(self): + assert truncate_records_newest_first([], 10) is None + assert truncate_records_newest_first(["", ""], 10) is None + + +class TestBoundSummary: + @pytest.mark.asyncio + async def test_within_budget_is_untouched(self, bedrock): + result = await bound_summary(["a", "b"], BUDGET, model_enabled=True, model_id="m") + assert result.text == "a\n\nb" and result.outcome == "within_budget" + bedrock.assert_not_called() + + @pytest.mark.asyncio + async def test_model_compresses_once_at_cut(self, bedrock): + bedrock.return_value = _model_reply("Standing instructions: cite APA. Open: intro draft.") + records = ["r" * 300, "s" * 300] + result = await bound_summary(records, BUDGET, model_enabled=True, model_id="m") + assert result.outcome == "model" + assert approx_tokens(result.text) <= BUDGET + assert bedrock.call_count == 1 + kwargs = bedrock.call_args.kwargs + assert kwargs["modelId"] == "m" + assert "Standing instructions" in kwargs["system"][0]["text"] + + @pytest.mark.asyncio + async def test_model_failure_falls_back_to_newest_first(self, bedrock): + bedrock.side_effect = RuntimeError("throttled") + records = ["old " * 100, "new " * 50] # 400 + 200 chars + result = await bound_summary(records, BUDGET, model_enabled=True, model_id="m") + assert result.outcome == "truncated_after_model" + assert result.text.startswith("new") and "old" not in result.text + assert approx_tokens(result.text) <= BUDGET + + @pytest.mark.asyncio + async def test_model_ceiling_hit_is_a_failure(self, bedrock): + bedrock.return_value = _model_reply("frag", stop="max_tokens") + result = await bound_summary(["r" * 900], BUDGET, model_enabled=True, model_id="m") + assert result.outcome == "truncated_after_model" + assert approx_tokens(result.text) <= BUDGET + + @pytest.mark.asyncio + async def test_model_overshoot_is_tail_trimmed(self, bedrock): + bedrock.return_value = _model_reply("y" * 2000 + "END") + result = await bound_summary(["r" * 900], BUDGET, model_enabled=True, model_id="m") + assert result.outcome == "truncated_after_model" + assert result.text.endswith("END") and approx_tokens(result.text) <= BUDGET + + @pytest.mark.asyncio + async def test_kill_switch_skips_model(self, bedrock): + result = await bound_summary(["r" * 900], BUDGET, model_enabled=False, model_id="m") + assert result.outcome == "truncated" + bedrock.assert_not_called() + + @pytest.mark.asyncio + async def test_empty(self, bedrock): + result = await bound_summary([], BUDGET, model_enabled=True, model_id="m") + assert result.text is None and result.outcome == "empty" + + def test_from_env(self, monkeypatch): + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_SUMMARY_TOKEN_BUDGET", "1234") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ENABLED", "false") + monkeypatch.setenv("AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ID", "us.amazon.nova-lite-v1:0") + cfg = CompactionConfig.from_env() + assert cfg.summary_token_budget == 1234 + assert cfg.summary_model_enabled is False + assert cfg.summary_model_id == "us.amazon.nova-lite-v1:0" + monkeypatch.delenv("AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ENABLED") + assert CompactionConfig.from_env().summary_model_enabled is True + + +class TestThroughUpdateAfterTurn: + """Acceptance: oversized LTM records → the persisted summary is ≤ budget and + the restore prepends the same bounded bytes.""" + + def _manager(self, make_session_manager, records, **cfg): + config = CompactionConfig(enabled=True, deferred_apply_enabled=False, token_threshold=1000, protected_turns=3, + summary_token_budget=BUDGET, **cfg) + mgr = make_session_manager(compaction_config=config) + mgr.compaction_state = CompactionState() + mgr._save_compaction_state = MagicMock() + mgr._retrieve_session_summaries = MagicMock(return_value=records) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8] + mgr._all_messages_for_summary = make_conversation(5) + return mgr + + @pytest.mark.asyncio + async def test_oversized_ltm_join_is_bounded_and_persisted(self, make_session_manager, bedrock): + bedrock.side_effect = RuntimeError("no model in tests") + records = [f"record {i} " + "z" * 600 for i in range(10)] # ~6k chars + mgr = self._manager(make_session_manager, records) + result = await mgr.update_after_turn(2000) + assert result is not None + state = mgr.compaction_state + assert approx_tokens(state.summary) <= BUDGET + # Nothing fits whole (each record ~610 chars vs a 400-char budget), so + # the tail of the NEWEST record is kept — never the oldest. + assert state.summary == records[-1][-BUDGET * 4:] + assert state.policy["summarySource"] == "ltm" + assert state.policy["summaryOutcome"] == "truncated_after_model" + assert state.policy["summaryTokensBefore"] > BUDGET >= state.policy["summaryTokensAfter"] + assert state.policy["summaryTokenBudget"] == BUDGET + # The restore path prepends exactly the persisted bytes. + restored = mgr._prepend_summary_to_first_message(make_conversation(2), state.summary) + assert state.summary in restored[0]["content"][0]["text"] + + @pytest.mark.asyncio + async def test_small_ltm_join_is_untouched(self, make_session_manager, bedrock): + mgr = self._manager(make_session_manager, ["LTM summary 1", "LTM summary 2"]) + await mgr.update_after_turn(2000) + assert mgr.compaction_state.summary == "LTM summary 1\n\nLTM summary 2" + assert mgr.compaction_state.policy["summaryOutcome"] == "within_budget" + bedrock.assert_not_called() + + @pytest.mark.asyncio + async def test_fallback_summary_is_labelled(self, make_session_manager, bedrock): + mgr = self._manager(make_session_manager, []) + await mgr.update_after_turn(2000) + assert mgr.compaction_state.policy["summarySource"] == "fallback" + assert "Previous conversation" in mgr.compaction_state.summary + + +class TestCompactionMetrics: + @pytest.mark.asyncio + async def test_cut_emits_one_content_free_emf_record(self, make_session_manager, bedrock, monkeypatch): + import apis.shared.observability.emf as emf + emitted = [] + monkeypatch.setattr(emf, "emit_emf_metrics", lambda ns, metrics, properties=None, units=None: emitted.append((ns, metrics, properties))) + monkeypatch.delenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", raising=False) + config = CompactionConfig(enabled=True, deferred_apply_enabled=False, token_threshold=1000, protected_turns=3, summary_token_budget=BUDGET) + mgr = make_session_manager(compaction_config=config) + mgr.compaction_state = CompactionState() + mgr._save_compaction_state = MagicMock() + mgr._retrieve_session_summaries = MagicMock(return_value=["tiny"]) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8] + mgr._all_messages_for_summary = make_conversation(5) + + await mgr.update_after_turn(2000) + + assert len(emitted) == 1 + ns, metrics, props = emitted[0] + assert ns == "AgentCoreStack/Compaction" + assert metrics["CompactionCut"] == 1 and metrics["CompactionForced"] == 0 + assert metrics["CompactionInputTokens"] == 2000 + # Tiny 5-turn conversation calibrated to 2000 tokens against a 250 + # floor with 3 protected turns: the tail cannot fit → unreachable. + assert metrics["CompactionFloorUnreachable"] == 1 + assert props["summaryOutcome"] == "within_budget" and props["policySource"] == "fixed" + # Content-free: no summary text, no message text in the record. + assert "tiny" not in str(props) + + @pytest.mark.asyncio + async def test_kill_switch_silences_metrics(self, make_session_manager, bedrock, monkeypatch): + import apis.shared.observability.emf as emf + emitted = [] + monkeypatch.setattr(emf, "emit_emf_metrics", lambda *a, **k: emitted.append(a)) + monkeypatch.setenv("PROMPT_CACHE_OBSERVABILITY_ENABLED", "false") + config = CompactionConfig(enabled=True, deferred_apply_enabled=False, token_threshold=1000, protected_turns=3) + mgr = make_session_manager(compaction_config=config) + mgr.compaction_state = CompactionState() + mgr._save_compaction_state = MagicMock() + mgr._retrieve_session_summaries = MagicMock(return_value=[]) + mgr._valid_cutoff_indices = [0, 2, 4, 6, 8] + mgr._all_messages_for_summary = make_conversation(5) + await mgr.update_after_turn(2000) + assert emitted == [] diff --git a/backend/tests/agents/main_agent/session/test_context_attribution_hook.py b/backend/tests/agents/main_agent/session/test_context_attribution_hook.py index 6d18321e2..ef519f793 100644 --- a/backend/tests/agents/main_agent/session/test_context_attribution_hook.py +++ b/backend/tests/agents/main_agent/session/test_context_attribution_hook.py @@ -11,6 +11,8 @@ from agents.main_agent.session.hooks.context_attribution import ( ContextAttributionHook, + clear_probe_baselines, + clear_split_memo, get_context_breakdown, ) @@ -80,15 +82,18 @@ async def test_computes_partitions_that_sum_to_total(self): assert sum(parts.values()) == bd["total"] @pytest.mark.asyncio - async def test_makes_exactly_two_count_calls_neither_with_tools(self): + async def test_makes_exactly_three_count_calls_none_with_tools_and_none_empty(self): model = FakeModel() agent = FakeAgent(model, messages=[{"role": "user", "content": [{"text": "hi"}]}]) await ContextAttributionHook()._on_before_model_call(_event(agent, projected=650)) assert model.calls == [ - {"n_messages": 0, "has_tools": False}, # system only + {"n_messages": 1, "has_tools": False}, # probe only (per-model baseline) + {"n_messages": 1, "has_tools": False}, # probe + system {"n_messages": 1, "has_tools": False}, # system + messages, no tools ] + # Bedrock rejects an empty conversation; the hook must never send one. + assert all(c["n_messages"] >= 1 for c in model.calls) @pytest.mark.asyncio async def test_tool_partition_absorbs_scaffolding(self): @@ -136,9 +141,9 @@ async def test_cold_start_counts_full_with_tools_when_projected_none(self): # full counted with tools = 100 + 10 + 500 = 610; tools = 610-110 = 500 assert bd["total"] == 610 assert parts == {"system": 100, "tools": 500, "messages": 10} - # 3 calls: system only, no-tools, then full WITH tools - assert len(model.calls) == 3 - assert model.calls[2]["has_tools"] is True + # 4 calls: probe baseline, probe+system, no-tools, then full WITH tools + assert len(model.calls) == 4 + assert model.calls[3]["has_tools"] is True @pytest.mark.asyncio async def test_warm_turn_without_projected_leaves_breakdown_untouched(self): @@ -164,3 +169,235 @@ async def test_count_failure_is_swallowed_and_yields_no_breakdown(self): def test_get_context_breakdown_is_none_when_absent(self): agent = FakeAgent(FakeModel(), messages=[]) assert get_context_breakdown(agent) is None + + +class TestInlineAttachmentGuard: + """``toolTokens`` is a residual between two independently sourced counts + (``full`` from Strands' projection, ``no_tools`` from our CountTokens call), + so any disagreement about how a content block is counted lands wholly in it. + + Measured on dev 2026-09-16 (session ``61de2256``): a call reported + ``toolTokens`` of 106,756 where the session's real tools prefix was 12,516 — + a 94,240 difference against a document measured at ~94,485, i.e. the entire + document attributed to tools. The split is therefore not computed while + inline bytes are in context. + """ + + def _doc_message(self): + return { + "role": "user", + "content": [ + {"text": "what does it say?"}, + {"document": {"format": "pdf", "name": "d_pdf", "source": {"bytes": b"%PDF-1.4"}}}, + ], + } + + @pytest.mark.asyncio + async def test_no_split_is_computed_while_a_document_is_inline(self): + model = FakeModel() + agent = FakeAgent(model, messages=[self._doc_message()]) + await ContextAttributionHook()._on_before_model_call(_event(agent, projected=100_000)) + + assert get_context_breakdown(agent) is None, "a contaminated split must not be published" + assert model.calls == [], "and it must not pay for CountTokens to compute one" + + @pytest.mark.asyncio + async def test_an_image_counts_too(self): + model = FakeModel() + agent = FakeAgent(model, messages=[{ + "role": "user", + "content": [{"image": {"format": "png", "source": {"bytes": b"\x89PNG"}}}], + }]) + await ContextAttributionHook()._on_before_model_call(_event(agent, projected=50_000)) + assert get_context_breakdown(agent) is None + + @pytest.mark.asyncio + async def test_a_digest_is_not_an_attachment_so_the_split_is_taken(self): + """The offload turns the document into text, which counts normally — + so an attachment session still gets ``prefixTokens`` from turn 2.""" + model = FakeModel(system=100, per_msg=10, tool_overhead=500) + agent = FakeAgent(model, messages=[{ + "role": "user", + "content": [{"text": ''}], + }]) + await ContextAttributionHook()._on_before_model_call(_event(agent, projected=650)) + assert _parts(get_context_breakdown(agent)) == {"system": 100, "tools": 540, "messages": 10} + + @pytest.mark.asyncio + async def test_a_later_clean_turn_computes_the_split(self): + """Deferred, not abandoned: the same agent takes the split once the + attachment has left the live context.""" + model = FakeModel(system=100, per_msg=10, tool_overhead=500) + agent = FakeAgent(model, messages=[self._doc_message()]) + hook = ContextAttributionHook() + await hook._on_before_model_call(_event(agent, projected=100_000)) + assert get_context_breakdown(agent) is None + + agent.messages = [{"role": "user", "content": [{"text": "follow-up"}]}] + await hook._on_before_model_call(_event(agent, projected=650)) + assert _parts(get_context_breakdown(agent)) == {"system": 100, "tools": 540, "messages": 10} + + @pytest.mark.asyncio + async def test_a_cached_split_is_still_used_when_a_document_arrives_later(self): + """The guard only defers the *computation*. An agent that already has a + trustworthy split keeps reporting against it.""" + model = FakeModel(system=100, per_msg=10, tool_overhead=500) + agent = FakeAgent(model, messages=[{"role": "user", "content": [{"text": "hi"}]}]) + hook = ContextAttributionHook() + await hook._on_before_model_call(_event(agent, projected=650)) + + agent.messages = [{"role": "user", "content": [{"text": "hi"}]}, self._doc_message()] + await hook._on_before_model_call(_event(agent, projected=95_000)) + parts = _parts(get_context_breakdown(agent)) + assert parts["system"] == 100 and parts["tools"] == 540 + assert parts["messages"] == 95_000 - 640, "the document lands in messages, where it belongs" + + +class TestSessionSplitMemo: + """A rebuilt Agent for the same session + configuration adopts the split + its predecessor measured instead of paying two CountTokens calls again. + + Sessions whose injected tools keep them out of the agent cache rebuild + their Agent every turn; without the memo that was 2 extra counts per turn + (docs/specs/load-test-assessment-2026-09.md §1 fix 1).""" + + def setup_method(self): + clear_split_memo() + + def teardown_method(self): + clear_split_memo() + + @pytest.mark.asyncio + async def test_second_agent_for_same_session_makes_no_count_calls(self): + msgs = [{"role": "user", "content": [{"text": "hi"}]}] + first_model = FakeModel(system=100, per_msg=10, tool_overhead=500) + first = FakeAgent(first_model, messages=list(msgs)) + await ContextAttributionHook(session_id="s1")._on_before_model_call(_event(first, projected=650)) + assert len(first_model.calls) == 3 + + second_model = FakeModel(system=100, per_msg=10, tool_overhead=500) + second = FakeAgent(second_model, messages=list(msgs) + [{"role": "assistant", "content": [{"text": "yo"}]}]) + await ContextAttributionHook(session_id="s1")._on_before_model_call(_event(second, projected=660)) + + assert second_model.calls == [] + assert _parts(get_context_breakdown(second)) == {"system": 100, "tools": 540, "messages": 20} + + @pytest.mark.asyncio + async def test_different_session_recounts(self): + msgs = [{"role": "user", "content": [{"text": "hi"}]}] + await ContextAttributionHook(session_id="s1")._on_before_model_call( + _event(FakeAgent(FakeModel(), messages=list(msgs)), projected=650) + ) + other_model = FakeModel() + await ContextAttributionHook(session_id="s2")._on_before_model_call( + _event(FakeAgent(other_model, messages=list(msgs)), projected=650) + ) + assert len(other_model.calls) == 3 + + @pytest.mark.asyncio + async def test_changed_tools_or_prompt_recounts(self): + msgs = [{"role": "user", "content": [{"text": "hi"}]}] + await ContextAttributionHook(session_id="s1")._on_before_model_call( + _event(FakeAgent(FakeModel(), messages=list(msgs)), projected=650) + ) + + tools_changed = FakeModel() + await ContextAttributionHook(session_id="s1")._on_before_model_call( + _event(FakeAgent(tools_changed, messages=list(msgs), tool_specs=[{"name": "t"}, {"name": "u"}]), projected=700) + ) + assert len(tools_changed.calls) == 3 + + prompt_changed = FakeModel() + await ContextAttributionHook(session_id="s1")._on_before_model_call( + _event(FakeAgent(prompt_changed, messages=list(msgs), system_prompt="OTHER"), projected=650) + ) + assert len(prompt_changed.calls) == 3 + + @pytest.mark.asyncio + async def test_no_session_id_means_instance_only(self): + msgs = [{"role": "user", "content": [{"text": "hi"}]}] + await ContextAttributionHook()._on_before_model_call( + _event(FakeAgent(FakeModel(), messages=list(msgs)), projected=650) + ) + again = FakeModel() + await ContextAttributionHook()._on_before_model_call( + _event(FakeAgent(again, messages=list(msgs)), projected=650) + ) + assert len(again.calls) == 3 + + @pytest.mark.asyncio + async def test_a_deferred_split_is_not_memoised(self): + # Inline attachment → the split is skipped, so nothing must be stored + # for a later clean agent to adopt. + pdf = {"role": "user", "content": [{"document": {"format": "pdf", "name": "d", "source": {"bytes": b"%PDF"}}}]} + skipped = FakeModel() + await ContextAttributionHook(session_id="s1")._on_before_model_call( + _event(FakeAgent(skipped, messages=[pdf]), projected=650) + ) + assert skipped.calls == [] + + clean = FakeModel() + await ContextAttributionHook(session_id="s1")._on_before_model_call( + _event(FakeAgent(clean, messages=[{"role": "user", "content": [{"text": "hi"}]}]), projected=650) + ) + assert len(clean.calls) == 3 + + def test_memo_is_bounded(self): + from agents.main_agent.session.hooks import context_attribution as ca + + for i in range(ca._SPLIT_MEMO_MAX + 50): + ca._memo_put((f"s{i}", "p", "t"), {"systemTokens": 1, "toolTokens": 1}) + assert len(ca._split_memo) == ca._SPLIT_MEMO_MAX + assert ca._memo_get(("s0", "p", "t")) is None + assert ca._memo_get((f"s{ca._SPLIT_MEMO_MAX + 49}", "p", "t")) is not None + + +class TestProbeBaseline: + """The system prompt is counted against a fixed probe user message because + Bedrock refuses an empty conversation. The probe's own weight is a + per-model constant, measured once per model id per process.""" + + def setup_method(self): + clear_probe_baselines() + + def teardown_method(self): + clear_probe_baselines() + + class ConfiguredModel(FakeModel): + def __init__(self, model_id, **kw): + super().__init__(**kw) + self.config = {"model_id": model_id} + + @pytest.mark.asyncio + async def test_probe_weight_is_measured_once_per_model_id(self): + msgs = [{"role": "user", "content": [{"text": "hi"}]}] + first = self.ConfiguredModel("us.anthropic.x", system=100, per_msg=10) + await ContextAttributionHook()._on_before_model_call(_event(FakeAgent(first, messages=list(msgs)), projected=650)) + assert len(first.calls) == 3 + + second = self.ConfiguredModel("us.anthropic.x", system=100, per_msg=10) + second_agent = FakeAgent(second, messages=list(msgs)) + await ContextAttributionHook()._on_before_model_call(_event(second_agent, projected=650)) + # Baseline reused: probe+system and no-tools only. + assert len(second.calls) == 2 + assert _parts(get_context_breakdown(second_agent))["system"] == 100 + + other = self.ConfiguredModel("us.anthropic.y", system=100, per_msg=10) + await ContextAttributionHook()._on_before_model_call(_event(FakeAgent(other, messages=list(msgs)), projected=650)) + assert len(other.calls) == 3 + + @pytest.mark.asyncio + async def test_system_partition_is_the_difference_not_the_probe(self): + # A probe that weighs 24 tokens on its own must not leak into system. + model = self.ConfiguredModel("m", system=7, per_msg=24) + agent = FakeAgent(model, messages=[{"role": "user", "content": [{"text": "hi"}]}]) + await ContextAttributionHook()._on_before_model_call(_event(agent, projected=600)) + assert _parts(get_context_breakdown(agent))["system"] == 7 + + @pytest.mark.asyncio + async def test_a_model_without_a_config_counts_the_probe_each_time(self): + msgs = [{"role": "user", "content": [{"text": "hi"}]}] + a, b = FakeModel(), FakeModel() + await ContextAttributionHook()._on_before_model_call(_event(FakeAgent(a, messages=list(msgs)), projected=650)) + await ContextAttributionHook()._on_before_model_call(_event(FakeAgent(b, messages=list(msgs)), projected=650)) + assert len(a.calls) == len(b.calls) == 3 diff --git a/backend/tests/agents/main_agent/session/test_context_ledger_hook.py b/backend/tests/agents/main_agent/session/test_context_ledger_hook.py new file mode 100644 index 000000000..039ac6b97 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_context_ledger_hook.py @@ -0,0 +1,109 @@ +"""`ContextLedgerHook` — per model call, the window's trimmed-message count and +the compaction decisions taken since the previous call. + +Same lifecycle contract as the tool census: keyed by the turn's Nth model +call, reset per turn, read (never drained) at turn end, off with the same +kill switch, and never able to break a turn. +""" + +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.hooks.context_ledger import _MAX_EVENTS_PER_CALL, ContextLedgerHook + + +@pytest.fixture(autouse=True) +def diagnostics_enabled(monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + + +class _Manager: + def __init__(self, events=None): + self._events = list(events or []) + + def drain_compaction_events(self): + events, self._events = self._events, [] + return events + + +def _agent(removed=None, manager=None): + agent = MagicMock() + agent.conversation_manager.removed_message_count = removed + agent._session_manager = manager + return agent + + +def _call(hook, agent): + hook._on_before_model_call(MagicMock(agent=agent)) + + +def test_records_the_window_count_and_drains_compaction_events_per_call(): + hook = ContextLedgerHook() + manager = _Manager([{"kind": "applied", "checkpoint": 12, "summaryTokens": 900}]) + hook._on_turn_start(MagicMock()) + _call(hook, _agent(removed=0, manager=manager)) + _call(hook, _agent(removed=4, manager=manager)) # nothing left to drain + + assert hook.ledger_for_call(0) == { + "windowRemovedMessages": 0, + "compactionEvents": [{"kind": "applied", "checkpoint": 12, "summaryTokens": 900}], + } + assert hook.ledger_for_call(1) == {"windowRemovedMessages": 4} + assert hook.ledger_for_call(2) is None + + +def test_a_new_turn_forgets_the_previous_one(): + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + _call(hook, _agent(removed=7)) + hook._on_turn_start(MagicMock()) + assert hook.ledger_for_call(0) is None + + +def test_reads_do_not_drain_and_do_not_alias(): + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + _call(hook, _agent(removed=1, manager=_Manager([{"kind": "checkpoint"}]))) + first = hook.ledger_for_call(0) + first["compactionEvents"].append({"kind": "tampered"}) + assert hook.ledger_for_call(0)["compactionEvents"] == [{"kind": "checkpoint"}] + + +def test_no_window_manager_and_no_session_manager_records_nothing(): + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + agent = MagicMock() + agent.conversation_manager = None + agent._session_manager = None + _call(hook, agent) + assert hook.ledger_for_call(0) is None + + +def test_kill_switch_records_nothing_and_reads_none(monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + _call(hook, _agent(removed=3, manager=_Manager([{"kind": "applied"}]))) + assert hook.ledger_for_call(0) is None + + +def test_event_list_is_bounded_and_malformed_counts_never_raise(): + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + many = [{"kind": "applied"}] * (_MAX_EVENTS_PER_CALL + 5) + _call(hook, _agent(removed="not-a-number", manager=_Manager(many))) + entry = hook.ledger_for_call(0) + assert "windowRemovedMessages" not in entry + assert len(entry["compactionEvents"]) == _MAX_EVENTS_PER_CALL + + +def test_a_raising_manager_is_swallowed(): + class Broken: + def drain_compaction_events(self): + raise RuntimeError("boom") + + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + _call(hook, _agent(removed=2, manager=Broken())) + assert hook.ledger_for_call(0) == {"windowRemovedMessages": 2} diff --git a/backend/tests/agents/main_agent/session/test_document_context.py b/backend/tests/agents/main_agent/session/test_document_context.py new file mode 100644 index 000000000..ff660fdc3 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_document_context.py @@ -0,0 +1,145 @@ +"""The document-lifecycle analytics (docs/specs/document-context-offload.md §6.1): + +- ``summarize_document_context`` — the attachment footprint of the live + context, persisted per cost row (content-free by construction). +- the ``document_stripped`` compaction-ledger event ``_strip_document_bytes`` + records, so the restore defect is measured before PR-3 fixes it. +- the ``documentReads`` ledger entry the ``ContextLedgerHook`` tallies from + ``document_read`` results. +""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session.compaction_policy import CHARS_PER_TOKEN, IMAGE_TOKEN_ESTIMATE +from agents.main_agent.session.document_context import summarize_document_context +from agents.main_agent.session.hooks.context_ledger import ContextLedgerHook + + +@pytest.fixture(autouse=True) +def diagnostics_enabled(monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + + +def _doc(name="a", fmt="pdf", size=4000): + return {"document": {"format": fmt, "name": name, "source": {"bytes": b"x" * size}}} + + +def _image(size=100): + return {"image": {"format": "png", "source": {"bytes": b"i" * size}}} + + +class TestSummary: + def test_empty_or_malformed_is_none(self): + assert summarize_document_context(None) is None + assert summarize_document_context([]) is None + + def test_counts_tokens_formats_digests_and_the_last_prompts_attachments(self): + messages = [ + {"role": "user", "content": [{"text": "q1"}, _doc("a", "pdf", 4000), _doc("b", "docx", 800), _image()]}, + {"role": "assistant", "content": [{"text": "a1"}]}, + {"role": "user", "content": [{"text": "q2"}, {"text": "[Document placeholder: name=c, format=pdf, original_size=9 bytes]"}]}, + {"role": "assistant", "content": [{"toolUse": {"toolUseId": "t", "name": "document_read", "input": {}}}]}, + {"role": "user", "content": [{"toolResult": {"toolUseId": "t", "status": "success", "content": [ + {"json": {"pages_returned": 2}}, _doc("slice", "pdf", 2000)]}}]}, + {"role": "user", "content": [{"text": " "}, {"text": "q3"}]}, + ] + assert summarize_document_context(messages) == { + "hasDocuments": True, + "documentCount": 3, + "documentTokens": 4000 // CHARS_PER_TOKEN + 800 // CHARS_PER_TOKEN + IMAGE_TOKEN_ESTIMATE, + "documentDigests": 2, + "documentsAttached": 0, # the last prompt (q3) attached nothing + "documentSlices": 1, + "documentSliceTokens": 2000 // CHARS_PER_TOKEN, + "documentMime": {"pdf": 1, "docx": 1, "image": 1}, + } + + def test_a_digest_only_context_reads_as_no_documents(self): + messages = [ + {"role": "user", "content": [{"text": "[Document placeholder: name=a, format=pdf, original_size=1 bytes]"}, {"text": "q"}]}, + {"role": "assistant", "content": [{"text": "a"}]}, + ] + summary = summarize_document_context(messages) + assert summary["hasDocuments"] is False + assert summary["documentCount"] == 0 and summary["documentDigests"] == 1 + + def test_attach_turn_counts_its_own_attachments(self): + messages = [{"role": "user", "content": [{"text": "q"}, _doc(), _doc("b")]}] + assert summarize_document_context(messages)["documentsAttached"] == 2 + + def test_never_emits_names_or_bytes(self): + summary = summarize_document_context([{"role": "user", "content": [_doc("Secret Contract")]}]) + assert "Secret" not in repr(summary) and b"x" not in repr(summary).encode() + + +class TestStripEvent: + """With no upload rows to match, restore falls back to the placeholder and + records ``document_stripped`` — the pre-PR-3 behavior, kept as the floor. + The rehydrated path is pinned in ``test_document_rehydration.py``.""" + + @pytest.fixture(autouse=True) + def no_upload_rows(self, monkeypatch): + monkeypatch.setattr( + "agents.main_agent.session.document_rehydration.load_session_documents", lambda *_: [] + ) + + def test_strip_records_one_content_free_event(self, make_session_manager): + manager = make_session_manager() + messages = [ + {"role": "user", "content": [{"text": "q"}, _doc("a", "pdf", 8000), _doc("b", "txt", 400)]}, + {"role": "assistant", "content": [{"text": "a"}]}, + ] + stripped = manager._strip_document_bytes(messages) + assert all("document" not in b for b in stripped[0]["content"]) + assert manager.drain_compaction_events() == [ + {"kind": "document_stripped", "documents": 2, "documentTokens": 8400 // CHARS_PER_TOKEN} + ] + + def test_nothing_to_strip_records_nothing(self, make_session_manager): + manager = make_session_manager() + manager._strip_document_bytes([{"role": "user", "content": [{"text": "q"}]}]) + assert manager.drain_compaction_events() == [] + + def test_kill_switch_still_strips_but_records_nothing(self, make_session_manager, monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + manager = make_session_manager() + stripped = manager._strip_document_bytes([{"role": "user", "content": [_doc()]}]) + assert "text" in stripped[0]["content"][0] + assert manager.drain_compaction_events() == [] + + +def _tool_event(name="document_read", status="success", pages=3, size=500): + content = [{"json": {"pages_returned": pages}}] + if size: + content.append(_doc("slice", size=size)) + return MagicMock(tool_use={"toolUseId": "t", "name": name}, result={"toolUseId": "t", "status": status, "content": content}) + + +class TestLedgerDocumentReads: + def test_reads_are_attributed_to_the_requesting_call(self): + hook = ContextLedgerHook() + agent = MagicMock() + agent.conversation_manager.removed_message_count = None + agent._session_manager = None + hook._on_turn_start(MagicMock()) + hook._on_before_model_call(MagicMock(agent=agent)) # call 0 + hook._on_after_tool_call(_tool_event(pages=3, size=500)) + hook._on_after_tool_call(_tool_event(pages=2, size=300)) + hook._on_before_model_call(MagicMock(agent=agent)) # call 1 + hook._on_after_tool_call(_tool_event(name="calculator")) + hook._on_after_tool_call(_tool_event(status="error")) + + assert hook.ledger_for_call(0) == {"documentReads": {"calls": 2, "pages": 5, "bytes": 800}} + assert hook.ledger_for_call(1) is None + + def test_kill_switch_records_nothing(self, monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + hook = ContextLedgerHook() + hook._on_turn_start(MagicMock()) + hook._on_before_model_call(MagicMock()) + hook._on_after_tool_call(_tool_event()) + assert hook.ledger_for_call(0) is None diff --git a/backend/tests/agents/main_agent/session/test_document_offload.py b/backend/tests/agents/main_agent/session/test_document_offload.py new file mode 100644 index 000000000..66571e925 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_document_offload.py @@ -0,0 +1,295 @@ +"""Document offload (offload spec §4C / §4D, PR-4). + +Pins the three rules — pinning, minimum size, and free-or-unavoidable — the +in-place mutation (never rebinding ``agent.messages``), byte-equality with +the restore transformation, slice ageing on both paths, the rollout bucket, +and the ledger event with its cache gap. +""" + +from __future__ import annotations + +from datetime import datetime, timedelta, timezone +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session import document_offload as do +from agents.main_agent.session.compaction_models import CompactionConfig, CompactionState +from apis.shared.files.models import FileMetadata, FileStatus + +REHYDRATION = "agents.main_agent.session.document_rehydration" + + +@pytest.fixture(autouse=True) +def _env(monkeypatch): + for var in ("COST_DIAGNOSTICS_ENABLED", "DOCUMENT_OFFLOAD_ENABLED", "DOCUMENT_OFFLOAD_ROLLOUT_PERCENT"): + monkeypatch.delenv(var, raising=False) + + +BIG = 40_000 # bytes → 10k estimated tokens, over the 5k floor +SMALL = 4_000 # 1k tokens, under it + +READY_DIGEST = { + "version": 1, "status": "ready", "format": "pdf", "unit": "page", "count": 12, + "sections": [{"start": 1, "title": "Intro"}], "abstract": "A policy.", "tokens": 40, +} + + +def _doc(name="policy_pdf", size=BIG, fmt="pdf"): + return {"document": {"format": fmt, "name": name, "source": {"bytes": b"x" * size}}} + + +def _slice(name="policy p4-7 abc123", size=2000): + return {"toolResult": {"toolUseId": "t", "status": "success", "content": [ + {"json": {"upload_id": "up-1", "filename": "policy.pdf", "pages_returned": 4}}, + _doc(name, size), + ]}} + + +def _prompt(text, *blocks): + return {"role": "user", "content": [{"text": text}, *blocks]} + + +def _reply(text="ok"): + return {"role": "assistant", "content": [{"text": text}]} + + +def _meta(upload_id="up-1", filename="policy.pdf", size=BIG, digest=READY_DIGEST): + return FileMetadata( + upload_id=upload_id, user_id="u1", session_id="s1", filename=filename, mime_type="application/pdf", + size_bytes=size, s3_key="k", s3_bucket="b", status=FileStatus.READY, digest=digest, + ) + + +# --------------------------------------------------------------------------- +# Pure rules +# --------------------------------------------------------------------------- + + +class TestPinning: + def test_attach_turn_and_the_next_are_pinned_then_released(self): + conv = [_prompt("read this", _doc()), _reply()] + assert do.pinned_document_names(conv, "next question") == {"policy_pdf"} # attach turn (1 of 2) + conv += [_prompt("next question"), _reply()] + assert do.pinned_document_names(conv, "and another") == {"policy_pdf"} # turn after (2 of 2) + conv += [_prompt("and another"), _reply()] + assert do.pinned_document_names(conv, "unrelated") == set() # released + + def test_naming_the_document_in_the_prompt_pins_it(self): + conv = [_prompt("q", _doc("BBR Policy_pdf")), _reply(), _prompt("a"), _reply(), _prompt("b"), _reply()] + assert do.pinned_document_names(conv, "what does the bbr policy say about limits?") == {"BBR Policy_pdf"} + assert do.pinned_document_names(conv, "unrelated") == set() + # The previous prompt counts too (the model is still answering it). + conv[-2] = _prompt("compare against the BBR policy") + assert do.pinned_document_names(conv, "go on") == {"BBR Policy_pdf"} + + def test_a_recent_document_read_result_pins_its_document(self): + conv = [ + _prompt("q", _doc()), _reply(), _prompt("a"), _reply(), _prompt("b"), _reply(), + _prompt("show page 4"), {"role": "assistant", "content": [{"toolUse": {"toolUseId": "t", "name": "document_read", "input": {}}}]}, + {"role": "user", "content": [_slice()]}, _reply(), + ] + assert do.pinned_document_names(conv, "thanks") == {"policy_pdf"} + + def test_short_stems_never_match_prompt_text(self): + conv = [_prompt("q", _doc("a_pdf")), _reply(), _prompt("x"), _reply(), _prompt("y"), _reply()] + assert do.pinned_document_names(conv, "a question about a thing") == set() + + def test_name_stem(self): + assert do.name_stem("BBR Policy_pdf_2") == "BBR Policy" + assert do.name_stem("notes_md") == "notes" + assert do.name_stem("plain") == "plain" + + +class TestCandidates: + def test_unpinned_and_large_only(self): + conv = [_prompt("q", _doc("big_pdf", BIG), _doc("small_pdf", SMALL), _doc("pinned_pdf", BIG)), _reply()] + cands = do.candidate_documents(conv, {"pinned_pdf"}) + assert [(c.message_index, c.block_index, c.name, c.tokens) for c in cands] == [(0, 1, "big_pdf", BIG // 4)] + + def test_digest_and_s3_blocks_are_not_candidates(self): + conv = [_prompt("q", {"text": ""}, + {"document": {"format": "pdf", "name": "n", "source": {"s3Location": {"uri": "s3://b/k"}}}})] + assert do.candidate_documents(conv, set()) == [] + + +class TestRollout: + def test_bucket_is_stable_and_flag_and_percent_gate(self, monkeypatch): + assert do.session_bucket("s1") == do.session_bucket("s1") and 0 <= do.session_bucket("s1") < 100 + assert do.offload_enabled_for("s1") is True and do.offload_enabled_for("") is False + monkeypatch.setenv("DOCUMENT_OFFLOAD_ROLLOUT_PERCENT", "0") + assert do.offload_enabled_for("s1") is False + monkeypatch.setenv("DOCUMENT_OFFLOAD_ROLLOUT_PERCENT", str(do.session_bucket("s1") + 1)) + assert do.offload_enabled_for("s1") is True + monkeypatch.setenv("DOCUMENT_OFFLOAD_ROLLOUT_PERCENT", "garbage") + assert do.rollout_percent() == 100 + monkeypatch.setenv("DOCUMENT_OFFLOAD_ENABLED", "false") + assert do.offload_enabled_for("s1") is False + + +# --------------------------------------------------------------------------- +# Mutations +# --------------------------------------------------------------------------- + + +class TestOffloadDocuments: + def test_replaces_in_place_with_the_restore_transformation(self, monkeypatch): + monkeypatch.setattr(f"{REHYDRATION}.load_session_documents", lambda *_: [_meta()]) + conv = [_prompt("q", _doc()), _reply(), _prompt("a"), _reply(), _prompt("b"), _reply()] + content_list = conv[0]["content"] + cands = do.candidate_documents(conv, set()) + result = do.offload_documents(conv, cands, session_id="s1", user_id="u1") + + assert result.offloaded == 1 and result.evicted_tokens == BIG // 4 and result.upload_ids == ["up-1"] + assert conv[0]["content"] is content_list # in place, never rebound + assert conv[0]["content"][1]["text"].startswith(' str: + return (datetime.now(timezone.utc) - timedelta(seconds=seconds_ago)).isoformat() + + +def _manager(make_session_manager, *, updated_seconds_ago=None, last_input=0, last_key=None, current_key=None, policy=None, enabled=True): + mgr = make_session_manager(compaction_config=CompactionConfig(enabled=enabled, cache_ttl_seconds=300)) + mgr.compaction_state = CompactionState( + updated_at=_iso(updated_seconds_ago) if updated_seconds_ago is not None else None, + last_input_tokens=last_input, last_prefix_key=last_key, policy=policy, + ) + mgr._current_prefix_key = current_key + mgr._adopt_persisted_compaction_state = lambda: None + mgr._save_compaction_state = lambda *a, **k: None + return mgr + + +def _agent(messages): + agent = MagicMock() + agent.messages = messages + return agent + + +def _cold_conversation(): + return [_prompt("q", _doc()), _reply(), _prompt("a"), _reply(), _prompt("b"), _reply()] + + +class TestApplyDocumentOffload: + @pytest.fixture(autouse=True) + def rows(self, monkeypatch): + monkeypatch.setattr(f"{REHYDRATION}.load_session_documents", lambda *_: [_meta()]) + + def test_waits_while_the_cache_is_live(self, make_session_manager): + mgr = _manager(make_session_manager, updated_seconds_ago=30, last_input=1000, policy={"ceiling": 100_000}) + agent = _agent(_cold_conversation()) + assert mgr.apply_document_offload(agent, prompt="x") is None + assert "document" in agent.messages[0]["content"][1] + assert mgr.drain_compaction_events() == [] + + def test_cache_expired_offloads_and_records_the_gap(self, make_session_manager): + mgr = _manager(make_session_manager, updated_seconds_ago=900, last_input=1000, policy={"ceiling": 100_000}) + messages = _cold_conversation() + agent = _agent(messages) + assert mgr.apply_document_offload(agent, prompt="x") == "cache_expired" + assert agent.messages is messages # never rebound + assert messages[0]["content"][1]["text"].startswith("= 899 and events[0]["digestTokens"] > 0 + + def test_prefix_change_and_over_ceiling_are_the_other_reasons(self, make_session_manager): + mgr = _manager(make_session_manager, updated_seconds_ago=10, last_input=1000, last_key="m|a", current_key="m|b", policy={"ceiling": 100_000}) + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") == "prefix_changed" + mgr = _manager(make_session_manager, updated_seconds_ago=10, last_input=150_000, last_key="m|a", current_key="m|a", policy={"ceiling": 100_000}) + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") == "over_ceiling" + + def test_pinned_documents_never_move_even_when_free(self, make_session_manager): + mgr = _manager(make_session_manager, updated_seconds_ago=900) + agent = _agent([_prompt("q", _doc()), _reply()]) # attach turn + assert mgr.apply_document_offload(agent, prompt="next") is None + assert "document" in agent.messages[0]["content"][1] + agent = _agent(_cold_conversation()) + assert mgr.apply_document_offload(agent, prompt="about the policy please") is None + + def test_compaction_off_uses_the_in_process_turn_stamp(self, make_session_manager): + mgr = _manager(make_session_manager, enabled=False) + mgr._last_turn_completed_at = _iso(900) + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") == "cache_expired" + mgr._last_turn_completed_at = _iso(5) + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") is None + + def test_kill_switch_and_bucket(self, make_session_manager, monkeypatch): + mgr = _manager(make_session_manager, updated_seconds_ago=900) + monkeypatch.setenv("DOCUMENT_OFFLOAD_ENABLED", "false") + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") is None + monkeypatch.delenv("DOCUMENT_OFFLOAD_ENABLED") + monkeypatch.setenv("DOCUMENT_OFFLOAD_ROLLOUT_PERCENT", "0") + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") is None + + def test_slices_age_on_the_live_path_under_the_same_gate(self, make_session_manager): + mgr = _manager(make_session_manager, updated_seconds_ago=900) + conv = [_prompt("p1"), _reply(), {"role": "user", "content": [_slice()]}, _reply(), _prompt("p2"), _reply(), _prompt("p3"), _reply()] + assert mgr.apply_document_offload(_agent(conv), prompt="x") == "cache_expired" + assert conv[2]["content"][0]["toolResult"]["content"][1]["text"].startswith("[Retrieved pages placeholder:") + assert mgr.drain_compaction_events()[0]["slices"] == 1 + + def test_never_raises(self, make_session_manager, monkeypatch): + mgr = _manager(make_session_manager, updated_seconds_ago=900) + monkeypatch.setattr("agents.main_agent.session.document_offload.candidate_documents", lambda *a, **k: (_ for _ in ()).throw(RuntimeError("boom"))) + assert mgr.apply_document_offload(_agent(_cold_conversation()), prompt="x") is None + + +class TestRestoreAgesSlices: + def test_initialize_stubs_old_slices_like_the_live_path(self, make_session_manager, monkeypatch): + monkeypatch.setattr(f"{REHYDRATION}.load_session_documents", lambda *_: []) + mgr = make_session_manager() + tool_use = {"role": "assistant", "content": [{"toolUse": {"toolUseId": "t", "name": "document_read", "input": {}}}]} + conv = [_prompt("p1"), tool_use, {"role": "user", "content": [_slice()]}, _reply(), _prompt("p2"), _reply(), _prompt("p3"), _reply()] + agent = MagicMock() + agent.agent_id = "default" + agent.messages = conv + mgr.read_agent = MagicMock(return_value=object()) + mgr.list_messages = MagicMock(return_value=[MagicMock(to_message=lambda m=m: m) for m in conv]) + mgr.initialize(agent) + stub = agent.messages[2]["content"][0]["toolResult"]["content"][1] + assert stub == {"text": do.slice_stub("policy p4-7 abc123")} diff --git a/backend/tests/agents/main_agent/session/test_document_read_flag_coupling.py b/backend/tests/agents/main_agent/session/test_document_read_flag_coupling.py new file mode 100644 index 000000000..dcf347c2f --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_document_read_flag_coupling.py @@ -0,0 +1,126 @@ +"""DOCUMENT_READ_ENABLED must reach every path that promises the tool. + +The gap this guards: the flag was read only at the tool-injection gate +(``_document_tools_gate``). Neither the restore path nor the live offload +consulted it, so pulling the one kill switch an operator would actually reach +for left restore emitting ```` handles for a tool +that would not be injected — and left the live path still evicting bytes, which +is strictly worse than the pre-offload world, where live bytes never left. + +Rules pinned here: + +* restore still renders a digest (it has to drop the bytes either way, and a + digest beats the placeholder) but **without** the ``upload_id`` handle; +* the live offload **stops entirely** — bytes stay inline; +* with the flag on, everything is byte-identical to before. +""" + +from __future__ import annotations + +from typing import Any, Dict + +import pytest + +from agents.main_agent.session import document_offload as do +from agents.main_agent.session import document_rehydration as rh +from apis.shared.files.document_digest import DigestSection, DocumentDigest, render_digest +from apis.shared.files.models import FileMetadata, FileStatus + +RAW = b"P" * 40_000 + + +def _meta() -> FileMetadata: + digest = DocumentDigest( + status="ready", format="pdf", unit="page", count=10, + abstract="An abstract.", sections=[DigestSection(start=1, title="Intro")], + ) + return FileMetadata( + upload_id="u1", user_id="user-1", filename="Policy.pdf", mime_type="application/pdf", + size_bytes=len(RAW), status=FileStatus.READY, s3_key="k/u1", s3_bucket="b", + session_id="s1", digest=digest.to_item(), + ) + + +class _Repo: + def list_session_files_sync(self, session_id, status=None): + return [_meta()] + + def update_file_digest_sync(self, *a, **k): + pass + + +@pytest.fixture(autouse=True) +def repo(monkeypatch): + monkeypatch.setattr("apis.shared.files.repository.get_file_upload_repository", lambda: _Repo()) + + +def _doc_block() -> Dict[str, Any]: + return {"document": {"format": "pdf", "name": "Policy_pdf", "source": {"bytes": RAW}}} + + +def _conversation() -> list: + """Attach on turn 1, three turns elapsed — unpinned and offloadable.""" + return [ + {"role": "user", "content": [{"text": "here"}, _doc_block()]}, + {"role": "assistant", "content": [{"text": "a"}]}, + {"role": "user", "content": [{"text": "q2"}]}, + {"role": "assistant", "content": [{"text": "b"}]}, + {"role": "user", "content": [{"text": "q3"}]}, + {"role": "assistant", "content": [{"text": "c"}]}, + ] + + +class TestRenderHandle: + def test_default_is_unchanged(self): + d = DocumentDigest.from_item(_meta().digest) + assert 'upload_id="u1"' in render_digest(d, filename="Policy.pdf", upload_id="u1") + + def test_handle_can_be_omitted_without_losing_content(self): + d = DocumentDigest.from_item(_meta().digest) + text = render_digest(d, filename="Policy.pdf", upload_id="u1", include_handle=False) + assert "upload_id" not in text + # ...but the content the model actually reasons from survives. + assert "An abstract." in text and "Intro" in text and 'pages="10"' in text + + +class TestRestorePath: + def test_tool_on_renders_the_handle(self, monkeypatch): + monkeypatch.delenv("DOCUMENT_READ_ENABLED", raising=False) + res = rh.rehydrate_documents([{"role": "user", "content": [_doc_block()]}], + session_id="s1", user_id="user-1") + assert 'upload_id="u1"' in res.messages[0]["content"][0]["text"] + + def test_tool_off_keeps_the_digest_but_drops_the_handle(self, monkeypatch): + monkeypatch.setenv("DOCUMENT_READ_ENABLED", "false") + res = rh.rehydrate_documents([{"role": "user", "content": [_doc_block()]}], + session_id="s1", user_id="user-1") + text = res.messages[0]["content"][0]["text"] + assert res.rehydrated == 1, "a digest still beats the placeholder" + assert "upload_id" not in text, "must not advertise a tool that is not injected" + assert "An abstract." in text + + +class TestLiveOffloadPath: + def test_tool_on_offloads(self, monkeypatch): + monkeypatch.delenv("DOCUMENT_READ_ENABLED", raising=False) + assert do.offload_enabled_for("s1") is True + msgs = _conversation() + cands = do.candidate_documents(msgs, do.pinned_document_names(msgs, None)) + assert do.offload_documents(msgs, cands, session_id="s1", user_id="user-1").offloaded == 1 + + def test_tool_off_is_disabled_for_every_session(self, monkeypatch): + monkeypatch.setenv("DOCUMENT_READ_ENABLED", "false") + assert do.offload_enabled_for("s1") is False + # ...even for a session the rollout bucket would otherwise treat. + monkeypatch.setenv("DOCUMENT_OFFLOAD_ROLLOUT_PERCENT", "100") + assert all(do.offload_enabled_for(f"session-{i}") is False for i in range(50)) + + def test_tool_off_leaves_the_bytes_inline(self, monkeypatch): + """The regression in one assertion: the live path must never evict a + document whose only recovery path is switched off.""" + monkeypatch.setenv("DOCUMENT_READ_ENABLED", "false") + msgs = _conversation() + if do.offload_enabled_for("s1"): # pragma: no cover - the gate above + cands = do.candidate_documents(msgs, do.pinned_document_names(msgs, None)) + do.offload_documents(msgs, cands, session_id="s1", user_id="user-1") + assert msgs[0]["content"][1]["document"]["source"]["bytes"] == RAW diff --git a/backend/tests/agents/main_agent/session/test_document_rehydration.py b/backend/tests/agents/main_agent/session/test_document_rehydration.py new file mode 100644 index 000000000..9f28c92dc --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_document_rehydration.py @@ -0,0 +1,185 @@ +"""Restore rehydrates stripped documents as digests (offload spec §4E, PR-3). + +Pins: block→upload matching (sanitized name incl. PromptBuilder's `_2` +suffix, size tiebreak, no double-claiming), the rendered digest replacing the +bytes with the `upload_id` handle inside, lazy outline-only digests for rows +without one (persisted, no model call), the placeholder fallback for +unmatched blocks, the never-raise contract, the kill switch, and the two +ledger events the session manager records. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +from agents.main_agent.session import document_rehydration as dr +from apis.shared.files.models import FileMetadata, FileStatus +from tests.shared.test_document_read import build_pdf + +MODULE = "agents.main_agent.session.document_rehydration" + + +@pytest.fixture(autouse=True) +def diagnostics_enabled(monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + monkeypatch.delenv("DOCUMENT_REHYDRATE_ENABLED", raising=False) + + +def _meta(upload_id="up-1", filename="BBR Policy.pdf", size=1000, mime="application/pdf", digest=None, user="u1"): + return FileMetadata( + upload_id=upload_id, user_id=user, session_id="s1", filename=filename, mime_type=mime, + size_bytes=size, s3_key="k", s3_bucket="b", status=FileStatus.READY, digest=digest, + ) + + +def _doc(name="BBR Policy_pdf", fmt="pdf", raw=b"x" * 1000): + """PromptBuilder sanitizes the WHOLE filename, dot included: `BBR Policy.pdf` → `BBR Policy_pdf`.""" + return {"document": {"format": fmt, "name": name, "source": {"bytes": raw}}} + + +READY_DIGEST = { + "version": 1, "status": "ready", "format": "pdf", "unit": "page", "count": 47, + "sections": [{"start": 1, "title": "Declarations"}, {"start": 4, "title": "Insuring Agreements"}], + "abstract": "A cyber liability policy form.", "tokens": 80, +} + + +class TestMatching: + def test_prefers_name_and_size_then_name_then_format_and_size(self): + a = _meta("a", "BBR Policy.pdf", 1000) + b = _meta("b", "BBR Policy.pdf", 2000) + c = _meta("c", "Other.pdf", 1000) + assert dr.match_document("BBR Policy_pdf", "pdf", 2000, [a, b, c], set()) is b + assert dr.match_document("BBR Policy_pdf", "pdf", 999, [a, b, c], set()) is a + assert dr.match_document("Renamed", "pdf", 1000, [b, c], set()) is c + assert dr.match_document("Renamed", "docx", 1000, [b, c], set()) is None + + def test_duplicate_suffix_and_claimed_rows(self): + first = _meta("a", "memo.pdf", 10) + second = _meta("b", "memo.pdf", 10) + used: set = set() + assert dr.match_document("memo_pdf", "pdf", 10, [first, second], used) is first + used.add("a") + assert dr.match_document("memo_pdf_2", "pdf", 10, [first, second], used) is second + used.add("b") + assert dr.match_document("memo_pdf_3", "pdf", 10, [first, second], used) is None + + +class TestRehydrate: + def test_replaces_bytes_with_the_rendered_digest_and_handle(self, monkeypatch): + meta = _meta(digest=READY_DIGEST) + monkeypatch.setattr(f"{MODULE}.load_session_documents", lambda *_: [meta]) + messages = [ + {"role": "user", "content": [{"text": "q\n\n[Attached files: BBR Policy.pdf]"}, _doc()]}, + {"role": "assistant", "content": [{"text": "a"}]}, + ] + result = dr.rehydrate_documents(messages, session_id="s1", user_id="u1") + + block = result.messages[0]["content"][1] + assert set(block) == {"text"} + assert block["text"].startswith('') + assert "A cyber liability policy form." in block["text"] + assert '
Insuring Agreements
' in block["text"] + assert (result.rehydrated, result.stripped, result.lazy_digests) == (1, 0, 0) + assert result.digest_tokens > 0 and result.upload_ids == ["up-1"] + # The originals are untouched (deep copy) and no bytes survive. + assert "document" in messages[0]["content"][1] + assert b"x" * 1000 not in repr(result.messages).encode() + + def test_lazy_digest_is_built_from_the_restored_bytes_and_persisted(self, monkeypatch): + meta = _meta(digest=None, size=5) + monkeypatch.setattr(f"{MODULE}.load_session_documents", lambda *_: [meta]) + repo = MagicMock() + monkeypatch.setattr("apis.shared.files.repository.get_file_upload_repository", lambda: repo) + raw = build_pdf(["1. Intro\nbody", "2. Terms"]) + messages = [{"role": "user", "content": [_doc(raw=raw)]}] + + result = dr.rehydrate_documents(messages, session_id="s1", user_id="u1") + + text = result.messages[0]["content"][0]["text"] + assert 'upload_id="up-1"' in text and '
2. Terms
' in text + assert "" not in text # outline only — no model call on the restore path + assert result.lazy_digests == 1 + repo.update_file_digest_sync.assert_called_once() + user, upload, item = repo.update_file_digest_sync.call_args.args + assert (user, upload, item["status"], item["count"]) == ("u1", "up-1", "ready", 2) + assert item["tokens"] > 0 + + def test_lazy_digest_persist_failure_still_serves_this_restore(self, monkeypatch): + meta = _meta(digest=None) + monkeypatch.setattr(f"{MODULE}.load_session_documents", lambda *_: [meta]) + repo = MagicMock() + repo.update_file_digest_sync.side_effect = RuntimeError("ddb") + monkeypatch.setattr("apis.shared.files.repository.get_file_upload_repository", lambda: repo) + result = dr.rehydrate_documents([{"role": "user", "content": [_doc(raw=build_pdf(["Hello"]))]}], session_id="s1", user_id="u1") + assert result.rehydrated == 1 and " 0 + assert events[1] == {"kind": "document_stripped", "documents": 1, "documentTokens": 100} + + def test_restore_output_is_stable_across_restores(self, make_session_manager, monkeypatch): + rows = [_meta("up-1", "BBR Policy.pdf", 1000, digest=READY_DIGEST)] + monkeypatch.setattr(f"{MODULE}.load_session_documents", lambda *_: rows) + manager = make_session_manager() + messages = [{"role": "user", "content": [_doc()]}] + first = manager._strip_document_bytes(messages) + second = manager._strip_document_bytes(messages) + assert first == second diff --git a/backend/tests/agents/main_agent/session/test_memory_retrieval_bounded.py b/backend/tests/agents/main_agent/session/test_memory_retrieval_bounded.py new file mode 100644 index 000000000..fcf38cce5 --- /dev/null +++ b/backend/tests/agents/main_agent/session/test_memory_retrieval_bounded.py @@ -0,0 +1,199 @@ +"""Bounded long-term memory retrieval (`TurnBasedSessionManager.retrieve_customer_context`). + +The SDK retrieves long-term memory on every user message, awaited before the +model call, through its shared read/write boto client with default retries — +so a throttled RetrieveMemoryRecords put retry backoff on first-token latency. +The override keeps the SDK's contract (namespaces from `retrieval_config`, +relevance filter, `` block prepended to the last user message) +and swaps in a dedicated one-attempt, short-timeout client. +""" + +from types import SimpleNamespace +from unittest.mock import patch + +import pytest +from botocore.exceptions import ClientError + +from agents.main_agent.session import turn_based_session_manager as tbsm + +PREFS = "/strategies/pref-1/actors/{actorId}" +FACTS = "/strategies/sem-1/actors/{actorId}" + + +class FakeRetrievalClient: + def __init__(self, results=None, raise_with=None): + self.results = results or {} + self.raise_with = raise_with + self.calls = [] + + def retrieve_memory_records(self, **kwargs): + self.calls.append(kwargs) + if self.raise_with is not None: + raise self.raise_with + return {"memoryRecordSummaries": self.results.get(kwargs["namespacePath"], [])} + + +def _throttle(): + return ClientError({"Error": {"Code": "ThrottlingException", "Message": "slow down"}}, "RetrieveMemoryRecords") + + +def _record(text, score=0.9): + return {"content": {"text": text}, "score": score} + + +def _event(text="what is my name?"): + agent = SimpleNamespace(messages=[ + {"role": "user", "content": [{"text": "earlier"}]}, + {"role": "assistant", "content": [{"text": "ok"}]}, + {"role": "user", "content": [{"text": text}]}, + ]) + return SimpleNamespace(agent=agent) + + +def _manager(make_session_manager, client, namespaces=(PREFS, FACTS), relevance=0.7): + mgr = make_session_manager() + mgr.config.retrieval_config = { + ns: SimpleNamespace(top_k=5, relevance_score=relevance, strategy_id=None) for ns in namespaces + } + mgr.config.context_tag = "user_context" + mgr._retrieval_client = client + return mgr + + +class TestRetrieval: + def test_prepends_context_from_every_namespace_and_resolves_templates(self, make_session_manager): + client = FakeRetrievalClient(results={ + "/strategies/pref-1/actors/test-actor": [_record("likes short answers")], + "/strategies/sem-1/actors/test-actor": [_record("name is Ada")], + }) + mgr = _manager(make_session_manager, client) + mgr.config.actor_id = "test-actor" + event = _event() + + mgr.retrieve_customer_context(event) + + assert len(client.calls) == 2 + assert {c["namespacePath"] for c in client.calls} == { + "/strategies/pref-1/actors/test-actor", "/strategies/sem-1/actors/test-actor", + } + assert all(c["searchCriteria"] == {"searchQuery": "what is my name?", "topK": 5} for c in client.calls) + assert all(c["memoryId"] == mgr.config.memory_id for c in client.calls) + first_block = event.agent.messages[-1]["content"][0]["text"] + assert first_block.startswith("") and first_block.endswith("") + assert "likes short answers" in first_block and "name is Ada" in first_block + # The user's own words stay last. + assert event.agent.messages[-1]["content"][-1] == {"text": "what is my name?"} + + def test_relevance_filter_applies(self, make_session_manager): + client = FakeRetrievalClient(results={ + "/strategies/pref-1/actors/test-actor": [_record("weak", score=0.2), _record("strong", score=0.95)], + }) + mgr = _manager(make_session_manager, client, namespaces=(PREFS,)) + mgr.config.actor_id = "test-actor" + event = _event() + + mgr.retrieve_customer_context(event) + + block = event.agent.messages[-1]["content"][0]["text"] + assert "strong" in block and "weak" not in block + + def test_nothing_retrieved_means_message_untouched(self, make_session_manager): + mgr = _manager(make_session_manager, FakeRetrievalClient()) + event = _event() + before = [dict(b) for b in event.agent.messages[-1]["content"]] + + mgr.retrieve_customer_context(event) + + assert event.agent.messages[-1]["content"] == before + + def test_skips_when_last_message_is_not_a_text_user_message(self, make_session_manager): + client = FakeRetrievalClient() + mgr = _manager(make_session_manager, client) + assistant_last = SimpleNamespace(agent=SimpleNamespace(messages=[{"role": "assistant", "content": [{"text": "x"}]}])) + tool_result = SimpleNamespace(agent=SimpleNamespace(messages=[{"role": "user", "content": [{"toolResult": {"toolUseId": "t", "content": []}}]}])) + + mgr.retrieve_customer_context(assistant_last) + mgr.retrieve_customer_context(tool_result) + mgr.retrieve_customer_context(SimpleNamespace(agent=SimpleNamespace(messages=[]))) + + assert client.calls == [] + + def test_no_retrieval_config_means_no_client(self, make_session_manager): + mgr = make_session_manager() + mgr.config.retrieval_config = {} + with patch.object(tbsm.TurnBasedSessionManager, "_get_retrieval_client") as get_client: + mgr.retrieve_customer_context(_event()) + get_client.assert_not_called() + + +class TestBounded: + def test_throttle_costs_one_attempt_and_the_turn_proceeds_without_context(self, make_session_manager): + client = FakeRetrievalClient(raise_with=_throttle()) + mgr = _manager(make_session_manager, client) + event = _event() + + mgr.retrieve_customer_context(event) # must not raise + + assert len(client.calls) == 2 # one per namespace, no retries at this layer + assert event.agent.messages[-1]["content"] == [{"text": "what is my name?"}] + + def test_one_failing_namespace_does_not_sink_the_other(self, make_session_manager): + class HalfBroken(FakeRetrievalClient): + def retrieve_memory_records(self, **kwargs): + self.calls.append(kwargs) + if "pref-1" in kwargs["namespacePath"]: + raise RuntimeError("boom") + return {"memoryRecordSummaries": [_record("name is Ada")]} + + mgr = _manager(make_session_manager, HalfBroken()) + event = _event() + + mgr.retrieve_customer_context(event) + + assert "name is Ada" in event.agent.messages[-1]["content"][0]["text"] + + def test_dedicated_client_is_lazy_and_bounded(self, make_session_manager, monkeypatch): + monkeypatch.delenv(tbsm.MEMORY_RETRIEVAL_TIMEOUT_ENV, raising=False) + monkeypatch.delenv(tbsm.MEMORY_RETRIEVAL_MAX_ATTEMPTS_ENV, raising=False) + mgr = make_session_manager() + assert getattr(mgr, "_retrieval_client", None) is None + + with patch("boto3.client") as boto_client: + boto_client.return_value = object() + first = mgr._get_retrieval_client() + second = mgr._get_retrieval_client() + + assert first is second + boto_client.assert_called_once() + assert boto_client.call_args.args == ("bedrock-agentcore",) + cfg = boto_client.call_args.kwargs["config"] + assert cfg.retries == {"total_max_attempts": 1, "mode": "standard"} + assert cfg.read_timeout == 2.0 and cfg.connect_timeout == 2.0 + assert boto_client.call_args.kwargs["region_name"] == mgr.region_name + + def test_env_overrides_and_garbage(self, monkeypatch): + monkeypatch.setenv(tbsm.MEMORY_RETRIEVAL_TIMEOUT_ENV, "0.75") + monkeypatch.setenv(tbsm.MEMORY_RETRIEVAL_MAX_ATTEMPTS_ENV, "2") + assert tbsm.memory_retrieval_timeout_seconds() == 0.75 + assert tbsm.memory_retrieval_max_attempts() == 2 + + monkeypatch.setenv(tbsm.MEMORY_RETRIEVAL_TIMEOUT_ENV, "later") + monkeypatch.setenv(tbsm.MEMORY_RETRIEVAL_MAX_ATTEMPTS_ENV, "0") + assert tbsm.memory_retrieval_timeout_seconds() == 2.0 + assert tbsm.memory_retrieval_max_attempts() == 1 + + +class TestSdkWiring: + def test_sdk_register_hooks_dispatches_to_our_override(self): + """The SDK registers `self.retrieve_customer_context` (both modes), so + the override is what runs. Guards against an SDK bump that starts + registering a private method instead.""" + import inspect + + from bedrock_agentcore.memory.integrations.strands.session_manager import ( + AgentCoreMemorySessionManager, + ) + + src = inspect.getsource(AgentCoreMemorySessionManager.register_hooks) + assert src.count("self.retrieve_customer_context") >= 2 + assert tbsm.TurnBasedSessionManager.retrieve_customer_context is not AgentCoreMemorySessionManager.retrieve_customer_context diff --git a/backend/tests/agents/main_agent/session/test_session_factory.py b/backend/tests/agents/main_agent/session/test_session_factory.py index c5aec739b..7efa65977 100644 --- a/backend/tests/agents/main_agent/session/test_session_factory.py +++ b/backend/tests/agents/main_agent/session/test_session_factory.py @@ -204,7 +204,9 @@ def test_uses_default_thresholds( aws_region="us-west-2", caching_enabled=True, ) - assert mock_retrieval.call_count == 3 + # preferences + facts; the session's own summary namespace is not + # retrieved per message unless explicitly re-enabled. + assert mock_retrieval.call_count == 2 for c in mock_retrieval.call_args_list: assert c == call(top_k=10, relevance_score=0.7) @@ -229,6 +231,46 @@ def test_reads_custom_thresholds_from_env( aws_region="us-west-2", caching_enabled=True, ) - assert mock_retrieval.call_count == 3 + assert mock_retrieval.call_count == 2 for c in mock_retrieval.call_args_list: assert c == call(top_k=20, relevance_score=0.85) + + @patch("agents.main_agent.session.session_factory.AGENTCORE_MEMORY_AVAILABLE", True) + @patch("agents.main_agent.session.session_factory._discover_strategy_ids") + @patch("agents.main_agent.session.session_factory.AgentCoreMemoryConfig") + @patch("agents.main_agent.session.session_factory.RetrievalConfig") + @patch("agents.main_agent.session.turn_based_session_manager.TurnBasedSessionManager", create=True) + def test_summary_namespace_is_off_by_default_and_opt_in( + self, mock_tbsm, mock_retrieval, mock_mem_config, mock_discover, monkeypatch + ): + """The current session's summary is a second copy of the conversation + the model already holds, at a RetrieveMemoryRecords call per message + against a 30/s quota — off unless explicitly re-enabled.""" + from agents.main_agent.session.session_factory import ( + MEMORY_SUMMARY_RETRIEVAL_ENV, + SessionFactory, + ) + + mock_discover.return_value = ("semantic-1", "pref-1", "sum-1") + mock_tbsm.return_value = MagicMock() + + def namespaces(): + return list(mock_mem_config.call_args.kwargs["retrieval_config"].keys()) + + monkeypatch.delenv(MEMORY_SUMMARY_RETRIEVAL_ENV, raising=False) + SessionFactory._create_cloud_session_manager( + memory_id="mem-1", session_id="s-1", user_id="u-1", + aws_region="us-west-2", caching_enabled=True, + ) + assert namespaces() == [ + "/strategies/pref-1/actors/{actorId}", + "/strategies/semantic-1/actors/{actorId}", + ] + + monkeypatch.setenv(MEMORY_SUMMARY_RETRIEVAL_ENV, "true") + SessionFactory._create_cloud_session_manager( + memory_id="mem-1", session_id="s-1", user_id="u-1", + aws_region="us-west-2", caching_enabled=True, + ) + assert namespaces()[-1] == "/strategies/sum-1/actors/{actorId}/sessions/{sessionId}" + assert len(namespaces()) == 3 diff --git a/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py b/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py new file mode 100644 index 000000000..dc19fabe2 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_agent_status_live_drain.py @@ -0,0 +1,238 @@ +"""The concurrent `agent_status` drain (docs/specs/agent-state-feedback.md PR-2). + +The status hook records model-call and tool-call boundaries from inside +Strands' event loop, which has no route to the SSE stream — so the coordinator +drains it. Draining only from the emit loop meant a transition could not leave +the container until the agent stream produced its next event, and during tool +execution the agent stream produces NOTHING. A `tool_start` therefore waited +out exactly the silence it existed to explain. + +That bug is invisible to an ordering assertion: the old drain ran just before +each event was yielded, so the frames came out in the right ORDER and hours +late. What these tests pin is TIMING — did the status frame reach the consumer +while the agent stream was still silent, or only once it spoke again — and they +do it without wall-clock thresholds, by asking the fake stream whether it had +yielded yet at the moment each frame arrived. +""" + +import asyncio +import json +from typing import Any, AsyncIterator, Dict, List, Optional + +import pytest + +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + + +class _Hook: + """The seams the coordinator uses, plus a way to record mid-silence.""" + + def __init__(self) -> None: + self._statuses: List[dict] = [] + self.drains = 0 + + def record(self, status: dict) -> None: + self._statuses.append(status) + + def drain_statuses(self) -> List[dict]: + self.drains += 1 + statuses, self._statuses = self._statuses, [] + return statuses + + def drain_batches(self) -> List[dict]: + return [] + + +class _Wrapper: + def __init__(self, hook: Optional[_Hook]) -> None: + if hook is not None: + self.agent_status_hook = hook + + +class _SessionManager: + def __init__(self) -> None: + self.cancelled = False + + async def update_after_turn(self, input_tokens, current_messages=None): + return None + + +class _StallingAgent: + """An agent stream that goes quiet exactly the way a tool call does. + + It yields one event, then records a `tool_start` and stalls — no events at + all while the "tool runs" — then records `tool_end` and speaks again. + `spoke_again` flips at the moment of that second yield, which is what lets + the tests tell "arrived during the silence" from "arrived after it". + """ + + def __init__(self, hook: _Hook, silence: float = 0.4) -> None: + self.messages = [{"role": "user", "content": [{"text": "hi"}]}] + self._hook = hook + self._silence = silence + self.spoke_again = False + + def stream_async(self, prompt: Any) -> AsyncIterator[Dict[str, Any]]: + async def _gen() -> AsyncIterator[Dict[str, Any]]: + yield {"event": {"messageStart": {"role": "assistant"}}} + + self._hook.record( + {"phase": "tool_start", "cycle": 1, "toolName": "list_assignments"} + ) + await asyncio.sleep(self._silence) + self._hook.record( + {"phase": "tool_end", "cycle": 1, "toolName": "list_assignments"} + ) + + self.spoke_again = True + yield {"event": {"messageStop": {"stopReason": "end_turn"}}} + + return _gen() + + +async def _collect_with_arrival(agent, wrapper) -> List[tuple]: + """Every frame, paired with whether the stream had spoken again yet.""" + coordinator = StreamCoordinator() + seen: List[tuple] = [] + async for sse in coordinator.stream_response( + agent=agent, + prompt="hi", + session_manager=_SessionManager(), + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=wrapper, + ): + seen.append((sse, agent.spoke_again)) + return seen + + +def _status_payloads(frames) -> List[dict]: + prefix = "event: agent_status\ndata: " + return [ + json.loads(f[len(prefix) :].strip()) + for f, _ in frames + if f.startswith(prefix) + ] + + +def _arrival_of_phase(frames, phase: str) -> bool: + """`spoke_again` at the moment the frame for `phase` arrived.""" + prefix = "event: agent_status\ndata: " + for sse, spoke_again in frames: + if not sse.startswith(prefix): + continue + if json.loads(sse[len(prefix) :].strip()).get("phase") == phase: + return spoke_again + raise AssertionError(f"no agent_status frame for phase {phase!r}") + + +class TestLiveDrain: + @pytest.mark.asyncio + async def test_tool_start_reaches_the_client_during_the_silence(self, monkeypatch): + """The whole point: narrate the tool WHILE it runs.""" + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() + agent = _StallingAgent(hook) + + frames = await _collect_with_arrival(agent, _Wrapper(hook)) + + assert _arrival_of_phase(frames, "tool_start") is False + + @pytest.mark.asyncio + async def test_the_kill_switch_restores_the_between_yields_drain(self, monkeypatch): + """Off, the frame waits for the stream to speak — the old behaviour.""" + monkeypatch.setenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", "false") + hook = _Hook() + agent = _StallingAgent(hook) + + frames = await _collect_with_arrival(agent, _Wrapper(hook)) + + # Still delivered, and still in order — just late, which is the bug. + assert _arrival_of_phase(frames, "tool_start") is True + phases = [p["phase"] for p in _status_payloads(frames)] + assert phases == ["tool_start", "tool_end"] + + @pytest.mark.asyncio + async def test_every_transition_is_delivered_exactly_once(self, monkeypatch): + """A status line that stutters backwards is worse than none.""" + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() + + frames = await _collect_with_arrival(_StallingAgent(hook), _Wrapper(hook)) + + phases = [p["phase"] for p in _status_payloads(frames)] + assert phases == ["tool_start", "tool_end"] + + @pytest.mark.asyncio + async def test_sessionid_is_stamped_on_every_frame(self, monkeypatch): + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() + + frames = await _collect_with_arrival(_StallingAgent(hook), _Wrapper(hook)) + + assert all(p["sessionId"] == "sess-1" for p in _status_payloads(frames)) + assert _status_payloads(frames) # not vacuous + + +class TestFailSoft: + @pytest.mark.asyncio + async def test_a_wrapper_without_the_hook_streams_normally(self, monkeypatch): + """Voice and tests build wrappers with no status hook.""" + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() # recorded into, but never reachable by the coordinator + agent = _StallingAgent(hook, silence=0.05) + + frames = await _collect_with_arrival(agent, _Wrapper(None)) + + assert _status_payloads(frames) == [] + assert any(f.startswith("event: done") for f, _ in frames) + + @pytest.mark.asyncio + async def test_a_failing_drain_does_not_break_the_turn(self, monkeypatch): + """Narration is a nicety; it must never be able to end a turn.""" + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + + class _BrokenHook(_Hook): + def drain_statuses(self): + raise RuntimeError("hook exploded") + + hook = _BrokenHook() + agent = _StallingAgent(hook, silence=0.05) + + frames = await _collect_with_arrival(agent, _Wrapper(hook)) + + assert _status_payloads(frames) == [] + assert any(f.startswith("event: done") for f, _ in frames) + + +class TestEarlyExit: + @pytest.mark.asyncio + async def test_a_cooperative_stop_mid_silence_ends_cleanly(self, monkeypatch): + """Stop abandons the merge, which must unwind the agent stream. + + The merge holds an in-flight `__anext__` across its poll timeouts, so + this is the path where that task gets cancelled. A generator left + half-closed here would surface as a hung turn or a leaked lease. + """ + monkeypatch.delenv("AGENT_STATUS_LIVE_DRAIN_ENABLED", raising=False) + hook = _Hook() + agent = _StallingAgent(hook, silence=0.3) + session_manager = _SessionManager() + + coordinator = StreamCoordinator() + frames: List[str] = [] + async for sse in coordinator.stream_response( + agent=agent, + prompt="hi", + session_manager=session_manager, + session_id="sess-1", + user_id="user-1", + main_agent_wrapper=_Wrapper(hook), + ): + frames.append(sse) + # Arm the stop as soon as the first status frame lands — i.e. while + # the agent stream is still stalled inside its sleep. + if sse.startswith("event: agent_status"): + session_manager.cancelled = True + + assert any(f.startswith("event: done") for f in frames) diff --git a/backend/tests/agents/main_agent/streaming/test_browser_login_events.py b/backend/tests/agents/main_agent/streaming/test_browser_login_events.py new file mode 100644 index 000000000..4da4bbba5 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_browser_login_events.py @@ -0,0 +1,215 @@ +"""`browser_login_required` extraction from a paused turn. + +Same shape as `test_user_question_events.py` — the extractor reads +`agent._interrupt_state` at the `done` event — so these concentrate on what is +specific to this flavor: + +* the conversation id is stamped on here, not in the tool, and must not be + confused with the browser session id; +* the D4 projection is written alongside the breadcrumb, because app-api reads + the metadata row and cannot read `agent.state`; +* neither the event nor the projection may carry a live-view URL; +* persistence is best-effort — a DynamoDB failure costs the viewer, never the + prompt. +""" + +from __future__ import annotations + +import json +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import pytest + +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator +from apis.shared.browser_takeover import assert_no_url + + +def _parse(sse: str) -> dict: + data_line = next(line for line in sse.splitlines() if line.startswith("data: ")) + return json.loads(data_line[len("data: ") :]) + + +def _coordinator() -> StreamCoordinator: + return StreamCoordinator.__new__(StreamCoordinator) + + +def _agent(*interrupts, activated: bool = True): + return SimpleNamespace( + _interrupt_state=SimpleNamespace( + activated=activated, + interrupts={f"i-{n}": i for n, i in enumerate(interrupts)}, + ) + ) + + +def _interrupt(reason, interrupt_id: str = "v1:tool_call:tu-1:abc"): + return SimpleNamespace(id=interrupt_id, reason=reason) + + +def _reason(**overrides): + payload = { + "type": "browser_login_required", + "toolUseId": "tu-1", + "reason": "Sign in to JSTOR so I can check the results page.", + "browserSessionId": "bs-1", + "browserId": "browser-abc", + "viewport": {"width": 1280, "height": 800}, + "controlState": "user", + "deadlineAt": "2026-09-18T12:08:00+00:00", + "targetUrl": "https://www.jstor.org/action/showLogin", + } + payload.update(overrides) + return payload + + +@pytest.fixture +def persistence(): + """Patch both writers at their definition sites.""" + with patch( + "apis.shared.sessions.metadata.add_pending_interrupt", + new_callable=AsyncMock, + ) as add_pending, patch( + "apis.shared.sessions.metadata.set_browser_session", + new_callable=AsyncMock, + ) as set_browser: + yield SimpleNamespace(add_pending=add_pending, set_browser=set_browser) + + +class TestEventEmission: + @pytest.mark.asyncio + async def test_emits_one_event_describing_the_handover(self, persistence): + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id="s1", user_id="u1" + ) + + assert len(events) == 1 + assert events[0].startswith("event: browser_login_required\n") + payload = _parse(events[0]) + assert payload["interruptId"] == "v1:tool_call:tu-1:abc" + assert payload["toolUseId"] == "tu-1" + assert payload["browserSessionId"] == "bs-1" + assert payload["viewport"] == {"width": 1280, "height": 800} + assert payload["reason"].startswith("Sign in to JSTOR") + + @pytest.mark.asyncio + async def test_session_id_is_the_conversation_not_the_browser(self, persistence): + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id="conv-42", user_id="u1" + ) + + payload = _parse(events[0]) + # app-api's live-view route is scoped by conversation owner and + # resolves the browser session server-side, so these must stay distinct. + assert payload["sessionId"] == "conv-42" + assert payload["browserSessionId"] == "bs-1" + + @pytest.mark.asyncio + async def test_no_live_view_url_reaches_the_client(self, persistence): + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id="s1", user_id="u1" + ) + + assert_no_url(_parse(events[0])) # must not raise + assert "liveViewUrl" not in _parse(events[0]) + + @pytest.mark.asyncio + async def test_the_viewport_rides_the_event_so_dcv_does_not_crop( + self, persistence + ): + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason(viewport={"width": 1920, "height": 1080}))), + session_id="s1", + user_id="u1", + ) + + assert _parse(events[0])["viewport"] == {"width": 1920, "height": 1080} + + @pytest.mark.asyncio + async def test_other_interrupt_flavors_are_ignored(self, persistence): + events = await _coordinator()._extract_browser_login_required_events( + _agent( + _interrupt({"type": "user_question_required", "questions": []}), + _interrupt({"type": "oauth_required", "providerId": "p1"}), + ), + session_id="s1", + user_id="u1", + ) + + assert events == [] + + @pytest.mark.asyncio + async def test_nothing_emitted_when_the_turn_did_not_pause(self, persistence): + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason()), activated=False), + session_id="s1", + user_id="u1", + ) + + assert events == [] + + @pytest.mark.asyncio + async def test_an_undescribable_session_is_dropped_not_emitted(self, persistence): + # An event the SPA cannot act on is worse than none: it would show a + # viewer affordance for a browser nothing can resolve. + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason(browserSessionId=None, viewport=None))), + session_id="s1", + user_id="u1", + ) + + assert events == [] + persistence.set_browser.assert_not_called() + + +class TestPersistence: + @pytest.mark.asyncio + async def test_projects_the_browser_session_onto_the_metadata_row( + self, persistence + ): + await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id="s1", user_id="u1" + ) + + persistence.set_browser.assert_awaited_once() + kwargs = persistence.set_browser.await_args.kwargs + assert kwargs["session_id"] == "s1" + assert kwargs["user_id"] == "u1" + assert kwargs["ref"]["browserSessionId"] == "bs-1" + assert_no_url(kwargs["ref"]) + + @pytest.mark.asyncio + async def test_writes_a_breadcrumb_so_the_prompt_survives_a_refresh( + self, persistence + ): + await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id="s1", user_id="u1" + ) + + interrupt = persistence.add_pending.await_args.kwargs["interrupt"] + assert interrupt.kind == "browser_login" + assert interrupt.tool_name == "request_user_login" + assert json.loads(interrupt.browser_session)["browserSessionId"] == "bs-1" + assert interrupt.reason.startswith("Sign in to JSTOR") + + @pytest.mark.asyncio + async def test_a_dynamo_failure_still_emits_the_prompt(self, persistence): + # Losing the breadcrumb costs a refresh; losing the event costs the turn. + persistence.add_pending.side_effect = RuntimeError("throttled") + persistence.set_browser.side_effect = RuntimeError("throttled") + + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id="s1", user_id="u1" + ) + + assert len(events) == 1 + + @pytest.mark.asyncio + async def test_anonymous_flows_emit_without_persisting(self, persistence): + events = await _coordinator()._extract_browser_login_required_events( + _agent(_interrupt(_reason())), session_id=None, user_id=None + ) + + assert len(events) == 1 + persistence.set_browser.assert_not_called() + persistence.add_pending.assert_not_called() diff --git a/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py b/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py index 32223b618..8eb0f28e7 100644 --- a/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py +++ b/backend/tests/agents/main_agent/streaming/test_compaction_sse_emit_once.py @@ -95,7 +95,11 @@ async def update_after_turn( self, input_tokens: int, current_messages: Optional[List[Dict]] = None, + context_window: Optional[int] = None, + history_tokens: Optional[int] = None, ) -> Optional[CompactionResult]: + # Same seam as TurnBasedSessionManager.update_after_turn — the policy + # inputs (context_window / history_tokens) are accepted and ignored. self.calls.append(input_tokens) return self._result @@ -158,6 +162,14 @@ async def test_compaction_sse_emitted_exactly_once_when_checkpoint_advances(): "newCheckpoint": 4, "summarizedTurns": 2, "inputTokens": 150_000, + # Model-relative policy fields (additive; the stub result leaves + # them at their defaults). + "contextWindow": None, + "ceiling": None, + "floor": None, + "hardCeiling": None, + "forced": False, + "retainedTokensEstimate": None, } diff --git a/backend/tests/agents/main_agent/streaming/test_context_ledger_attach.py b/backend/tests/agents/main_agent/streaming/test_context_ledger_attach.py new file mode 100644 index 000000000..9b98a00ad --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_context_ledger_attach.py @@ -0,0 +1,95 @@ +"""The stream coordinator persists the context ledger and the prefix split on +the call's cost row, as extra fields next to `toolCalls`.""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +from agents.main_agent.session.hooks.context_attribution import _SPLIT_ATTR +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + + +def _coordinator() -> StreamCoordinator: + return object.__new__(StreamCoordinator) + + +def _usage_metadata(): + return {"usage": {"inputTokens": 100, "outputTokens": 20, "totalTokens": 120}} + + +def _wrapper(strands_agent): + # A plain namespace, not a MagicMock: the coordinator reads + # `model_config.model_id` into a pydantic model and a mock is not a str. + return SimpleNamespace( + agent=strands_agent, + model_config=SimpleNamespace(model_id="claude-haiku-4-5", model_name="Claude Haiku 4.5"), + ) + + +def _wrapper_with_split(system=12_000, tools=48_000): + strands_agent = SimpleNamespace() + setattr(strands_agent, _SPLIT_ATTR, {"systemTokens": system, "toolTokens": tools}) + return _wrapper(strands_agent) + + +async def _store(monkeypatch, **kwargs): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + store = AsyncMock() + with patch("apis.shared.sessions.metadata.store_message_metadata", store), \ + patch("agents.main_agent.streaming.stream_coordinator.get_prefix_fingerprint", return_value=None): + await _coordinator()._store_message_metadata( + session_id="s1", user_id="u1", message_id=3, + accumulated_metadata=_usage_metadata(), + stream_start_time=0.0, stream_end_time=1.0, first_token_time=0.5, + call_index=0, **kwargs, + ) + store.assert_awaited_once() + return store.await_args.kwargs["message_metadata"] + + +@pytest.mark.asyncio +async def test_ledger_and_prefix_split_are_attached(monkeypatch): + stored = await _store( + monkeypatch, + agent=_wrapper_with_split(), + context_ledger={ + "windowRemovedMessages": 6, + "compactionEvents": [{"kind": "applied", "summaryTokens": 1200}], + }, + ) + extra = stored.model_extra + assert extra["windowRemovedMessages"] == 6 + assert extra["compactionEvents"] == [{"kind": "applied", "summaryTokens": 1200}] + assert extra["prefixTokens"] == {"system": 12_000, "tools": 48_000} + dumped = stored.model_dump(by_alias=True) + assert dumped["prefixTokens"]["tools"] == 48_000 + + +@pytest.mark.asyncio +async def test_absent_ledger_and_split_leave_no_fields(monkeypatch): + stored = await _store(monkeypatch, agent=_wrapper(SimpleNamespace()), context_ledger=None) + for key in ("windowRemovedMessages", "compactionEvents", "prefixTokens"): + assert key not in (stored.model_extra or {}) + + +@pytest.mark.asyncio +async def test_kill_switch_drops_the_prefix_split_too(monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + store = AsyncMock() + with patch("apis.shared.sessions.metadata.store_message_metadata", store), \ + patch("agents.main_agent.streaming.stream_coordinator.get_prefix_fingerprint", return_value=None): + await _coordinator()._store_message_metadata( + session_id="s1", user_id="u1", message_id=3, + accumulated_metadata=_usage_metadata(), + stream_start_time=0.0, stream_end_time=1.0, first_token_time=0.5, + agent=_wrapper_with_split(), call_index=0, context_ledger=None, + ) + stored = store.await_args.kwargs["message_metadata"] + assert "prefixTokens" not in (stored.model_extra or {}) + + +@pytest.mark.asyncio +async def test_the_argument_is_optional_for_the_interrupt_path(monkeypatch): + stored = await _store(monkeypatch, agent=None) + assert "windowRemovedMessages" not in (stored.model_extra or {}) diff --git a/backend/tests/agents/main_agent/streaming/test_event_formatter.py b/backend/tests/agents/main_agent/streaming/test_event_formatter.py deleted file mode 100644 index b7823f2b0..000000000 --- a/backend/tests/agents/main_agent/streaming/test_event_formatter.py +++ /dev/null @@ -1,232 +0,0 @@ -""" -Tests for StreamEventFormatter — SSE event formatting and final-result extraction. - -Validates: Requirements 12.1–12.8 -""" - -import json -from types import SimpleNamespace -from unittest.mock import MagicMock - -import pytest - -from agents.main_agent.streaming.event_formatter import StreamEventFormatter - - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -def _parse_sse(sse_string: str) -> dict: - """Strip the SSE envelope and return the parsed JSON payload.""" - assert sse_string.startswith("data: ") - assert sse_string.endswith("\n\n") - return json.loads(sse_string[len("data: "):-2]) - - -# --------------------------------------------------------------------------- -# 12.1 format_sse_event produces "data: {json}\n\n" -# --------------------------------------------------------------------------- - -class TestFormatSseEvent: - """Validates: Requirement 12.1, 12.2""" - - def test_basic_format(self): - """format_sse_event wraps a dict in the SSE wire format.""" - result = StreamEventFormatter.format_sse_event({"type": "ping"}) - assert result.startswith("data: ") - assert result.endswith("\n\n") - payload = json.loads(result[len("data: "):-2]) - assert payload == {"type": "ping"} - - def test_complex_payload(self): - """Nested dicts and lists serialise correctly.""" - data = {"type": "response", "items": [1, 2, 3], "nested": {"a": True}} - payload = _parse_sse(StreamEventFormatter.format_sse_event(data)) - assert payload == data - - # 12.2 Non-serializable objects → error event (no exception) - def test_non_serializable_returns_error_event(self): - """Non-serializable values produce an error event instead of raising.""" - bad_data = {"obj": object()} - result = StreamEventFormatter.format_sse_event(bad_data) - payload = _parse_sse(result) - assert payload["type"] == "error" - assert "Serialization error" in payload["message"] - - -# --------------------------------------------------------------------------- -# 12.3 create_init_event -# --------------------------------------------------------------------------- - -class TestCreateInitEvent: - """Validates: Requirement 12.3""" - - def test_init_event_type(self): - payload = _parse_sse(StreamEventFormatter.create_init_event()) - assert payload["type"] == "init" - - def test_init_event_has_message(self): - payload = _parse_sse(StreamEventFormatter.create_init_event()) - assert "message" in payload - - -# --------------------------------------------------------------------------- -# 12.4 create_response_event -# --------------------------------------------------------------------------- - -class TestCreateResponseEvent: - """Validates: Requirement 12.4""" - - def test_response_event_type_and_text(self): - payload = _parse_sse(StreamEventFormatter.create_response_event("hello")) - assert payload["type"] == "response" - assert payload["text"] == "hello" - - def test_response_event_empty_text(self): - payload = _parse_sse(StreamEventFormatter.create_response_event("")) - assert payload["type"] == "response" - assert payload["text"] == "" - - -# --------------------------------------------------------------------------- -# 12.5 create_tool_use_event -# --------------------------------------------------------------------------- - -class TestCreateToolUseEvent: - """Validates: Requirement 12.5""" - - def test_tool_use_event_fields(self): - tool_use = { - "toolUseId": "tu-123", - "name": "calculator", - "input": {"expression": "2+2"}, - } - payload = _parse_sse(StreamEventFormatter.create_tool_use_event(tool_use)) - assert payload["type"] == "tool_use" - assert payload["toolUseId"] == "tu-123" - assert payload["name"] == "calculator" - assert payload["input"] == {"expression": "2+2"} - - def test_tool_use_event_missing_input_defaults_empty(self): - tool_use = {"toolUseId": "tu-456", "name": "search"} - payload = _parse_sse(StreamEventFormatter.create_tool_use_event(tool_use)) - assert payload["input"] == {} - - -# --------------------------------------------------------------------------- -# 12.6 create_complete_event -# --------------------------------------------------------------------------- - -class TestCreateCompleteEvent: - """Validates: Requirement 12.6""" - - def test_complete_event_basic(self): - payload = _parse_sse(StreamEventFormatter.create_complete_event("Done")) - assert payload["type"] == "complete" - assert payload["message"] == "Done" - - def test_complete_event_with_images(self): - images = [{"format": "png", "data": "abc123"}] - payload = _parse_sse( - StreamEventFormatter.create_complete_event("Done", images=images) - ) - assert payload["images"] == images - - def test_complete_event_with_usage(self): - usage = {"inputTokens": 100, "outputTokens": 50} - payload = _parse_sse( - StreamEventFormatter.create_complete_event("Done", usage=usage) - ) - assert payload["usage"] == usage - - def test_complete_event_no_optional_fields(self): - payload = _parse_sse(StreamEventFormatter.create_complete_event("Done")) - assert "images" not in payload - assert "usage" not in payload - - -# --------------------------------------------------------------------------- -# 12.7 create_error_event -# --------------------------------------------------------------------------- - -class TestCreateErrorEvent: - """Validates: Requirement 12.7""" - - def test_error_event_type_and_message(self): - payload = _parse_sse( - StreamEventFormatter.create_error_event("something broke") - ) - assert payload["type"] == "error" - assert payload["message"] == "something broke" - - -# --------------------------------------------------------------------------- -# 12.8 extract_final_result_data -# --------------------------------------------------------------------------- - -class TestExtractFinalResultData: - """Validates: Requirement 12.8""" - - def test_extracts_text_parts(self): - """Text items in message.content are joined into result_text.""" - content = [{"text": "Hello"}, {"text": "World"}] - final_result = SimpleNamespace( - message=SimpleNamespace(content=content) - ) - images, text = StreamEventFormatter.extract_final_result_data(final_result) - assert text == "Hello World" - assert images == [] - - def test_extracts_image_data(self): - """Image items are extracted with format and data.""" - content = [ - { - "image": { - "format": "png", - "source": {"data": "base64data"}, - } - } - ] - final_result = SimpleNamespace( - message=SimpleNamespace(content=content) - ) - images, text = StreamEventFormatter.extract_final_result_data(final_result) - assert len(images) == 1 - assert images[0]["format"] == "png" - assert images[0]["data"] == "base64data" - - def test_mixed_text_and_images(self): - """Both text and images are extracted from mixed content.""" - content = [ - {"text": "Here is an image:"}, - { - "image": { - "format": "jpeg", - "source": {"data": "imgdata"}, - } - }, - ] - final_result = SimpleNamespace( - message=SimpleNamespace(content=content) - ) - images, text = StreamEventFormatter.extract_final_result_data(final_result) - assert text == "Here is an image:" - assert len(images) == 1 - assert images[0]["format"] == "jpeg" - - def test_no_message_attribute_falls_back_to_str(self): - """Objects without .message.content fall back to str(final_result).""" - images, text = StreamEventFormatter.extract_final_result_data("plain string") - assert text == "plain string" - assert images == [] - - def test_empty_content_list(self): - """Empty content list falls back to str(final_result).""" - final_result = SimpleNamespace( - message=SimpleNamespace(content=[]) - ) - images, text = StreamEventFormatter.extract_final_result_data(final_result) - assert images == [] - # No text_parts collected, so result_text stays as str(final_result) - assert isinstance(text, str) diff --git a/backend/tests/agents/main_agent/streaming/test_static_prefix_ttl_cost.py b/backend/tests/agents/main_agent/streaming/test_static_prefix_ttl_cost.py new file mode 100644 index 000000000..49dac86c3 --- /dev/null +++ b/backend/tests/agents/main_agent/streaming/test_static_prefix_ttl_cost.py @@ -0,0 +1,41 @@ +"""The coordinator sizes the 1h static prefix from the context breakdown (PR-5).""" + +from types import SimpleNamespace + +from agents.main_agent.streaming.stream_coordinator import StreamCoordinator + + +class _Strands: + pass + + +def _wrapper(long_ttl): + return SimpleNamespace(model_config=SimpleNamespace(long_ttl_static_prefix=lambda: long_ttl)) + + +def _with_breakdown(system, tools): + from agents.main_agent.session.hooks import context_attribution as ca + agent = _Strands() + setattr(agent, ca._BREAKDOWN_ATTR, { + "total": system + tools + 500, + "partitions": [ + {"key": "system", "label": "System prompt", "tokens": system}, + {"key": "tools", "label": "Tools", "tokens": tools}, + {"key": "messages", "label": "Messages", "tokens": 500}, + ], + }) + return agent + + +def test_returns_system_plus_tools_when_arm_is_on(): + assert StreamCoordinator._long_ttl_static_prefix_tokens(_wrapper(True), _with_breakdown(8_000, 3_000)) == 11_000 + + +def test_none_when_arm_is_off(): + assert StreamCoordinator._long_ttl_static_prefix_tokens(_wrapper(False), _with_breakdown(8_000, 3_000)) is None + + +def test_none_without_breakdown_or_wrapper(): + assert StreamCoordinator._long_ttl_static_prefix_tokens(_wrapper(True), _Strands()) is None + assert StreamCoordinator._long_ttl_static_prefix_tokens(None, _with_breakdown(1, 1)) is None + assert StreamCoordinator._long_ttl_static_prefix_tokens(_wrapper(True), None) is None diff --git a/backend/tests/agents/main_agent/streaming/test_tool_result_processor.py b/backend/tests/agents/main_agent/streaming/test_tool_result_processor.py deleted file mode 100644 index 41e499ad9..000000000 --- a/backend/tests/agents/main_agent/streaming/test_tool_result_processor.py +++ /dev/null @@ -1,261 +0,0 @@ -""" -Tests for ToolResultProcessor — text/image extraction from tool results. - -Validates: Requirements 13.1–13.4 -""" - -import base64 -import json - -import pytest - -from agents.main_agent.streaming.tool_result_processor import ToolResultProcessor - - -# --------------------------------------------------------------------------- -# 13.1 Text content extraction -# --------------------------------------------------------------------------- - -class TestTextExtraction: - """Validates: Requirement 13.1""" - - def test_extracts_plain_text(self): - """process_tool_result extracts text from a simple text content item.""" - tool_result = {"content": [{"text": "Hello world"}]} - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert text == "Hello world" - assert images == [] - - def test_extracts_multiple_text_items(self): - """Multiple text items are concatenated.""" - tool_result = { - "content": [ - {"text": "Part one. "}, - {"text": "Part two."}, - ] - } - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert text == "Part one. Part two." - assert images == [] - - def test_handles_string_tool_result(self): - """A plain string tool_result is wrapped into content automatically.""" - text, images = ToolResultProcessor.process_tool_result("raw text result") - assert "raw text result" in text - assert images == [] - - def test_handles_json_string_tool_result(self): - """A JSON-encoded string tool_result is parsed.""" - payload = json.dumps({"content": [{"text": "from json"}]}) - text, images = ToolResultProcessor.process_tool_result(payload) - assert "from json" in text - - -# --------------------------------------------------------------------------- -# 13.2 Image content extraction -# --------------------------------------------------------------------------- - -class TestImageExtraction: - """Validates: Requirement 13.2""" - - def test_extracts_image_with_data_field(self): - """Image content with base64 data in source.data is extracted.""" - b64 = base64.b64encode(b"fake-png-data").decode() - tool_result = { - "content": [ - { - "image": { - "format": "png", - "source": {"data": b64}, - } - } - ] - } - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert len(images) == 1 - assert images[0]["format"] == "png" - assert images[0]["data"] == b64 - - def test_extracts_image_with_bytes_field(self): - """Image content with raw bytes in source.bytes is base64-encoded.""" - raw_bytes = b"fake-image-bytes" - tool_result = { - "content": [ - { - "image": { - "format": "jpeg", - "source": {"bytes": raw_bytes}, - } - } - ] - } - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert len(images) == 1 - assert images[0]["format"] == "jpeg" - assert images[0]["data"] == base64.b64encode(raw_bytes).decode("utf-8") - - def test_extracts_image_with_string_bytes_field(self): - """Image content with a string in source.bytes is used as-is.""" - tool_result = { - "content": [ - { - "image": { - "format": "gif", - "source": {"bytes": "string-bytes-data"}, - } - } - ] - } - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert len(images) == 1 - assert images[0]["data"] == "string-bytes-data" - - def test_image_defaults_format_to_png(self): - """Missing format in image content defaults to 'png'.""" - tool_result = { - "content": [ - { - "image": { - "source": {"data": "abc123"}, - } - } - ] - } - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert len(images) == 1 - assert images[0]["format"] == "png" - - def test_mixed_text_and_images(self): - """Both text and image items are extracted from the same result.""" - tool_result = { - "content": [ - {"text": "Here is the chart:"}, - { - "image": { - "format": "png", - "source": {"data": "chartdata"}, - } - }, - ] - } - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert "Here is the chart:" in text - assert len(images) == 1 - assert images[0]["format"] == "png" - - -# --------------------------------------------------------------------------- -# 13.3 JSON content with embedded images -# --------------------------------------------------------------------------- - -class TestJsonContentWithImages: - """Validates: Requirement 13.3""" - - def test_extracts_screenshot_from_json(self): - """_process_json_content extracts screenshot images from JSON text.""" - json_data = { - "result": "success", - "screenshot": { - "format": "png", - "data": "base64screenshotdata", - }, - } - json_text = json.dumps(json_data) - images, cleaned = ToolResultProcessor._process_json_content(json_text) - assert len(images) == 1 - assert images[0]["format"] == "png" - assert images[0]["data"] == "base64screenshotdata" - # Cleaned text should not contain the raw base64 data - assert "base64screenshotdata" not in cleaned - - def test_extracts_image_field_from_json(self): - """_process_json_content extracts 'image' field from JSON.""" - json_data = { - "image": { - "format": "jpeg", - "data": "jpegdata", - } - } - images, cleaned = ToolResultProcessor._process_json_content(json.dumps(json_data)) - assert len(images) == 1 - assert images[0]["format"] == "jpeg" - - def test_extracts_images_array_from_json(self): - """_process_json_content extracts from 'images' array field.""" - json_data = { - "images": [ - {"format": "png", "data": "img1"}, - {"format": "jpeg", "data": "img2"}, - ] - } - images, cleaned = ToolResultProcessor._process_json_content(json.dumps(json_data)) - assert len(images) == 2 - - def test_cleans_text_removes_large_base64(self): - """Cleaned text replaces base64 data with metadata note.""" - json_data = { - "chart": { - "format": "png", - "data": "A" * 1000, - }, - "summary": "Chart generated", - } - images, cleaned = ToolResultProcessor._process_json_content(json.dumps(json_data)) - assert len(images) == 1 - parsed_cleaned = json.loads(cleaned) - assert parsed_cleaned["chart"]["note"] == "Image data extracted and displayed separately" - assert parsed_cleaned["summary"] == "Chart generated" - - def test_non_json_text_returns_unchanged(self): - """Non-JSON text passes through _process_json_content unchanged.""" - plain_text = "This is not JSON" - images, cleaned = ToolResultProcessor._process_json_content(plain_text) - assert images == [] - assert cleaned == plain_text - - def test_json_without_images_returns_unchanged(self): - """JSON without image fields returns empty images and original text.""" - json_text = json.dumps({"status": "ok", "count": 42}) - images, cleaned = ToolResultProcessor._process_json_content(json_text) - assert images == [] - assert cleaned == json_text - - def test_skips_optimized_screenshot_format(self): - """Optimized screenshot references (available + description) are skipped.""" - json_data = { - "screenshot": { - "available": True, - "description": "Page loaded successfully", - } - } - images, cleaned = ToolResultProcessor._process_json_content(json.dumps(json_data)) - assert images == [] - - -# --------------------------------------------------------------------------- -# 13.4 Empty / no content -# --------------------------------------------------------------------------- - -class TestEmptyContent: - """Validates: Requirement 13.4""" - - def test_empty_content_list(self): - """Tool result with empty content list returns empty text and images.""" - tool_result = {"content": []} - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert text == "" - assert images == [] - - def test_no_content_key(self): - """Tool result without content key returns empty text and images.""" - tool_result = {"toolUseId": "tu-123"} - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert text == "" - assert images == [] - - def test_content_with_empty_dict_items(self): - """Content items that are empty dicts produce no text or images.""" - tool_result = {"content": [{}]} - text, images = ToolResultProcessor.process_tool_result(tool_result) - assert text == "" - assert images == [] diff --git a/backend/tests/agents/main_agent/utils/test_timezone.py b/backend/tests/agents/main_agent/utils/test_timezone.py index 2ba49e4a9..741906914 100644 --- a/backend/tests/agents/main_agent/utils/test_timezone.py +++ b/backend/tests/agents/main_agent/utils/test_timezone.py @@ -1,29 +1,50 @@ """ Tests for timezone utilities. Requirements: 21.1–21.3 + +The rendered string is part of the Bedrock prompt-cache prefix, so it must be +byte-stable within a Pacific day: date, weekday and timezone only — no hour. """ import re +from datetime import datetime from unittest.mock import patch - -import pytest +from zoneinfo import ZoneInfo from agents.main_agent.utils.timezone import get_current_date_pacific -# Regex for "YYYY-MM-DD (DayName) HH:00 TZ" +# Regex for "YYYY-MM-DD (DayName) TZ" DATE_FORMAT_RE = re.compile( r"^\d{4}-\d{2}-\d{2} " # YYYY-MM-DD r"\([A-Z][a-z]+\) " # (DayName) - r"\d{2}:00 " # HH:00 r"[A-Z]{3,4}$" # TZ abbreviation ) VALID_DAYS = {"Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"} +PACIFIC = ZoneInfo("America/Los_Angeles") + + +class _FrozenDatetime(datetime): + """datetime subclass whose now() returns a fixed instant (in the requested tz).""" + + frozen: datetime + + @classmethod + def now(cls, tz=None): + return cls.frozen.astimezone(tz) if tz else cls.frozen + + +def _at(instant: datetime) -> str: + """Call get_current_date_pacific() as if it were `instant` right now.""" + _FrozenDatetime.frozen = instant + with patch("agents.main_agent.utils.timezone.datetime", _FrozenDatetime): + return get_current_date_pacific() + class TestGetCurrentDatePacific: - """Req 21.1: Verify format matches 'YYYY-MM-DD (DayName) HH:00 TZ'.""" + """Req 21.1: Verify format matches 'YYYY-MM-DD (DayName) TZ'.""" def test_format_matches_expected_pattern(self): result = get_current_date_pacific() @@ -34,12 +55,32 @@ def test_contains_valid_day_name(self): day = re.search(r"\((\w+)\)", result).group(1) assert day in VALID_DAYS, f"Day '{day}' is not a valid day name" - def test_hour_is_zero_padded_with_00_minutes(self): + def test_has_no_time_of_day_component(self): + # Prompt-cache contract: the line must not change at hour boundaries. result = get_current_date_pacific() - hour_match = re.search(r"(\d{2}):00", result) - assert hour_match is not None - hour = int(hour_match.group(1)) - assert 0 <= hour <= 23 + assert not re.search(r"\d{1,2}:\d{2}", result), f"Unexpected time-of-day in '{result}'" + assert len(result.split()) == 3 + + +class TestDayStability: + """The rendered string must be identical for every hour of the same Pacific day + and change only when the Pacific calendar date changes.""" + + def test_identical_across_every_hour_of_a_day(self): + base = datetime(2026, 9, 15, 0, 0, tzinfo=PACIFIC) + rendered = {_at(base.replace(hour=h, minute=m)) for h in range(24) for m in (0, 59)} + assert rendered == {"2026-09-15 (Tuesday) PDT"}, rendered + + def test_changes_at_pacific_midnight(self): + before = _at(datetime(2026, 9, 15, 23, 59, tzinfo=PACIFIC)) + after = _at(datetime(2026, 9, 16, 0, 0, tzinfo=PACIFIC)) + assert before == "2026-09-15 (Tuesday) PDT" + assert after == "2026-09-16 (Wednesday) PDT" + + def test_uses_pacific_date_not_utc_date(self): + # 03:00 UTC on the 16th is still 20:00 PDT on the 15th. + utc_instant = datetime(2026, 9, 16, 3, 0, tzinfo=ZoneInfo("UTC")) + assert _at(utc_instant) == "2026-09-15 (Tuesday) PDT" class TestTimezoneAbbreviation: @@ -50,6 +91,9 @@ def test_timezone_is_pst_or_pdt(self): tz = result.split()[-1] assert tz in ("PST", "PDT"), f"Timezone '{tz}' is not PST or PDT" + def test_standard_time_renders_pst(self): + assert _at(datetime(2026, 1, 15, 12, 0, tzinfo=PACIFIC)) == "2026-01-15 (Thursday) PST" + class TestUTCFallback: """Req 21.3: Verify UTC fallback when timezone libraries unavailable.""" @@ -63,7 +107,7 @@ def test_falls_back_to_utc_when_timezone_unavailable(self): def test_utc_fallback_format_matches(self): with patch("agents.main_agent.utils.timezone.TIMEZONE_AVAILABLE", False): result = get_current_date_pacific() - # Should still have YYYY-MM-DD (DayName) HH:00 UTC + # Should still have YYYY-MM-DD (DayName) UTC — and no hour parts = result.split() - assert len(parts) == 4 - assert parts[3] == "UTC" + assert len(parts) == 3 + assert parts[2] == "UTC" diff --git a/backend/tests/apis/app_api/admin/costs/test_diagnoses.py b/backend/tests/apis/app_api/admin/costs/test_diagnoses.py index 306d18819..8a436ae0a 100644 --- a/backend/tests/apis/app_api/admin/costs/test_diagnoses.py +++ b/backend/tests/apis/app_api/admin/costs/test_diagnoses.py @@ -107,13 +107,28 @@ def test_agent_switch_churn_threshold(): assert dg.agent_switch_churn(_known(agent_switch_count=3)).severity == "info" -def test_agent_cache_bypass_names_only_the_non_key_described_injected_ids(): - # create_artifact is key-described (promoted); analyze_spreadsheet is not. - facts = _known(enabled_tools=["calculator", "create_artifact", "analyze_spreadsheet", "workspace_files"]) +def test_agent_cache_bypass_names_only_the_non_key_described_injected_ids(monkeypatch): + # The rule's mechanics, pinned against fixed sets so the test does not + # re-litigate which families are promoted today (that is + # tests/shared/test_injected_tool_cache_eligibility.py's job). + monkeypatch.setattr(dg, "INJECTED_TOOL_IDS", frozenset({"a_tool", "b_tool", "c_tool"})) + monkeypatch.setattr(dg, "KEY_DESCRIBED_INJECTED_TOOL_IDS", frozenset({"a_tool"})) + facts = _known(enabled_tools=["calculator", "a_tool", "c_tool", "b_tool"]) d = dg.agent_cache_bypass(facts) assert d is not None - assert d.evidence["bypassingToolIds"] == ["analyze_spreadsheet", "workspace_files"] - assert dg.agent_cache_bypass(_known(enabled_tools=["calculator", "create_artifact"])) is None + assert d.evidence["bypassingToolIds"] == ["b_tool", "c_tool"] # sorted, registry ids excluded + assert dg.agent_cache_bypass(_known(enabled_tools=["calculator", "a_tool"])) is None + + +def test_agent_cache_bypass_tracks_the_live_promotion_state(): + # Every enabled_tools-gated injected family is promoted (spreadsheet + # analysis last, once assistant_id joined the cache key), so the rule is + # silent for any toolset today. It stays wired for the next family that + # closes over something the key does not carry. + promoted = ["create_artifact", "create_word_document", "create_excel_spreadsheet", + "create_powerpoint_presentation", "workspace_files", + "analyze_spreadsheet", "list_spreadsheets"] + assert dg.agent_cache_bypass(_known(enabled_tools=["calculator", *promoted])) is None def test_large_toolset_counts_catalog_ids(): diff --git a/backend/tests/apis/app_api/admin/costs/test_document_fields.py b/backend/tests/apis/app_api/admin/costs/test_document_fields.py new file mode 100644 index 000000000..9dde52bb8 --- /dev/null +++ b/backend/tests/apis/app_api/admin/costs/test_document_fields.py @@ -0,0 +1,138 @@ +"""Document-context fields on the admin cost surfaces +(docs/specs/document-context-offload.md §6.1). + +Rows carry the attachment footprint at each call; the anatomy projects it +per call, the profile rolls it up (digest-vs-full turn shares, peak document +tokens, document_read totals) and reports coverage. The session-row rollups +the write path ADDs are pinned too, so the two never disagree on a definition. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from apis.app_api.admin.costs.models import CompactionEvent +from apis.app_api.admin.costs.service import AdminCostService, _call_ledger +from apis.shared.sessions.metadata import DOCUMENT_ROLLUP_ATTRS, _document_rollups + + +def _record(i, **extra): + rec = { + "timestamp": f"2026-09-16T00:00:{i:02d}Z", + "messageId": i, + "tokenUsage": {"inputTokens": 100, "outputTokens": 5, "cacheReadInputTokens": 0, "cacheWriteInputTokens": 0}, + "modelInfo": {"modelId": "m1"}, + "cost": {"total": 0.01}, + "cacheStatus": "hit", + } + rec.update(extra) + return rec + + +FULL = dict(hasDocuments=True, documentCount=2, documentTokens=12_000, documentDigests=0, + documentsAttached=2, documentSlices=0, documentSliceTokens=0, documentMime={"pdf": 2}) +DIGEST = dict(hasDocuments=False, documentCount=0, documentTokens=0, documentDigests=2, + documentsAttached=0, documentSlices=1, documentSliceTokens=900, documentMime={}, + documentReads={"calls": 1, "pages": 4, "bytes": 3600}) + + +class TestCallLedger: + def test_absent_fields_stay_untracked(self): + ledger = _call_ledger(_record(0), None) + assert ledger.documents is None and ledger.document_reads is None + + def test_fields_decode_with_decimal_coercion(self): + from decimal import Decimal + + rec = _record(0, hasDocuments=True, documentCount=Decimal("1"), documentTokens=Decimal("500"), + documentMime={"pdf": Decimal("1")}, documentReads={"calls": Decimal("2"), "pages": 3}) + ledger = _call_ledger(rec, None) + assert ledger.documents == {"hasDocuments": True, "documentCount": 1, "documentTokens": 500, "documentMime": {"pdf": 1}} + assert (ledger.document_reads.calls, ledger.document_reads.pages, ledger.document_reads.bytes) == (2, 3, 0) + + def test_compaction_event_accepts_the_document_kinds(self): + event = CompactionEvent(kind="document_offload", documents=1, documentTokens=9000, cacheGapSeconds=420) + assert (event.documents, event.document_tokens, event.cache_gap_seconds) == (1, 9000, 420) + + +def _service(records, row=None, files=None): + service = AdminCostService.__new__(AdminCostService) + service.storage = AsyncMock() + service.storage.get_session_cost_records = AsyncMock(return_value=records) + service.storage.get_session_diagnostic_row = AsyncMock(return_value=row) + service.storage.get_user_cost_summary = AsyncMock(return_value=None) + service._file_repository = AsyncMock() + service._file_repository.list_session_file_stats = AsyncMock(return_value=files or []) + return service + + +class TestAnatomy: + @pytest.mark.asyncio + async def test_rows_project_the_document_fields_or_null(self): + anatomy = await _service([_record(0, **FULL), _record(1, **DIGEST), _record(2)]).get_session_cost_anatomy("s1") + full, digest, bare = anatomy.calls + assert full.has_documents is True and full.document_tokens == 12_000 and full.document_mime == {"pdf": 2} + assert full.document_reads is None + assert digest.has_documents is False and digest.document_digests == 2 + assert digest.document_reads.pages == 4 and digest.document_slice_tokens == 900 + assert bare.has_documents is None and bare.document_reads is None + wire = anatomy.model_dump(by_alias=True)["calls"][1] + assert wire["documentReads"] == {"calls": 1, "pages": 4, "bytes": 3600} + + +def _row(**overrides): + row = { + "sessionId": "s1", "userId": "u1", "status": "active", "messageCount": 4, "totalCost": 0.5, + "lastContextTokens": 20_000, "contextWindow": 200_000, "totalCacheReadTokens": 1, "totalCacheWriteTokens": 1, + "preferences": {"lastModel": "m1", "enabledTools": []}, "compaction": {}, + } + row.update(overrides) + return row + + +class TestProfile: + @pytest.mark.asyncio + async def test_rollups_from_rows(self): + records = [_record(0, **FULL), _record(1, **{**FULL, "documentTokens": 15_000}), _record(2, **DIGEST), _record(3)] + profile = await _service(records, row=_row()).get_session_profile("s1") + assert profile.data_coverage.documents is True + assert (profile.full_document_calls, profile.digest_only_calls) == (2, 1) + assert profile.peak_document_tokens == 15_000 + assert (profile.document_read_calls, profile.document_read_pages) == (1, 4) + + @pytest.mark.asyncio + async def test_session_row_rollups_cover_rows_that_expired(self): + row = _row(fullDocumentCalls=7, digestOnlyCalls=2, documentReadCalls=3, documentReadPages=11) + profile = await _service([_record(0)], row=row).get_session_profile("s1") + assert profile.data_coverage.documents is True + assert (profile.full_document_calls, profile.digest_only_calls) == (7, 2) + assert (profile.document_read_calls, profile.document_read_pages) == (3, 11) + assert profile.peak_document_tokens is None + + @pytest.mark.asyncio + async def test_untracked_reads_as_not_tracked(self): + profile = await _service([_record(0)], row=_row()).get_session_profile("s1") + assert profile.data_coverage.documents is False + assert profile.full_document_calls == 0 and profile.peak_document_tokens is None + + +class TestWriteSideRollups: + def test_attrs_are_the_ones_the_profile_reads(self): + assert DOCUMENT_ROLLUP_ATTRS == ("fullDocumentCalls", "digestOnlyCalls", "documentReadCalls", "documentReadPages") + + @pytest.mark.parametrize( + "extra,expected", + [ + ({}, (0, 0, 0, 0)), + (FULL, (1, 0, 0, 0)), + (DIGEST, (0, 1, 1, 4)), + ({"hasDocuments": True, "documentDigests": 3}, (1, 0, 0, 0)), # inline wins over digests + ({"documentReads": {"calls": "x", "pages": None}}, (0, 0, 0, 0)), + ], + ) + def test_deltas_from_the_calls_extras(self, extra, expected): + rollups = _document_rollups(SimpleNamespace(model_extra=dict(extra))) + assert tuple(rollups[a] for a in DOCUMENT_ROLLUP_ATTRS) == expected diff --git a/backend/tests/apis/app_api/admin/costs/test_drilldown_routes.py b/backend/tests/apis/app_api/admin/costs/test_drilldown_routes.py index 7f05d0798..036f84919 100644 --- a/backend/tests/apis/app_api/admin/costs/test_drilldown_routes.py +++ b/backend/tests/apis/app_api/admin/costs/test_drilldown_routes.py @@ -102,7 +102,16 @@ def test_session_profile_returns_200(): assert resp.status_code == 200 body = resp.json() assert body["sessionId"] == "s1" and body["compactionThreshold"] == 100_000 - assert body["dataCoverage"] == {"toolCensus": False, "compactionCount": False, "fingerprints": False, "cost": False} + assert body["dataCoverage"] == { + "toolCensus": False, "compactionCount": False, "fingerprints": False, "cost": False, + "prefixTokens": False, "windowTrim": False, "compactionEvents": False, + "feedback": False, + "documents": False, + } + assert body["feedback"] == { + "up": 0, "down": 0, "byTurnClass": None, "reasons": {}, "unjoined": 0, + "retried": 0, "reworkUsd": None, "implicit": None, "evaluations": None, + } service.get_session_profile.assert_awaited_once_with("s1") diff --git a/backend/tests/apis/app_api/admin/costs/test_session_profile_service.py b/backend/tests/apis/app_api/admin/costs/test_session_profile_service.py index 83a10ab11..001430b34 100644 --- a/backend/tests/apis/app_api/admin/costs/test_session_profile_service.py +++ b/backend/tests/apis/app_api/admin/costs/test_session_profile_service.py @@ -193,16 +193,287 @@ async def test_share_of_user_period_uses_the_current_month(): @pytest.mark.asyncio -async def test_agent_cache_bypass_names_the_offending_ids(): +async def test_agent_cache_bypass_names_the_offending_ids(monkeypatch): + # Every injected family is cache-eligible today, so the rule is silent on + # the live sets; un-promote spreadsheet analysis for this test to prove the + # profile still surfaces the diagnosis when a family does bypass. + from apis.app_api.admin.costs import diagnoses as dg + + monkeypatch.setattr(dg, "KEY_DESCRIBED_INJECTED_TOOL_IDS", frozenset()) p = await _service(_row(), []).get_session_profile("s1") bypass = next(d for d in p.diagnoses if d.code == "AGENT_CACHE_BYPASS") assert bypass.evidence["bypassingToolIds"] == ["analyze_spreadsheet"] assert p.enabled_tool_ids == ["analyze_spreadsheet", "calculator"] +@pytest.mark.asyncio +async def test_agent_cache_bypass_is_silent_when_every_family_is_promoted(): + p = await _service(_row(), []).get_session_profile("s1") + assert not [d for d in p.diagnoses if d.code == "AGENT_CACHE_BYPASS"] + + @pytest.mark.asyncio async def test_the_whole_profile_serializes_without_content_bearing_keys(): from apis.shared.observability.content_policy import content_bearing_paths files = [{"uploadId": "a", "mimeType": "application/pdf", "sizeBytes": 1}] p = await _service(_row(), [_call(0, tool_calls={"t": {"calls": 1, "errors": 0}})], files=files).get_session_profile("s1") assert content_bearing_paths(p.model_dump(by_alias=True)) == [] + + +@pytest.mark.asyncio +async def test_context_ledger_is_decoded_diffed_and_covered(): + records = [ + _call(0, read=10_000), + _call(1, read=10_000), + _call(2, read=10_000), + _call(3, read=10_000), + ] + # Rows carry the ledger from call 1 on: a stable prefix split, a window + # count that rises before call 3 (a trim), and one compaction decision + # with the summary's size at that moment. + records[1]["prefixTokens"] = {"system": 12_000, "tools": 48_000} + records[1]["windowRemovedMessages"] = 0 + records[2]["windowRemovedMessages"] = 0 + records[3]["windowRemovedMessages"] = 8 + records[3]["compactionEvents"] = [{"kind": "applied", "checkpoint": 12, "summaryTokens": 2_300}] + + p = await _service(_row(), records).get_session_profile("s1") + + assert p.prefix_tokens.system == 12_000 and p.prefix_tokens.tools == 48_000 + assert p.window_trim_calls == 1 + assert p.window_removed_messages == 8 + assert p.compaction_event_counts == {"applied": 1} + assert p.last_summary_tokens == 2_300 + assert p.data_coverage.prefix_tokens and p.data_coverage.window_trim and p.data_coverage.compaction_events + trimmed = [pt.window_trimmed for pt in p.context_trajectory] + assert trimmed == [None, 0, 0, 8] + assert p.context_trajectory[3].compaction == ["applied"] + + +@pytest.mark.asyncio +async def test_rows_without_a_ledger_read_not_tracked(): + p = await _service(_row(), [_call(0), _call(1)]).get_session_profile("s1") + assert p.prefix_tokens is None + assert p.window_removed_messages is None and p.window_trim_calls == 0 + assert p.compaction_event_counts == {} and p.last_summary_tokens is None + assert not p.data_coverage.prefix_tokens and not p.data_coverage.window_trim + assert not p.data_coverage.compaction_events + assert all(pt.window_trimmed is None and pt.compaction is None for pt in p.context_trajectory) + + +@pytest.mark.asyncio +async def test_session_row_counters_alone_mark_compaction_events_as_tracked(): + p = await _service(_row(compactionAppliedCount=0), [_call(0)]).get_session_profile("s1") + assert p.data_coverage.compaction_events + assert p.session.compaction_applied_count == 0 + + +# ── feedback join (document-context offload PR-7) ─────────────────────────── + + +def _feedback(message_id, value, reason=None): + row = {"sessionId": "s1", "messageId": message_id, "value": value, "updatedAt": "2026-09-16T00:00:00Z"} + if reason: + row["reason"] = reason + return row + + +def _service_with_feedback(row, records, feedback): + service = _service(row, records) + service.storage.get_session_feedback_rows = AsyncMock(return_value=feedback) + return service + + +@pytest.mark.asyncio +async def test_feedback_joins_the_turn_class_when_the_rows_carry_it(): + records = [ + _call(0), # attach turn: full document inline + _call(1), # follow-up: digest only + _call(2), # follow-up that read pages back + _call(3), # no documents at all + ] + records[0]["hasDocuments"] = True + records[1]["hasDocuments"] = False + records[1]["documentDigests"] = 1 + records[2]["hasDocuments"] = False + records[2]["documentDigests"] = 1 + records[2]["documentReads"] = {"calls": 1, "pages": 4, "bytes": 1000} + records[3]["hasDocuments"] = False + records[3]["documentDigests"] = 0 + feedback = [ + _feedback(0, 1), + _feedback(1, -1, "wrong"), + _feedback(2, 1), + _feedback(3, -1, "tool_failed"), + _feedback(9, -1), # no cost row for this message + ] + p = await _service_with_feedback(_row(), records, feedback).get_session_profile("s1") + + assert (p.feedback.up, p.feedback.down, p.feedback.unjoined) == (2, 3, 1) + by = p.feedback.by_turn_class + assert by is not None + assert (by.full.up, by.full.down) == (1, 0) + assert (by.digest_only.up, by.digest_only.down) == (0, 1) + assert (by.retrieved.up, by.retrieved.down) == (1, 0) + assert (by.none.up, by.none.down) == (0, 1) + assert p.data_coverage.feedback is True + # Wire shape the SPA reads. + wire = p.model_dump(by_alias=True)["feedback"] + assert wire["byTurnClass"]["digestOnly"] == {"up": 0, "down": 1} + + +@pytest.mark.asyncio +async def test_feedback_counts_without_turn_class_when_rows_predate_1137(): + records = [_call(0), _call(1)] # no hasDocuments / documentDigests / documentReads + feedback = [_feedback(0, 1), _feedback(1, -1, "instructions")] + p = await _service_with_feedback(_row(), records, feedback).get_session_profile("s1") + assert (p.feedback.up, p.feedback.down) == (1, 1) + assert p.feedback.by_turn_class is None, "turn class is 'not tracked', not 'none'" + assert p.feedback.unjoined == 0 + assert p.data_coverage.feedback is True + + +@pytest.mark.asyncio +async def test_no_feedback_rows_falls_back_to_rollups_and_coverage_is_honest(): + p = await _service_with_feedback(_row(), [_call(0)], []).get_session_profile("s1") + assert (p.feedback.up, p.feedback.down) == (0, 0) + assert p.data_coverage.feedback is False + + p = await _service_with_feedback(_row(thumbsUp=2, thumbsDown=1), [_call(0)], []).get_session_profile("s1") + assert (p.feedback.up, p.feedback.down) == (2, 1) + assert p.data_coverage.feedback is True + assert p.feedback.by_turn_class is None + + +@pytest.mark.asyncio +async def test_retries_are_counted_and_rework_is_priced_from_the_cost_rows(): + # Turn A: assistant 1 (thumbed down, $0.10). Retry sent as user message 2; + # its turn is assistant 3 + 4 ($0.20 + $0.05, a tool round trip). User 5, + # assistant 6 ($0.99) is the NEXT turn and must not be counted. + records = [_call(1, cost=0.10), _call(3, cost=0.20), _call(4, cost=0.05), _call(6, cost=0.99)] + feedback = [{**_feedback(1, -1, "wrong"), "retryMessageId": 2}, _feedback(6, 1)] + p = await _service_with_feedback(_row(), records, feedback).get_session_profile("s1") + assert p.feedback.retried == 1 + assert p.feedback.rework_usd == 0.35 + assert (p.feedback.up, p.feedback.down) == (1, 1) + + # A retry whose turn has no cost rows yet (still streaming) counts, but prices only the thumbed side. + feedback = [{**_feedback(1, -1, "wrong"), "retryMessageId": 7}] + p = await _service_with_feedback(_row(), records, feedback).get_session_profile("s1") + assert p.feedback.retried == 1 and p.feedback.rework_usd == 0.10 + + # Nothing priced at all → None, not 0. + feedback = [{**_feedback(9, -1), "retryMessageId": 10}] + p = await _service_with_feedback(_row(), records, feedback).get_session_profile("s1") + assert p.feedback.retried == 1 and p.feedback.rework_usd is None + + +@pytest.mark.asyncio +async def test_implicit_signal_rows_are_never_summed_into_the_thumb_counts(): + records = [_call(0)] + records[0]["hasDocuments"] = True + feedback = [_feedback(0, 1), {**_feedback(0, -1), "signal": "implicit"}, {**_feedback(0, -1), "signal": "explicit"}] + p = await _service_with_feedback(_row(), records, feedback).get_session_profile("s1") + assert (p.feedback.up, p.feedback.down) == (1, 1) + assert (p.feedback.by_turn_class.full.up, p.feedback.by_turn_class.full.down) == (1, 1) + + +@pytest.mark.asyncio +async def test_implicit_signals_are_counted_as_messages_touched_per_kind(): + def implicit(message_id, kind, count): + return {"sessionId": "s1", "messageId": message_id, "signal": "implicit", "kind": kind, "count": count, "updatedAt": "t"} + + feedback = [implicit(0, "copy", 3), implicit(2, "copy", 1), implicit(2, "continue", 1), implicit(4, "weird", 1)] + p = await _service_with_feedback(_row(), [_call(0), _call(2)], feedback).get_session_profile("s1") + assert p.feedback.implicit is not None + assert (p.feedback.implicit.copied, p.feedback.implicit.continued) == (2, 1) + assert (p.feedback.up, p.feedback.down) == (0, 0) + assert p.data_coverage.feedback is True + assert p.model_dump(by_alias=True)["feedback"]["implicit"] == {"copied": 2, "continued": 1} + + p = await _service_with_feedback(_row(), [_call(0)], [_feedback(0, 1)]).get_session_profile("s1") + assert p.feedback.implicit is None + + +@pytest.mark.asyncio +async def test_feedback_reader_failure_never_breaks_the_profile(): + service = _service(_row(), [_call(0)]) + service.storage.get_session_feedback_rows = AsyncMock(side_effect=RuntimeError("boom")) + p = await service.get_session_profile("s1") + assert p is not None and p.feedback.up == 0 and p.data_coverage.feedback is False + + +@pytest.mark.asyncio +async def test_judged_thumbs_aggregate_per_evaluator_and_corroboration(): + def judged(message_id, reason, scores=None, corroborated=None): + row = _feedback(message_id, -1, reason) + row["evaluatedAt"] = "t" + row["evaluation"] = {"reason": reason, "evaluators": list((scores or {}).keys())} + if scores: + row["evaluation"]["scores"] = {k: {"value": v, "n": 1} for k, v in scores.items()} + if corroborated is not None: + row["evaluation"]["toolFailureCorroborated"] = corroborated + return row + + feedback = [ + judged(0, "wrong", {"Builtin.Correctness": 1.0, "Builtin.Faithfulness": 0.5}), + judged(1, "wrong", {"Builtin.Correctness": 0.0}), + judged(2, "tool_failed", corroborated=True), + judged(3, "tool_failed", corroborated=False), + _feedback(4, -1, "other"), # not judged yet + ] + p = await _service_with_feedback(_row(), [_call(i) for i in range(5)], feedback).get_session_profile("s1") + ev = p.feedback.evaluations + assert ev is not None and ev.judged == 4 + assert ev.by_evaluator["Builtin.Correctness"].n == 2 and ev.by_evaluator["Builtin.Correctness"].mean == 0.5 + assert ev.by_evaluator["Builtin.Faithfulness"].mean == 0.5 + assert (ev.tool_failures_reported, ev.tool_failures_corroborated) == (2, 1) + assert p.feedback.down == 5 + wire = p.model_dump(by_alias=True)["feedback"]["evaluations"] + assert wire["byEvaluator"]["Builtin.Correctness"] == {"n": 2, "mean": 0.5} + + p = await _service_with_feedback(_row(), [_call(0)], [_feedback(0, -1)]).get_session_profile("s1") + assert p.feedback.evaluations is None + + +class TestDownThumbReasons: + """The per-session half of the reason split the fleet view reports (#1152): + the fleet says *how much*, this says *why this conversation*. + + A closed set of codes, never free text — the write path types ``reason`` as + a ``Literal``, so anything else is a row from a future schema and is dropped + rather than surfaced. + """ + + def _rows(self, *reasons): + return [ + {"messageId": i, "value": -1, **({"reason": r} if r else {})} + for i, r in enumerate(reasons, start=1) + ] + + def test_reasons_are_counted(self): + from apis.app_api.admin.costs.service import _join_feedback + + profile = _join_feedback([], self._rows("tool_failed", "tool_failed", "length")) + assert profile.reasons == {"tool_failed": 2, "length": 1} + assert profile.down == 3 + + def test_a_reasonless_down_thumb_still_counts_as_one(self): + from apis.app_api.admin.costs.service import _join_feedback + + profile = _join_feedback([], self._rows("wrong", None)) + assert profile.reasons == {"wrong": 1} + assert profile.down == 2 + + def test_unknown_codes_are_dropped(self): + from apis.app_api.admin.costs.service import _join_feedback + + profile = _join_feedback([], self._rows("from-a-future-schema")) + assert profile.reasons == {} and profile.down == 1 + + def test_up_thumbs_carry_no_reason(self): + from apis.app_api.admin.costs.service import _join_feedback + + profile = _join_feedback([], [{"messageId": 1, "value": 1, "reason": "wrong"}]) + assert profile.reasons == {}, "a reason on an up-thumb is meaningless" + assert profile.up == 1 diff --git a/backend/tests/apis/app_api/admin/costs/test_user_sessions_service.py b/backend/tests/apis/app_api/admin/costs/test_user_sessions_service.py index 811926d4e..dfb2c53c7 100644 --- a/backend/tests/apis/app_api/admin/costs/test_user_sessions_service.py +++ b/backend/tests/apis/app_api/admin/costs/test_user_sessions_service.py @@ -114,7 +114,7 @@ async def test_all_time_drops_the_period_and_the_share(): assert resp.period is None and resp.user_period_cost is None assert resp.sessions[0].share_of_user_period is None # No period → no active_since filter reaches storage. - service.storage.get_user_session_diagnostics.assert_awaited_once_with(user_id="u1", active_since=None) + service.storage.get_user_session_diagnostics.assert_awaited_once_with(user_id="u1", active_since=None, include_deleted=True) service.storage.get_user_cost_summary.assert_not_awaited() @@ -123,7 +123,7 @@ async def test_period_scoping_passes_the_month_start_to_storage(): service = _service([]) await service.get_user_sessions("u1", period="2026-09") service.storage.get_user_session_diagnostics.assert_awaited_once_with( - user_id="u1", active_since="2026-09-01" + user_id="u1", active_since="2026-09-01", include_deleted=True ) @@ -160,3 +160,43 @@ async def test_a_broken_period_cost_lookup_degrades_to_no_share(): resp = await service.get_user_sessions("u1", period="2026-09") assert resp.user_period_cost is None assert resp.sessions[0].share_of_user_period is None + + +@pytest.mark.asyncio +async def test_deleted_conversations_are_listed_flagged_and_rolled_up(): + service = _service( + [ + _row("live", totalCost=0.5), + _row("gone", totalCost=2.25, deleted=True, status="deleted"), + _row("legacy-tombstone", totalCost=1.0, deleted=True, status="active"), + _row("gone-unpriced", totalCost=None, deleted=True, status="deleted"), + ], + period_cost=4.0, + ) + + response = await service.get_user_sessions(user_id="u1", period="2026-09") + + # The storage reader is asked for tombstones explicitly — the default + # reader hides them, which is what left a $20 month showing one $3 row. + service.storage.get_user_session_diagnostics.assert_awaited_once() + assert service.storage.get_user_session_diagnostics.await_args.kwargs["include_deleted"] is True + + by_id = {s.session_id: s for s in response.sessions} + assert by_id["live"].status == "active" + assert by_id["gone"].status == "deleted" + # A legacy tombstone carries `deleted` without the status flip; the page + # gets one normalised signal. + assert by_id["legacy-tombstone"].status == "deleted" + assert response.total == 4 + assert response.deleted_session_count == 3 + assert response.deleted_session_cost == pytest.approx(3.25) + # Deleted rows still take their share of the period total. + assert by_id["gone"].share_of_user_period == pytest.approx(56.25) + + +@pytest.mark.asyncio +async def test_no_deleted_conversations_reports_zero(): + service = _service([_row("live")], period_cost=1.0) + response = await service.get_user_sessions(user_id="u1", period="2026-09") + assert response.deleted_session_count == 0 + assert response.deleted_session_cost == 0.0 diff --git a/backend/tests/apis/app_api/admin/test_feedback_eval_routes.py b/backend/tests/apis/app_api/admin/test_feedback_eval_routes.py new file mode 100644 index 000000000..6f453d94a --- /dev/null +++ b/backend/tests/apis/app_api/admin/test_feedback_eval_routes.py @@ -0,0 +1,76 @@ +"""Admin eval-sampling routes (spec §11 PR-4): + +- GET /admin/feedback/evaluations → 200, content-free queue with verdicts +- POST /admin/feedback/evaluations/run → 202 and a background batch with the injected judge; + 404 while FEEDBACK_EVAL_SAMPLING_ENABLED is off +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from apis.app_api.admin.feedback import routes as feedback_routes +from apis.shared.observability.content_policy import content_bearing_paths +from tests.conftest import override_admin_auth + + +def _app(storage, judge=None): + app = FastAPI() + app.include_router(feedback_routes.router) + override_admin_auth(app, lambda: SimpleNamespace(user_id="admin", email="a@x", roles=["system_admin"])) + app.dependency_overrides[feedback_routes.get_storage] = lambda: storage + app.dependency_overrides[feedback_routes.get_judge] = lambda: judge or SimpleNamespace(judge=lambda *a, **k: []) + return app + + +def test_queue_lists_recent_down_thumbs_with_verdicts(monkeypatch): + monkeypatch.delenv("FEEDBACK_EVAL_SAMPLING_ENABLED", raising=False) + storage = SimpleNamespace(get_recent_down_thumbs=AsyncMock(return_value=[ + {"sessionId": "s1", "messageId": 3, "reason": "wrong", "updatedAt": "t2", "evaluatedAt": "t3", + "evaluation": {"reason": "wrong", "evaluators": ["Builtin.Correctness"], + "scores": {"Builtin.Correctness": {"value": 0.5, "n": 2, "tokens": 100, "rating": "Meh"}}}}, + {"sessionId": "s2", "messageId": 1, "updatedAt": "t1"}, + {"garbage": True}, + ])) + resp = TestClient(_app(storage)).get("/feedback/evaluations", params={"limit": 5}) + assert resp.status_code == 200 + body = resp.json() + assert body["samplingEnabled"] is False and body["pending"] == 1 + assert [i["sessionId"] for i in body["items"]] == ["s1", "s2"] + assert body["items"][0]["evaluation"]["scores"]["Builtin.Correctness"]["rating"] == "Meh" + assert content_bearing_paths(body) == [] + storage.get_recent_down_thumbs.assert_awaited_once_with(limit=5) + + +def test_run_is_404_while_off_and_202_with_a_batch_when_on(monkeypatch): + calls = {} + + async def fake_batch(table, judge, *, limit, cost_row_lookup=None): + calls["limit"] = limit + calls["judge"] = judge + return SimpleNamespace() + + monkeypatch.setattr("apis.shared.feedback_eval.sampler.run_sampling_batch", fake_batch) + storage = SimpleNamespace(sessions_metadata_table=object(), get_recent_down_thumbs=AsyncMock(return_value=[])) + judge = SimpleNamespace(judge=lambda *a, **k: []) + + monkeypatch.setenv("FEEDBACK_EVAL_SAMPLING_ENABLED", "false") + assert TestClient(_app(storage, judge)).post("/feedback/evaluations/run").status_code == 404 + assert calls == {} + + monkeypatch.setenv("FEEDBACK_EVAL_SAMPLING_ENABLED", "true") + resp = TestClient(_app(storage, judge)).post("/feedback/evaluations/run", params={"limit": 7}) + assert resp.status_code == 202 + assert resp.json()["accepted"] is True and resp.json()["limit"] == 7 + # TestClient runs background tasks before returning. + assert calls["limit"] == 7 and calls["judge"] is judge + + +def test_run_limit_is_bounded(monkeypatch): + monkeypatch.setenv("FEEDBACK_EVAL_SAMPLING_ENABLED", "true") + storage = SimpleNamespace(sessions_metadata_table=object()) + assert TestClient(_app(storage)).post("/feedback/evaluations/run", params={"limit": 500}).status_code == 422 diff --git a/backend/tests/apis/app_api/admin/test_feedback_fleet.py b/backend/tests/apis/app_api/admin/test_feedback_fleet.py new file mode 100644 index 000000000..863e75dde --- /dev/null +++ b/backend/tests/apis/app_api/admin/test_feedback_fleet.py @@ -0,0 +1,212 @@ +"""Fleet-level feedback attribution (response-feedback spec §7, compaction +spec §7.2): the arm math, and the route that assembles it. + +The rules under test are §9's, because they are what stop this surface +becoming the quality KPI the spec forbids: every arm carries its n, an arm +below the floor reports no rate at all, coverage is reported rather than +implied, and no fleet-wide rate field exists. +""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from apis.app_api.admin.feedback import fleet, routes as feedback_routes +from apis.shared.observability.content_policy import content_bearing_paths +from tests.conftest import override_admin_auth + +WINDOW = {"start": "2026-09-01T00:00:00Z", "end": "2026-09-18T00:00:00Z", "days": 17} + + +def _thumb(session_id, message_id, value, reason=None, signal="explicit"): + row = {"sessionId": session_id, "messageId": message_id, "value": value, "signal": signal} + if reason: + row["reason"] = reason + return row + + +def _call(message_id, *, model="haiku", switched=False, compaction=False, has_documents=None): + row = {"messageId": message_id, "modelInfo": {"modelId": model}, "agentSwitched": switched} + if compaction: + row["compactionEvents"] = [{"kind": "applied"}] + if has_documents is not None: + row["hasDocuments"] = has_documents + return row + + +# ── arm math ─────────────────────────────────────────────────────────────── + + +def test_an_arm_below_the_floor_reports_no_rate_at_all(): + arm = fleet.ArmCounts(up=1, down=1) + assert arm.to_dict(floor=20) == {"up": 1, "down": 1, "n": 2, "downRate": None, "belowFloor": True} + big = fleet.ArmCounts(up=15, down=5) + assert big.to_dict(floor=20) == {"up": 15, "down": 5, "n": 20, "downRate": 0.25, "belowFloor": False} + + +def test_the_floor_is_env_tunable_but_never_zero(monkeypatch): + monkeypatch.delenv("FEEDBACK_ARM_MINIMUM_N", raising=False) + assert fleet.minimum_n() == fleet.DEFAULT_MINIMUM_N + monkeypatch.setenv("FEEDBACK_ARM_MINIMUM_N", "5") + assert fleet.minimum_n() == 5 + monkeypatch.setenv("FEEDBACK_ARM_MINIMUM_N", "0") + assert fleet.minimum_n() == 1 + monkeypatch.setenv("FEEDBACK_ARM_MINIMUM_N", "nonsense") + assert fleet.minimum_n() == fleet.DEFAULT_MINIMUM_N + + +def test_distance_from_the_last_compaction_cut_buckets_by_calls(): + # positions: 0 1 2 3 4 5 + records = [_call(0), _call(1, compaction=True), _call(2), _call(3), _call(4), _call(5)] + index = fleet.index_session(records) + assert fleet.calls_since_compaction(index, 0) == "never" # before any cut + assert fleet.calls_since_compaction(index, 1) == "same call" + assert fleet.calls_since_compaction(index, 2) == "1-3 calls" + assert fleet.calls_since_compaction(index, 4) == "1-3 calls" + assert fleet.calls_since_compaction(index, 5) == "4+ calls" + # A session that was never compacted is the control arm, not an omission. + plain = fleet.index_session([_call(0), _call(1)]) + assert fleet.calls_since_compaction(plain, 1) == "never" + # An unknown message cannot be placed. + assert fleet.calls_since_compaction(index, 99) == "never" + + +def test_a_turns_tool_round_trips_share_a_message_id_and_the_last_row_wins(): + records = [_call(3, model="haiku"), _call(3, model="sonnet", compaction=True)] + index = fleet.index_session(records) + assert index.by_message[3]["modelInfo"]["modelId"] == "sonnet" + assert index.position[3] == 1 + + +def test_report_splits_every_dimension_and_counts_what_it_could_not_join(): + thumbs = [ + _thumb("s1", 0, -1, "wrong"), + _thumb("s1", 1, 1), + _thumb("s1", 2, -1, "instructions"), + _thumb("s2", 0, -1, "wrong"), + _thumb("s9", 0, -1), # session never joined + _thumb("s1", 77, 1), # no cost row for that message + _thumb("s1", 3, -1, signal="implicit"), # §10 row: never a thumb + ] + records = { + "s1": [_call(0, model="haiku"), _call(1, model="haiku", compaction=True), _call(2, model="sonnet", switched=True)], + "s2": [_call(0, model="sonnet", has_documents=True)], + } + report = fleet.build_fleet_report(thumbs, records, window=WINDOW, floor=1) + + assert report["totals"] == {"up": 2, "down": 4, "thumbs": 6} + assert report["coverage"]["joined"] == 4 + assert report["coverage"]["unjoined"] == 2 + assert report["coverage"]["joinRate"] == round(4 / 6, 4) + assert report["coverage"]["sessionsWithFeedback"] == 3 + assert report["coverage"]["sessionsJoined"] == 2 + + by_key = {a["key"]: a for a in report["arms"]["model"]} + assert by_key["haiku"]["n"] == 2 and by_key["haiku"]["down"] == 1 + assert by_key["sonnet"]["n"] == 2 and by_key["sonnet"]["down"] == 2 + switch = {a["key"]: a for a in report["arms"]["agentSwitch"]} + assert switch["switched agent"]["down"] == 1 and switch["same agent"]["n"] == 3 + turn = {a["key"]: a for a in report["arms"]["turnClass"]} + assert turn["full"]["down"] == 1 + assert report["reasons"] == {"wrong": 2, "instructions": 1, "unspecified": 1} + assert report["minimumN"] == 1 + + +def test_no_fleet_wide_rate_field_exists_to_be_quoted(): + report = fleet.build_fleet_report([_thumb("s1", 0, -1)], {}, window=WINDOW, floor=1) + assert "downRate" not in report + assert "downRate" not in report["totals"] + assert "score" not in repr(report) + + +def test_compaction_arm_keeps_distance_order_others_lead_with_the_biggest_sample(): + thumbs = [_thumb("s1", i, -1) for i in range(6)] + [_thumb("s1", 6, 1)] + records = {"s1": [_call(0, compaction=True), _call(1), _call(2), _call(3), _call(4), _call(5), _call(6, model="zzz")]} + report = fleet.build_fleet_report(thumbs, records, window=WINDOW, floor=1) + assert [a["key"] for a in report["arms"]["callsSinceCompaction"]] == ["same call", "1-3 calls", "4+ calls"] + assert [a["key"] for a in report["arms"]["model"]][0] == "haiku" # 6 thumbs beats zzz's 1 + + +def test_an_empty_window_is_zeroes_not_a_crash(): + report = fleet.build_fleet_report([], {}, window=WINDOW) + assert report["totals"] == {"up": 0, "down": 0, "thumbs": 0} + assert report["coverage"]["joinRate"] is None + assert all(arm == [] for arm in report["arms"].values()) + + +# ── route ────────────────────────────────────────────────────────────────── + + +def _app(storage): + app = FastAPI() + app.include_router(feedback_routes.router) + override_admin_auth(app, lambda: SimpleNamespace(user_id="admin", email="a@x", roles=["system_admin"])) + app.dependency_overrides[feedback_routes.get_storage] = lambda: storage + app.dependency_overrides[feedback_routes.get_judge] = lambda: SimpleNamespace(judge=lambda *a, **k: []) + return app + + +def test_fleet_route_windows_joins_and_stays_content_free(): + thumbs = [_thumb("s1", 0, -1, "wrong"), _thumb("s1", 1, 1), _thumb("s2", 0, -1, "length")] + storage = SimpleNamespace( + get_feedback_in_window=AsyncMock(return_value=thumbs), + get_session_cost_records=AsyncMock(side_effect=lambda sid: { + "s1": [_call(0, model="haiku"), _call(1, model="haiku", compaction=True)], + "s2": [_call(0, model="sonnet")], + }.get(sid, [])), + ) + resp = TestClient(_app(storage)).get("/feedback/fleet", params={"days": 7}) + + assert resp.status_code == 200 + body = resp.json() + assert body["window"]["days"] == 7 + assert body["totals"] == {"up": 1, "down": 2, "thumbs": 3} + assert body["coverage"]["joined"] == 3 and body["coverage"]["truncated"] is False + assert {a["key"] for a in body["arms"]["model"]} == {"haiku", "sonnet"} + # Below the default floor of 20, every arm reports counts but no rate. + assert all(a["downRate"] is None and a["belowFloor"] for a in body["arms"]["model"]) + assert body["minimumN"] == 20 + assert content_bearing_paths(body) == [] + # The window passed to storage is the one reported back. + kwargs = storage.get_feedback_in_window.await_args.kwargs + assert kwargs["start"] == body["window"]["start"] and kwargs["end"] == body["window"]["end"] + + +def test_fleet_route_reports_truncation_and_omitted_sessions(monkeypatch): + monkeypatch.setattr(feedback_routes, "MAX_THUMBS_PER_WINDOW", 3) + monkeypatch.setattr(feedback_routes, "MAX_SESSIONS_JOINED", 2) + thumbs = [_thumb(f"s{i}", 0, -1) for i in range(3)] + storage = SimpleNamespace( + get_feedback_in_window=AsyncMock(return_value=thumbs), + get_session_cost_records=AsyncMock(return_value=[_call(0)]), + ) + body = TestClient(_app(storage)).get("/feedback/fleet").json() + assert body["coverage"]["truncated"] is True + assert body["coverage"]["sessionsOmitted"] == 1 + assert body["coverage"]["sessionsJoined"] == 2 + assert body["coverage"]["unjoined"] == 1 + + +def test_a_session_whose_rows_fail_to_read_degrades_to_unjoined(): + storage = SimpleNamespace( + get_feedback_in_window=AsyncMock(return_value=[_thumb("s1", 0, -1), _thumb("s2", 0, -1)]), + get_session_cost_records=AsyncMock(side_effect=lambda sid: (_ for _ in ()).throw(RuntimeError("boom")) + if sid == "s2" else [_call(0)]), + ) + body = TestClient(_app(storage)).get("/feedback/fleet").json() + assert body["coverage"]["joined"] == 1 and body["coverage"]["unjoined"] == 1 + + +def test_fleet_route_bounds_the_window_and_maps_failure_to_500(): + storage = SimpleNamespace(get_feedback_in_window=AsyncMock(return_value=[]), get_session_cost_records=AsyncMock(return_value=[])) + client = TestClient(_app(storage)) + assert client.get("/feedback/fleet", params={"days": 0}).status_code == 422 + assert client.get("/feedback/fleet", params={"days": 400}).status_code == 422 + + storage.get_feedback_in_window = AsyncMock(side_effect=RuntimeError("dynamo down")) + assert TestClient(_app(storage)).get("/feedback/fleet").status_code == 500 diff --git a/backend/tests/apis/app_api/test_browser_live_view_route.py b/backend/tests/apis/app_api/test_browser_live_view_route.py new file mode 100644 index 000000000..d3a498f9c --- /dev/null +++ b/backend/tests/apis/app_api/test_browser_live_view_route.py @@ -0,0 +1,293 @@ +"""`POST /sessions/{id}/browser/live-view` — mint a Live View URL (spec D2). + +The route exists because a Live View URL is SigV4 *query*-signed and lives at +most 300 seconds: it cannot be minted once and reused, so the client sends only +a conversation id and app-api signs a fresh one per call. + +What these hold down: + +* **The client never names the browser session.** It sends a conversation id; + the browser session is resolved server-side from the metadata row. A route + that accepted a browser session id would let any signed-in user stream any + browser in the account. +* **Ownership is the metadata read**, which is user-scoped, so another user's + conversation is indistinguishable from a missing one. +* **404 and 409 mean different things** — "no viewer here" vs "the session + ended" — and the SPA says different things for each. +* **The signed URL never reaches a log line.** + +Auth is the cookie-aware `get_current_user_from_session`, per the SPA-facing +route rule in CLAUDE.md. +""" + +from __future__ import annotations + +import logging +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from apis.app_api.sessions import routes as session_routes +from apis.app_api.sessions.services import browser_live_view +from apis.app_api.sessions.services.browser_live_view import LiveViewUnavailable +from apis.shared.auth.dependencies import get_current_user_from_session +from apis.shared.auth.models import User + +SIGNED_URL = ( + "https://bedrock-agentcore.us-west-2.amazonaws.com/browser-streams/b-1/" + "sessions/bs-1/live?X-Amz-Signature=deadbeefdeadbeef&X-Amz-Expires=300" +) + +REF = { + "browserSessionId": "bs-1", + "browserId": "browser-abc", + "viewport": {"width": 1280, "height": 800}, + "controlState": "user", +} + + +def _user() -> User: + return User( + user_id="user-1", email="u@example.com", name="U", + roles=["default"], raw_token="tok", + ) + + +def _client(monkeypatch, *, metadata=..., mint=None) -> TestClient: + monkeypatch.delenv("BROWSER_TAKEOVER_ENABLED", raising=False) + if metadata is ...: + metadata = SimpleNamespace(browser_session=dict(REF)) + monkeypatch.setattr( + session_routes, "get_session_metadata", AsyncMock(return_value=metadata) + ) + monkeypatch.setattr( + browser_live_view, + "mint_live_view", + mint + or AsyncMock( + return_value={ + "url": SIGNED_URL, + "expiresAt": "2026-09-18T12:05:00+00:00", + "viewport": {"width": 1280, "height": 800}, + "controlState": "user", + } + ), + ) + app = FastAPI() + app.include_router(session_routes.router) + app.dependency_overrides[get_current_user_from_session] = _user + return TestClient(app) + + +class TestMinting: + def test_returns_a_fresh_url_with_its_expiry_and_viewport(self, monkeypatch): + client = _client(monkeypatch) + + resp = client.post("/sessions/s1/browser/live-view") + + assert resp.status_code == 200 + body = resp.json() + assert body["url"] == SIGNED_URL + assert body["expiresAt"] == "2026-09-18T12:05:00+00:00" + # Carried, not re-declared in the viewer: DCV crops on a mismatch. + assert body["viewport"] == {"width": 1280, "height": 800} + assert body["controlState"] == "user" + + def test_the_client_sends_only_a_conversation_id(self, monkeypatch): + mint = AsyncMock( + return_value={ + "url": SIGNED_URL, + "expiresAt": "t", + "viewport": {"width": 1280, "height": 800}, + "controlState": "user", + } + ) + client = _client(monkeypatch, mint=mint) + + client.post("/sessions/s1/browser/live-view") + + # The browser session came from the metadata row, not the request. A + # route that took it from the client would let any signed-in user + # stream any browser session in the account. + assert mint.await_args.args[0]["browserSessionId"] == "bs-1" + + def test_the_lookup_is_scoped_to_the_calling_user(self, monkeypatch): + client = _client(monkeypatch) + + client.post("/sessions/s1/browser/live-view") + + lookup = session_routes.get_session_metadata.await_args + assert lookup.args == ("s1", "user-1") + + +class TestRefusals: + def test_unknown_or_other_users_conversation_is_404(self, monkeypatch): + client = _client(monkeypatch, metadata=None) + + assert client.post("/sessions/s1/browser/live-view").status_code == 404 + + def test_a_conversation_with_no_browser_session_is_404(self, monkeypatch): + client = _client(monkeypatch, metadata=SimpleNamespace(browser_session=None)) + + resp = client.post("/sessions/s1/browser/live-view") + + assert resp.status_code == 404 + assert "no browser session" in resp.json()["detail"] + + def test_an_ended_session_is_409_not_404(self, monkeypatch): + # Distinct on purpose: the SPA says "the session ended", not "no + # viewer here". Collapsing them would make a timed-out takeover look + # like a missing feature. + mint = AsyncMock( + side_effect=LiveViewUnavailable("The browser session is gone.", code=409) + ) + client = _client(monkeypatch, mint=mint) + + resp = client.post("/sessions/s1/browser/live-view") + + assert resp.status_code == 409 + assert "gone" in resp.json()["detail"] + + def test_404_while_the_kill_switch_is_off(self, monkeypatch): + client = _client(monkeypatch) + monkeypatch.setenv("BROWSER_TAKEOVER_ENABLED", "false") + + assert client.post("/sessions/s1/browser/live-view").status_code == 404 + + def test_the_kill_switch_short_circuits_before_any_lookup(self, monkeypatch): + client = _client(monkeypatch) + monkeypatch.setenv("BROWSER_TAKEOVER_ENABLED", "false") + session_routes.get_session_metadata.reset_mock() + + client.post("/sessions/s1/browser/live-view") + + session_routes.get_session_metadata.assert_not_awaited() + + def test_a_signing_failure_does_not_leak_its_message(self, monkeypatch): + # The exception text from a partially-built signed URL can contain the + # URL itself. + mint = AsyncMock(side_effect=RuntimeError(f"failed building {SIGNED_URL}")) + client = _client(monkeypatch, mint=mint) + + resp = client.post("/sessions/s1/browser/live-view") + + assert resp.status_code == 500 + assert "X-Amz-Signature" not in resp.text + + def test_it_is_a_post_so_the_signature_stays_out_of_history(self, monkeypatch): + client = _client(monkeypatch) + + assert client.get("/sessions/s1/browser/live-view").status_code == 405 + + +class TestLogging: + def test_the_signed_url_never_reaches_the_logs(self, monkeypatch, caplog): + client = _client(monkeypatch) + + with caplog.at_level(logging.DEBUG): + client.post("/sessions/s1/browser/live-view") + + assert "X-Amz-Signature" not in caplog.text + assert SIGNED_URL not in caplog.text + + def test_the_unavailable_log_describes_the_session_without_a_url( + self, monkeypatch, caplog + ): + mint = AsyncMock(side_effect=LiveViewUnavailable("gone", code=409)) + client = _client(monkeypatch, mint=mint) + + with caplog.at_level(logging.INFO): + client.post("/sessions/s1/browser/live-view") + + assert "bs-1" in caplog.text + assert "X-Amz-Signature" not in caplog.text + + +class TestMintLiveView: + """The service, against a fake `BrowserClient`.""" + + @pytest.fixture + def fake_client(self, monkeypatch): + calls = SimpleNamespace(expires=None, identifier=None, session_id=None) + + class FakeBrowserClient: + def __init__(self, **_): + self.identifier = None + self.session_id = None + + def generate_live_view_url(self, expires): + calls.expires = expires + calls.identifier = self.identifier + calls.session_id = self.session_id + return SIGNED_URL + + monkeypatch.setattr( + "bedrock_agentcore.tools.browser_client.BrowserClient", FakeBrowserClient + ) + return calls + + @pytest.mark.asyncio + async def test_signs_against_the_referenced_session(self, fake_client): + minted = await browser_live_view.mint_live_view(dict(REF)) + + assert minted["url"] == SIGNED_URL + assert fake_client.identifier == "browser-abc" + assert fake_client.session_id == "bs-1" + + @pytest.mark.asyncio + async def test_never_asks_for_more_than_the_services_cap(self, fake_client): + # `generate_live_view_url` RAISES above 300 rather than clamping, so + # an over-large env value must be clamped here or every mint fails. + await browser_live_view.mint_live_view(dict(REF)) + + assert fake_client.expires <= browser_live_view.MAX_EXPIRY_SECONDS + + @pytest.mark.asyncio + async def test_a_missing_reference_is_404_not_a_crash(self, fake_client): + with pytest.raises(LiveViewUnavailable) as exc: + await browser_live_view.mint_live_view({"viewport": {"width": 1, "height": 1}}) + + assert exc.value.code == 404 + + @pytest.mark.asyncio + async def test_a_signing_refusal_reads_as_409(self, monkeypatch): + class Boom: + def __init__(self, **_): + self.identifier = None + self.session_id = None + + def generate_live_view_url(self, expires): + raise RuntimeError("ResourceNotFoundException") + + monkeypatch.setattr( + "bedrock_agentcore.tools.browser_client.BrowserClient", Boom + ) + + with pytest.raises(LiveViewUnavailable) as exc: + await browser_live_view.mint_live_view(dict(REF)) + + assert exc.value.code == 409 + + @pytest.mark.asyncio + async def test_an_unusable_viewport_falls_back_rather_than_failing( + self, fake_client + ): + minted = await browser_live_view.mint_live_view( + {**REF, "viewport": {"width": "wide"}} + ) + + assert minted["viewport"] == browser_live_view.DEFAULT_VIEWPORT + + @pytest.mark.asyncio + async def test_expiry_is_measured_from_the_request_not_the_read(self, fake_client): + from datetime import datetime, timezone + + before = datetime.now(timezone.utc) + minted = await browser_live_view.mint_live_view(dict(REF)) + expires = datetime.fromisoformat(minted["expiresAt"]) + + window = (expires - before).total_seconds() + assert 0 < window <= browser_live_view.LIVE_VIEW_EXPIRY_SECONDS + 1 diff --git a/backend/tests/apis/app_api/test_file_digest_scheduling.py b/backend/tests/apis/app_api/test_file_digest_scheduling.py new file mode 100644 index 000000000..3916f9992 --- /dev/null +++ b/backend/tests/apis/app_api/test_file_digest_scheduling.py @@ -0,0 +1,119 @@ +"""DocumentDigest scheduling on upload completion (files/service.py). + +``complete_upload`` queues a digest build for document uploads only, off the +request path, with a strong task reference; the build reads S3, calls the +builder and persists the map; nothing on this path can fail the upload or +raise out of the task. +""" + +from __future__ import annotations + +import asyncio +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from apis.app_api.files.service import FileUploadService +from apis.shared.files.document_digest import DocumentDigest +from apis.shared.files.models import FileMetadata, FileStatus + +DOCX = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + + +def _meta(mime="application/pdf", filename="policy.pdf", status=FileStatus.PENDING): + return FileMetadata( + upload_id="up-1", user_id="u1", session_id="s1", filename=filename, mime_type=mime, + size_bytes=10, s3_key="k", s3_bucket="b", status=status, + ) + + +def _service(meta, body=b"%PDF"): + repository = MagicMock() + repository.get_file = AsyncMock(return_value=meta) + repository.update_file_status = AsyncMock() + repository.increment_quota = AsyncMock() + repository.update_file_digest = AsyncMock(return_value=meta) + s3 = MagicMock() + s3.get_object.return_value = {"Body": MagicMock(read=MagicMock(return_value=body))} + return FileUploadService(repository=repository, s3_client=s3, bucket_name="b") + + +@pytest.fixture(autouse=True) +def _flag(monkeypatch): + monkeypatch.delenv("DOCUMENT_DIGEST_ENABLED", raising=False) + + +async def _drain(service): + tasks = list(service._digest_tasks) + if tasks: + await asyncio.gather(*tasks) + return tasks + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "mime,filename,scheduled", + [ + ("application/pdf", "a.pdf", True), + (DOCX, "a.docx", True), + ("text/markdown", "a.md", True), + ("text/csv", "a.csv", False), + ("image/png", "a.png", False), + ("application/vnd.openxmlformats-officedocument.presentationml.presentation", "a.pptx", False), + ], +) +async def test_complete_upload_schedules_a_digest_for_documents_only(monkeypatch, mime, filename, scheduled): + built = DocumentDigest(status="ready", format="pdf", count=1, tokens=50) + build = AsyncMock(return_value=built) + monkeypatch.setattr("apis.shared.files.document_digest.build_digest", build) + service = _service(_meta(mime, filename)) + + response = await service.complete_upload("u1", "up-1") + assert response.status == "ready" + tasks = await _drain(service) + + assert bool(tasks) is scheduled + if scheduled: + build.assert_awaited_once() + assert build.await_args.kwargs["raw"] == b"%PDF" + service.repository.update_file_digest.assert_awaited_once_with("u1", "up-1", built.to_item()) + else: + service.repository.update_file_digest.assert_not_awaited() + assert not service._digest_tasks # done-callback dropped the strong ref + + +@pytest.mark.asyncio +async def test_kill_switch_skips_scheduling(monkeypatch): + monkeypatch.setenv("DOCUMENT_DIGEST_ENABLED", "false") + service = _service(_meta()) + await service.complete_upload("u1", "up-1") + assert service._schedule_digest(_meta()) is None and not service._digest_tasks + + +@pytest.mark.asyncio +async def test_build_failures_never_escape_the_task(monkeypatch): + monkeypatch.setattr("apis.shared.files.document_digest.build_digest", AsyncMock(side_effect=RuntimeError("boom"))) + service = _service(_meta()) + await service.complete_upload("u1", "up-1") + await _drain(service) # gather would re-raise if the task did + service.repository.update_file_digest.assert_not_awaited() + + service.repository.get_file = AsyncMock(return_value=_meta()) + service._s3_client.get_object.side_effect = RuntimeError("s3 down") + monkeypatch.setattr("apis.shared.files.document_digest.build_digest", AsyncMock()) + await service.complete_upload("u1", "up-1") + await _drain(service) + + +@pytest.mark.asyncio +async def test_a_deleted_row_is_tolerated(monkeypatch): + monkeypatch.setattr("apis.shared.files.document_digest.build_digest", AsyncMock(return_value=DocumentDigest())) + service = _service(_meta()) + service.repository.update_file_digest = AsyncMock(return_value=None) + await service.complete_upload("u1", "up-1") + await _drain(service) + + +def test_scheduling_without_a_running_loop_is_a_noop(): + service = _service(_meta()) + assert service._schedule_digest(_meta()) is None diff --git a/backend/tests/apis/app_api/test_message_feedback_routes.py b/backend/tests/apis/app_api/test_message_feedback_routes.py new file mode 100644 index 000000000..3b6f60158 --- /dev/null +++ b/backend/tests/apis/app_api/test_message_feedback_routes.py @@ -0,0 +1,113 @@ +"""Routes for the thumbs signal: + +- PUT /sessions/{id}/messages/{messageId}/feedback → 200 with the stored thumb +- DELETE /sessions/{id}/messages/{messageId}/feedback → 204 +- 404 for another user's session, 404 while RESPONSE_FEEDBACK_ENABLED=false, + 422 for anything but ±1 / a fixed reason code, 503 with no table. + +Auth is the cookie-aware `get_current_user_from_session` (SPA-facing route); +storage is swapped through the module's own function names. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from apis.app_api.sessions import routes as session_routes +from apis.shared.auth.dependencies import get_current_user_from_session +from apis.shared.auth.models import User +from apis.shared.sessions.feedback import SessionNotOwned +from apis.shared.sessions.models import MessageFeedback + + +def _user() -> User: + return User(user_id="user-1", email="u@example.com", name="U", roles=["default"], raw_token="tok") + + +def _client(monkeypatch, put=None, delete=None, signal=None) -> TestClient: + monkeypatch.delenv("RESPONSE_FEEDBACK_ENABLED", raising=False) + monkeypatch.setattr(session_routes, "record_implicit_signal", signal or AsyncMock(return_value=None)) + monkeypatch.setattr(session_routes, "put_message_feedback", put or AsyncMock( + return_value=MessageFeedback(value=1, updated_at="2026-09-16T00:00:00Z") + )) + monkeypatch.setattr(session_routes, "delete_message_feedback", delete or AsyncMock(return_value=True)) + app = FastAPI() + app.include_router(session_routes.router) + app.dependency_overrides[get_current_user_from_session] = _user + return TestClient(app) + + +def test_put_stores_and_echoes_the_thumb(monkeypatch): + put = AsyncMock(return_value=MessageFeedback(value=-1, reason="instructions", updated_at="2026-09-16T00:00:00Z")) + client = _client(monkeypatch, put=put) + + resp = client.put("/sessions/s1/messages/3/feedback", json={"value": -1, "reason": "instructions"}) + + assert resp.status_code == 200 + assert resp.json() == {"value": -1, "reason": "instructions", "updatedAt": "2026-09-16T00:00:00Z"} + put.assert_awaited_once_with(session_id="s1", user_id="user-1", message_id=3, value=-1, reason="instructions", retry_message_id=None) + + +def test_put_forwards_the_retry_link(monkeypatch): + put = AsyncMock(return_value=MessageFeedback(value=-1, retry_message_id=4, updated_at="t")) + client = _client(monkeypatch, put=put) + resp = client.put("/sessions/s1/messages/3/feedback", json={"value": -1, "retryMessageId": 4}) + assert resp.status_code == 200 and resp.json()["retryMessageId"] == 4 + assert put.await_args.kwargs["retry_message_id"] == 4 + assert client.put("/sessions/s1/messages/3/feedback", json={"value": -1, "retryMessageId": -1}).status_code == 422 + + +def test_put_rejects_free_text_and_out_of_range_values(monkeypatch): + client = _client(monkeypatch) + assert client.put("/sessions/s1/messages/3/feedback", json={"value": 1, "reason": "it lied to me"}).status_code == 422 + assert client.put("/sessions/s1/messages/3/feedback", json={"value": 0}).status_code == 422 + assert client.put("/sessions/s1/messages/3/feedback", json={"value": 5}).status_code == 422 + assert client.put("/sessions/s1/messages/three/feedback", json={"value": 1}).status_code == 422 + assert client.put("/sessions/s1/messages/-1/feedback", json={"value": 1}).status_code == 422 + + +def test_delete_returns_204(monkeypatch): + delete = AsyncMock(return_value=True) + client = _client(monkeypatch, delete=delete) + resp = client.delete("/sessions/s1/messages/3/feedback") + assert resp.status_code == 204 + delete.assert_awaited_once_with(session_id="s1", user_id="user-1", message_id=3) + + +def test_another_users_session_is_404(monkeypatch): + client = _client( + monkeypatch, + put=AsyncMock(side_effect=SessionNotOwned("s1")), + delete=AsyncMock(side_effect=SessionNotOwned("s1")), + ) + assert client.put("/sessions/s1/messages/3/feedback", json={"value": 1}).status_code == 404 + assert client.delete("/sessions/s1/messages/3/feedback").status_code == 404 + + +def test_kill_switch_hides_the_surface(monkeypatch): + put = AsyncMock() + client = _client(monkeypatch, put=put) + monkeypatch.setenv("RESPONSE_FEEDBACK_ENABLED", "false") + assert client.put("/sessions/s1/messages/3/feedback", json={"value": 1}).status_code == 404 + assert client.delete("/sessions/s1/messages/3/feedback").status_code == 404 + put.assert_not_awaited() + + +def test_implicit_signal_is_fire_and_forget_204(monkeypatch): + signal = AsyncMock(return_value=None) + client = _client(monkeypatch, signal=signal) + assert client.post("/sessions/s1/messages/3/signals", json={"kind": "copy"}).status_code == 204 + signal.assert_awaited_once_with(session_id="s1", user_id="user-1", message_id=3, kind="copy") + assert client.post("/sessions/s1/messages/3/signals", json={"kind": "abandon"}).status_code == 422 + assert client.post("/sessions/s1/messages/3/signals", json={"kind": "I copied it"}).status_code == 422 + monkeypatch.setenv("RESPONSE_FEEDBACK_ENABLED", "false") + assert client.post("/sessions/s1/messages/3/signals", json={"kind": "copy"}).status_code == 404 + + +def test_missing_table_is_503_not_500(monkeypatch): + client = _client(monkeypatch, put=AsyncMock(side_effect=RuntimeError("DYNAMODB_SESSIONS_METADATA_TABLE_NAME"))) + assert client.put("/sessions/s1/messages/3/feedback", json={"value": 1}).status_code == 503 diff --git a/backend/tests/apis/app_api/test_middleware_stack.py b/backend/tests/apis/app_api/test_middleware_stack.py new file mode 100644 index 000000000..b82a4b219 --- /dev/null +++ b/backend/tests/apis/app_api/test_middleware_stack.py @@ -0,0 +1,74 @@ +"""The app-api middleware stack is ordered on purpose — lock the order down. + +`main.py` documents the intended request-side order: + + ProxiedRedirect → GZip → SessionRefresh → CSRF → AgentCoreContext + → CORS → router + +Starlette's `add_middleware` prepends, so that order is the reverse of the +call order in the module and easy to break by moving a call. Two placements +in particular carry a reason: + +* ProxiedRedirect stays outermost, as its own docstring requires — it has to + see the final response headers. +* GZip sits directly inside it, so it compresses everything the app emits + (including the error bodies CSRF and SessionRefresh return) while leaving + ProxiedRedirect's `Location` rewriting untouched. +""" + +from __future__ import annotations + +from fastapi.middleware.cors import CORSMiddleware +from fastapi.testclient import TestClient + +from apis.app_api.main import app +from apis.shared.middleware.agentcore_context import AgentCoreContextMiddleware +from apis.shared.middleware.compression import StreamSafeGZipMiddleware +from apis.shared.middleware.csrf import CSRFMiddleware +from apis.shared.middleware.proxied_redirect import ( + FORWARDED_PREFIX_HEADER, + ProxiedRedirectMiddleware, +) +from apis.shared.middleware.session_refresh import SessionRefreshMiddleware + +#: Outermost first — `app.user_middleware` is in request order. +EXPECTED_ORDER = [ + ProxiedRedirectMiddleware, + StreamSafeGZipMiddleware, + SessionRefreshMiddleware, + CSRFMiddleware, + AgentCoreContextMiddleware, + CORSMiddleware, +] + + +def test_middleware_order_matches_the_documented_stack() -> None: + assert [m.cls for m in app.user_middleware] == EXPECTED_ORDER + + +def test_compression_is_configured_for_json_not_for_cpu_burn() -> None: + gzip_middleware = next( + m for m in app.user_middleware if m.cls is StreamSafeGZipMiddleware + ) + + # Level 9 (Starlette's default) costs ~4x the CPU of level 6 for ~3% + # more compression on this repo's largest payloads. + assert gzip_middleware.kwargs["compresslevel"] == 6 + assert gzip_middleware.kwargs["minimum_size"] == 500 + + +def test_redirect_rewriting_survives_the_compression_layer() -> None: + """ProxiedRedirect still owns `Location` with GZip underneath it.""" + client = TestClient(app, follow_redirects=False) + + response = client.get( + "/agents/", + headers={ + "Host": "api.dev.boisestate.ai", + "Accept-Encoding": "gzip", + FORWARDED_PREFIX_HEADER: "/api", + }, + ) + + assert response.status_code == 307 + assert response.headers["location"] == "/api/agents" diff --git a/backend/tests/apis/app_api/test_sheet_preview.py b/backend/tests/apis/app_api/test_sheet_preview.py new file mode 100644 index 000000000..ea935694f --- /dev/null +++ b/backend/tests/apis/app_api/test_sheet_preview.py @@ -0,0 +1,251 @@ +"""Tests for the .xlsx reader behind GET /files/{id}/sheet-preview.""" + +from datetime import datetime +from io import BytesIO + +import pytest +from openpyxl import Workbook + +from apis.app_api.files.sheet_preview import ( + MAX_COLUMNS_PER_SHEET, + MAX_ROWS_PER_SHEET, + MAX_SHEETS, + MAX_WORKBOOK_BYTES, + WorkbookTooLargeError, + WorkbookUnreadableError, + read_workbook_preview, +) + + +def build(*, sheets: dict[str, list[list]] | None = None) -> bytes: + """Serialise a workbook the way openpyxl itself would write one.""" + wb = Workbook() + wb.remove(wb.active) + for name, rows in (sheets or {"Sheet1": [["a"], [1]]}).items(): + ws = wb.create_sheet(title=name) + for row in rows: + ws.append(row) + buffer = BytesIO() + wb.save(buffer) + return buffer.getvalue() + + +class TestBasicReading: + def test_reads_headers_and_rows(self): + data = build(sheets={"Budget": [["Item", "Cost"], ["Rent", 1200], ["Food", 300]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.name == "Budget" + assert sheet.headers == ["Item", "Cost"] + assert sheet.rows == [["Rent", "1200"], ["Food", "300"]] + assert sheet.truncated is False + + def test_reads_every_visible_sheet_in_order(self): + data = build( + sheets={ + "First": [["a"], ["1"]], + "Second": [["b"], ["2"]], + } + ) + + sheets = read_workbook_preview(data) + + assert [s.name for s in sheets] == ["First", "Second"] + + def test_skips_hidden_sheets(self): + # Excel does not show them either; surfacing a scratch sheet would + # misrepresent the workbook. + wb = Workbook() + wb.remove(wb.active) + visible = wb.create_sheet(title="Visible") + visible.append(["a"]) + visible.append([1]) + hidden = wb.create_sheet(title="Scratch") + hidden.append(["secret"]) + hidden.sheet_state = "hidden" + buffer = BytesIO() + wb.save(buffer) + + sheets = read_workbook_preview(buffer.getvalue()) + + assert [s.name for s in sheets] == ["Visible"] + + def test_substitutes_a_column_letter_for_a_blank_header(self): + data = build(sheets={"S": [[None, "name"], [1, "widget"]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.headers == ["A", "name"] + + def test_pads_a_short_row_to_the_grid_width(self): + data = build(sheets={"S": [["a", "b", "c"], [1, 2]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [["1", "2", ""]] + + def test_reports_an_empty_sheet_without_failing(self): + data = build(sheets={"Empty": []}) + + [sheet] = read_workbook_preview(data) + + assert sheet.headers == [] + assert sheet.rows == [] + + +class TestFormulas: + """The trap that motivated reading the workbook twice.""" + + def test_shows_formula_text_when_no_value_was_cached(self): + # openpyxl performs no evaluation, so every formula in a workbook + # it wrote reads back as None under data_only=True. A values-only + # preview would show a blank exactly where the total belongs. + wb = Workbook() + ws = wb.active + ws.append(["Item", "Cost"]) + ws.append(["Rent", 1200]) + ws.append(["Total", "=SUM(B2:B2)"]) + buffer = BytesIO() + wb.save(buffer) + + [sheet] = read_workbook_preview(buffer.getvalue()) + + assert sheet.rows[1] == ["Total", "=SUM(B2:B2)"] + + def test_prefers_a_cached_value_over_the_formula(self): + # A workbook saved by Excel carries the computed value, which is + # what the user should see. + wb = Workbook() + wb.active.append(["total"]) + wb.active.append(["=SUM(A1:A1)"]) + buffer = BytesIO() + wb.save(buffer) + without_cache = read_workbook_preview(buffer.getvalue())[0] + assert without_cache.rows == [["=SUM(A1:A1)"]] + + # Simulate the cached-value case by writing the literal instead. + cached = read_workbook_preview(build(sheets={"Sheet": [["total"], [42]]}))[0] + assert cached.rows == [["42"]] + + +class TestCellFormatting: + @pytest.mark.parametrize( + "value,expected", + [ + (None, ""), + (True, "TRUE"), + (False, "FALSE"), + (7, "7"), + (-3, "-3"), + ("plain", "plain"), + ], + ) + def test_scalar_values(self, value, expected): + data = build(sheets={"S": [["h"], [value]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [[expected]] + + def test_a_whole_float_loses_its_trailing_zero(self): + data = build(sheets={"S": [["h"], [3.0]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [["3"]] + + def test_float_noise_is_not_shown(self): + # 0.1 + 0.2 must not preview as 0.30000000000000004. + data = build(sheets={"S": [["h"], [0.1 + 0.2]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [["0.3"]] + + def test_a_date_does_not_gain_a_midnight_it_never_had(self): + # openpyxl returns a datetime for every date-formatted cell, so + # rendering it whole would invent precision the file lacks. + data = build(sheets={"S": [["when"], [datetime(2026, 3, 15)]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [["2026-03-15"]] + + def test_a_real_timestamp_keeps_its_time(self): + data = build(sheets={"S": [["when"], [datetime(2026, 3, 15, 9, 30)]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [["2026-03-15 09:30:00"]] + + def test_a_numeric_string_stays_a_string(self): + # Same contract as the csv grid: the preview must not reinterpret + # what the file will hand downstream. + data = build(sheets={"S": [["zip"], ["007"]]}) + + [sheet] = read_workbook_preview(data) + + assert sheet.rows == [["007"]] + + +class TestCaps: + def test_stops_at_the_row_cap_and_reports_the_real_total(self): + rows = [["n"]] + [[i] for i in range(MAX_ROWS_PER_SHEET + 50)] + data = build(sheets={"S": rows}) + + [sheet] = read_workbook_preview(data) + + assert len(sheet.rows) == MAX_ROWS_PER_SHEET + assert sheet.truncated is True + assert sheet.truncated_by == "rows" + # The denominator is what the file claims, not what we read. + assert sheet.total_rows == MAX_ROWS_PER_SHEET + 50 + + def test_stops_at_the_column_cap(self): + wide = list(range(MAX_COLUMNS_PER_SHEET + 10)) + data = build(sheets={"S": [wide, wide]}) + + [sheet] = read_workbook_preview(data) + + assert len(sheet.headers) == MAX_COLUMNS_PER_SHEET + assert len(sheet.rows[0]) == MAX_COLUMNS_PER_SHEET + assert sheet.truncated_by == "columns" + + def test_stops_at_the_sheet_cap(self): + data = build( + sheets={f"S{i}": [["a"], [i]] for i in range(MAX_SHEETS + 5)} + ) + + sheets = read_workbook_preview(data) + + assert len(sheets) <= MAX_SHEETS + + def test_drops_trailing_empty_columns(self): + # A stray format on an empty cell is enough to stretch a sheet's + # used range, and the grid should not open on a screen of blanks. + wb = Workbook() + ws = wb.active + ws.append(["a", "b"]) + ws.append([1, 2]) + ws["H5"] = None + buffer = BytesIO() + wb.save(buffer) + + [sheet] = read_workbook_preview(buffer.getvalue()) + + assert sheet.headers == ["a", "b"] + + def test_refuses_a_workbook_past_the_byte_cap(self): + with pytest.raises(WorkbookTooLargeError): + read_workbook_preview(b"x" * (MAX_WORKBOOK_BYTES + 1)) + + +class TestFailures: + def test_rejects_bytes_that_are_not_a_workbook(self): + with pytest.raises(WorkbookUnreadableError): + read_workbook_preview(b"this is not a zip file") + + def test_rejects_an_empty_body(self): + with pytest.raises(WorkbookUnreadableError): + read_workbook_preview(b"") diff --git a/backend/tests/apis/inference_api/test_attachment_tool_autoenable.py b/backend/tests/apis/inference_api/test_attachment_tool_autoenable.py new file mode 100644 index 000000000..cca861dd5 --- /dev/null +++ b/backend/tests/apis/inference_api/test_attachment_tool_autoenable.py @@ -0,0 +1,197 @@ +"""Spreadsheet Analysis auto-enable on attachment (routes.py, P2-E of +docs/specs/load-test-assessment-2026-09.md). + +CSV/XLSX never go inline and the analysis tools are opt-in in the picker, so +on the default tool set an attached spreadsheet was a dead end. A session +that holds one now gets the tools injected for the turn — gated on the +caller's RBAC grant (enables, never grants) and sticky across the session so +the agent-cache key does not flip between the attach turn and the follow-up. +""" + +from types import SimpleNamespace +from unittest.mock import AsyncMock, patch + +import pytest + +from apis.inference_api.chat import routes +from apis.shared.feature_flags import attachment_tool_autoenable_enabled +from apis.shared.tools.injected import SPREADSHEET_TOOL_IDS + +USER = SimpleNamespace(user_id="u1") + + +@pytest.fixture(autouse=True) +def _clean_memo(): + routes._TABULAR_SESSIONS.clear() + yield + routes._TABULAR_SESSIONS.clear() + + +def _role_service(allowed): + svc = SimpleNamespace() + svc.can_access_tool = AsyncMock(side_effect=lambda user, tool_id: tool_id in allowed or "*" in allowed) + return svc + + +class TestFlag: + def test_defaults_on(self, monkeypatch): + monkeypatch.delenv("ATTACHMENT_TOOL_AUTOENABLE_ENABLED", raising=False) + assert attachment_tool_autoenable_enabled() + + def test_empty_is_on_and_false_is_off(self, monkeypatch): + monkeypatch.setenv("ATTACHMENT_TOOL_AUTOENABLE_ENABLED", "") + assert attachment_tool_autoenable_enabled() + monkeypatch.setenv("ATTACHMENT_TOOL_AUTOENABLE_ENABLED", "FALSE") + assert not attachment_tool_autoenable_enabled() + + +class TestWithAutoEnabledTools: + def test_appends_missing_ids_in_the_order_given(self): + assert routes._with_auto_enabled_tools(["calculator"], ["analyze_spreadsheet", "list_spreadsheets"]) == [ + "calculator", "analyze_spreadsheet", "list_spreadsheets", + ] + + def test_none_input_becomes_just_the_auto_ids(self): + assert routes._with_auto_enabled_tools(None, ["analyze_spreadsheet"]) == ["analyze_spreadsheet"] + + def test_returns_the_same_object_when_nothing_to_add(self): + original = ["analyze_spreadsheet", "list_spreadsheets", "calculator"] + assert routes._with_auto_enabled_tools(original, ["list_spreadsheets"]) is original + assert routes._with_auto_enabled_tools(None, []) is None + + def test_does_not_duplicate(self): + assert routes._with_auto_enabled_tools(["analyze_spreadsheet"], ["analyze_spreadsheet", "list_spreadsheets"]) == [ + "analyze_spreadsheet", "list_spreadsheets", + ] + + +class TestSessionHasTabular: + @pytest.mark.asyncio + async def test_this_turns_attachment_answers_without_a_query_and_is_remembered(self): + with patch("apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock()) as lookup: + assert await routes._session_has_tabular("s1", "u1", turn_has_tabular=True) + assert await routes._session_has_tabular("s1", "u1") # follow-up turn: sticky + lookup.assert_not_awaited() + assert routes._TABULAR_SESSIONS.get("s1") is True + + @pytest.mark.asyncio + async def test_queries_once_and_memoizes_a_positive_answer(self): + with patch("apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock(return_value=True)) as lookup: + assert await routes._session_has_tabular("s1", "u1") + assert await routes._session_has_tabular("s1", "u1") + assert lookup.await_count == 1 + + @pytest.mark.asyncio + async def test_negative_answers_are_not_memoized(self): + with patch("apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock(return_value=False)) as lookup: + assert not await routes._session_has_tabular("s1", "u1") + assert not await routes._session_has_tabular("s1", "u1") + assert lookup.await_count == 2 + assert "s1" not in routes._TABULAR_SESSIONS + + @pytest.mark.asyncio + async def test_lookup_failure_is_fail_closed(self): + with patch("apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock(side_effect=RuntimeError("ddb"))): + assert not await routes._session_has_tabular("s1", "u1") + + @pytest.mark.asyncio + async def test_missing_identity_is_false_without_a_query(self): + with patch("apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock()) as lookup: + assert not await routes._session_has_tabular("", "u1") + assert not await routes._session_has_tabular("s1", "") + lookup.assert_not_awaited() + + def test_memo_is_bounded(self, monkeypatch): + monkeypatch.setattr(routes, "_TABULAR_SESSIONS_MAX", 3) + for sid in ("s1", "s2", "s3", "s4"): + routes._remember_tabular_session(sid) + assert list(routes._TABULAR_SESSIONS) == ["s2", "s3", "s4"] + + +class TestAutoEnabledIds: + @pytest.mark.asyncio + async def test_grants_every_spreadsheet_id_the_role_admits_in_fixed_order(self): + with patch.object(routes, "get_app_role_service", return_value=_role_service(set(SPREADSHEET_TOOL_IDS))): + ids = await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1", turn_has_tabular=True) + assert ids == sorted(SPREADSHEET_TOOL_IDS) + + @pytest.mark.asyncio + async def test_wildcard_grant_admits_all(self): + with patch.object(routes, "get_app_role_service", return_value=_role_service({"*"})): + ids = await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1", turn_has_tabular=True) + assert ids == sorted(SPREADSHEET_TOOL_IDS) + + @pytest.mark.asyncio + async def test_enables_never_grants(self): + # A role that carries neither id gets nothing; one that carries one id gets that one. + with patch.object(routes, "get_app_role_service", return_value=_role_service(set())): + assert await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1", turn_has_tabular=True) == [] + with patch.object(routes, "get_app_role_service", return_value=_role_service({"analyze_spreadsheet"})): + assert await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1", turn_has_tabular=True) == [ + "analyze_spreadsheet" + ] + + @pytest.mark.asyncio + async def test_no_spreadsheet_in_session_means_no_rbac_call(self): + svc = _role_service({"*"}) + with patch.object(routes, "get_app_role_service", return_value=svc), patch( + "apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock(return_value=False) + ): + assert await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1") == [] + svc.can_access_tool.assert_not_awaited() + + @pytest.mark.asyncio + async def test_kill_switch(self, monkeypatch): + monkeypatch.setenv("ATTACHMENT_TOOL_AUTOENABLE_ENABLED", "false") + svc = _role_service({"*"}) + with patch.object(routes, "get_app_role_service", return_value=svc): + assert await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1", turn_has_tabular=True) == [] + svc.can_access_tool.assert_not_awaited() + + @pytest.mark.asyncio + async def test_rbac_failure_does_not_fail_the_turn(self): + svc = SimpleNamespace(can_access_tool=AsyncMock(side_effect=RuntimeError("rbac down"))) + with patch.object(routes, "get_app_role_service", return_value=svc): + assert await routes._auto_enabled_attachment_tool_ids(USER, "s1", "u1", turn_has_tabular=True) == [] + + +class TestApply: + @pytest.mark.asyncio + async def test_attach_turn_then_follow_up_produce_the_same_effective_list(self): + """The cache key hashes enabled_tools; the attach turn and the next + turn (no upload on it) must agree or the slot flips.""" + with patch.object(routes, "get_app_role_service", return_value=_role_service({"*"})): + attach_turn = await routes._apply_attachment_tool_autoenable( + ["calculator"], USER, "s1", "u1", turn_has_tabular=True + ) + follow_up = await routes._apply_attachment_tool_autoenable(["calculator"], USER, "s1", "u1") + assert attach_turn == follow_up == ["calculator", "analyze_spreadsheet", "list_spreadsheets"] + + @pytest.mark.asyncio + async def test_picker_already_on_is_untouched(self): + original = ["list_spreadsheets", "analyze_spreadsheet"] + with patch.object(routes, "get_app_role_service", return_value=_role_service({"*"})): + result = await routes._apply_attachment_tool_autoenable(original, USER, "s1", "u1", turn_has_tabular=True) + assert result is original + + @pytest.mark.asyncio + async def test_no_spreadsheet_means_list_passes_through_untouched(self): + with patch("apis.shared.files.document_read.session_has_tabular_files", new=AsyncMock(return_value=False)): + assert await routes._apply_attachment_tool_autoenable(None, USER, "s1", "u1") is None + + +class TestSharedLookup: + @pytest.mark.asyncio + async def test_session_has_tabular_files_matches_csv_and_xlsx_for_the_owner_only(self): + from apis.shared.files import document_read as dr + + rows = [ + SimpleNamespace(user_id="u1", filename="notes.pdf", mime_type="application/pdf"), + SimpleNamespace(user_id="u2", filename="theirs.csv", mime_type="text/csv"), + SimpleNamespace(user_id="u1", filename="mine.csv", mime_type="text/csv"), + ] + repo = SimpleNamespace(list_session_files=AsyncMock(return_value=rows)) + with patch.object(dr, "get_file_upload_repository", return_value=repo): + assert await dr.session_has_tabular_files("u1", "s1") + rows.pop() # only the other user's CSV and a PDF remain + assert not await dr.session_has_tabular_files("u1", "s1") diff --git a/backend/tests/apis/inference_api/test_attachment_turn_guard.py b/backend/tests/apis/inference_api/test_attachment_turn_guard.py new file mode 100644 index 000000000..08c8bca48 --- /dev/null +++ b/backend/tests/apis/inference_api/test_attachment_turn_guard.py @@ -0,0 +1,426 @@ +"""A turn's attachments are held to one message's budget — never to +``SessionException`` (docs/specs/document-context-offload.md §4E, PR-6). + +The stored unit is the MESSAGE, not the file. A turn's inline attachments are +persisted as one AgentCore Memory event; anything over the SDK's ~72 KB +conversational limit is written as a base64 ``blob`` (4/3 inflation) bounded by +the 10 MB event quota, so ~7.5 MB of raw attachment bytes per turn is the break +point. Past it, ``create_message`` re-raises ``SessionException`` — a hole in +history. The per-file 4 MB gate and the SPA's 5-file cap do not protect: prod +measurement (validation doc, Claim 7) found ~1.3–1.4% of attachment turns over +quota, several with only 3–4 files. + +Two helpers, composed in the chat route: + +- ``_apply_message_file_cap`` — the server side of the SPA's + ``MAX_FILES_PER_MESSAGE``, applied *before* the S3 fetch so a sixth file is + reported to the user instead of silently truncated by the resolver. +- ``_apply_inline_byte_budget`` — first-fit in attachment order against + ``INLINE_ATTACHMENTS_MAX_TOTAL_BYTES``; images count. +""" + +import base64 +import math + +import pytest + +from apis.inference_api.chat.routes import ( + _apply_inline_byte_budget, + _apply_message_file_cap, + _attachment_marker_names, + _build_attachment_guidance, + _emit_attachment_over_quota_metric, + _estimate_decoded_size, + _partition_attachments, +) +from apis.shared.feature_flags import attachment_turn_guard_enabled +from apis.shared.files import models as file_models + +XLSX_MIME = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" +EVENT_QUOTA_BYTES = 10_000_000 # AgentCore Memory event payload quota + + +def _b64_of_size(n: int) -> str: + """Base64 text whose decoded size is exactly ``n`` bytes.""" + return base64.b64encode(b"x" * n).decode() + + +class _Attachment: + """Minimal stand-in for FileContent — the helpers read these three.""" + + def __init__(self, filename: str, content_type: str = "application/pdf", size: int = 0): + self.filename = filename + self.content_type = content_type + self.bytes = _b64_of_size(size) + + +MB = 1024 * 1024 + + +class TestInlineByteBudget: + def test_under_the_budget_keeps_everything_in_order(self): + files = [_Attachment("a.pdf", size=3 * MB), _Attachment("b.pdf", size=3 * MB)] + kept, over, requested = _apply_inline_byte_budget(files, 7_500_000) + + assert kept == files + assert over == [] + assert requested == 6 * MB + + def test_over_the_budget_moves_the_file_that_breaks_it(self): + files = [_Attachment("a.pdf", size=4 * MB), _Attachment("b.pdf", size=4 * MB)] + kept, over, requested = _apply_inline_byte_budget(files, 7_500_000) + + assert [f.filename for f in kept] == ["a.pdf"] + assert [f.filename for f in over] == ["b.pdf"] + assert requested == 8 * MB + + def test_first_fit_earlier_files_win_and_a_later_small_file_still_rides(self): + # A(4) fits, B(4) would break the budget, C(1) still fits after A. + files = [ + _Attachment("a.pdf", size=4 * MB), + _Attachment("b.pdf", size=4 * MB), + _Attachment("c.pdf", size=1 * MB), + ] + kept, over, _ = _apply_inline_byte_budget(files, 7_500_000) + + assert [f.filename for f in kept] == ["a.pdf", "c.pdf"] + assert [f.filename for f in over] == ["b.pdf"] + + def test_order_is_deterministic_attachment_order_in_both_lists(self): + files = [ + _Attachment("z.pdf", size=3 * MB), + _Attachment("y.pdf", size=3 * MB), + _Attachment("x.pdf", size=3 * MB), + _Attachment("w.pdf", size=3 * MB), + ] + kept, over, _ = _apply_inline_byte_budget(files, 7_500_000) + + assert [f.filename for f in kept] == ["z.pdf", "y.pdf"] + assert [f.filename for f in over] == ["x.pdf", "w.pdf"] + + def test_images_count_toward_the_budget(self): + # The per-file document gate skips images; the message-level budget + # must not, because the image bytes are in the same persisted event. + files = [ + _Attachment("photo.png", content_type="image/png", size=4 * MB), + _Attachment("report.pdf", size=4 * MB), + ] + kept, over, _ = _apply_inline_byte_budget(files, 7_500_000) + + assert [f.filename for f in kept] == ["photo.png"] + assert [f.filename for f in over] == ["report.pdf"] + + def test_a_single_file_over_the_budget_is_moved_not_raised(self): + files = [_Attachment("huge.png", content_type="image/png", size=8 * MB)] + kept, over, _ = _apply_inline_byte_budget(files, 7_500_000) + + assert kept == [] + assert over == files + + def test_exactly_at_the_budget_is_allowed(self): + files = [_Attachment("a.pdf", size=7_500_000)] + kept, over, _ = _apply_inline_byte_budget(files, 7_500_000) + assert kept == files and over == [] + + def test_non_positive_budget_disables_the_cap(self): + files = [_Attachment("a.pdf", size=4 * MB), _Attachment("b.pdf", size=4 * MB)] + kept, over, requested = _apply_inline_byte_budget(files, 0) + + assert kept == files + assert over == [] + assert requested == 8 * MB + + def test_empty_input(self): + assert _apply_inline_byte_budget([], 7_500_000) == ([], [], 0) + + +class TestNothingReachesTheSessionExceptionPath: + """The property the whole PR exists for: whatever survives the budget is + writable as one AgentCore Memory event. + + The SDK serialises the message as base64 inside a ``blob`` payload. If the + kept set's encoded size is under the 10 MB event quota, ``create_message`` + cannot fail on size, so the turn degrades to the guidance note rather than + to ``SessionException``. + """ + + @pytest.mark.parametrize( + "sizes", + [ + [4 * MB, 4 * MB], # the classic two-file case + [4 * MB, 4 * MB, 4 * MB, 4 * MB, 4 * MB], # SPA-legal five files + [3 * MB, 3 * MB, 3 * MB], # 3 files, each fine + [2_600_000, 2_600_000, 2_600_000], # the "3–4 files" prod shape + [7_500_000, 1], + [1] * 5 + [4 * MB] * 5, + ], + ) + def test_kept_set_encodes_under_the_event_quota(self, sizes): + files = [_Attachment(f"f{i}.pdf", size=s) for i, s in enumerate(sizes)] + default_cap = file_models.INLINE_ATTACHMENTS_MAX_TOTAL_BYTES + kept, over, _ = _apply_inline_byte_budget(files, default_cap) + + encoded = sum(len(f.bytes) for f in kept) + assert encoded <= math.ceil(default_cap * 4 / 3) + 4 * len(kept) # padding slack + assert encoded <= EVENT_QUOTA_BYTES + # Nothing is lost silently: every input is in exactly one bucket. + assert sorted(f.filename for f in kept + over) == sorted(f.filename for f in files) + + def test_partition_then_budget_chain_on_files_that_each_pass_the_per_file_gate(self): + # Each file is under INLINE_DOCUMENT_MAX_BYTES (4 MiB) so the + # per-file gate keeps all three; only the aggregate budget catches + # the turn. This is exactly the hole the per-file gate left open. + per_file = file_models.INLINE_DOCUMENT_MAX_BYTES - 1024 + files = [_Attachment(f"f{i}.pdf", size=per_file) for i in range(3)] + sheet = _Attachment("data.xlsx", content_type=XLSX_MIME, size=6 * MB) + + inline, tabular, _, oversized = _partition_attachments(files + [sheet]) + assert oversized == [] and inline == files and tabular == [sheet] + + kept, over, requested = _apply_inline_byte_budget( + inline, file_models.INLINE_ATTACHMENTS_MAX_TOTAL_BYTES + ) + assert [f.filename for f in kept] == ["f0.pdf"] + assert [f.filename for f in over] == ["f1.pdf", "f2.pdf"] + assert requested == 3 * per_file + # The diverted spreadsheet never counted: it is not in the message. + assert sum(len(f.bytes) for f in kept) <= EVENT_QUOTA_BYTES + + def test_default_cap_is_the_documented_derivation(self): + # 10 MB event quota × 3/4 (base64) = 7.5 MB raw is the *break point*, + # not a safe cap: encoding exactly that lands ON the quota, and the + # event's JSON envelope (role, content keys, the prompt text block, + # per-file metadata, the wrapper) is then added on top. The default + # therefore sits below it, with room for the envelope. + assert len(_b64_of_size(7_500_000)) == EVENT_QUOTA_BYTES, "the break point is real" + assert file_models.INLINE_ATTACHMENTS_MAX_TOTAL_BYTES == 7_000_000 + assert len(_b64_of_size(file_models.INLINE_ATTACHMENTS_MAX_TOTAL_BYTES)) < EVENT_QUOTA_BYTES + + +class TestMessageFileCap: + def test_under_the_cap_is_untouched(self): + direct = [_Attachment("a.pdf")] + ids = ["u1", "u2"] + assert _apply_message_file_cap(direct, ids, 5) == (direct, ids, [], 0) + + def test_upload_ids_beyond_the_cap_are_counted_not_fetched(self): + # The old resolver truncated at 5 silently; now the route knows. + ids = [f"u{i}" for i in range(7)] + kept_direct, kept_ids, names, total = _apply_message_file_cap([], ids, 5) + + assert kept_direct == [] + assert kept_ids == ids[:5] + assert names == [] + assert total == 2 + + def test_direct_files_beyond_the_cap_are_named(self): + direct = [_Attachment(f"d{i}.pdf") for i in range(6)] + kept_direct, kept_ids, names, total = _apply_message_file_cap(direct, [], 5) + + assert kept_direct == direct[:5] + assert kept_ids == [] + assert names == ["d5.pdf"] + assert total == 1 + + def test_direct_files_come_first_and_ids_fill_the_remainder(self): + direct = [_Attachment("d0.pdf"), _Attachment("d1.pdf")] + ids = ["u0", "u1", "u2", "u3", "u4"] + kept_direct, kept_ids, names, total = _apply_message_file_cap(direct, ids, 5) + + assert kept_direct == direct + assert kept_ids == ["u0", "u1", "u2"] + assert names == [] + assert total == 2 + + def test_direct_overflow_leaves_no_budget_for_ids(self): + direct = [_Attachment(f"d{i}.pdf") for i in range(6)] + kept_direct, kept_ids, names, total = _apply_message_file_cap(direct, ["u0"], 5) + + assert len(kept_direct) == 5 + assert kept_ids == [] + assert names == ["d5.pdf"] + assert total == 2 + + def test_non_positive_cap_disables(self): + direct = [_Attachment(f"d{i}.pdf") for i in range(9)] + ids = [f"u{i}" for i in range(9)] + assert _apply_message_file_cap(direct, ids, 0) == (direct, ids, [], 0) + + def test_default_cap_matches_the_spa(self): + assert file_models.MAX_FILES_PER_MESSAGE == 5 + + +class TestGuidanceText: + def test_aggregate_case_names_the_files_and_says_follow_up(self): + over = [_Attachment("b.pdf"), _Attachment("c.pdf")] + text = _build_attachment_guidance([], [], [], None, over_budget=over) + + assert "`b.pdf`, `c.pdf`" in text + assert "together exceed the combined size limit" in text + assert "follow-up message" in text + + def test_aggregate_and_per_file_notes_are_different_sentences(self): + # The remedy differs: per-file → smaller file; aggregate → next message. + huge = [_Attachment("huge.pdf")] + over = [_Attachment("b.pdf")] + text = _build_attachment_guidance([], [], huge, None, over_budget=over) + + per_file, aggregate = text.split("\n\n") + assert "`huge.pdf`" in per_file and "Try a smaller file" in per_file + assert "`b.pdf`" in aggregate and "follow-up message" in aggregate + assert "`b.pdf`" not in per_file + + def test_count_cap_note_with_names(self): + text = _build_attachment_guidance( + [], [], [], None, + dropped_over_count_names=["f.pdf"], dropped_over_count_total=1, max_files=5, + ) + assert "Only the first 5 files per message are attached" in text + assert "`f.pdf` were not" in text + + def test_count_cap_note_with_names_and_unnamed_ids(self): + text = _build_attachment_guidance( + [], [], [], None, + dropped_over_count_names=["f.pdf"], dropped_over_count_total=3, max_files=5, + ) + assert "`f.pdf` and 2 more were not" in text + + def test_count_cap_note_count_only_when_ids_were_dropped(self): + text = _build_attachment_guidance( + [], [], [], None, dropped_over_count_total=2, max_files=5, + ) + assert "2 more files were not" in text + assert "follow-up message" in text + + def test_count_cap_note_singular(self): + text = _build_attachment_guidance( + [], [], [], None, dropped_over_count_total=1, max_files=5, + ) + assert "1 more file was not" in text + + def test_silent_when_nothing_was_trimmed(self): + assert _build_attachment_guidance([], [], [], None) == "" + assert _build_attachment_guidance( + [], [], [], None, over_budget=[], dropped_over_count_names=[], dropped_over_count_total=0 + ) == "" + + def test_existing_notes_are_unchanged_by_the_new_defaults(self): + huge = [_Attachment("huge.pdf")] + assert _build_attachment_guidance([], [], huge, None) == _build_attachment_guidance( + [], [], huge, None, over_budget=None, dropped_over_count_total=0 + ) + + +class TestMarkerNames: + def test_over_budget_files_are_excluded_like_oversized_ones(self): + # Both were dropped from the turn; the SPA must not rebuild a card. + a, b, c = _Attachment("a.pdf"), _Attachment("b.pdf"), _Attachment("c.pdf") + assert _attachment_marker_names([a, b, c], [c] + [b]) == ["a.pdf"] + + +class TestKillSwitch: + @pytest.mark.parametrize("value", [None, "", "true", "TRUE", "yes", "0"]) + def test_default_on(self, monkeypatch, value): + if value is None: + monkeypatch.delenv("ATTACHMENT_TURN_GUARD_ENABLED", raising=False) + else: + monkeypatch.setenv("ATTACHMENT_TURN_GUARD_ENABLED", value) + assert attachment_turn_guard_enabled() is True + + @pytest.mark.parametrize("value", ["false", "FALSE", " False "]) + def test_only_literal_false_disables(self, monkeypatch, value): + monkeypatch.setenv("ATTACHMENT_TURN_GUARD_ENABLED", value) + assert attachment_turn_guard_enabled() is False + + +class TestOverQuotaMetric: + def test_emits_one_content_free_record_in_bytes(self, monkeypatch): + from apis.shared.observability import emf, prompt_cache + + calls = [] + monkeypatch.setattr(prompt_cache, "prompt_cache_observability_enabled", lambda: True) + monkeypatch.setattr( + emf, "emit_emf_metrics", + lambda namespace, metrics, properties=None, units=None: calls.append( + (namespace, metrics, properties, units) + ), + ) + _emit_attachment_over_quota_metric( + requested_bytes=8 * MB, cap_bytes=7_500_000, inline_count=2, dropped_count=1 + ) + + assert calls == [( + "AgentCoreStack/Compaction", + {"AttachmentTurnOverQuota": 8 * MB}, + {"capBytes": 7_500_000, "inlineFileCount": 2, "droppedFileCount": 1}, + {"AttachmentTurnOverQuota": "Bytes"}, + )] + # Content-free: no filename, no user, no session in the record. + assert not any("name" in k.lower() and "file" not in k.lower() for k in calls[0][2]) + + def test_silenced_with_the_observability_layer(self, monkeypatch): + from apis.shared.observability import emf, prompt_cache + + monkeypatch.setattr(prompt_cache, "prompt_cache_observability_enabled", lambda: False) + monkeypatch.setattr( + emf, "emit_emf_metrics", lambda *a, **k: pytest.fail("must not emit") + ) + _emit_attachment_over_quota_metric(1, 1, 1, 1) + + def test_never_raises(self, monkeypatch): + from apis.shared.observability import emf, prompt_cache + + monkeypatch.setattr(prompt_cache, "prompt_cache_observability_enabled", lambda: True) + + def boom(*a, **k): + raise RuntimeError("emf down") + + monkeypatch.setattr(emf, "emit_emf_metrics", boom) + _emit_attachment_over_quota_metric(1, 1, 1, 1) + + +class TestEstimateDecodedSize: + def test_matches_real_decoded_length(self): + for n in (0, 1, 2, 3, 4, 100, 7_500_000): + f = _Attachment("x", size=n) + assert _estimate_decoded_size(f) == n + + +class TestBudgetHeadroom: + """The guard exists so a turn's inline attachments never exceed AgentCore's + 10 MB *event* quota — past which ``create_message`` raises + ``SessionException`` and leaves a hole in history. + + Its first default did not achieve that. base64 of N raw bytes is + ``4*ceil(N/3)``, so 7,500,000 encoded to **exactly 10,000,000** — the quota + itself, with nothing left for the event's JSON envelope. A guard whose + default sits precisely on the break point it exists to stay under does not + prevent the failure it was written for. + """ + + QUOTA_BYTES = 10_000_000 + + def _encoded(self, raw_bytes: int) -> int: + return 4 * math.ceil(raw_bytes / 3) + + def test_the_default_leaves_room_for_the_event_envelope(self): + from apis.shared.files.models import INLINE_ATTACHMENTS_MAX_TOTAL_BYTES as cap + + encoded = self._encoded(cap) + headroom = self.QUOTA_BYTES - encoded + assert headroom > 0, f"budget encodes to {encoded:,} against a {self.QUOTA_BYTES:,} quota" + # Enough for the role, content keys, prompt text and per-file metadata + # many times over, without being so conservative it trims real turns + # (prod p90 attachment cluster is 2.58 MB). + assert headroom >= 250_000, f"only {headroom:,} bytes of headroom" + + def test_the_encoding_math_matches_base64(self): + """Pin the 4/3 inflation the budget is derived from, against the real + encoder rather than the arithmetic in a comment.""" + for raw in (3, 3_000, 7_000_000): + assert len(base64.b64encode(b"\0" * raw)) == self._encoded(raw) + + def test_a_turn_at_the_budget_still_fits_the_quota(self): + from apis.shared.files.models import INLINE_ATTACHMENTS_MAX_TOTAL_BYTES as cap + + assert len(base64.b64encode(b"\0" * cap)) < self.QUOTA_BYTES diff --git a/backend/tests/apis/inference_api/test_chat_service.py b/backend/tests/apis/inference_api/test_chat_service.py index 27e0534ec..061ebb11c 100644 --- a/backend/tests/apis/inference_api/test_chat_service.py +++ b/backend/tests/apis/inference_api/test_chat_service.py @@ -548,3 +548,103 @@ async def test_a_newly_cacheable_agent_still_shares_the_session_conversation( ) assert mentioned.agent.messages, "the second live Agent forked the conversation" + + +def test_adopting_the_conversation_also_adopts_the_compaction_offset(monkeypatch): + """The live list's coordinate system travels with it (thresholds spec §3.5). + + Compaction expresses its checkpoint as ``_live_offset + index into the + list`` and the pending-cut apply slices the list in place at that offset, + so an instance that adopts another instance's list must adopt its offset. + """ + live_inner = SimpleNamespace(messages=[{"role": "user", "t": 1}, {"role": "assistant", "t": 2}]) + live_wrapper = SimpleNamespace(agent=live_inner, session_manager=SimpleNamespace(_live_offset=7)) + monkeypatch.setattr(service, "_agent_cache", {("s", "key-a"): live_wrapper}) + + fresh_inner = SimpleNamespace(messages=[{"role": "user", "t": 1}]) + fresh = SimpleNamespace(agent=fresh_inner, session_manager=SimpleNamespace(_live_offset=0)) + + service._adopt_session_conversation(fresh, "s") + + assert fresh.agent.messages is live_inner.messages + assert fresh.session_manager._live_offset == 7 + + +def test_create_cache_key_includes_assistant_id(): + """Spreadsheet-analysis tools close over the assistant, so two assistants + must not share a cache slot; no assistant hashes to a stable empty element + so pre-field keys are unchanged.""" + base = dict( + session_id="s", + user_id="u", + enabled_tools=["analyze_spreadsheet"], + model_id="m", + inference_params={}, + system_prompt=None, + caching_enabled=False, + provider="bedrock", + freshness_hash="f", + agent_type="chat", + ) + k_a = service._create_cache_key(**base, assistant_id="asst-a") + k_b = service._create_cache_key(**base, assistant_id="asst-b") + k_none = service._create_cache_key(**base) + assert k_a != k_b + assert k_a != k_none + assert "asst-a" in k_a + assert "" in k_none and "asst-a" not in k_none + # skills_hash stays the trailing element (other tests index it as [-1]). + assert k_none[-1] == "" + + +@pytest.mark.asyncio +async def test_assistant_id_is_stamped_on_the_construction_snapshot( + mock_create_agent, mock_freshness_hash +): + agent = await service.get_agent( + session_id="s1", user_id="u1", is_resume=False, assistant_id="asst-a" + ) + assert agent._construction_snapshot["assistant_id"] == "asst-a" + + bare = await service.get_agent(session_id="s2", user_id="u1", is_resume=False) + assert bare._construction_snapshot["assistant_id"] is None + + +@pytest.mark.asyncio +async def test_resume_replays_assistant_id_onto_the_same_slot( + mock_create_agent, mock_freshness_hash +): + """A spreadsheet turn against an assistant pauses; resume feeds the + snapshot's assistant_id back and must land on the paused agent. A + different assistant (or none) must NOT.""" + first = await service.get_agent( + session_id="s1", + user_id="u1", + enabled_tools=["analyze_spreadsheet"], + extra_tools=[object()], + extra_tools_key_described=True, + is_resume=False, + assistant_id="asst-a", + ) + first.agent._interrupt_state.activated = True + replay = first._construction_snapshot["assistant_id"] + + same = await service.get_agent( + session_id="s1", + user_id="u1", + enabled_tools=["analyze_spreadsheet"], + is_resume=True, + assistant_id=replay, + ) + assert same is first + assert mock_create_agent.call_count == 1 + + other = await service.get_agent( + session_id="s1", + user_id="u1", + enabled_tools=["analyze_spreadsheet"], + is_resume=True, + assistant_id="asst-b", + ) + assert other is not first + assert mock_create_agent.call_count == 2 diff --git a/backend/tests/apis/inference_api/test_middleware_stack.py b/backend/tests/apis/inference_api/test_middleware_stack.py new file mode 100644 index 000000000..b58023795 --- /dev/null +++ b/backend/tests/apis/inference_api/test_middleware_stack.py @@ -0,0 +1,38 @@ +"""inference-api must not go back to stock `GZipMiddleware`. + +This service exists to serve one route — the `/invocations` SSE turn — and +Starlette's stock gzip middleware withholds `http.response.start` until the +first body chunk arrives. On an SSE turn that chunk is the model's first +token, so the response headers landed behind the agent's whole thinking time, +with or without `Accept-Encoding: gzip`. app-api's chat proxy reads this +response's `content-type` before it can open its own stream to the SPA, so +the delay reached the browser. + +The swap to `StreamSafeGZipMiddleware` is easy to undo by accident — the two +have the same constructor and compress identically — so it is pinned here. +""" + +from __future__ import annotations + +from starlette.middleware.gzip import GZipMiddleware + +from apis.inference_api.main import app +from apis.shared.middleware.compression import StreamSafeGZipMiddleware + + +def test_compression_is_the_stream_safe_variant() -> None: + installed = [m.cls for m in app.user_middleware] + + assert StreamSafeGZipMiddleware in installed + # `is not` rather than membership: the stream-safe class *subclasses* + # `GZipMiddleware`, so an `in` check would pass either way. + assert not any(cls is GZipMiddleware for cls in installed) + + +def test_compression_settings_are_unchanged_by_the_swap() -> None: + gzip_middleware = next( + m for m in app.user_middleware if m.cls is StreamSafeGZipMiddleware + ) + + assert gzip_middleware.kwargs["minimum_size"] == 1000 + assert gzip_middleware.kwargs["compresslevel"] == 6 diff --git a/backend/tests/apis/inference_api/test_preparing_phase.py b/backend/tests/apis/inference_api/test_preparing_phase.py new file mode 100644 index 000000000..d2817f157 --- /dev/null +++ b/backend/tests/apis/inference_api/test_preparing_phase.py @@ -0,0 +1,253 @@ +"""The deferred, narrated agent build (docs/specs/agent-state-feedback.md PR-3). + +Measured on dev: a cold agent-cache miss spends ~1500ms inside `get_agent`, and +because FastAPI flushes response headers when the handler returns its +`StreamingResponse`, every millisecond of that is dead air. PR-3 defers the +build into the stream generator so the response opens first and the wait can be +narrated. + +**The timing decision is NOT made here.** It used to be: the generator raced the +build against a 250ms timer and emitted the frame only if the build was still +running, so a warm build never flashed a phase nobody can read. That cannot +work — `create_agent` is synchronous, so a cold build occupies the event loop +for its whole duration and `asyncio.wait` never fires its timeout. Verified on +dev: a 1548ms build, six times the threshold, emitted nothing. The frame is now +unconditional and the SPA holds it for 250ms before rendering. + +So what is pinned here is what the server still owns: the frame goes out BEFORE +the build, a failed build surfaces instead of hanging, and the lease is released +either way. +""" + +import asyncio +import json +from pathlib import Path +from typing import Any, AsyncGenerator, List, Optional + +import pytest + +# Module level on purpose. `backend/tests/apis/__init__.py` makes `tests/apis` +# a package ALSO named `apis`, and pytest puts `backend/tests` on sys.path, so +# a file whose first `apis.` import happens later — inside a fixture, say — +# can bind `apis` to the TEST package and fail on `apis.shared.caching`, which +# only exists under `src`. Importing here binds it to the real one first, which +# is why every other test file in this tree does the same. +import apis.inference_api.chat.routes as routes_module + + +def _frames_of(kind: str, frames: List[str]) -> List[dict]: + prefix = f"event: {kind}\ndata: " + return [ + json.loads(f[len(prefix) :].strip()) + for f in frames + if f.startswith(prefix) + ] + + +class _Harness: + """A faithful copy of the generator's build-and-narrate preamble. + + The real `_guarded_stream` is a closure over ~40 locals inside a + 1700-line handler; reproducing its *decision* here keeps the test on the + behaviour under change instead of on FastAPI wiring. Kept in step with + `routes.py` by `TestRouteContract`. + """ + + def __init__(self, build_seconds: float, fails: bool = False) -> None: + self.build_seconds = build_seconds + self.fails = fails + self.released = False + self.built = False + + async def _build(self) -> Any: + await asyncio.sleep(self.build_seconds) + if self.fails: + raise RuntimeError("tool registry exploded") + self.built = True + return object() + + async def stream(self) -> AsyncGenerator[str, None]: + agent: Optional[Any] = None + try: + yield ( + "event: agent_status\ndata: " + + json.dumps( + { + "type": "agent_status", + "sessionId": "sess-1", + "phase": "preparing", + } + ) + + "\n\n" + ) + try: + agent = await self._build() + except Exception: + yield 'event: stream_error\ndata: {"code": "AGENT_ERROR"}\n\n' + yield "event: done\ndata: {}\n\n" + return + assert agent is not None + yield ( + "event: agent_status\ndata: " + + json.dumps( + { + "type": "agent_status", + "sessionId": "sess-1", + "phase": "prepared", + "durationMs": int(self.build_seconds * 1000), + } + ) + + "\n\n" + ) + yield 'event: message_start\ndata: {"role": "assistant"}\n\n' + yield "event: done\ndata: {}\n\n" + finally: + self.released = True + + +class TestTheFrame: + @pytest.mark.asyncio + async def test_is_emitted_for_every_deferred_build(self): + """Unconditional by design — the server cannot time its own build. + + It is always paired with `prepared`: the SPA needs the start to show + the label and the end to suppress it on a fast build. + """ + harness = _Harness(build_seconds=0.01) + + frames = [f async for f in harness.stream()] + + assert [s["phase"] for s in _frames_of("agent_status", frames)] == [ + "preparing", + "prepared", + ] + assert harness.built + + @pytest.mark.asyncio + async def test_precedes_the_build_it_explains(self): + """After the build it would describe a wait that had already ended.""" + harness = _Harness(build_seconds=0.3) + + frames = [f async for f in harness.stream()] + + preparing = next(i for i, f in enumerate(frames) if "preparing" in f) + message_start = next( + i for i, f in enumerate(frames) if f.startswith("event: message_start") + ) + assert preparing < message_start + + @pytest.mark.asyncio + async def test_carries_no_cycle(self): + """`preparing` precedes the event loop, so there is no cycle to number. + + The SPA validator accepts it on that basis; a fabricated cycle would + make the two disagree about what the phase means. + """ + harness = _Harness(build_seconds=0.01) + + frames = [f async for f in harness.stream()] + + assert "cycle" not in _frames_of("agent_status", frames)[0] + + +class TestTheEndFrame: + """`prepared` is what lets the SPA suppress a fast build. + + Without it the client can only infer the build ended from `thinking`, + which does not arrive until the head-of-turn work and the event loop's + startup have also run — well past the 250ms the SPA waits. Measured on + dev, builds of 0ms, 1ms and 40ms all rendered "Getting ready…" because of + exactly that gap. + """ + + @pytest.mark.asyncio + async def test_follows_the_build(self): + harness = _Harness(build_seconds=0.05) + + frames = [f async for f in harness.stream()] + + assert [s["phase"] for s in _frames_of("agent_status", frames)] == [ + "preparing", + "prepared", + ] + + @pytest.mark.asyncio + async def test_precedes_the_answer(self): + """It must land before `message_start`, or the label it clears is + already competing with streamed text.""" + harness = _Harness(build_seconds=0.05) + + frames = [f async for f in harness.stream()] + + prepared = next(i for i, f in enumerate(frames) if "prepared" in f) + message_start = next( + i for i, f in enumerate(frames) if f.startswith("event: message_start") + ) + assert prepared < message_start + + @pytest.mark.asyncio + async def test_carries_the_build_duration(self): + """A bare marker would end the label; the duration also says how long + the wait the user was told about actually took.""" + harness = _Harness(build_seconds=0.05) + + frames = [f async for f in harness.stream()] + + prepared = _frames_of("agent_status", frames)[1] + assert prepared["durationMs"] >= 0 + + @pytest.mark.asyncio + async def test_is_not_emitted_when_the_build_fails(self): + """Nothing was prepared. The error frame is what ends the label.""" + harness = _Harness(build_seconds=0.01, fails=True) + + frames = [f async for f in harness.stream()] + + assert [s["phase"] for s in _frames_of("agent_status", frames)] == [ + "preparing" + ] + assert any(f.startswith("event: stream_error") for f in frames) + + +class TestFailure: + @pytest.mark.asyncio + async def test_a_failed_build_surfaces_as_a_conversational_error(self): + """The handler has already returned, so its `except` arms cannot see + this. Without the in-generator catch it is a silent hung stream.""" + harness = _Harness(build_seconds=0.01, fails=True) + + frames = [f async for f in harness.stream()] + + assert any(f.startswith("event: stream_error") for f in frames) + assert any(f.startswith("event: done") for f in frames) + assert not any(f.startswith("event: message_start") for f in frames) + + @pytest.mark.asyncio + async def test_a_failed_build_still_releases_the_lease(self): + harness = _Harness(build_seconds=0.01, fails=True) + + [f async for f in harness.stream()] + + assert harness.released + + +class TestRouteContract: + def test_route_still_matches_this_shape(self): + """Guards the harness above against the route drifting away from it.""" + source = Path(routes_module.__file__).read_text() + + assert '"phase": "preparing"' in source + # The end frame, without which a 1ms build still renders the label. + assert '"phase": "prepared"' in source + assert "Deferred agent build failed" in source + + def test_the_route_no_longer_races_its_own_build(self): + """The regression this file exists to prevent a second time. + + A server-side timer around a synchronous build cannot fire, so any + reappearance of one here means the frame has silently stopped being + sent again. + """ + source = Path(routes_module.__file__).read_text() + + assert "_PREPARING_NOTICE_SECONDS" not in source diff --git a/backend/tests/apis/inference_api/test_turn_timing.py b/backend/tests/apis/inference_api/test_turn_timing.py new file mode 100644 index 000000000..1f4c105e1 --- /dev/null +++ b/backend/tests/apis/inference_api/test_turn_timing.py @@ -0,0 +1,383 @@ +"""Pre-stream turn timing (docs/specs/agent-state-feedback.md). + +This exists to answer one question — which stage owns the 3.75s before the +first SSE byte — so what is worth pinning is that it cannot lie about that +(stages in order, deltas not cumulative totals) and cannot break the turn it +measures. +""" + +import json +import logging + +from apis.inference_api.chat.turn_timing import TurnPrelude, _metric_name + + +class TestMarks: + def test_stages_are_recorded_in_order(self): + prelude = TurnPrelude() + for stage in ("preamble", "rag", "tools", "agent_build"): + prelude.mark(stage) + + payload = _emitted(prelude) + + assert list(payload["stages"].keys()) == [ + "preamble", + "rag", + "tools", + "agent_build", + ] + + def test_each_stage_is_a_delta_not_a_running_total(self, monkeypatch): + """The slowest stage is the answer, so a cumulative number would point + at the last stage every time.""" + clock = iter([0.0, 1.0, 1.5, 4.5, 4.6]) + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", + lambda: next(clock), + ) + + prelude = TurnPrelude() # consumes 0.0 + prelude.mark("preamble") # 1.0 -> 1000ms + prelude.mark("rag") # 1.5 -> 500ms + prelude.mark("agent_build") # 4.5 -> 3000ms + + payload = _emitted(prelude) # total_ms consumes 4.6 + + assert payload["stages"] == { + "preamble": 1000, + "rag": 500, + "agent_build": 3000, + } + assert payload["totalMs"] == 4600 + + def test_total_covers_the_whole_prelude(self, monkeypatch): + clock = iter([0.0, 2.0, 7.0]) + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", + lambda: next(clock), + ) + + prelude = TurnPrelude() + prelude.mark("preamble") + + assert _emitted(prelude)["totalMs"] == 7000 + + +class TestStartedAt: + """The value handed to the stream coordinator for the turn recap.""" + + def test_is_wall_clock_not_perf_counter(self): + """Mixing clock domains yields a meaningless number, not a close one. + + The coordinator subtracts this from a `time.time()` reading. A + `perf_counter` value — seconds since an arbitrary origin — would make + the recap read as decades, or negative. + """ + import time + + before = time.time() + prelude = TurnPrelude() + after = time.time() + + assert before <= prelude.started_at <= after + + def test_marks_are_immune_to_the_wall_clock_moving(self, monkeypatch): + """The stage deltas must not inherit the wall clock. + + NTP can step `time.time()` backwards mid-turn, which would produce a + negative stage. The marks use `perf_counter`, so a wall clock that + jumps a decade must change nothing. + """ + prelude = TurnPrelude() + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.time", lambda: 0.0 + ) + prelude.mark("preamble") + + payload = _emitted(prelude) + + assert payload["stages"]["preamble"] >= 0 + assert payload["totalMs"] >= 0 + + +class TestPayload: + def test_carries_the_session_and_the_caller_s_extras(self): + prelude = TurnPrelude() + prelude.mark("preamble") + + payload = _emitted( + prelude, session_id="sess-1", extra={"isResume": False} + ) + + assert payload["sessionId"] == "sess-1" + assert payload["streamKind"] == "agent" + assert payload["isResume"] is False + + def test_carries_no_message_content(self): + """A latency record is held to the same content-free rule as the cost + rows — there is no field here for a prompt, and none should be added.""" + prelude = TurnPrelude() + prelude.mark("preamble") + + payload = _emitted(prelude) + + assert set(payload) == {"sessionId", "streamKind", "totalMs", "stages"} + + def test_a_flat_prelude_carries_no_groups_key(self): + """A group of one is noise — the number is already in `stages`.""" + prelude = TurnPrelude() + prelude.mark("rag") + + assert "groups" not in _emitted(prelude) + + +class TestFailSoft: + def test_a_mark_that_cannot_be_taken_never_raises(self, monkeypatch): + prelude = TurnPrelude() + + def _boom(): + raise RuntimeError("clock exploded") + + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", _boom + ) + prelude.mark("preamble") # must not raise + + def test_an_unserializable_extra_never_raises(self, caplog): + prelude = TurnPrelude() + prelude.mark("preamble") + + class _Hostile: + def __repr__(self): + raise RuntimeError("no") + + # `default=str` calls repr on the way out; the emit must swallow it + # rather than take the turn down with it. + prelude.emit( + session_id="s", stream_kind="agent", extra={"bad": _Hostile()} + ) + + def test_emit_writes_exactly_one_line(self, caplog): + prelude = TurnPrelude() + prelude.mark("preamble") + + with caplog.at_level(logging.INFO, logger="apis.inference_api.chat.turn_timing"): + prelude.emit(session_id="s", stream_kind="agent") + + lines = [r for r in caplog.records if r.getMessage().startswith("turn_prelude ")] + assert len(lines) == 1 + + +class TestGroups: + """Sub-stages sum back into the stage they decompose. + + docs/specs/turn-latency-preamble.md splits `preamble` into five. The only + measurement anyone has of the pre-stream window is stated in terms of the + single coarse number, so the split has to keep reproducing it or it + discards its own baseline. + """ + + def test_dotted_substages_sum_into_their_prefix(self, monkeypatch): + clock = iter([0.0, 0.1, 0.4, 1.0, 1.1]) + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", + lambda: next(clock), + ) + + prelude = TurnPrelude() # consumes 0.0 + prelude.mark("preamble.ownership") # 100ms + prelude.mark("preamble.quota") # 300ms + prelude.mark("rag") # 600ms + + payload = _emitted(prelude) # total_ms consumes 1.1 + + assert payload["groups"] == {"preamble": 400} + + def test_the_substages_stay_visible_alongside_the_group(self): + """The group is an addition, not a replacement: the whole point of the + split is knowing WHICH sub-stage owns the time.""" + prelude = TurnPrelude() + prelude.mark("preamble.ownership") + prelude.mark("preamble.quota") + + payload = _emitted(prelude) + + assert list(payload["stages"]) == ["preamble.ownership", "preamble.quota"] + + def test_undotted_stages_are_not_grouped(self, monkeypatch): + clock = iter([0.0, 0.2, 0.5, 0.6]) + monkeypatch.setattr( + "apis.inference_api.chat.turn_timing.time.perf_counter", + lambda: next(clock), + ) + + prelude = TurnPrelude() + prelude.mark("preamble.files") # 200ms + prelude.mark("agent_build") # 300ms + + assert _emitted(prelude)["groups"] == {"preamble": 200} + + def test_a_group_never_breaks_the_turn_it_measures(self, monkeypatch): + """Same fail-soft contract as `mark`: a stage name that is not a string + must cost a log line, not the turn.""" + prelude = TurnPrelude() + prelude._marks.append((None, 5.0)) # type: ignore[arg-type] + + prelude.emit(session_id="s", stream_kind="agent") # must not raise + + +class TestMetricNames: + """The EMF metric name is derived from the stage name, not looked up. + + A hand-kept table drifts silently: the dashboard widget renders an empty + graph, which is indistinguishable from "that stage never ran". + """ + + def test_dotted_and_snake_stages_become_camel_case_ms(self): + assert _metric_name("preamble.session_state") == "PreambleSessionStateMs" + assert _metric_name("preamble.ownership") == "PreambleOwnershipMs" + assert _metric_name("agent_build") == "AgentBuildMs" + assert _metric_name("rag") == "RagMs" + + def test_a_group_prefix_renders_the_same_either_way(self): + """`groups.preamble` and a flat `preamble` mark must not disagree — + they are the same quantity, so the same name is correct.""" + assert _metric_name("preamble") == "PreambleMs" + + +class TestMetrics: + """One EMF record per turn, so the fleet has percentiles. + + The log line answers "where did THIS turn go"; only an aggregate can + answer "did the change help", which is the question + docs/specs/turn-latency-preamble.md exists to make answerable. + """ + + def test_emits_every_stage_plus_the_group_and_the_total(self, monkeypatch): + record = _emf_record(monkeypatch, ["preamble.ownership", "preamble.quota", "agent_build"]) + + assert set(record["metrics"]) == { + "PreludeTotalMs", + "PreambleOwnershipMs", + "PreambleQuotaMs", + "PreambleMs", + "AgentBuildMs", + } + + def test_every_metric_is_milliseconds(self, monkeypatch): + record = _emf_record(monkeypatch, ["preamble.quota"]) + + assert set(record["units"].values()) == {"Milliseconds"} + + def test_the_group_total_rides_along_with_its_substages(self, monkeypatch): + """Both, not either: the sub-stages answer the new question and the + group keeps the pre-split baseline comparable.""" + record = _emf_record(monkeypatch, ["preamble.ownership", "preamble.quota"]) + + assert "PreambleMs" in record["metrics"] + assert "PreambleOwnershipMs" in record["metrics"] + + def test_turn_shape_rides_as_properties_not_dimensions(self, monkeypatch): + """Dimensions multiply metric streams AND invite reading a p99 off a + slice too thin to have one. The Logs Insights widgets slice instead.""" + record = _emf_record( + monkeypatch, + ["preamble.quota"], + extra={"isResume": True, "deferredBuild": False, "hasAssistant": True}, + ) + + assert record["properties"]["isResume"] is True + assert record["properties"]["deferredBuild"] is False + assert record["properties"]["streamKind"] == "agent" + + def test_carries_no_message_content(self, monkeypatch): + """Same content-free rule as the log line and the cost rows.""" + record = _emf_record(monkeypatch, ["preamble.quota"]) + + assert set(record["properties"]) <= { + "streamKind", + "sessionId", + "isResume", + "deferredBuild", + "hasAssistant", + } + + def test_the_kill_switch_suppresses_only_the_metrics(self, monkeypatch, caplog): + """The marks stay ungated — they cost a perf_counter. The metrics cost + log ingestion and custom-metric charges, hence the switch.""" + monkeypatch.setenv("TURN_LATENCY_METRICS_ENABLED", "false") + calls = [] + monkeypatch.setattr( + "apis.shared.observability.emf.emit_emf_metrics", + lambda **kw: calls.append(kw), + ) + + prelude = TurnPrelude() + prelude.mark("preamble.quota") + with caplog.at_level(logging.INFO, logger="apis.inference_api.chat.turn_timing"): + prelude.emit(session_id="s", stream_kind="agent") + + assert calls == [] + assert [r for r in caplog.records if r.getMessage().startswith("turn_prelude ")] + + def test_an_empty_flag_value_leaves_metrics_on(self, monkeypatch): + """House rule: a workflow env var can materialize as "" and that must + not read as off.""" + monkeypatch.setenv("TURN_LATENCY_METRICS_ENABLED", "") + + assert _emf_record(monkeypatch, ["preamble.quota"])["metrics"] + + def test_a_metrics_failure_never_costs_the_log_line(self, monkeypatch, caplog): + """Two independent readers of the same turn; neither may take the + other down, and neither may take the turn down.""" + def _boom(**_kwargs): + raise RuntimeError("cloudwatch exploded") + + monkeypatch.setattr( + "apis.shared.observability.emf.emit_emf_metrics", _boom + ) + + prelude = TurnPrelude() + prelude.mark("preamble.quota") + with caplog.at_level(logging.INFO, logger="apis.inference_api.chat.turn_timing"): + prelude.emit(session_id="s", stream_kind="agent") # must not raise + + assert [r for r in caplog.records if r.getMessage().startswith("turn_prelude ")] + + +def _emf_record(monkeypatch, stages, *, extra=None): + """Capture the kwargs the prelude hands to `emit_emf_metrics`.""" + captured = {} + monkeypatch.setattr( + "apis.shared.observability.emf.emit_emf_metrics", + lambda **kw: captured.update(kw), + ) + + prelude = TurnPrelude() + for stage in stages: + prelude.mark(stage) + prelude.emit(session_id="s", stream_kind="agent", extra=extra) + return captured + + +def _emitted(prelude, *, session_id="s", extra=None): + """The JSON payload the emit would log, parsed back.""" + captured = {} + + class _Sink(logging.Handler): + def emit(self, record): + message = record.getMessage() + if message.startswith("turn_prelude "): + captured.update(json.loads(message[len("turn_prelude ") :])) + + logger = logging.getLogger("apis.inference_api.chat.turn_timing") + handler = _Sink() + logger.addHandler(handler) + previous = logger.level + logger.setLevel(logging.INFO) + try: + prelude.emit(session_id=session_id, stream_kind="agent", extra=extra) + finally: + logger.removeHandler(handler) + logger.setLevel(previous) + return captured diff --git a/backend/tests/apis/inference_api/test_warmup.py b/backend/tests/apis/inference_api/test_warmup.py new file mode 100644 index 000000000..c7a330517 --- /dev/null +++ b/backend/tests/apis/inference_api/test_warmup.py @@ -0,0 +1,111 @@ +"""Startup warm-up — pulls the first turn's lazy imports and boto service-model +loads forward to container start (apis/inference_api/warmup.py). + +Nothing here touches the network: boto3.client is patched, and the module list +is exercised with stand-ins so the tests don't depend on sympy import time. +""" + +import sys +import threading +import types +from unittest.mock import patch + +import pytest + +from apis.inference_api import warmup + + +class TestKillSwitch: + def test_defaults_on(self, monkeypatch): + monkeypatch.delenv(warmup.WARMUP_ENABLED_ENV, raising=False) + assert warmup.warmup_enabled() + + def test_empty_string_is_on(self, monkeypatch): + monkeypatch.setenv(warmup.WARMUP_ENABLED_ENV, "") + assert warmup.warmup_enabled() + + @pytest.mark.parametrize("value", ["false", "FALSE", " False "]) + def test_false_is_off(self, monkeypatch, value): + monkeypatch.setenv(warmup.WARMUP_ENABLED_ENV, value) + assert not warmup.warmup_enabled() + + def test_disabled_starts_no_thread(self, monkeypatch): + monkeypatch.setenv(warmup.WARMUP_ENABLED_ENV, "false") + with patch.object(warmup, "run_warmup") as run: + assert warmup.start_warmup_in_background() is None + run.assert_not_called() + + +class TestWarmModules: + def test_imports_each_module_once(self, monkeypatch): + fake = types.ModuleType("fake_warm_target") + fake.loaded = 0 + monkeypatch.setitem(sys.modules, "fake_warm_target", fake) + + with patch.object(warmup.importlib, "import_module", wraps=warmup.importlib.import_module) as imp: + warmup.warm_modules(["fake_warm_target"]) + + imp.assert_called_once_with("fake_warm_target") + + def test_a_missing_module_is_logged_not_raised(self): + # Must not raise: a warm-up failure is never a startup failure. + warmup.warm_modules(["definitely_not_a_real_module_xyz"]) + + def test_the_default_list_names_the_calculator_first(self): + # sympy behind the calculator is the single heaviest lazy import on the + # first-turn path; it must be the first thing warmed. + assert warmup.WARM_MODULES[0] == "strands_tools.calculator" + + +class TestWarmBotoClients: + def test_builds_one_client_per_service_in_the_configured_region(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + with patch("boto3.client") as client: + warmup.warm_boto_clients(["bedrock-runtime", "s3"]) + + assert [c.args[0] for c in client.call_args_list] == ["bedrock-runtime", "s3"] + assert all(c.kwargs["region_name"] == "us-west-2" for c in client.call_args_list) + + def test_skips_without_a_region(self, monkeypatch): + monkeypatch.delenv("AWS_REGION", raising=False) + monkeypatch.delenv("AWS_DEFAULT_REGION", raising=False) + with patch("boto3.client") as client: + warmup.warm_boto_clients(["s3"]) + client.assert_not_called() + + def test_a_failing_client_does_not_stop_the_rest(self, monkeypatch): + monkeypatch.setenv("AWS_REGION", "us-west-2") + calls = [] + + def flaky(service, **kwargs): + calls.append(service) + if service == "bedrock-runtime": + raise RuntimeError("no such service") + return object() + + with patch("boto3.client", side_effect=flaky): + warmup.warm_boto_clients(["bedrock-runtime", "s3"]) + + assert calls == ["bedrock-runtime", "s3"] + + +class TestBackground: + def test_runs_on_a_daemon_thread_and_returns_immediately(self, monkeypatch): + monkeypatch.delenv(warmup.WARMUP_ENABLED_ENV, raising=False) + started = threading.Event() + release = threading.Event() + + def slow(): + started.set() + release.wait(timeout=5) + + with patch.object(warmup, "run_warmup", side_effect=slow): + thread = warmup.start_warmup_in_background() + assert thread is not None + assert thread.daemon + assert started.wait(timeout=5) + # Caller is not blocked while the warm-up is still running. + assert thread.is_alive() + release.set() + thread.join(timeout=5) + assert not thread.is_alive() diff --git a/backend/tests/apis/shared/costs/__init__.py b/backend/tests/apis/shared/costs/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/backend/tests/apis/shared/costs/test_calculator_long_ttl.py b/backend/tests/apis/shared/costs/test_calculator_long_ttl.py new file mode 100644 index 000000000..78ed45bcc --- /dev/null +++ b/backend/tests/apis/shared/costs/test_calculator_long_ttl.py @@ -0,0 +1,53 @@ +"""1h static-prefix cache writes are billed at 2x base (thresholds spec §3.6, PR-5).""" + +import pytest + +from apis.shared.costs.calculator import CostCalculator + +PRICING = { + "inputPricePerMtok": 2.0, # base + "outputPricePerMtok": 10.0, + "cacheReadPricePerMtok": 0.2, # 0.1x + "cacheWritePricePerMtok": 2.5, # 1.25x (5m) +} + + +def _write_cost(read, write, static=None): + usage = {"inputTokens": 0, "outputTokens": 0, "cacheReadInputTokens": read, "cacheWriteInputTokens": write} + _, b = CostCalculator.calculate_message_cost(usage, PRICING, long_ttl_static_prefix_tokens=static) + return round(b.cache_write_cost, 9) + + +def test_multiplier_is_bedrocks_1h_premium(): + assert CostCalculator.LONG_TTL_CACHE_WRITE_MULTIPLIER == 2.0 + + +def test_default_prices_every_write_at_5m(): + assert _write_cost(read=0, write=30_000) == pytest.approx(30_000 / 1e6 * 2.5) + assert _write_cost(read=0, write=30_000, static=None) == pytest.approx(30_000 / 1e6 * 2.5) + + +def test_cold_everything_bills_static_at_2x_and_history_at_5m(): + # 10k static + 20k history all written, nothing read. + expected = 10_000 / 1e6 * 4.0 + 20_000 / 1e6 * 2.5 + assert _write_cost(read=0, write=30_000, static=10_000) == pytest.approx(expected) + + +def test_static_read_means_history_only_write_at_5m(): + # Static segment was read (1h entry alive); only history re-wrote. + assert _write_cost(read=10_000, write=20_000, static=10_000) == pytest.approx(20_000 / 1e6 * 2.5) + assert _write_cost(read=12_000, write=20_000, static=10_000) == pytest.approx(20_000 / 1e6 * 2.5) + + +def test_partially_read_static_bills_the_unread_remainder_at_2x(): + # Read 4k of a 10k static segment (tools hit, system missed) → 6k at 2x. + expected = 6_000 / 1e6 * 4.0 + 14_000 / 1e6 * 2.5 + assert _write_cost(read=4_000, write=20_000, static=10_000) == pytest.approx(expected) + + +def test_write_smaller_than_static_is_capped_by_the_write(): + assert _write_cost(read=0, write=3_000, static=10_000) == pytest.approx(3_000 / 1e6 * 4.0) + + +def test_no_write_no_cost(): + assert _write_cost(read=10_000, write=0, static=10_000) == 0.0 diff --git a/backend/tests/apis/shared/middleware/test_compression_middleware.py b/backend/tests/apis/shared/middleware/test_compression_middleware.py new file mode 100644 index 000000000..a8edc03f4 --- /dev/null +++ b/backend/tests/apis/shared/middleware/test_compression_middleware.py @@ -0,0 +1,293 @@ +"""Tests for StreamSafeGZipMiddleware. + +app-api compresses its JSON surface, and the entire risk of doing so lands on +the one surface it must never touch: `text/event-stream`. Compressing an SSE +stream — or merely *buffering its response headers* — degrades or breaks the +chat path, which is the product's main path. + +So most of what follows is about what the middleware refuses to do. The +sharpest test is `test_excluded_response_start_is_sent_before_first_chunk`: +Starlette's stock responders withhold `http.response.start` until the first +body message arrives, which on a chat turn is the model's first token. That +would put the agent's thinking time in front of every client's "stream is +open" transition, so this middleware forwards the start message the moment +it can see the response is excluded. +""" + +from __future__ import annotations + +import asyncio +import gzip +import json + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from starlette.middleware.gzip import DEFAULT_EXCLUDED_CONTENT_TYPES +from starlette.responses import Response, StreamingResponse +from starlette.types import Message, Receive, Scope, Send + +from apis.shared.middleware.compression import ( + EXCLUDED_CONTENT_TYPES, + StreamSafeGZipMiddleware, +) + +GZIP = {"Accept-Encoding": "gzip"} + +#: Comfortably over the 500-byte floor these tests configure, and +#: compressible enough that the result is smaller than the input. +BIG_PAYLOAD = {"rows": [{"name": f"row {i}", "status": "OK"} for i in range(200)]} + + +@pytest.fixture +def app() -> FastAPI: + app = FastAPI() + app.add_middleware(StreamSafeGZipMiddleware, minimum_size=500, compresslevel=6) + + @app.get("/big") + def big() -> dict: + return BIG_PAYLOAD + + @app.get("/small") + def small() -> dict: + return {"ok": True} + + @app.get("/sse") + def sse() -> StreamingResponse: + async def events(): + for i in range(50): + yield f"data: {json.dumps({'i': i, 'text': 'x' * 100})}\n\n".encode() + + return StreamingResponse(events(), media_type="text/event-stream") + + @app.get("/zip") + def archive() -> Response: + return Response(b"PK\x03\x04" + b"\x00" * 4000, media_type="application/zip") + + @app.get("/pre-encoded") + def pre_encoded() -> Response: + # A handler that compressed its own body — e.g. a proxied upstream + # response relayed with its encoding intact. + return Response( + gzip.compress(b"already encoded " * 500), + media_type="application/json", + headers={"Content-Encoding": "gzip"}, + ) + + return app + + +@pytest.fixture +def client(app: FastAPI) -> TestClient: + return TestClient(app) + + +# --- the JSON surface: this is the payoff --------------------------------- + + +def test_large_json_is_compressed_and_round_trips(client: TestClient) -> None: + raw = client.get("/big", headers={"Accept-Encoding": "identity"}) + compressed = client.get("/big", headers=GZIP) + + assert compressed.headers["content-encoding"] == "gzip" + # httpx decodes transparently; the decoded body must be byte-identical. + assert compressed.json() == BIG_PAYLOAD + assert int(compressed.headers["content-length"]) < len(raw.content) + assert "accept-encoding" in compressed.headers["vary"].lower() + + +def test_small_json_is_left_alone(client: TestClient) -> None: + response = client.get("/small", headers=GZIP) + + assert "content-encoding" not in response.headers + assert response.json() == {"ok": True} + + +def test_client_without_gzip_gets_an_uncompressed_body(client: TestClient) -> None: + response = client.get("/big", headers={"Accept-Encoding": "identity"}) + + assert "content-encoding" not in response.headers + assert response.json() == BIG_PAYLOAD + # Still varies, so a cache can't hand this body to a gzip client. + assert "accept-encoding" in response.headers["vary"].lower() + + +# --- the SSE surface: this is the risk ------------------------------------ + + +def test_sse_is_never_compressed(client: TestClient) -> None: + response = client.get("/sse", headers=GZIP) + + assert response.status_code == 200 + assert "content-encoding" not in response.headers + assert response.headers["content-type"].startswith("text/event-stream") + # Every frame intact, in order, and readable as plain text. + frames = [f for f in response.text.split("\n\n") if f] + assert len(frames) == 50 + assert json.loads(frames[0].removeprefix("data: "))["i"] == 0 + assert json.loads(frames[-1].removeprefix("data: "))["i"] == 49 + + +async def _drive(middleware: StreamSafeGZipMiddleware, scope: Scope) -> list[Message]: + """Run `middleware` over `scope`, returning the messages it sent.""" + sent: list[Message] = [] + + async def send(message: Message) -> None: + sent.append(message) + + async def receive() -> Message: # pragma: no cover - never awaited + return {"type": "http.disconnect"} + + await middleware(scope, receive, send) + return sent + + +@pytest.mark.parametrize("accept_encoding", [b"gzip", b"identity"]) +def test_excluded_response_start_is_sent_before_first_chunk( + accept_encoding: bytes, +) -> None: + """Headers must not wait on the agent's first token. + + The stub below opens an SSE response and then stalls, exactly like a turn + that spends its first seconds on a tool call. Stock `GZipMiddleware` + would hold `http.response.start` for the whole stall; this must not. + """ + first_chunk_released = asyncio.Event() + + async def stalling_sse(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": 200, + "headers": [(b"content-type", b"text/event-stream; charset=utf-8")], + } + ) + await first_chunk_released.wait() + await send({"type": "http.response.body", "body": b"data: hi\n\n"}) + + middleware = StreamSafeGZipMiddleware(stalling_sse, minimum_size=500) + scope: Scope = { + "type": "http", + "headers": [(b"accept-encoding", accept_encoding)], + } + + async def scenario() -> list[Message]: + sent: list[Message] = [] + + async def send(message: Message) -> None: + sent.append(message) + + async def receive() -> Message: # pragma: no cover - never awaited + return {"type": "http.disconnect"} + + task = asyncio.create_task(middleware(scope, receive, send)) + # Let the stub reach its stall, then look at what the client has. + await asyncio.sleep(0) + await asyncio.sleep(0) + headers_before_body = list(sent) + + first_chunk_released.set() + await task + return headers_before_body + + headers_before_body = asyncio.run(scenario()) + + assert [m["type"] for m in headers_before_body] == ["http.response.start"] + assert not any( + name.lower() == b"content-encoding" + for name, _ in headers_before_body[0]["headers"] + ) + + +def test_compressible_streaming_response_still_withholds_start() -> None: + """The eager path is scoped to excluded responses only. + + A compressible stream *must* keep buffering its start message — that is + where `Content-Encoding` gets added and `Content-Length` dropped. + """ + released = asyncio.Event() + + async def stalling_json(scope: Scope, receive: Receive, send: Send) -> None: + await send( + { + "type": "http.response.start", + "status": 200, + "headers": [(b"content-type", b"application/json")], + } + ) + await released.wait() + await send( + {"type": "http.response.body", "body": b"x" * 2000, "more_body": True} + ) + await send({"type": "http.response.body", "body": b""}) + + middleware = StreamSafeGZipMiddleware(stalling_json, minimum_size=500) + scope: Scope = {"type": "http", "headers": [(b"accept-encoding", b"gzip")]} + + async def scenario() -> tuple[list[Message], list[Message]]: + sent: list[Message] = [] + + async def send(message: Message) -> None: + sent.append(message) + + async def receive() -> Message: # pragma: no cover - never awaited + return {"type": "http.disconnect"} + + task = asyncio.create_task(middleware(scope, receive, send)) + await asyncio.sleep(0) + await asyncio.sleep(0) + before = list(sent) + + released.set() + await task + return before, sent + + before, after = asyncio.run(scenario()) + + assert before == [] + assert after[0]["type"] == "http.response.start" + assert (b"content-encoding", b"gzip") in after[0]["headers"] + + +# --- other pass-throughs --------------------------------------------------- + + +def test_pre_encoded_body_is_not_re_encoded(client: TestClient) -> None: + response = client.get("/pre-encoded", headers=GZIP) + + assert response.headers["content-encoding"] == "gzip" + # One layer of gzip, not two: httpx strips exactly one. + assert response.content == b"already encoded " * 500 + + +def test_already_compressed_payload_types_are_skipped(client: TestClient) -> None: + response = client.get("/zip", headers=GZIP) + + assert "content-encoding" not in response.headers + assert response.content.startswith(b"PK\x03\x04") + + +def test_non_http_scopes_are_passed_straight_through() -> None: + """Voice mode is a WebSocket proxy; it must never meet a responder.""" + seen: list[Scope] = [] + + async def inner(scope: Scope, receive: Receive, send: Send) -> None: + seen.append(scope) + + middleware = StreamSafeGZipMiddleware(inner) + scope: Scope = {"type": "websocket", "headers": [(b"accept-encoding", b"gzip")]} + + asyncio.run(_drive(middleware, scope)) + + assert seen == [scope] + + +def test_starlette_exclusions_are_carried_forward() -> None: + """Drift guard on the pinned Starlette. + + `text/event-stream` is excluded here *because* it is excluded upstream — + if a version bump ever renames or empties that constant, this fails + rather than silently un-excluding SSE. + """ + assert "text/event-stream" in DEFAULT_EXCLUDED_CONTENT_TYPES + assert set(DEFAULT_EXCLUDED_CONTENT_TYPES) <= set(EXCLUDED_CONTENT_TYPES) diff --git a/backend/tests/architecture/test_admin_scope_coverage.py b/backend/tests/architecture/test_admin_scope_coverage.py index b93e4a1af..3b0d81c20 100644 --- a/backend/tests/architecture/test_admin_scope_coverage.py +++ b/backend/tests/architecture/test_admin_scope_coverage.py @@ -44,6 +44,7 @@ "routes.py": "admin.models", "quota/routes.py": "admin.quota", "costs/routes.py": "admin.costs", + "feedback/routes.py": "admin.costs", # eval-sampling queue sits beside the cost rows "users/routes.py": "admin.users", "tools/routes.py": "admin.tools", "skills/routes.py": "admin.skills", @@ -54,6 +55,7 @@ "user_menu_links/routes.py": "admin.user_menu_links", "announcements/routes.py": "admin.announcements", "system_prompts/routes.py": "admin.system_prompts", + "agent_templates/routes.py": "admin.agent_templates", "fine_tuning/routes.py": "admin.fine_tuning", } diff --git a/backend/tests/architecture/test_source_tree_not_written.py b/backend/tests/architecture/test_source_tree_not_written.py new file mode 100644 index 000000000..23fd06952 --- /dev/null +++ b/backend/tests/architecture/test_source_tree_not_written.py @@ -0,0 +1,152 @@ +"""The services must not write into their own source tree. + +The container images ship ``/app/src`` and ``/app/.venv`` root-owned, so the +runtime user cannot rewrite the code it executes. That only holds as long as +nothing creates directories or files inside the source tree at runtime. + +Something did, until it was deleted: both API lifespans ran ``os.makedirs`` +against a ``base_dir`` derived from ``__file__`` (landing in ``/apis``), +and ``agents/utils/config.py`` did the same one level up to hold local tool +output. That local-output mechanism was dead — its only writer could fire for +two tool names that no longer exist anywhere in the repo, and the static mounts +that served it were never referenced by the frontend. Output goes to S3. + +These tests keep it from coming back. If one fails, the container is about to +fail at boot — or, worse, boot and then fail the first time it tries to write — +so failing here is the cheap version. +""" + +import ast +from pathlib import Path + +_BACKEND_SRC = Path(__file__).resolve().parent.parent.parent / "src" + +# Runs on SageMaker / in the code-interpreter sandbox, not in our containers. +_OUT_OF_CONTAINER = ( + "fine_tuning/sagemaker_scripts", + "documents/ingestion/generate_random_doc.py", +) + +# Directory names the deleted mechanism used. A new one of these joined onto a +# path derived from __file__ is the regression we are guarding against. +_WRITABLE_NAMES = {"output", "uploads", "generated_images"} + + +def _container_python_files(): + for path in sorted(_BACKEND_SRC.rglob("*.py")): + rel = path.relative_to(_BACKEND_SRC).as_posix() + if any(marker in rel for marker in _OUT_OF_CONTAINER): + continue + yield path + + +def _root_name(expr): + """Walk `a.b.c` back to `a`, returning the bare name or None.""" + while isinstance(expr, ast.Attribute): + expr = expr.value + return expr.id if isinstance(expr, ast.Name) else None + + +def test_no_writable_dir_is_derived_from_file(): + """Catch a writable directory built from ``__file__``. + + This is the exact pattern that made the source tree writable: + + base_dir = Path(__file__).parent.parent # lands in the source tree + output_dir = os.path.join(base_dir, "output") + + Reading a file relative to ``__file__`` stays fine (templates, .env). What + this flags is joining one of the writable directory names onto a path + derived from ``__file__`` — precise enough that it does not fire on every + module that merely mentions both. + """ + offenders = [] + + for path in _container_python_files(): + try: + source = path.read_text(encoding="utf-8") + tree = ast.parse(source, filename=str(path)) + except (SyntaxError, UnicodeDecodeError): + continue + if "__file__" not in source: + continue + + # Names bound to something derived from __file__. + file_derived: set[str] = set() + for node in ast.walk(tree): + if isinstance(node, ast.Assign): + try: + value_src = ast.get_source_segment(source, node.value) or "" + except Exception: # pragma: no cover - defensive + value_src = "" + if "__file__" in value_src: + for target in node.targets: + if isinstance(target, ast.Name): + file_derived.add(target.id) + if not file_derived: + continue + + def _is_writable_const(expr): + return isinstance(expr, ast.Constant) and expr.value in _WRITABLE_NAMES + + for node in ast.walk(tree): + # os.path.join(base_dir, "output") + if isinstance(node, ast.Call) and len(node.args) >= 2: + func = node.func + name = func.attr if isinstance(func, ast.Attribute) else getattr(func, "id", "") + if name == "join" and _root_name(node.args[0]) in file_derived: + if any(_is_writable_const(a) for a in node.args[1:]): + offenders.append(f"{path.relative_to(_BACKEND_SRC)}:{node.lineno}") + # base_dir / "output" + if isinstance(node, ast.BinOp) and isinstance(node.op, ast.Div): + if _root_name(node.left) in file_derived and _is_writable_const(node.right): + offenders.append(f"{path.relative_to(_BACKEND_SRC)}:{node.lineno}") + + assert not offenders, ( + "These build a writable directory from __file__, which puts it inside the " + "source tree the image ships read-only. Generated output belongs in S3:\n " + + "\n ".join(sorted(set(offenders))) + ) + + +def test_no_makedirs_under_the_source_tree(): + """Catch a hard-coded write under /app/src (the read-only source tree).""" + offenders = [] + for path in _container_python_files(): + source = path.read_text(encoding="utf-8") + for lineno, line in enumerate(source.splitlines(), start=1): + stripped = line.strip() + if stripped.startswith("#"): + continue + if "/app/src" in stripped and ("makedirs" in stripped or "mkdir" in stripped): + offenders.append(f"{path.relative_to(_BACKEND_SRC)}:{lineno}") + + assert not offenders, ( + "Hard-coded writes under /app/src (the read-only source tree):\n " + + "\n ".join(offenders) + ) + + +def test_the_deleted_local_output_mechanism_stays_deleted(): + """The static mounts and their directories are gone; keep them gone. + + ``app.mount("/output", StaticFiles(...))`` served directories that nothing + wrote to and nothing linked to, backed by a saver that could not fire. + Reintroducing a StaticFiles mount over a source-tree path would make the + image need a writable /app again — and would expose whatever it serves on + an unauthenticated route. + """ + offenders = [] + for path in _container_python_files(): + source = path.read_text(encoding="utf-8") + if "StaticFiles" not in source: + continue + for lineno, line in enumerate(source.splitlines(), start=1): + if "StaticFiles" in line and not line.strip().startswith("#"): + offenders.append(f"{path.relative_to(_BACKEND_SRC)}:{lineno}: {line.strip()}") + + assert not offenders, ( + "StaticFiles mounts are back. They serve from the source tree the image " + "ships read-only, on an unauthenticated route. Generated output goes to " + "S3:\n " + "\n ".join(offenders) + ) diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 8d9038c95..47a37ce03 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -222,3 +222,122 @@ def override_admin_auth(app, impl) -> None: "did the router fail to mount, or was the app built before a module " "reload replaced its dependencies?" ) + + +# --------------------------------------------------------------------------- +# No off-box sockets. Every AWS call in this suite is supposed to be mocked +# (moto), and moto never opens a real socket — so "connected to something that +# is not localhost" is a precise detector for a test that escaped the mock. +# +# It is worth enforcing because the failure is otherwise *invisible*. Service +# code here is deliberately fail-open (a user should not lose their session +# because a table blipped), so a test that mocks one dependency and misses a +# second gets a real DynamoDB client, a real request, a swallowed exception, +# and a green assertion. That is not hypothetical: 25 cases across 6 files were +# doing it, and the suite reported the same 9196 passed with and without the +# connections. On a developer machine `~/.aws/config` carries a `[default]` +# profile, so those were *authenticated* requests to real AWS from a unit test; +# in CI one occasionally stalled in TLS and botocore's connect timeout × +# retries turned a silent escape into a 72-minute hang. +# +# Hooking botocore would be ambiguous — moto intercepts `before-send` itself — +# so the guard sits at the socket layer, below every SDK. +# +# Two halves, because raising is not enough: the fail-open code under test +# swallows the error, so the violation is also recorded and asserted at +# teardown. `AWS_TEST_ALLOW_OFF_BOX_SOCKETS=1` disables it for the rare test +# that genuinely needs the network. +# --------------------------------------------------------------------------- +import socket as _socket + +_OFF_BOX_ALLOWED_HOSTS = frozenset({"127.0.0.1", "::1", "localhost", "0.0.0.0", ""}) +_off_box_attempts: list = [] +_real_socket_connect = _socket.socket.connect +_real_socket_connect_ex = _socket.socket.connect_ex + + +def _is_off_box(sock, address) -> "str | None": + """The destination host when this is an off-box TCP connect, else ``None``. + + Only AF_INET/AF_INET6 **stream** sockets are guarded: UDP is left alone so + DNS resolution keeps working, and AF_UNIX has no host to check. + """ + if getattr(sock, "type", None) != _socket.SOCK_STREAM: + return None + if getattr(sock, "family", None) not in (_socket.AF_INET, _socket.AF_INET6): + return None + if not isinstance(address, tuple) or not address: + return None + host = str(address[0]) + if host in _OFF_BOX_ALLOWED_HOSTS or host.startswith("127."): + return None + return host + + +def _guard_connect(self, address): + host = _is_off_box(self, address) + if host is not None: + _off_box_attempts.append(host) + raise RuntimeError( + f"Blocked an off-box connection to {host!r}. Tests must not reach real " + "AWS — mock it (moto, or patch the repository/client this code path " + "builds). See the 'No off-box sockets' note in tests/conftest.py." + ) + return _real_socket_connect(self, address) + + +def _guard_connect_ex(self, address): + host = _is_off_box(self, address) + if host is not None: + _off_box_attempts.append(host) + raise RuntimeError(f"Blocked an off-box connection to {host!r} (see tests/conftest.py).") + return _real_socket_connect_ex(self, address) + + +# Warm tiktoken's BPE vocabulary *before* the guard arms. `csv_chunker` calls +# `tiktoken.get_encoding("cl100k_base")`, which downloads the vocabulary from an +# external CDN on first use and caches it on disk. That is a legitimate asset +# fetch, not an escaped AWS call — but it is also a real network dependency of +# the test run, which is why it only showed up on CI (cold cache) and never +# locally (warm one). Fetching it here keeps the guarded window hermetic without +# widening the allowlist, and makes the dependency explicit rather than +# incidental. Best-effort: offline, the CSV chunker tests fail on their own terms +# rather than on a confusing socket error. +try: # noqa: SIM105 + import tiktoken as _tiktoken + + _tiktoken.get_encoding("cl100k_base") +except Exception: # noqa: BLE001 - never block collection on a cache warm-up + pass + + +if os.environ.get("AWS_TEST_ALLOW_OFF_BOX_SOCKETS") != "1": + _socket.socket.connect = _guard_connect + _socket.socket.connect_ex = _guard_connect_ex + + + + +@pytest.fixture(autouse=True) +def _fail_on_off_box_sockets(): + """Fail a test that tried to leave the box, *even if it swallowed the error*. + + The raise above stops the connection; this is what makes it visible. Without + it a fail-open code path turns the block into a silent no-op and the test + still passes — which is exactly how this went unnoticed. + """ + _off_box_attempts.clear() + try: + yield + finally: + attempted = list(_off_box_attempts) + _off_box_attempts.clear() + if not attempted: + return + hosts = ", ".join(sorted(set(attempted))) + pytest.fail( + f"This test opened {len(attempted)} connection(s) off-box ({hosts}). " + "Something it exercises built a real AWS client. Mock that dependency; " + "a fail-open except block hides the failure but the call still happens. " + "See the 'No off-box sockets' note above." + ) diff --git a/backend/tests/costs/test_content_free_projections.py b/backend/tests/costs/test_content_free_projections.py index ffaf7085f..d58721c31 100644 --- a/backend/tests/costs/test_content_free_projections.py +++ b/backend/tests/costs/test_content_free_projections.py @@ -148,12 +148,115 @@ async def test_top_sessions_reader_keeps_title_by_decision(storage): assert "compaction" not in rows[0] # and it never widened into the diagnostic fields -@pytest.mark.asyncio -async def test_deleted_sessions_are_excluded_from_the_diagnostic_list(storage): - _seed(storage) +def _seed_deleted(storage): storage.sessions_metadata_table.put_item(Item={ "PK": f"USER#{USER_ID}", "SK": "S#gone", "GSI_PK": "SESSION#gone", "GSI_SK": "META", - "sessionId": "gone", "userId": USER_ID, "deleted": True, "totalCost": Decimal("9"), + "sessionId": "gone", "userId": USER_ID, "deleted": True, "status": "deleted", + "totalCost": Decimal("9"), "title": "DELETED TITLE", }) + + +@pytest.mark.asyncio +async def test_deleted_sessions_are_excluded_from_the_diagnostic_list_by_default(storage): + _seed(storage) + _seed_deleted(storage) rows = await storage.get_user_session_diagnostics(USER_ID) assert [r["sessionId"] for r in rows] == [SESSION_ID] + + +@pytest.mark.asyncio +async def test_deleted_sessions_are_listed_on_request_and_stay_content_free(storage): + # A delete is a tombstone, not a refund: the row's cost survives it, so + # the audit must be able to see it to account for the period total. + _seed(storage) + _seed_deleted(storage) + rows = await storage.get_user_session_diagnostics(USER_ID, include_deleted=True) + by_id = {r["sessionId"]: r for r in rows} + assert set(by_id) == {SESSION_ID, "gone"} + assert by_id["gone"]["deleted"] is True + assert by_id["gone"]["status"] == "deleted" + assert by_id["gone"]["totalCost"] == 9 + assert "title" not in by_id["gone"] + + +@pytest.mark.asyncio +async def test_feedback_rows_come_back_content_free_and_keyed_to_the_call(storage): + _seed(storage) + storage.sessions_metadata_table.put_item(Item={ + "PK": f"USER#{USER_ID}", "SK": f"F#{SESSION_ID}#3", + "GSI_PK": f"SESSION#{SESSION_ID}", "GSI_SK": "F#3", + "sessionId": SESSION_ID, "messageId": Decimal(3), "userId": USER_ID, + "value": Decimal(-1), "reason": "wrong", "signal": "explicit", "updatedAt": "2026-09-16T00:00:00Z", + "ttl": Decimal(1_800_000_000), + # A stray content-bearing attribute must never leave the reader even + # if something wrote one (the writer cannot, but the reader is the guard). + "displayText": "SECRET", + }) + rows = await storage.get_session_feedback_rows(SESSION_ID) + assert len(rows) == 1 + row = rows[0] + assert content_bearing_paths(row) == [] + assert row == {"sessionId": SESSION_ID, "messageId": 3, "value": -1, "reason": "wrong", "signal": "explicit", "updatedAt": "2026-09-16T00:00:00Z"} + # Joins the C# row on messageId. + records = await storage.get_session_cost_records(SESSION_ID) + assert records[0]["messageId"] == row["messageId"] + assert await storage.get_session_feedback_rows("no-such-session") == [] + + +@pytest.mark.asyncio +async def test_recent_down_thumbs_queue_is_content_free_and_newest_first(storage): + _seed(storage) + for i, ts in ((1, "2026-09-16T00:00:01Z"), (2, "2026-09-16T00:00:02Z")): + storage.sessions_metadata_table.put_item(Item={ + "PK": f"USER#{USER_ID}", "SK": f"F#{SESSION_ID}#{i}", + "GSI_PK": f"SESSION#{SESSION_ID}", "GSI_SK": f"F#{i}", + "GSI1PK": "FEEDBACK#down", "GSI1SK": ts, + "sessionId": SESSION_ID, "messageId": Decimal(i), "userId": USER_ID, + "value": Decimal(-1), "reason": "wrong", "signal": "explicit", "updatedAt": ts, + "evaluation": {"reason": "wrong", "scores": {"Builtin.Correctness": {"value": Decimal("0.5"), "n": Decimal(1), "explanation": "SECRET"}}}, + "displayText": "SECRET", + }) + rows = await storage.get_recent_down_thumbs(limit=10) + assert [r["messageId"] for r in rows] == [2, 1] + assert all(content_bearing_paths(r) == [] for r in rows) + assert "userId" not in rows[0] and "displayText" not in rows[0] + assert rows[0]["evaluation"]["scores"]["Builtin.Correctness"] == {"value": 0.5, "n": 1} + + +def _seed_thumb(storage, message_id, value, ts, *, reason=None, signal="explicit", session_id=SESSION_ID): + item = { + "PK": f"USER#{USER_ID}", "SK": f"F#{session_id}#{message_id}", + "GSI_PK": f"SESSION#{session_id}", "GSI_SK": f"F#{message_id}", + "GSI1PK": f"FEEDBACK#{'down' if value == -1 else 'up'}", "GSI1SK": ts, + "sessionId": session_id, "messageId": Decimal(message_id), "userId": USER_ID, + "value": Decimal(value), "signal": signal, "updatedAt": ts, + "displayText": "SECRET", + } + if reason: + item["reason"] = reason + storage.sessions_metadata_table.put_item(Item=item) + + +@pytest.mark.asyncio +async def test_feedback_window_reads_both_polarities_content_free_and_time_bounded(storage): + _seed(storage) + _seed_thumb(storage, 1, -1, "2026-09-02T00:00:00Z", reason="wrong") + _seed_thumb(storage, 2, 1, "2026-09-03T00:00:00Z") + _seed_thumb(storage, 3, -1, "2026-08-01T00:00:00Z") # before the window + _seed_thumb(storage, 4, -1, "2026-10-01T00:00:00Z") # after the window + + rows = await storage.get_feedback_in_window(start="2026-09-01T00:00:00Z", end="2026-09-30T00:00:00Z") + + assert sorted(r["messageId"] for r in rows) == [1, 2] + assert {r["value"] for r in rows} == {-1, 1} + # No user id reaches an aggregate surface, and no content of any kind. + assert all(content_bearing_paths(r) == [] for r in rows) + assert all("userId" not in r and "displayText" not in r for r in rows) + assert [r for r in rows if r["value"] == -1][0]["reason"] == "wrong" + + +@pytest.mark.asyncio +async def test_feedback_window_is_empty_when_nothing_falls_in_it(storage): + _seed(storage) + _seed_thumb(storage, 1, -1, "2026-08-01T00:00:00Z") + assert await storage.get_feedback_in_window(start="2026-09-01T00:00:00Z", end="2026-09-30T00:00:00Z") == [] diff --git a/backend/tests/fine_tuning/conftest.py b/backend/tests/fine_tuning/conftest.py index 1a24dbdc7..054bf45bb 100644 --- a/backend/tests/fine_tuning/conftest.py +++ b/backend/tests/fine_tuning/conftest.py @@ -16,8 +16,16 @@ def aws(monkeypatch): monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "testing") monkeypatch.setenv("AWS_SECURITY_TOKEN", "testing") monkeypatch.setenv("AWS_SESSION_TOKEN", "testing") + # Drop any boto3 client cached under a PREVIOUS test's moto backend, and + # again on the way out so nothing built here leaks into the next test. + # `mock_aws()` is per-test, so a cached client outliving its mock would + # talk to a torn-down backend — an order-dependent failure. + from apis.shared.aws_clients import reset_cached_clients + + reset_cached_clients() with mock_aws(): yield + reset_cached_clients() @pytest.fixture() diff --git a/backend/tests/lambdas/conftest.py b/backend/tests/lambdas/conftest.py index 0cf359feb..a0956a73b 100644 --- a/backend/tests/lambdas/conftest.py +++ b/backend/tests/lambdas/conftest.py @@ -21,8 +21,16 @@ def aws(monkeypatch): monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "testing") monkeypatch.setenv("AWS_SECURITY_TOKEN", "testing") monkeypatch.setenv("AWS_SESSION_TOKEN", "testing") + # Drop any boto3 client cached under a PREVIOUS test's moto backend, and + # again on the way out so nothing built here leaks into the next test. + # `mock_aws()` is per-test, so a cached client outliving its mock would + # talk to a torn-down backend — an order-dependent failure. + from apis.shared.aws_clients import reset_cached_clients + + reset_cached_clients() with mock_aws(): yield + reset_cached_clients() def _gsi(index_name, hash_key, range_key): diff --git a/backend/tests/rbac/test_app_role_service.py b/backend/tests/rbac/test_app_role_service.py index 5f786a2c1..49aeff174 100644 --- a/backend/tests/rbac/test_app_role_service.py +++ b/backend/tests/rbac/test_app_role_service.py @@ -24,6 +24,28 @@ from apis.shared.rbac.service import AppRoleService +@pytest.fixture(autouse=True) +def _no_live_tool_catalog(): + """Stub the tool-catalog snapshot the RBAC path consults. + + ``can_access_tool`` / ``filter_requested_tools`` fall through to + ``tools.freshness._get_snapshot`` -> ``get_tool_catalog_repository().list_tools()``, + which builds a real DynamoDB client. ``freshness`` memoizes on a module-level + TTL, so whether this fires depends on which test warmed it first — these pass + alone and reached AWS only in full-suite order. Fail-open, so the failure was + swallowed. See the off-box socket guard in ``tests/conftest.py``. + """ + from unittest.mock import AsyncMock as _AsyncMock, MagicMock as _MagicMock, patch as _patch + + repo = _MagicMock() + repo.list_tools = _AsyncMock(return_value=[]) + with _patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ): + yield + + @pytest.fixture def user(): """A test user with two JWT roles.""" diff --git a/backend/tests/routes/conftest.py b/backend/tests/routes/conftest.py index d23567815..e1704415b 100644 --- a/backend/tests/routes/conftest.py +++ b/backend/tests/routes/conftest.py @@ -238,3 +238,54 @@ def _admin_client(app: FastAPI) -> TestClient: return TestClient(app) return _admin_client + + +@pytest.fixture(autouse=True) +def _no_live_infrastructure_reads(monkeypatch): + """Stub the infrastructure lookups the route paths make behind whatever the + test itself mocked. + + Every one of these is fail-open — the route keeps serving if the table is + unreachable — so a live DynamoDB call from a unit test was swallowed and the + assertion passed anyway. They are stubbed to benign defaults here rather than + per file because the same handful recur across the whole directory. A test + that cares about one of them patches it itself, and that patch wins (it is + applied inside this one). + + See the off-box socket guard in ``tests/conftest.py``. + """ + from unittest.mock import AsyncMock as _AsyncMock, patch as _patch + + stubs = [ + # Inference invocation path. + ("apis.inference_api.chat.system_prompt_resolver.get_session_metadata", None), + ("apis.shared.files.document_read.session_has_documents", False), + ("apis.shared.files.document_read.session_has_tabular_files", False), + # Converse path: model routing, the rate-limit window, and the quota + # override lookup. The 429 test drives quota through `get_quota_checker`, + # which is a different seam, so stubbing these does not weaken it. + ("apis.shared.models.managed_models.list_managed_models", []), + ("apis.shared.rate_limit.RateLimiter.check_rate_limit", True), + ("agents.main_agent.quota.repository.QuotaRepository.get_active_override", None), + ("agents.main_agent.quota.repository.QuotaRepository.query_user_assignment", None), + ("agents.main_agent.quota.repository.QuotaRepository.query_role_assignments", []), + ("agents.main_agent.quota.repository.QuotaRepository.list_assignments_by_type", []), + # RBAC role resolution. Tests that care about authorization override the + # auth dependency itself, which never reaches this. + ("apis.shared.rbac.repository.AppRoleRepository.get_roles_for_jwt_role", []), + ("apis.shared.rbac.repository.AppRoleRepository.get_role", None), + # Agent-detail label resolution. + ("apis.shared.memory.repository.MemorySpaceRepository.get_space", None), + # Session delete cascades into artifact share cleanup. Whether it fires + # depends on whether an earlier test left artifacts "configured", which + # is why these passed alone and only failed in full-suite order. + ("apis.app_api.artifacts.service.ArtifactShareService.delete_for_session", 0), + ] + patchers = [_patch(target, new=_AsyncMock(return_value=value)) for target, value in stubs] + for patcher in patchers: + patcher.start() + try: + yield + finally: + for patcher in reversed(patchers): + patcher.stop() diff --git a/backend/tests/routes/test_converse_cost_accounting.py b/backend/tests/routes/test_converse_cost_accounting.py index 297d74ccd..27800ba6d 100644 --- a/backend/tests/routes/test_converse_cost_accounting.py +++ b/backend/tests/routes/test_converse_cost_accounting.py @@ -129,12 +129,19 @@ def mock_store_metadata(): yield m -@pytest.fixture +@pytest.fixture(autouse=True) def mock_pricing(): """Patch create_pricing_snapshot to return canned pricing (avoids DynamoDB). The fixed converse_routes.py calls create_pricing_snapshot inside _record_cost; without this mock the call hits DynamoDB and fails. + + ``autouse`` since 2026-09-18: it was opt-in, and the tests that did not + request it were issuing live DynamoDB calls — ``_record_cost`` is fail-open, + so the failure was logged and swallowed and the assertions passed anyway. + Tests that want the canned values still take it as an argument; this only + guarantees no test reaches the real table. See the off-box socket guard in + ``tests/conftest.py``. """ pricing = { "inputPricePerMtok": 1.0, diff --git a/backend/tests/routes/test_pbt_model_access_preservation.py b/backend/tests/routes/test_pbt_model_access_preservation.py index 99f601ce0..62aea76e1 100644 --- a/backend/tests/routes/test_pbt_model_access_preservation.py +++ b/backend/tests/routes/test_pbt_model_access_preservation.py @@ -16,6 +16,7 @@ os.environ.setdefault("AWS_REGION", "us-east-1") os.environ.setdefault("AWS_DEFAULT_REGION", "us-east-1") +import pytest from unittest.mock import AsyncMock, MagicMock, patch from fastapi import FastAPI @@ -42,6 +43,28 @@ # Helpers # --------------------------------------------------------------------------- +@pytest.fixture(autouse=True) +def _no_live_session_reads(): + """Stub the two session-scoped reads the invocation path makes. + + ``system_prompt_resolver.resolve_active_prompt_text`` reads session + metadata, and ``_session_has_documents`` asks the file index whether to + inject ``document_read``. Both build real DynamoDB clients, and both are + fail-open — so these property tests were issuing live AWS calls on every + Hypothesis example while still passing. See the off-box socket guard in + ``tests/conftest.py``. + """ + with patch( + "apis.inference_api.chat.system_prompt_resolver.get_session_metadata", + new=AsyncMock(return_value=None), + ), patch( + # Imported inside the function, so patch it at the source module. + "apis.shared.files.document_read.session_has_documents", + new=AsyncMock(return_value=False), + ): + yield + + def _make_mock_user() -> User: return User( email="test@example.com", diff --git a/backend/tests/routes/test_role_agent_pins.py b/backend/tests/routes/test_role_agent_pins.py index 90f2335f4..2fd572092 100644 --- a/backend/tests/routes/test_role_agent_pins.py +++ b/backend/tests/routes/test_role_agent_pins.py @@ -39,6 +39,38 @@ ADMIN_ROUTES = "apis.app_api.admin.roles.agent_pins" +@pytest.fixture(autouse=True) +def _no_live_catalog_reads(): + """Stub the two catalogs the pin surfaces read behind the mocked services. + + ``role_pins.count_locked_outside`` -> ``get_app_role_admin_service().list_roles()`` + -> ``rbac.repository.list_roles`` builds a real DynamoDB client. These tests + mock the pin services only, so that call used to leave the box; it is + fail-open, so the failure was swallowed and the assertions passed anyway. + The same applies to ``agent_detail._tool_labels`` (-> ``ToolCatalogService.get_tool``) + and ``_memory_labels`` (-> ``MemorySpaceService.resolve_permission``). All three + are fail-open, so the failures were swallowed and the assertions passed anyway. See the off-box socket guard + in ``tests/conftest.py``. + """ + role_service = MagicMock() + role_service.list_roles = AsyncMock(return_value=[]) + with patch( + "apis.shared.rbac.admin_service.get_app_role_admin_service", + return_value=role_service, + ), patch( + # Patched on the service, not the repository: ``_tool_labels`` receives + # an already-constructed ``ToolCatalogService``, so swapping the repo + # accessor comes too late. + "apis.app_api.tools.service.ToolCatalogService.get_tool", + new=AsyncMock(return_value=None), + ), patch( + # Same shape: ``_memory_labels`` gets a constructed service. + "apis.shared.memory.service.MemorySpaceService.resolve_permission", + new=MagicMock(return_value=(None, None)), + ): + yield + + def _make_assistant(**overrides) -> Assistant: defaults = dict( assistantId="ast-001", diff --git a/backend/tests/routes/test_sheet_preview_route.py b/backend/tests/routes/test_sheet_preview_route.py new file mode 100644 index 000000000..2623e37ef --- /dev/null +++ b/backend/tests/routes/test_sheet_preview_route.py @@ -0,0 +1,199 @@ +"""End-to-end tests for GET /files/{upload_id}/sheet-preview. + +Unlike `tests/apis/app_api/test_sheet_preview.py`, which exercises the +reader on its own, these drive the whole chain: route -> auth dependency +-> FileUploadService -> S3 body -> openpyxl -> response model. The +service is real; only the repository and the S3 client are stubs, so a +mismatch between what openpyxl returns and what the response model +accepts fails here rather than in production. +""" + +from datetime import datetime, timezone +from io import BytesIO +from unittest.mock import AsyncMock, MagicMock + +import pytest +from fastapi import FastAPI +from openpyxl import Workbook + +from apis.app_api.files.routes import router +from apis.app_api.files.service import FileUploadService, get_file_upload_service +from apis.app_api.files.sheet_preview import MAX_WORKBOOK_BYTES +from apis.shared.files.models import FileMetadata, FileStatus + +from tests.routes.conftest import mock_service + +XLSX_MIME = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + + +def workbook_bytes(rows=None, *, sheet_name="Sheet1") -> bytes: + wb = Workbook() + ws = wb.active + ws.title = sheet_name + for row in rows if rows is not None else [["Item", "Cost"], ["Rent", 1200]]: + ws.append(row) + buffer = BytesIO() + wb.save(buffer) + return buffer.getvalue() + + +def metadata(**overrides) -> FileMetadata: + base = dict( + upload_id="up1", + user_id="user-1", + session_id="sess-1", + filename="budget.xlsx", + mime_type=XLSX_MIME, + size_bytes=4096, + s3_key="users/user-1/up1/budget.xlsx", + s3_bucket="test-bucket", + status=FileStatus.READY, + created_at=datetime.now(timezone.utc), + ) + base.update(overrides) + return FileMetadata(**base) + + +@pytest.fixture +def s3_body(): + """Mutable holder for whatever the stub S3 client returns.""" + return {"data": workbook_bytes()} + + +@pytest.fixture +def file_meta(): + return {"value": metadata()} + + +@pytest.fixture +def app(s3_body, file_meta): + s3 = MagicMock() + s3.get_object.side_effect = lambda **_: { + "Body": BytesIO(s3_body["data"]) + } + + repository = MagicMock() + repository.get_file = AsyncMock(side_effect=lambda *_: file_meta["value"]) + + service = FileUploadService( + repository=repository, + s3_client=s3, + bucket_name="test-bucket", + thumbnail_renderer=MagicMock(), + ) + + _app = FastAPI() + _app.include_router(router) + mock_service(_app, get_file_upload_service, service) + return _app + + +class TestSuccess: + def test_returns_sheets_as_rows_of_strings( + self, app, make_user, authenticated_client + ): + client = authenticated_client(app, make_user(user_id="user-1")) + + response = client.get("/files/up1/sheet-preview") + + assert response.status_code == 200 + body = response.json() + assert body["filename"] == "budget.xlsx" + assert body["truncated"] is False + [sheet] = body["sheets"] + assert sheet["name"] == "Sheet1" + assert sheet["headers"] == ["Item", "Cost"] + assert sheet["rows"] == [["Rent", "1200"]] + + def test_serialises_with_camelCase_aliases( + self, app, make_user, authenticated_client + ): + # The SPA's SheetPreview interface reads these names directly. + client = authenticated_client(app, make_user(user_id="user-1")) + + [sheet] = client.get("/files/up1/sheet-preview").json()["sheets"] + + assert "totalRows" in sheet + assert "truncatedBy" in sheet + assert "total_rows" not in sheet + + def test_the_workbook_bytes_never_leave_the_server( + self, app, make_user, authenticated_client + ): + # The whole point of this route: no presigned URL, no bytes. + client = authenticated_client(app, make_user(user_id="user-1")) + + raw = client.get("/files/up1/sheet-preview").content + + assert b"PK\x03\x04" not in raw # the zip magic an .xlsx starts with + + def test_shows_formula_text_where_no_value_was_cached( + self, app, make_user, authenticated_client, s3_body + ): + # openpyxl does not evaluate, so a workbook it wrote carries no + # cached value — a values-only read would show a blank total. + s3_body["data"] = workbook_bytes( + [["Item", "Cost"], ["Rent", 1200], ["Total", "=SUM(B2:B2)"]] + ) + client = authenticated_client(app, make_user(user_id="user-1")) + + [sheet] = client.get("/files/up1/sheet-preview").json()["sheets"] + + assert sheet["rows"][1] == ["Total", "=SUM(B2:B2)"] + + +class TestRejections: + def test_401_without_a_session(self, app, unauthenticated_client): + client = unauthenticated_client(app) + + assert client.get("/files/up1/sheet-preview").status_code == 401 + + def test_404_when_the_file_is_not_the_callers( + self, app, make_user, authenticated_client, file_meta + ): + file_meta["value"] = None + client = authenticated_client(app, make_user(user_id="user-1")) + + assert client.get("/files/up1/sheet-preview").status_code == 404 + + def test_404_while_the_upload_is_still_pending( + self, app, make_user, authenticated_client, file_meta + ): + file_meta["value"] = metadata(status=FileStatus.PENDING) + client = authenticated_client(app, make_user(user_id="user-1")) + + assert client.get("/files/up1/sheet-preview").status_code == 404 + + def test_415_for_a_format_that_is_not_a_workbook( + self, app, make_user, authenticated_client, file_meta + ): + # .xls shares nothing with OOXML; the UI should never have asked. + file_meta["value"] = metadata( + filename="old.xls", mime_type="application/vnd.ms-excel" + ) + client = authenticated_client(app, make_user(user_id="user-1")) + + assert client.get("/files/up1/sheet-preview").status_code == 415 + + def test_413_before_downloading_an_oversized_workbook( + self, app, make_user, authenticated_client, file_meta + ): + # Checked against recorded metadata, so an oversized file costs a + # metadata read rather than a transfer into memory. + file_meta["value"] = metadata(size_bytes=MAX_WORKBOOK_BYTES + 1) + client = authenticated_client(app, make_user(user_id="user-1")) + + assert client.get("/files/up1/sheet-preview").status_code == 413 + + def test_422_for_bytes_that_are_not_a_workbook( + self, app, make_user, authenticated_client, s3_body + ): + s3_body["data"] = b"not a workbook at all" + client = authenticated_client(app, make_user(user_id="user-1")) + + response = client.get("/files/up1/sheet-preview") + + assert response.status_code == 422 + # openpyxl's own message names internal XML parts; the user gets + # something they can act on instead. + assert "corrupt or password-protected" in response.json()["detail"] diff --git a/backend/tests/shared/conftest.py b/backend/tests/shared/conftest.py index da41e44a3..a0ff7a155 100644 --- a/backend/tests/shared/conftest.py +++ b/backend/tests/shared/conftest.py @@ -20,8 +20,16 @@ def aws(monkeypatch): monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "testing") monkeypatch.setenv("AWS_SECURITY_TOKEN", "testing") monkeypatch.setenv("AWS_SESSION_TOKEN", "testing") + # Drop any boto3 client cached under a PREVIOUS test's moto backend, and + # again on the way out so nothing built here leaks into the next test. + # `mock_aws()` is per-test, so a cached client outliving its mock would + # talk to a torn-down backend — an order-dependent failure. + from apis.shared.aws_clients import reset_cached_clients + + reset_cached_clients() with mock_aws(): yield + reset_cached_clients() # =================================================================== diff --git a/backend/tests/shared/test_agent_templates.py b/backend/tests/shared/test_agent_templates.py new file mode 100644 index 000000000..537c45d95 --- /dev/null +++ b/backend/tests/shared/test_agent_templates.py @@ -0,0 +1,258 @@ +"""Tests for the agent_templates shared module (models + repository + service + seed).""" + +import boto3 +import pytest + +from apis.shared.agent_templates.models import ( + AgentTemplate, + AgentTemplateCreate, + AgentTemplateUpdate, + BindingPayload, + ModelConfigPayload, + TemplateBinding, + TemplateModelConfig, +) +from apis.shared.agent_templates.repository import ( + AgentTemplatesRepository, + slugify, +) +from apis.shared.agent_templates.service import AgentTemplatesService +from apis.shared.agent_templates.seed import seed_default_templates +from apis.shared.caching import config_cache + +AWS_REGION = "us-east-1" +TABLE_NAME = "test-agent-templates" + + +@pytest.fixture() +def templates_table(aws, monkeypatch): + ddb = boto3.client("dynamodb", region_name=AWS_REGION) + ddb.create_table( + TableName=TABLE_NAME, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + monkeypatch.setenv("DYNAMODB_AGENT_TEMPLATES_TABLE_NAME", TABLE_NAME) + monkeypatch.setenv("AWS_REGION", AWS_REGION) + # Clear the process-wide config cache between tests so a Scan is re-issued. + config_cache.invalidate(config_cache.AGENT_TEMPLATES) + return boto3.resource("dynamodb", region_name=AWS_REGION).Table(TABLE_NAME) + + +@pytest.fixture() +def repo(templates_table): + return AgentTemplatesRepository(table_name=TABLE_NAME, region=AWS_REGION) + + +@pytest.fixture() +def service(repo): + return AgentTemplatesService(repo) + + +def _create_data(**kw) -> AgentTemplateCreate: + defaults = dict( + template_id="course-helper", + name="Course Helper", + description="A study assistant for one course.", + emoji="🎓", + instructions="You are a course helper.", + tags=["demo"], + starters=["What topics does this cover?"], + model_cfg=ModelConfigPayload(model_id="us.anthropic.claude-sonnet-5", params={}), + bindings=[BindingPayload(kind="tool", ref="some_tool", config={})], + pitch="Answers from your course materials.", + status="enabled", + sort_order=10, + ) + defaults.update(kw) + return AgentTemplateCreate(**defaults) + + +# --------------------------------------------------------------------------- +# Model tests +# --------------------------------------------------------------------------- + + +class TestAgentTemplateModel: + def test_dynamo_roundtrip(self): + t = AgentTemplate( + template_id="abc-123", + name="Test", + description="A test template", + emoji="🤖", + instructions="Be helpful.", + tags=["a", "b"], + starters=["Hello?"], + model_config_=TemplateModelConfig(model_id="model-x", params={"temperature": 1}), + bindings=[TemplateBinding(kind="tool", ref="t1", config={"k": "v"})], + pitch="A pitch.", + status="enabled", + sort_order=5, + created_at="2026-01-01T00:00:00Z", + updated_at="2026-01-01T00:00:00Z", + created_by="admin@example.com", + ) + item = t.to_dynamo_item() + assert item["PK"] == "TEMPLATE#abc-123" + assert item["SK"] == "METADATA" + assert item["modelConfig"]["modelId"] == "model-x" + assert item["bindings"][0]["ref"] == "t1" + + back = AgentTemplate.from_dynamo_item(item) + assert back.template_id == "abc-123" + assert back.model_config_.model_id == "model-x" + assert back.bindings[0].kind == "tool" + assert back.sort_order == 5 + + def test_from_item_defaults_missing_status_to_enabled(self): + t = AgentTemplate.from_dynamo_item( + { + "PK": "TEMPLATE#x", + "SK": "METADATA", + "templateId": "x", + "name": "X", + "createdAt": "2026-01-01T00:00:00Z", + "updatedAt": "2026-01-01T00:00:00Z", + } + ) + assert t.status == "enabled" + assert t.model_config_.model_id is None + assert t.bindings == [] + + def test_from_item_unknown_status_is_disabled(self): + t = AgentTemplate.from_dynamo_item( + { + "PK": "TEMPLATE#x", + "SK": "METADATA", + "templateId": "x", + "name": "X", + "status": "weird", + "createdAt": "2026-01-01T00:00:00Z", + "updatedAt": "2026-01-01T00:00:00Z", + } + ) + assert t.status == "disabled" + + +class TestSlugify: + def test_basic(self): + assert slugify("Course Helper") == "course-helper" + + def test_punctuation_collapses(self): + assert slugify("Q&A Assistant!!") == "q-a-assistant" + + def test_empty_falls_back_to_uuid(self): + assert slugify(" ").startswith("template-") + + +# --------------------------------------------------------------------------- +# Repository / service tests (moto-backed) +# --------------------------------------------------------------------------- + + +class TestRepository: + @pytest.mark.asyncio + async def test_create_and_get(self, repo): + created = await repo.create_template(_create_data(), created_by="admin@example.com") + assert created.template_id == "course-helper" + fetched = await repo.get_template("course-helper") + assert fetched is not None + assert fetched.name == "Course Helper" + assert fetched.model_config_.model_id == "us.anthropic.claude-sonnet-5" + + @pytest.mark.asyncio + async def test_create_slugifies_when_id_omitted(self, repo): + created = await repo.create_template( + _create_data(template_id=None, name="My New Template") + ) + assert created.template_id == "my-new-template" + + @pytest.mark.asyncio + async def test_duplicate_id_raises(self, repo): + await repo.create_template(_create_data()) + with pytest.raises(ValueError): + await repo.create_template(_create_data()) + + @pytest.mark.asyncio + async def test_list_sorts_by_sort_order(self, repo): + await repo.create_template(_create_data(template_id="b", name="B", sort_order=20)) + await repo.create_template(_create_data(template_id="a", name="A", sort_order=10)) + listed = await repo.list_templates() + assert [t.template_id for t in listed] == ["a", "b"] + + @pytest.mark.asyncio + async def test_list_enabled_only(self, repo): + await repo.create_template(_create_data(template_id="on", name="On", status="enabled")) + await repo.create_template(_create_data(template_id="off", name="Off", status="disabled")) + listed = await repo.list_templates(enabled_only=True) + assert [t.template_id for t in listed] == ["on"] + + @pytest.mark.asyncio + async def test_update_partial(self, repo): + await repo.create_template(_create_data()) + updated = await repo.update_template( + "course-helper", + AgentTemplateUpdate(name="Renamed", status="disabled"), + ) + assert updated is not None + assert updated.name == "Renamed" + assert updated.status == "disabled" + # Untouched fields survive. + assert updated.emoji == "🎓" + + @pytest.mark.asyncio + async def test_update_replaces_nested_bindings_and_model(self, repo): + await repo.create_template(_create_data()) + updated = await repo.update_template( + "course-helper", + AgentTemplateUpdate( + model_cfg=ModelConfigPayload(model_id=None, params={}), + bindings=[], + ), + ) + assert updated is not None + assert updated.model_config_.model_id is None + assert updated.bindings == [] + + @pytest.mark.asyncio + async def test_update_missing_returns_none(self, repo): + assert await repo.update_template("nope", AgentTemplateUpdate(name="x")) is None + + @pytest.mark.asyncio + async def test_delete(self, repo): + await repo.create_template(_create_data()) + assert await repo.delete_template("course-helper") is True + assert await repo.get_template("course-helper") is None + assert await repo.delete_template("course-helper") is False + + +# --------------------------------------------------------------------------- +# Seed tests +# --------------------------------------------------------------------------- + + +class TestSeed: + @pytest.mark.asyncio + async def test_seed_is_idempotent(self, service): + first = await seed_default_templates(service) + assert first == 3 + # Re-running seeds nothing new. + second = await seed_default_templates(service) + assert second == 0 + + @pytest.mark.asyncio + async def test_seed_templates_are_fork_safe(self, service): + await seed_default_templates(service) + templates = await service.list_templates() + assert len(templates) == 3 + for t in templates: + # Fork-safety invariants: no org-specific tool refs, platform-default model. + assert t.bindings == [], f"{t.template_id} must bind no tools" + assert t.model_config_.model_id is None, f"{t.template_id} must use default model" diff --git a/backend/tests/shared/test_agent_templates_routes.py b/backend/tests/shared/test_agent_templates_routes.py new file mode 100644 index 000000000..1c56a69cd --- /dev/null +++ b/backend/tests/shared/test_agent_templates_routes.py @@ -0,0 +1,172 @@ +"""Route tests for agent-templates endpoints (admin CRUD + public read). + +Admin wire is snake_case; the public endpoint returns the client +``TemplateCatalogEntry`` shape (camelCase ``draft`` + ``pitch``), so the +existing picker consumes it unchanged. +""" + +import boto3 +import pytest +from fastapi import APIRouter, FastAPI +from fastapi.testclient import TestClient + +from apis.shared.auth import get_current_user_from_session +from tests.conftest import override_admin_auth +from apis.shared.auth.models import User +from apis.shared.agent_templates import repository as repo_module +from apis.shared.agent_templates import service as service_module +from apis.shared.caching import config_cache + +AWS_REGION = "us-east-1" +TABLE_NAME = "test-agent-templates-routes" + + +def _make_user(email: str = "user@example.com", roles=None) -> User: + return User( + email=email, + user_id="user-001", + name="Test User", + roles=roles if roles is not None else ["User"], + ) + + +@pytest.fixture() +def templates_table(aws, monkeypatch): + ddb = boto3.client("dynamodb", region_name=AWS_REGION) + ddb.create_table( + TableName=TABLE_NAME, + KeySchema=[ + {"AttributeName": "PK", "KeyType": "HASH"}, + {"AttributeName": "SK", "KeyType": "RANGE"}, + ], + AttributeDefinitions=[ + {"AttributeName": "PK", "AttributeType": "S"}, + {"AttributeName": "SK", "AttributeType": "S"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + monkeypatch.setenv("DYNAMODB_AGENT_TEMPLATES_TABLE_NAME", TABLE_NAME) + monkeypatch.setenv("AWS_REGION", AWS_REGION) + monkeypatch.setattr(repo_module, "_repository", None) + monkeypatch.setattr(service_module, "_service", None) + config_cache.invalidate(config_cache.AGENT_TEMPLATES) + return boto3.resource("dynamodb", region_name=AWS_REGION).Table(TABLE_NAME) + + +def _build_admin_app() -> FastAPI: + from apis.app_api.admin.agent_templates.routes import router as admin_router + + app = FastAPI() + parent = APIRouter(prefix="/admin") + parent.include_router(admin_router) + app.include_router(parent) + return app + + +def _build_user_app() -> FastAPI: + from apis.app_api.agent_templates.routes import router as user_router + + app = FastAPI() + app.include_router(user_router) + return app + + +_PAYLOAD = { + "template_id": "course-helper", + "name": "Course Helper", + "description": "A study assistant for one course.", + "emoji": "🎓", + "instructions": "You are a course helper.", + "tags": [], + "starters": ["What topics does this cover?"], + "modelConfig": {"modelId": "us.anthropic.claude-sonnet-5", "params": {}}, + "bindings": [{"kind": "tool", "ref": "some_tool", "config": {}}], + "pitch": "Answers from your course materials.", + "status": "enabled", + "sort_order": 10, +} + + +@pytest.fixture() +def admin_client(templates_table): + app = _build_admin_app() + admin = _make_user(roles=["system_admin"]) + override_admin_auth(app, lambda: admin) + return TestClient(app) + + +@pytest.fixture() +def user_client(templates_table): + app = _build_user_app() + user = _make_user() + app.dependency_overrides[get_current_user_from_session] = lambda: user + return TestClient(app) + + +class TestAdminRoutes: + def test_create_returns_201(self, admin_client): + resp = admin_client.post("/admin/agent-templates/", json=_PAYLOAD) + assert resp.status_code == 201 + body = resp.json() + assert body["template_id"] == "course-helper" + assert body["name"] == "Course Helper" + # Admin wire exposes modelConfig as the camel alias. + assert body["modelConfig"]["modelId"] == "us.anthropic.claude-sonnet-5" + assert body["created_at"] + + def test_duplicate_id_conflicts(self, admin_client): + admin_client.post("/admin/agent-templates/", json=_PAYLOAD) + resp = admin_client.post("/admin/agent-templates/", json=_PAYLOAD) + assert resp.status_code == 409 + + def test_list_all_includes_disabled(self, admin_client): + admin_client.post("/admin/agent-templates/", json=_PAYLOAD) + admin_client.post( + "/admin/agent-templates/", + json={**_PAYLOAD, "template_id": "off", "name": "Off", "status": "disabled"}, + ) + resp = admin_client.get("/admin/agent-templates/") + assert resp.status_code == 200 + assert resp.json()["total"] == 2 + + def test_update_and_delete(self, admin_client): + admin_client.post("/admin/agent-templates/", json=_PAYLOAD) + patched = admin_client.patch( + "/admin/agent-templates/course-helper", json={"name": "Renamed"} + ) + assert patched.status_code == 200 + assert patched.json()["name"] == "Renamed" + + deleted = admin_client.delete("/admin/agent-templates/course-helper") + assert deleted.status_code == 204 + assert admin_client.get("/admin/agent-templates/course-helper").status_code == 404 + + +class TestPublicRoute: + def test_empty_catalog_returns_empty_list(self, user_client): + resp = user_client.get("/templates/") + assert resp.status_code == 200 + body = resp.json() + assert body == {"templates": [], "total": 0} + + def test_returns_enabled_only_in_client_shape(self, admin_client, user_client): + admin_client.post("/admin/agent-templates/", json=_PAYLOAD) + admin_client.post( + "/admin/agent-templates/", + json={**_PAYLOAD, "template_id": "off", "name": "Off", "status": "disabled"}, + ) + resp = user_client.get("/templates/") + assert resp.status_code == 200 + body = resp.json() + assert body["total"] == 1 + entry = body["templates"][0] + # Public shape mirrors the client TemplateCatalogEntry { draft, pitch }. + assert set(entry.keys()) == {"draft", "pitch"} + draft = entry["draft"] + assert draft["templateId"] == "course-helper" + assert draft["modelConfig"]["modelId"] == "us.anthropic.claude-sonnet-5" + assert draft["bindings"][0]["ref"] == "some_tool" + # Catalog/audit fields must NOT leak into the public draft. + assert "status" not in draft + assert "sort_order" not in draft + assert "createdBy" not in draft diff --git a/backend/tests/shared/test_aws_clients.py b/backend/tests/shared/test_aws_clients.py new file mode 100644 index 000000000..eb5fb3a64 --- /dev/null +++ b/backend/tests/shared/test_aws_clients.py @@ -0,0 +1,86 @@ +"""Process-cached boto3 clients (docs/specs/turn-latency-preamble.md PR-3). + +The saving is small per call — ~1.5ms — and the risk is not. These tests pin +the two things that make it safe: the cache actually caches, and it can be +torn down so a client built under one `moto` backend never serves the next +test. +""" + +import pytest + +from apis.shared import aws_clients + + +@pytest.fixture(autouse=True) +def _clean_cache(): + aws_clients.reset_cached_clients() + yield + aws_clients.reset_cached_clients() + + +class TestCaching: + def test_the_same_service_returns_the_identical_resource(self, aws): + first = aws_clients.get_resource("dynamodb") + second = aws_clients.get_resource("dynamodb") + + assert first is second, "the whole point is not rebuilding it" + + def test_clients_cache_independently_of_resources(self, aws): + resource = aws_clients.get_resource("dynamodb") + client = aws_clients.get_client("dynamodb") + + assert client is not resource + + def test_region_is_part_of_the_key(self, aws): + """Two regions are two endpoints; collapsing them would silently send + writes to the wrong one.""" + default = aws_clients.get_resource("dynamodb") + west = aws_clients.get_resource("dynamodb", region_name="us-west-2") + east = aws_clients.get_resource("dynamodb", region_name="us-east-1") + + assert west is not east + assert default is not west or default is not east + + def test_a_table_handle_is_bound_to_the_cached_resource(self, aws): + """The Table is deliberately not cached — the resource behind it is.""" + a = aws_clients.get_dynamodb_table("some-table") + b = aws_clients.get_dynamodb_table("some-table") + + assert a is not b + assert a.meta.client is b.meta.client + + +class TestResetProtectsMoto: + """`mock_aws()` is entered per test. A cached client outliving its mock + talks to a torn-down backend — or a real AWS endpoint — and the failure is + order-dependent, which is the worst kind to debug.""" + + def test_reset_drops_everything(self, aws): + before = aws_clients.get_resource("dynamodb") + + aws_clients.reset_cached_clients() + + assert aws_clients.get_resource("dynamodb") is not before + + def test_the_aws_fixture_leaves_no_client_behind(self): + """Outside any `aws` fixture the cache must be empty, which is what + stops the previous test's mock backend leaking into the next one.""" + assert aws_clients._resources == {} + assert aws_clients._clients == {} + + +class TestMetadataUsesIt: + @pytest.mark.asyncio + async def test_session_reads_go_through_one_cached_resource( + self, sessions_metadata_table + ): + """The preamble's point: six helpers, one client build between them.""" + from apis.shared.sessions.metadata import load_session_meta + + aws_clients.reset_cached_clients() + + await load_session_meta("s1", "u1") + after_first = aws_clients.get_resource("dynamodb") + await load_session_meta("s2", "u1") + + assert aws_clients.get_resource("dynamodb") is after_first diff --git a/backend/tests/shared/test_browser_takeover_models.py b/backend/tests/shared/test_browser_takeover_models.py new file mode 100644 index 000000000..1c4255ec3 --- /dev/null +++ b/backend/tests/shared/test_browser_takeover_models.py @@ -0,0 +1,214 @@ +"""Wire-contract tests for the browser-takeover interrupt. + +What these protect is the two rules that are easy to state and easy to regress: +no URL may travel on this path, and a resume payload must never be able to +raise. Everything else about the flow is exercised in the tool and streaming +tests. +""" + +from __future__ import annotations + +import json + +import pytest + +from apis.shared.browser_takeover import ( + BrowserLoginRequiredEvent, + BrowserSessionRef, + assert_no_url, + decode_ref, + encode_ref, + format_outcome, + parse_outcome, +) + + +def _ref(**overrides) -> BrowserSessionRef: + payload = { + "browserSessionId": "bs-1", + "browserId": "browser-abc", + "viewport": {"width": 1280, "height": 800}, + "controlState": "user", + "deadlineAt": "2026-09-18T12:00:00+00:00", + } + payload.update(overrides) + return BrowserSessionRef.model_validate(payload) + + +def _event(**overrides) -> BrowserLoginRequiredEvent: + payload = { + "interruptId": "v1:tool_call:tu-1:abc", + "toolUseId": "tu-1", + "sessionId": "conv-1", + "browserSessionId": "bs-1", + "browserId": "browser-abc", + "viewport": {"width": 1280, "height": 800}, + } + payload.update(overrides) + return BrowserLoginRequiredEvent.model_validate(payload) + + +# --------------------------------------------------------------------------- +# No URL, anywhere (PR #1101's rule, enforced rather than remembered) +# --------------------------------------------------------------------------- + + +class TestNoUrlReachesTheClient: + def test_the_event_carries_identifiers_not_a_live_view_url(self) -> None: + payload = _event().model_dump(by_alias=True, exclude_none=True) + + assert payload["browserSessionId"] == "bs-1" + assert_no_url(payload) # must not raise + + def test_a_presigned_url_smuggled_onto_the_payload_is_caught(self) -> None: + payload = _event().model_dump(by_alias=True, exclude_none=True) + payload["liveViewUrl"] = ( + "https://bedrock-agentcore.us-west-2.amazonaws.com/live?X-Amz-Signature=deadbeef" + ) + + with pytest.raises(ValueError, match="minted by app-api"): + assert_no_url(payload) + + def test_a_url_nested_deeper_is_still_caught(self) -> None: + with pytest.raises(ValueError): + assert_no_url({"a": [{"b": "wss://example.com/stream"}]}) + + def test_target_url_is_the_one_allowed_url(self) -> None: + # The page the agent already browsed, shown so the user knows what they + # are signing into. It is an ordinary navigational URL, not a signed + # one, and the model put it there in the first place. + assert_no_url( + _event(targetUrl="https://www.jstor.org/action/showLogin").model_dump( + by_alias=True, exclude_none=True + ) + ) + + def test_the_sandbox_origin_is_allowed_but_a_signed_url_is_not(self) -> None: + # The origin is a deployment constant with no credential in it, and the + # SPA needs it to know where to frame the viewer from. The guard exists + # for presigned URLs, which are credentials in URL form — so widening + # it for the origin must not widen it for those. + payload = _event(sandboxOrigin="https://mcp-sandbox.example.edu").model_dump( + by_alias=True, exclude_none=True + ) + assert_no_url(payload) + + payload["someOtherField"] = "https://example.com/x?X-Amz-Signature=abc" + with pytest.raises(ValueError): + assert_no_url(payload) + + def test_an_absent_sandbox_origin_reads_as_no_viewer(self) -> None: + # Not an error: an environment without the sandbox origin deployed + # shows the prompt without a viewer rather than framing nothing. + assert _event().sandbox_origin == "" + + def test_sse_frame_names_the_event_and_carries_json(self) -> None: + frame = _event().to_sse_format() + + assert frame.startswith("event: browser_login_required\n") + assert frame.endswith("\n\n") + data = json.loads(frame.split("data: ", 1)[1]) + assert data["type"] == "browser_login_required" + assert data["sessionId"] == "conv-1" + + def test_session_id_is_the_conversation_and_browser_is_separate(self) -> None: + # The two are deliberately distinct fields: app-api scopes the + # live-view route by conversation owner, and resolves the browser + # session server-side. A single ambiguous `sessionId` would collide. + data = json.loads(_event().to_sse_format().split("data: ", 1)[1]) + + assert data["sessionId"] != data["browserSessionId"] + + +# --------------------------------------------------------------------------- +# Resume payloads +# --------------------------------------------------------------------------- + + +class TestParseOutcome: + @pytest.mark.parametrize( + "response,expected", + [ + ({"completed": True}, "completed"), + ({"skipped": True}, "skipped"), + ({"expired": True}, "expired"), + ({"outcome": "completed"}, "completed"), + ({"outcome": "failed"}, "failed"), + ("completed", "completed"), + ("expired", "expired"), + ], + ) + def test_recognized_shapes(self, response, expected) -> None: + assert parse_outcome(response)[0] == expected + + @pytest.mark.parametrize( + "response", + [None, 42, [], {}, {"nonsense": True}, "gibberish", {"outcome": "weird"}], + ) + def test_anything_unrecognized_reads_as_skipped_and_never_raises( + self, response + ) -> None: + # The user's only route out of a paused turn must not be able to fail + # on a shape mismatch — same reasoning as `parse_answers`. + assert parse_outcome(response)[0] == "skipped" + + def test_a_note_rides_along_and_is_bounded(self) -> None: + outcome, note = parse_outcome({"completed": True, "note": "x" * 5000}) + + assert outcome == "completed" + assert note is not None and len(note) == 500 + + def test_blank_note_is_dropped(self) -> None: + assert parse_outcome({"completed": True, "note": " "})[1] is None + + +class TestFormatOutcome: + @pytest.mark.parametrize( + "outcome", ["completed", "skipped", "expired", "failed"] + ) + def test_every_outcome_renders_one_short_line(self, outcome) -> None: + text = format_outcome(outcome) + + assert text and "\n" not in text + # This text re-enters the conversation on every subsequent turn, so it + # stays small by contract, not by luck. + assert len(text) < 300 + + def test_completed_tells_the_model_the_session_is_authenticated(self) -> None: + assert "authenticated" in format_outcome("completed") + + def test_declined_tells_the_model_not_to_ask_again(self) -> None: + assert "not ask again" in format_outcome("skipped") + + def test_a_note_is_appended(self) -> None: + assert "(User note: no account)" in format_outcome("skipped", "no account") + + def test_no_outcome_line_contains_a_url(self) -> None: + for outcome in ("completed", "skipped", "expired", "failed"): + assert_no_url(format_outcome(outcome)) + + +# --------------------------------------------------------------------------- +# Persistence round trip +# --------------------------------------------------------------------------- + + +class TestEncoding: + def test_ref_round_trips_through_the_dynamo_string(self) -> None: + decoded = decode_ref(encode_ref(_ref())) + + assert decoded is not None + assert decoded["browserSessionId"] == "bs-1" + assert decoded["viewport"] == {"width": 1280, "height": 800} + + @pytest.mark.parametrize("encoded", [None, "", "not json", "[1,2,3]"]) + def test_unreadable_rows_decode_to_none_rather_than_raising(self, encoded) -> None: + assert decode_ref(encoded) is None + + def test_viewport_survives_so_the_dcv_stream_does_not_crop(self) -> None: + # remoteWidth/remoteHeight must match the session viewport exactly; + # carrying it rather than re-declaring 1280x800 in the SPA is what + # stops the two drifting. + ref = _ref(viewport={"width": 1920, "height": 1080}) + + assert decode_ref(encode_ref(ref))["viewport"]["width"] == 1920 diff --git a/backend/tests/shared/test_build_stages.py b/backend/tests/shared/test_build_stages.py new file mode 100644 index 000000000..f9fe72d6b --- /dev/null +++ b/backend/tests/shared/test_build_stages.py @@ -0,0 +1,133 @@ +"""Agent-build stage recorder (docs/specs/turn-latency-preamble.md PR-4). + +`agent_build` is the largest number left in the prelude — ~2950ms cold against +1-47ms warm — and nothing said which part of it that was. These pin the +recorder's contract, which is entirely about NOT being able to break the turn +it measures. +""" + +import pytest + +from apis.shared.observability import build_stages + + +@pytest.fixture(autouse=True) +def _no_recorder(): + """Every test starts with nothing installed, which is also the state every + caller outside the inference-api turn path is in.""" + token = build_stages.set_stage_recorder(None) + yield + build_stages.reset_stage_recorder(token) + + +class TestRecording: + def test_marks_reach_the_installed_recorder(self): + seen = [] + token = build_stages.set_stage_recorder(seen.append) + + build_stages.mark_stage("tools") + build_stages.mark_stage("hooks") + + build_stages.reset_stage_recorder(token) + assert seen == ["tools", "hooks"] + + def test_marking_without_a_recorder_is_a_silent_no_op(self): + """The agent is constructed from tests, the scheduled-runs Lambda and + app-api, none of which install one. A mark there must cost nothing and + raise nothing.""" + build_stages.mark_stage("tools") # must not raise + + def test_reset_restores_the_previous_recorder(self): + outer = [] + inner = [] + outer_token = build_stages.set_stage_recorder(outer.append) + inner_token = build_stages.set_stage_recorder(inner.append) + + build_stages.mark_stage("inner") + build_stages.reset_stage_recorder(inner_token) + build_stages.mark_stage("outer") + build_stages.reset_stage_recorder(outer_token) + + assert inner == ["inner"] + assert outer == ["outer"] + + def test_after_reset_nothing_is_recorded(self): + """A build that finishes must not keep marking. A leaked recorder would + attribute the NEXT turn's stages to a finished prelude.""" + seen = [] + token = build_stages.set_stage_recorder(seen.append) + build_stages.reset_stage_recorder(token) + + build_stages.mark_stage("stray") + + assert seen == [] + + +class TestFailSoft: + def test_a_recorder_that_raises_never_breaks_the_build(self): + """The whole module is a measurement. A measurement that can break the + thing it measures is worse than no measurement — the same contract + `TurnPrelude.mark` already holds.""" + def _boom(_stage): + raise RuntimeError("clock exploded") + + token = build_stages.set_stage_recorder(_boom) + try: + build_stages.mark_stage("tools") # must not raise + finally: + build_stages.reset_stage_recorder(token) + + def test_a_bad_reset_token_never_raises(self): + class _Hostile: + pass + + build_stages.reset_stage_recorder(_Hostile()) # must not raise + + +class TestPreludeIntegration: + def test_sub_stages_group_under_agent_build(self): + """`groups.agent_build` has to reproduce the pre-split number, or the + decomposition throws away the only series anyone has — the same + contract the preamble split holds.""" + import json + import logging + + from apis.inference_api.chat.turn_timing import TurnPrelude + + prelude = TurnPrelude() + prelude.mark("preamble.quota") + + token = build_stages.set_stage_recorder( + lambda stage: prelude.mark(f"agent_build.{stage}") + ) + for stage in ("prompt", "registry", "session_mgr", "tools"): + build_stages.mark_stage(stage) + build_stages.reset_stage_recorder(token) + prelude.mark("agent_build.rest") + + captured = {} + + class _Sink(logging.Handler): + def emit(self, record): + message = record.getMessage() + if message.startswith("turn_prelude "): + captured.update(json.loads(message[len("turn_prelude ") :])) + + logger = logging.getLogger("apis.inference_api.chat.turn_timing") + handler = _Sink() + logger.addHandler(handler) + logger.setLevel(logging.INFO) + try: + prelude.emit(session_id="s", stream_kind="agent") + finally: + logger.removeHandler(handler) + + assert "agent_build" in captured["groups"] + assert "preamble" in captured["groups"] + assert [k for k in captured["stages"] if k.startswith("agent_build.")] == [ + "agent_build.prompt", + "agent_build.registry", + "agent_build.session_mgr", + "agent_build.tools", + "agent_build.rest", + ] diff --git a/backend/tests/shared/test_context_ledger_persistence.py b/backend/tests/shared/test_context_ledger_persistence.py new file mode 100644 index 000000000..1ceb884c3 --- /dev/null +++ b/backend/tests/shared/test_context_ledger_persistence.py @@ -0,0 +1,102 @@ +"""Context-ledger extras land on the `C#` row and compaction decisions roll up +by kind on the session row (moto).""" + +from decimal import Decimal + +import pytest + +from apis.shared.sessions.models import MessageMetadata, ModelInfo, TokenUsage + + +def _meta(**extra): + return MessageMetadata( + token_usage=TokenUsage(inputTokens=100, outputTokens=50, totalTokens=150), + model_info=ModelInfo(modelId="claude-haiku-4-5", modelName="Claude Haiku 4.5"), + cost=0.01, + **extra, + ) + + +def _seed_session(table, session_id="s1", user_id="u1"): + table.put_item(Item={ + "PK": f"USER#{user_id}", "SK": f"S#{session_id}", + "GSI_PK": f"SESSION#{session_id}", "GSI_SK": "META", + "sessionId": session_id, "userId": user_id, "status": "active", + "createdAt": "2026-09-01T00:00:00Z", "lastMessageAt": "2026-09-01T00:00:00Z", + "messageCount": Decimal(0), + }) + + +def _session_row(table, session_id="s1", user_id="u1"): + return table.get_item(Key={"PK": f"USER#{user_id}", "SK": f"S#{session_id}"})["Item"] + + +@pytest.mark.asyncio +async def test_ledger_lands_on_the_cost_row_and_events_roll_up_by_kind(sessions_metadata_table, monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + from apis.shared.sessions.metadata import store_message_metadata + + _seed_session(sessions_metadata_table) + await store_message_metadata( + session_id="s1", user_id="u1", message_id=1, + message_metadata=_meta( + prefixTokens={"system": 12_000, "tools": 48_000}, + windowRemovedMessages=0, + compactionEvents=[ + {"kind": "applied", "checkpoint": 12, "summaryTokens": 900}, + {"kind": "checkpoint", "checkpoint": 12, "summaryTokens": 900}, + ], + ), + ) + await store_message_metadata( + session_id="s1", user_id="u1", message_id=2, + message_metadata=_meta(windowRemovedMessages=4, compactionEvents=[{"kind": "forced"}]), + ) + + cost_rows = sorted( + (i for i in sessions_metadata_table.scan()["Items"] if i["SK"].startswith("C#")), + key=lambda r: r["SK"], + ) + assert len(cost_rows) == 2 + first = [r for r in cost_rows if "prefixTokens" in r][0] + assert first["prefixTokens"] == {"system": Decimal(12_000), "tools": Decimal(48_000)} + assert first["windowRemovedMessages"] == Decimal(0) + assert first["compactionEvents"][0]["kind"] == "applied" + assert [r["windowRemovedMessages"] for r in cost_rows if "prefixTokens" not in r] == [Decimal(4)] + + row = _session_row(sessions_metadata_table) + assert row["compactionAppliedCount"] == Decimal(1) + assert row["compactionForcedCount"] == Decimal(1) + assert row["compactionFloorUnreachableCount"] == Decimal(0) + # `checkpoint` is counted by `_save_compaction_state(record_event=True)`, + # never here — two counters for one event would disagree. + assert "compactionCheckpointCount" not in row + + +@pytest.mark.asyncio +async def test_kill_switch_writes_no_counters(sessions_metadata_table, monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + from apis.shared.sessions.metadata import store_message_metadata + + _seed_session(sessions_metadata_table) + await store_message_metadata( + session_id="s1", user_id="u1", message_id=1, + message_metadata=_meta(compactionEvents=[{"kind": "applied"}]), + ) + row = _session_row(sessions_metadata_table) + for attr in ("compactionAppliedCount", "compactionForcedCount", "compactionFloorUnreachableCount"): + assert attr not in row + + +@pytest.mark.asyncio +async def test_malformed_events_count_as_zero_not_as_a_failure(sessions_metadata_table, monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + from apis.shared.sessions.metadata import store_message_metadata + + _seed_session(sessions_metadata_table) + await store_message_metadata( + session_id="s1", user_id="u1", message_id=1, + message_metadata=_meta(compactionEvents=["applied", {"nokind": 1}, {"kind": "applied"}]), + ) + row = _session_row(sessions_metadata_table) + assert row["compactionAppliedCount"] == Decimal(1) diff --git a/backend/tests/shared/test_document_digest.py b/backend/tests/shared/test_document_digest.py new file mode 100644 index 000000000..b0f9a0b87 --- /dev/null +++ b/backend/tests/shared/test_document_digest.py @@ -0,0 +1,247 @@ +"""DocumentDigest (apis/shared/files/document_digest.py). + +The deterministic outline (headings, counts, table/figure mentions, sampled +sections), the rendered block and its hard token budget — including the +spec's gate, a 200-page PDF digest under 1,500 tokens — the fail-open +abstract, and ``build_digest``'s never-raise contract. Fixtures are the same +hand-built PDFs and DOCX archives the document_read tests use. +""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from apis.shared.files import document_digest as dd +from tests.shared.test_document_read import build_docx, build_pdf + +MODULE = "apis.shared.files.document_digest" +DOCX = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + + +class TestHeadings: + @pytest.mark.parametrize( + "line,expected", + [ + ("1. Introduction", True), + ("2.3 Coverage Limits", True), + ("ARTICLE IV", True), + ("Section 12 Termination", True), + ("DEFINITIONS", True), + ("Named Insured And Limits", True), + ("This policy covers the named insured for losses.", False), + ("the quick brown fox jumps over the lazy dog again", False), + ("ok", False), + ("Total:", False), + ], + ) + def test_heuristic(self, line, expected): + assert dd.looks_like_heading(line) is expected + + +class TestOutline: + def test_pdf_outline_carries_page_numbers_and_structure_counts(self): + pages = ["1. Declarations\nNamed insured Acme", "body text only here.", "2. Insuring Agreements\nsee Table 1 and Figure 2"] + outline = dd.extract_outline("pdf", build_pdf(pages)) + assert (outline.unit, outline.count) == ("page", 3) + assert [(s.start, s.title) for s in outline.sections] == [(1, "1. Declarations"), (3, "2. Insuring Agreements")] + assert (outline.tables, outline.figures) == (1, 1) + assert outline.chars > 0 and outline.abstract is None + + def test_pdf_without_headings_anchors_on_first_lines(self): + outline = dd.extract_outline("pdf", build_pdf(["scanned text one.", "", "scanned text three."])) + assert [(s.start, s.title) for s in outline.sections] == [(1, "scanned text one."), (3, "scanned text three.")] + + def test_docx_outline_uses_paragraph_numbers(self): + raw = build_docx(["Executive Summary", "The budget grows by 4 million this year.", "Risks", "Some risk text."]) + outline = dd.extract_outline("docx", raw) + assert (outline.unit, outline.count) == ("paragraph", 4) + assert [(s.start, s.title) for s in outline.sections] == [(1, "Executive Summary"), (3, "Risks")] + + def test_markdown_headings_win_and_html_tags_are_stripped(self): + md = dd.extract_outline("md", b"# Title\nbody\n## Second part\nmore") + assert [(s.start, s.title) for s in md.sections] == [(1, "Title"), (3, "Second part")] + html = dd.extract_outline("html", b"

OVERVIEW

\n

lower case body sentence.

") + assert [s.title for s in html.sections] == ["OVERVIEW"] + + def test_sections_are_sampled_evenly_past_the_cap(self, monkeypatch): + monkeypatch.setattr(dd, "DOCUMENT_DIGEST_MAX_SECTIONS", 5) + outline = dd.extract_outline("pdf", build_pdf([f"{i}. Heading {i}" for i in range(1, 21)])) + assert len(outline.sections) == 5 + assert [s.start for s in outline.sections] == [1, 5, 9, 13, 17] + + def test_text_sample_spreads_across_pages(self): + sample = dd.text_sample("pdf", build_pdf(["first page words", "middle page words", "last page words"]), limit=200) + assert "first" in sample and "middle" in sample and "last" in sample + assert len(dd.text_sample("md", b"x" * 50_000, limit=100)) == 100 + + +class TestRender: + def _digest(self, sections=3, abstract="An abstract."): + return dd.DocumentDigest( + format="pdf", unit="page", count=47, tables=2, figures=0, abstract=abstract, + sections=[dd.DigestSection(start=i + 1, title=f"Section {i + 1}") for i in range(sections)], + ) + + def test_block_shape(self): + text = dd.render_digest(self._digest(), filename='Policy <"A&B">.pdf', upload_id="up-1") + assert text.startswith('') + assert " An abstract." in text + assert '
Section 3
' in text + assert text.endswith("") + + def test_budget_drops_sections_first_then_trims_the_abstract(self): + digest = self._digest(sections=40, abstract="word " * 400) + full = dd.render_digest(digest, filename="a.pdf", upload_id="u") + assert dd.estimate_tokens(full) > 300 + tight = dd.render_digest(digest, filename="a.pdf", upload_id="u", budget_tokens=300) + assert dd.estimate_tokens(tight) <= 300 + assert '
' not in tight or "…" in tight + tiny = dd.render_digest(digest, filename="a.pdf", upload_id="u", budget_tokens=40) + assert dd.estimate_tokens(tiny) <= 40 + 4 # header always fits, may exceed a tiny budget by a line + assert tiny.startswith(" 0 and digest.extractor_ms >= 0 + item = digest.to_item() + assert item["version"] == dd.DIGEST_VERSION and "error" not in item + assert dd.DocumentDigest.from_item(item).sections[1].title == "2. Terms" + + @pytest.mark.asyncio + async def test_abstract_failure_keeps_the_outline(self, monkeypatch): + _bedrock(monkeypatch, fail=True) + digest = await dd.build_digest(raw=build_pdf(["1. Intro"]), mime_type="application/pdf", filename="p.pdf", upload_id="u1") + assert digest.status == "ready" and digest.abstract is None and digest.model_id is None + assert digest.sections[0].title == "1. Intro" + + @pytest.mark.asyncio + async def test_extraction_failure_is_a_failed_digest_with_a_class_name_only(self): + digest = await dd.build_digest(raw=b"not a pdf", mime_type="application/pdf", filename="p.pdf", upload_id="u1", with_abstract=False) + assert digest.status == "failed" and digest.error == "DocumentReadError" + assert "not a pdf" not in digest.to_item().values().__repr__() + + @pytest.mark.asyncio + async def test_non_documents_are_refused(self): + digest = await dd.build_digest(raw=b"a,b", mime_type="text/csv", filename="d.csv", upload_id="u1") + assert (digest.status, digest.error) == ("failed", "NotADocument") + + def test_malformed_rows_read_as_no_digest(self): + assert dd.DocumentDigest.from_item("nope") is None + assert dd.DocumentDigest.from_item({"count": "many"}) is None + + def test_metric_is_content_free(self, monkeypatch): + calls = [] + monkeypatch.setattr("apis.shared.observability.emf.emit_emf_metrics", lambda ns, metrics, properties=None, units=None: calls.append((ns, metrics, properties))) + monkeypatch.setattr("apis.shared.observability.prompt_cache.prompt_cache_observability_enabled", lambda: True) + dd.record_digest(dd.DocumentDigest(format="pdf", tokens=900, extractor_ms=1200, abstract="SECRET")) + assert calls == [("AgentCoreStack/Compaction", {"DocumentDigestGenerated": 1, "DocumentDigestTokens": 900, "DocumentDigestMs": 1200}, {"format": "pdf", "outcome": "ready"})] + assert "SECRET" not in repr(calls) + + def test_kill_switch(self, monkeypatch): + monkeypatch.setenv("DOCUMENT_DIGEST_ENABLED", "false") + assert dd.document_digest_enabled() is False + monkeypatch.setenv("DOCUMENT_DIGEST_ENABLED", "") + assert dd.document_digest_enabled() is True + + + +class TestRenderBudgetIsHard: + """``render_digest`` escaped the abstract *after* slicing it to the room + that was left, so ``&`` -> ``&`` could render up to 5x longer than the + slice it was measured on. Measured: an all-``&`` abstract rendered 388 + tokens against a 100-token budget. The cap is the contract (spec PR-2 + decision #15 — ``digest.tokens`` is a stored fact per file), so it is now + measured on the rendered text, and the abstract is dropped rather than + allowed to overshoot. + """ + + def _digest(self, abstract: str) -> dd.DocumentDigest: + return dd.DocumentDigest( + status="ready", format="pdf", unit="page", count=10, + abstract=abstract, sections=[dd.DigestSection(start=1, title="Intro")], + ) + + @pytest.mark.parametrize( + "abstract", + [ + "&" * 4000, # worst case: every char expands 5x + " & " * 400, + "R&D spending rose. " * 300, + "Ordinary prose with no entities at all. " * 200, + "& already-escaped-looking text " * 200, + ], + ids=["all-amps", "angle-and-amp", "r-and-d", "plain-prose", "looks-escaped"], + ) + @pytest.mark.parametrize("budget", [40, 100, 400, 1_500]) + def test_never_exceeds_the_budget(self, abstract, budget): + text = dd.render_digest(self._digest(abstract), filename="a.pdf", upload_id="u", budget_tokens=budget) + assert dd.estimate_tokens(text) <= budget, dd.estimate_tokens(text) + + def test_entities_are_never_split(self): + text = dd.render_digest(self._digest("R&D " * 500), filename="a.pdf", upload_id="u", budget_tokens=60) + # A truncated "&" would leave a bare "&" or a fragment like "&am". + assert text.count("&") == text.count("&") + + def test_the_handle_always_survives(self): + """Only the opening tag may outlive a budget this small — it is what + makes the document retrievable at all.""" + text = dd.render_digest(self._digest("&" * 4000), filename="a.pdf", upload_id="u-keep", budget_tokens=1) + assert 'upload_id="u-keep"' in text + + def test_a_normal_digest_is_unchanged(self): + digest = self._digest("A short, ordinary abstract of the policy.") + text = dd.render_digest(digest, filename="a.pdf", upload_id="u", budget_tokens=1_500) + assert "A short, ordinary abstract of the policy." in text + assert "Intro" in text diff --git a/backend/tests/shared/test_document_read.py b/backend/tests/shared/test_document_read.py new file mode 100644 index 000000000..c8fe0342f --- /dev/null +++ b/backend/tests/shared/test_document_read.py @@ -0,0 +1,356 @@ +"""Document read service (apis/shared/files/document_read.py). + +The retrieval primitive behind ``document_read``: PDF page ranges re-assembled +as native document blocks (bounded by ``max_pages``), pattern search over the +text layer with page numbers, bounded text for DOCX and text-family files, the +session listing, and the tool-injection gate. Fixtures are hand-built PDFs +and DOCX archives — never user files — rendered through the same pypdfium2 +the service uses, so page identity under re-assembly is exercised for real. +""" + +from __future__ import annotations + +import io +import zipfile +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from apis.shared.files import document_read as dr +from apis.shared.files.models import FileMetadata, FileStatus +from apis.shared.files.workspace import WorkspaceFileNotFoundError, WorkspaceValidationError + +MODULE = "apis.shared.files.document_read" + + +# --------------------------------------------------------------------------- +# Fixtures +# --------------------------------------------------------------------------- + + +def build_pdf(page_texts): + """A minimal multi-page PDF with one Helvetica text line per page.""" + objs = [] + + def add(body: bytes) -> int: + objs.append(body) + return len(objs) + + font = add(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>") + contents = [] + for text in page_texts: + esc = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)") + stream = f"BT /F1 12 Tf 72 720 Td ({esc}) Tj ET".encode() + contents.append(add(b"<< /Length %d >>\nstream\n" % len(stream) + stream + b"\nendstream")) + pages_id = len(objs) + len(page_texts) + 1 + page_ids = [ + add( + f"<< /Type /Page /Parent {pages_id} 0 R /MediaBox [0 0 612 792] " + f"/Resources << /Font << /F1 {font} 0 R >> >> /Contents {c} 0 R >>".encode() + ) + for c in contents + ] + kids = " ".join(f"{p} 0 R" for p in page_ids) + assert add(f"<< /Type /Pages /Kids [{kids}] /Count {len(page_ids)} >>".encode()) == pages_id + catalog = add(f"<< /Type /Catalog /Pages {pages_id} 0 R >>".encode()) + out = io.BytesIO() + out.write(b"%PDF-1.4\n") + offsets = [] + for i, body in enumerate(objs, 1): + offsets.append(out.tell()) + out.write(f"{i} 0 obj\n".encode() + body + b"\nendobj\n") + xref = out.tell() + out.write(f"xref\n0 {len(objs) + 1}\n".encode() + b"0000000000 65535 f \n") + for o in offsets: + out.write(f"{o:010d} 00000 n \n".encode()) + out.write( + f"trailer\n<< /Size {len(objs) + 1} /Root {catalog} 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode() + ) + return out.getvalue() + + +def build_docx(paragraphs): + ns = 'xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"' + body = "".join(f"{p}" for p in paragraphs) + xml = f'{body}' + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w") as z: + z.writestr("[Content_Types].xml", "") + z.writestr("word/document.xml", xml) + return buf.getvalue() + + +PAGES = [f"Page {i} alpha beta" for i in range(1, 11)] + ["The termination clause survives", "Page 12 end"] + + +def _meta(upload_id="up-1", filename="policy.pdf", mime="application/pdf", size=1000, user="u1", session="s1"): + return FileMetadata( + upload_id=upload_id, user_id=user, session_id=session, filename=filename, mime_type=mime, + size_bytes=size, s3_key=f"user-files/{user}/{session}/{upload_id}/{filename}", s3_bucket="b", + status=FileStatus.READY, + ) + + +@pytest.fixture +def stored(monkeypatch): + """Patch the metadata lookup and the S3 fetch; returns a setter.""" + state = {} + + async def _owned(user_id, upload_id): + meta = state.get("meta") + if meta is None or meta.user_id != user_id or meta.upload_id != upload_id: + raise WorkspaceFileNotFoundError(f"No file with id '{upload_id}'") + return meta + + monkeypatch.setattr(f"{MODULE}._get_owned_ready_file", _owned) + monkeypatch.setattr(f"{MODULE}._get_object_bytes", lambda bucket, key: state["raw"]) + + def _set(meta, raw): + state["meta"], state["raw"] = meta, raw + + return _set + + +# --------------------------------------------------------------------------- +# Classification and argument parsing +# --------------------------------------------------------------------------- + + +class TestClassification: + @pytest.mark.parametrize( + "mime,filename,expected", + [ + ("application/pdf", "a.pdf", "pdf"), + ("application/vnd.openxmlformats-officedocument.wordprocessingml.document", "a.docx", "docx"), + ("text/plain", "a.txt", "txt"), + ("text/markdown", "a.md", "md"), + ("text/html", "a.html", "html"), + ("text/csv", "a.csv", None), # spreadsheet tools + ("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", "a.xlsx", None), + ("application/vnd.openxmlformats-officedocument.presentationml.presentation", "a.pptx", None), + ("image/png", "a.png", None), + ], + ) + def test_document_class(self, mime, filename, expected): + assert dr.document_format_for(mime, filename) == expected + assert dr.is_document_class(mime, filename) is (expected is not None) + + @pytest.mark.parametrize( + "value,expected", + [("4-7", (4, 7)), ("4", (4, 4)), (" 4 .. 7 ", (4, 7)), ("4:7", (4, 7)), ({"start": 2, "end": 3}, (2, 3)), + ([5, 6], (5, 6)), (3, (3, 3)), ("", None), (None, None)], + ) + def test_parse_page_range(self, value, expected): + assert dr.parse_page_range(value) == expected + + @pytest.mark.parametrize("value", ["0-3", "7-4", "abc", "1-2-3"]) + def test_bad_page_range_is_a_validation_error(self, value): + with pytest.raises(WorkspaceValidationError): + dr.parse_page_range(value) + + def test_max_pages_is_clamped_to_the_hard_cap(self): + assert dr.clamp_max_pages(None) == dr.DOCUMENT_READ_MAX_PAGES + assert dr.clamp_max_pages(0) == 1 + assert dr.clamp_max_pages(10_000) == dr.DOCUMENT_READ_HARD_MAX_PAGES + assert dr.clamp_max_pages("x") == dr.DOCUMENT_READ_MAX_PAGES + + +# --------------------------------------------------------------------------- +# PDF +# --------------------------------------------------------------------------- + + +class TestPdf: + @pytest.mark.asyncio + async def test_page_range_returns_only_those_pages_as_a_native_block(self, stored): + stored(_meta(), build_pdf(PAGES)) + result = await dr.read_document("u1", "s1", "up-1", page_range="4-7") + + assert result.mode == "pages" + assert result.pages_returned == 4 + assert result.payload["pages"] == {"start": 4, "end": 7} + assert result.payload["page_count"] == 12 + assert result.payload["truncated_to_max_pages"] is False + block = result.document_block + assert block["document"]["format"] == "pdf" + assert block["document"]["name"].startswith("policy p4-7 ") + # The slice really is pages 4–7 of the original, in order. + import pypdfium2 as pdfium + + sub = pdfium.PdfDocument(io.BytesIO(block["document"]["source"]["bytes"])) + assert len(sub) == 4 + assert dr._pdf_page_text(sub, 0) == "Page 4 alpha beta" + assert dr._pdf_page_text(sub, 3) == "Page 7 alpha beta" + sub.close() + assert "page k is original page 4 + k - 1" in result.payload["page_numbering"] + + @pytest.mark.asyncio + async def test_max_pages_caps_a_range_and_points_at_the_continuation(self, stored): + stored(_meta(), build_pdf(PAGES)) + result = await dr.read_document("u1", "s1", "up-1", page_range="1-12", max_pages=3) + assert result.pages_returned == 3 + assert result.payload["pages"] == {"start": 1, "end": 3} + assert result.payload["truncated_to_max_pages"] is True + assert result.payload["next_start"] == 4 + + @pytest.mark.asyncio + async def test_hard_cap_holds_whatever_the_model_asks(self, stored): + stored(_meta(), build_pdf([f"p{i}" for i in range(1, 41)])) + result = await dr.read_document("u1", "s1", "up-1", page_range="1-40", max_pages=999) + assert result.pages_returned == dr.DOCUMENT_READ_HARD_MAX_PAGES + + @pytest.mark.asyncio + async def test_range_past_the_end_clips_or_rejects(self, stored): + stored(_meta(), build_pdf(PAGES)) + result = await dr.read_document("u1", "s1", "up-1", page_range="11-30") + assert result.payload["pages"] == {"start": 11, "end": 12} + with pytest.raises(WorkspaceValidationError): + await dr.read_document("u1", "s1", "up-1", page_range="13-14") + + @pytest.mark.asyncio + async def test_pattern_returns_page_numbers_and_context(self, stored): + stored(_meta(), build_pdf(PAGES)) + result = await dr.read_document("u1", "s1", "up-1", pattern="TERMINATION clause") + assert result.mode == "pattern" + assert result.document_block is None + assert result.payload["pages_matched"] == [11] + assert result.payload["matches"][0]["page"] == 11 + assert "termination" in result.payload["matches"][0]["text"].lower() + + @pytest.mark.asyncio + async def test_invalid_regex_falls_back_to_a_literal_search(self, stored): + stored(_meta(), build_pdf(["cost (usd", "other"])) + result = await dr.read_document("u1", "s1", "up-1", pattern="cost (usd") + assert result.payload["pages_matched"] == [1] + + @pytest.mark.asyncio + async def test_no_range_no_pattern_returns_a_bounded_page_index(self, stored, monkeypatch): + monkeypatch.setattr(dr, "DOCUMENT_READ_INDEX_PAGES", 5) + stored(_meta(), build_pdf(PAGES)) + result = await dr.read_document("u1", "s1", "up-1") + assert result.mode == "index" + assert result.payload["page_count"] == 12 + assert len(result.payload["page_index"]) == 5 + assert result.payload["page_index"][0] == {"page": 1, "snippet": "Page 1 alpha beta"} + assert result.payload["index_truncated"] is True + + @pytest.mark.asyncio + async def test_corrupt_pdf_is_a_conversational_error(self, stored): + stored(_meta(), b"not a pdf") + with pytest.raises(dr.DocumentReadError): + await dr.read_document("u1", "s1", "up-1", page_range="1") + + def test_slice_names_are_bedrock_safe_and_never_collide(self): + a = dr._unique_document_name("BBR 5.0 Policy_Form (final).pdf", "p4-7") + b = dr._unique_document_name("BBR 5.0 Policy_Form (final).pdf", "p4-7") + assert a != b + assert a.startswith("BBR 5 0 Policy Form (final) p4-7 ") + import re + + assert re.fullmatch(r"[a-zA-Z0-9\s\-\(\)\[\]]+", a) and " " not in a + + +# --------------------------------------------------------------------------- +# DOCX and text +# --------------------------------------------------------------------------- + + +class TestDocxAndText: + DOCX_MIME = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + + @pytest.mark.asyncio + async def test_docx_text_is_bounded_with_offset_continuation(self, stored, monkeypatch): + monkeypatch.setattr(dr, "WORKSPACE_READ_MAX_BYTES", 20) + stored(_meta(filename="memo.docx", mime=self.DOCX_MIME), build_docx(["Alpha paragraph", "Beta paragraph"])) + first = await dr.read_document("u1", "s1", "up-1") + assert first.mode == "text" and first.payload["paragraph_count"] == 2 + assert first.payload["truncated"] is True and first.payload["next_offset"] == 20 + second = await dr.read_document("u1", "s1", "up-1", offset=first.payload["next_offset"]) + assert first.payload["content"] + second.payload["content"] == "Alpha paragraph\n\nBeta paragraph" + assert second.payload["truncated"] is False + + @pytest.mark.asyncio + async def test_docx_pattern_reports_paragraph_numbers(self, stored): + stored(_meta(filename="memo.docx", mime=self.DOCX_MIME), build_docx(["Intro", "Budget is 4 million", "Close"])) + result = await dr.read_document("u1", "s1", "up-1", pattern=r"\d+ million") + assert result.payload["unit"] == "paragraph" + assert result.payload["matches"] == [ + {"line": 2, "text": "Budget is 4 million", "before": ["Intro"], "after": ["Close"]} + ] + assert result.format == "docx" + + @pytest.mark.asyncio + async def test_page_range_on_a_non_pdf_is_rejected(self, stored): + stored(_meta(filename="memo.docx", mime=self.DOCX_MIME), build_docx(["x"])) + with pytest.raises(WorkspaceValidationError): + await dr.read_document("u1", "s1", "up-1", page_range="1-2") + + @pytest.mark.asyncio + async def test_text_family_delegates_to_the_workspace_read(self, stored, monkeypatch): + stored(_meta(filename="notes.md", mime="text/markdown"), b"# heading\nbody line\n") + delegate = AsyncMock(return_value={"content": "# heading", "offset": 0, "truncated": True, "next_offset": 9}) + monkeypatch.setattr(f"{MODULE}.read_workspace_file", delegate) + result = await dr.read_document("u1", "s1", "up-1") + delegate.assert_awaited_once_with("u1", "up-1", offset=0) + assert result.payload["content"] == "# heading" and result.payload["next_offset"] == 9 + + @pytest.mark.asyncio + async def test_text_family_pattern_greps_lines(self, stored): + stored(_meta(filename="notes.txt", mime="text/plain"), b"one\ntwo Fish\nthree\n") + result = await dr.read_document("u1", "s1", "up-1", pattern="fish") + assert result.payload["matches"][0]["line"] == 2 + + @pytest.mark.asyncio + async def test_non_document_files_are_refused(self, stored): + stored(_meta(filename="data.csv", mime="text/csv"), b"a,b\n") + with pytest.raises(dr.DocumentReadError): + await dr.read_document("u1", "s1", "up-1") + + @pytest.mark.asyncio + async def test_missing_upload_id_and_negative_offset_are_validation_errors(self): + with pytest.raises(WorkspaceValidationError): + await dr.read_document("u1", "s1", "") + with pytest.raises(WorkspaceValidationError): + await dr.read_document("u1", "s1", "up-1", offset=-1) + + +# --------------------------------------------------------------------------- +# Listing and the injection gate +# --------------------------------------------------------------------------- + + +class TestListingAndGate: + def _repo(self, monkeypatch, files): + repo = MagicMock() + repo.list_session_files = AsyncMock(return_value=files) + monkeypatch.setattr(f"{MODULE}.get_file_upload_repository", lambda: repo) + return repo + + @pytest.mark.asyncio + async def test_listing_keeps_only_this_users_readable_documents(self, monkeypatch): + repo = self._repo(monkeypatch, [ + _meta("up-1", "policy.pdf"), + _meta("up-2", "data.xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"), + _meta("up-3", "other.pdf", user="someone-else"), + _meta("up-4", "notes.md", "text/markdown"), + ]) + listing = await dr.list_session_documents("u1", "s1") + repo.list_session_files.assert_awaited_once_with("s1", status=FileStatus.READY) + assert [d["upload_id"] for d in listing["documents"]] == ["up-1", "up-4"] + assert listing["documents"][0]["modes"] == ["page_range", "pattern"] + assert listing["documents"][1]["modes"] == ["pattern", "text"] + assert listing["count"] == 2 + + @pytest.mark.asyncio + async def test_gate_is_true_only_with_a_readable_document(self, monkeypatch): + self._repo(monkeypatch, [_meta("up-2", "data.csv", "text/csv")]) + assert await dr.session_has_documents("u1", "s1") is False + self._repo(monkeypatch, [_meta("up-2", "data.csv", "text/csv"), _meta("up-1", "a.pdf")]) + assert await dr.session_has_documents("u1", "s1") is True + + @pytest.mark.asyncio + async def test_identity_is_mandatory(self): + with pytest.raises(Exception): + await dr.list_session_documents("", "s1") + with pytest.raises(Exception): + await dr.session_has_documents("u1", "") diff --git a/backend/tests/shared/test_document_read_pattern_safety.py b/backend/tests/shared/test_document_read_pattern_safety.py new file mode 100644 index 000000000..6faee3dc6 --- /dev/null +++ b/backend/tests/shared/test_document_read_pattern_safety.py @@ -0,0 +1,214 @@ +"""ReDoS safety for ``document_read`` pattern mode. + +The defect: ``_compile`` passed a model-supplied regex straight to +``re.compile``. A nested-quantifier pattern then runs in exponential time — +measured on the real engine, ``(a+)+$`` against ``"a"*n + "!"`` takes 0.87 s at +n=24, 3.5 s at 26, **14 s at 28**. Extracted PDF lines run 60–100 characters, so +such a pattern never returns; the search happens in ``asyncio.to_thread``, which +CPython gives no way to cancel, so the turn hangs to the 600 s SSE timeout. + +Two defences, tested here: + +* ``catastrophic_pattern`` refuses the nested-quantifier family up front and the + pattern is searched literally instead — the same graceful degradation an + invalid regex already got — with a note carried to the model so a different + result is never presented as the requested one. +* ``_Budget`` bounds the walk across lines and pages, so a *merely slow* pattern + the detector does not flag degrades to a partial answer instead of hanging. +""" + +from __future__ import annotations + +import time + +import pytest + +from apis.shared.files import document_read as dr + +from .test_document_read import build_pdf + +# Shapes that make the backtracking engine go exponential. +CATASTROPHIC = [ + "(a+)+$", + r"(\w+\s*)+", + r"(\s*\w+)*", + "(a*)*", + "(a+|b){2,}", + r"([a-z]+)+@", + "(x+)+y", +] + +# Patterns a model would plausibly write against a real document. None may be +# refused: a false positive silently turns a regex search into a literal one. +LEGITIMATE = [ + r"\d+", + r"\w+\s+total", + "(invoice|receipt)", + "(foo)+", + r"section \d+", + r"(?:\d{4})-(?:\d{2})", + "retention", + r"[A-Z]{2,}\s", + r"\$[\d,]+\.\d{2}", + "(a|b)+", + r"a{2,5}", + r"\(\w+\)+", # escaped parens are literals, not a group + r"[(]a+[)]+", # ...and so are parens inside a character class +] + + +class TestCatastrophicPattern: + @pytest.mark.parametrize("pattern", CATASTROPHIC) + def test_flags_nested_quantifiers(self, pattern): + assert dr.catastrophic_pattern(pattern) is True + + @pytest.mark.parametrize("pattern", LEGITIMATE) + def test_never_flags_a_legitimate_pattern(self, pattern): + assert dr.catastrophic_pattern(pattern) is False + + @pytest.mark.parametrize("pattern", ["", "(", ")", "(((", "a{", "[a+)+", "\\"]) + def test_malformed_input_never_raises(self, pattern): + assert dr.catastrophic_pattern(pattern) in (True, False) + + +class TestCompileFallback: + def test_catastrophic_pattern_is_searched_literally_with_a_note(self): + regex, note = dr._compile("(a+)+$") + assert note is not None and "exponential" in note + # Literal: it matches its own text, and not an exponential input. + assert regex.search("see (a+)+$ here") is not None + assert regex.search("aaaaaaaa!") is None + + def test_the_measured_attack_is_now_instant(self): + """n=28 took 14 s before this change; n=80 must be immediate now.""" + regex, _ = dr._compile("(a+)+$") + t0 = time.monotonic() + regex.search("a" * 80 + "!") + assert time.monotonic() - t0 < 0.05 + + def test_invalid_regex_still_degrades_to_literal(self): + regex, note = dr._compile("([") + assert note is not None and "not a valid regular expression" in note + assert regex.search("a ([ bracket") is not None + + def test_a_good_pattern_compiles_as_a_regex_with_no_note(self): + regex, note = dr._compile(r"section \d+") + assert note is None + assert regex.search("SECTION 44 here") is not None # IGNORECASE preserved + + +class TestBudget: + def test_disabled_budget_never_expires(self): + clock = dr._Budget(0) + assert clock.out_of_time() is False + + def test_expires_and_latches(self): + clock = dr._Budget(0.01) + time.sleep(0.02) + assert clock.out_of_time() is True + assert clock.expired is True + + def test_grep_stops_on_an_exhausted_budget(self): + clock = dr._Budget(0.01) + time.sleep(0.02) + import re + out = dr._grep_lines(["match"] * 100, re.compile("match"), 40, clock) + assert out == [] + + def test_grep_without_a_budget_is_unchanged(self): + import re + out = dr._grep_lines(["alpha", "beta", "alpha"], re.compile("alpha"), 40) + assert [m["line"] for m in out] == [1, 3] + + +class TestBudgetCoversTheScanOnly: + """The budget bounds the *scan*, not the setup that precedes it. + + ``_pdf_pattern`` used to start the clock before ``_open_pdf``, so lazily + importing the pypdfium2 native library and parsing the document were charged + to the scan. A cold container paying enough of the budget on the open + reported ``timed_out`` with ``pages_searched: 0`` — a timeout on a search + that never started, and an empty answer to an ordinary pattern. + """ + + def test_the_clock_starts_after_the_pdf_is_opened(self, monkeypatch): + """Structural, so it cannot flake: the open happens first.""" + order: list[str] = [] + real_open, real_budget = dr._open_pdf, dr._Budget + + class RecordingBudget(real_budget): # type: ignore[misc, valid-type] + def __init__(self, seconds: float = 0.0) -> None: + order.append("clock") + super().__init__(seconds) + + monkeypatch.setattr(dr, "_open_pdf", lambda raw: (order.append("open"), real_open(raw))[1]) + monkeypatch.setattr(dr, "_Budget", RecordingBudget) + dr._pdf_pattern(build_pdf(["alpha"]), "alpha", {"filename": "d.pdf", "format": "pdf"}) + assert order == ["open", "clock"] + + def test_a_slow_open_does_not_consume_the_scan_budget(self, monkeypatch): + """Behavioural: an open that outlasts the whole budget still leaves the + scan its full window. ``sleep`` only guarantees a lower bound, so the + pre-fix failure is certain while the margin the scan needs is ~600x.""" + real_open = dr._open_pdf + + def slow_open(raw: bytes): + time.sleep(0.5) + return real_open(raw) + + monkeypatch.setattr(dr, "_open_pdf", slow_open) + monkeypatch.setattr(dr, "DOCUMENT_READ_PATTERN_BUDGET_SECONDS", 0.2) + raw = build_pdf([f"page {i} retention clause" for i in range(1, 4)]) + res = dr._pdf_pattern(raw, "retention", {"filename": "d.pdf", "format": "pdf"}) + assert "timed_out" not in res.payload + assert res.payload["match_count"] == 3 + + def test_text_pattern_splits_the_lines_before_starting_the_clock(self, monkeypatch): + order: list[str] = [] + real_budget = dr._Budget + + class RecordingBudget(real_budget): # type: ignore[misc, valid-type] + def __init__(self, seconds: float = 0.0) -> None: + order.append("clock") + super().__init__(seconds) + + class Text(str): + def splitlines(self, *a, **kw): # type: ignore[override] + order.append("split") + return str.splitlines(self, *a, **kw) + + monkeypatch.setattr(dr, "_Budget", RecordingBudget) + dr._text_pattern(Text("alpha\nbeta"), "alpha", {"filename": "d.txt", "format": "txt"}, "line") + assert order == ["split", "clock"] + + +class TestEndToEnd: + def test_pdf_pattern_carries_the_note_and_still_answers(self): + raw = build_pdf(["the (a+)+$ literal lives here", "nothing on this page"]) + res = dr._pdf_pattern(raw, "(a+)+$", {"filename": "d.pdf", "format": "pdf"}) + assert res.payload["pattern_note"] + assert res.payload["match_count"] == 1 + assert res.payload["pages_matched"] == [1] + + def test_pdf_pattern_over_a_long_document_finishes_promptly(self): + """The regression in one assertion: a catastrophic pattern against a + 60-page document used to never return.""" + raw = build_pdf([f"SECTION {i} " + "a" * 70 for i in range(1, 61)]) + t0 = time.monotonic() + res = dr._pdf_pattern(raw, r"(\w+\s*)+$", {"filename": "d.pdf", "format": "pdf"}) + assert time.monotonic() - t0 < 5.0 + assert res.payload["pattern_note"] + + def test_text_pattern_reports_a_timeout_with_partial_matches(self, monkeypatch): + monkeypatch.setattr(dr, "DOCUMENT_READ_PATTERN_BUDGET_SECONDS", 1e-9) + text = "\n".join(f"line {i} alpha" for i in range(500)) + res = dr._text_pattern(text, "alpha", {"filename": "d.txt", "format": "txt"}, "line") + assert res.payload["timed_out"] is True + assert "ran out of time" in res.payload["hint"] + + def test_a_normal_search_reports_no_timeout_and_no_note(self): + raw = build_pdf([f"SECTION {i} retention clause" for i in range(1, 21)]) + res = dr._pdf_pattern(raw, "retention", {"filename": "d.pdf", "format": "pdf"}) + assert "timed_out" not in res.payload + assert "pattern_note" not in res.payload + assert res.payload["match_count"] == 20 diff --git a/backend/tests/shared/test_document_tokens.py b/backend/tests/shared/test_document_tokens.py new file mode 100644 index 000000000..46a3563d7 --- /dev/null +++ b/backend/tests/shared/test_document_tokens.py @@ -0,0 +1,77 @@ +"""Document token estimation (apis/shared/files/document_tokens.py). + +The regression this guards: ``bytes/4`` scored a 27,578-byte, 60-page PDF at +6,894 tokens when the measured first write was ~94K (dev session ``61de2256``, +2026-09-16). Bedrock dual-encodes each PDF page as an image, so the dominant +term is page count, not file size. Fixtures are hand-built PDFs — never user +files — so the page-driven estimate is exercised for real. +""" + +from __future__ import annotations + +import io + +import pytest + +from apis.shared.files import document_tokens as dt + +from .test_document_read import build_pdf + + +class TestPdfPageCount: + def test_counts_pages(self): + assert dt.pdf_page_count(build_pdf([f"p{i}" for i in range(1, 13)])) == 12 + + @pytest.mark.parametrize("raw", [b"", b"not a pdf", b"%PDF-1.4\ngarbage", None, 12, "str"]) + def test_unparseable_is_none_never_raises(self, raw): + assert dt.pdf_page_count(raw) is None + + +class TestEstimateDocumentTokens: + def test_pdf_is_driven_by_pages_not_bytes(self): + """The defect, stated as a test: a byte-small PDF with many pages.""" + raw = build_pdf([f"SECTION {i}" for i in range(1, 61)]) + tokens = dt.estimate_document_tokens("pdf", raw) + assert tokens == 60 * dt.PDF_PAGE_TOKEN_ESTIMATE + # ...and that is far above what bytes/4 would have said. + assert tokens > 10 * (len(raw) // dt.CHARS_PER_TOKEN) + + def test_byte_heavy_pdf_is_never_scored_below_bytes_over_four(self): + """max(), not replace: a scanned PDF whose bytes dominate keeps the + old floor so the estimate can only move up.""" + raw = build_pdf(["one page"]) + b"%" + b"\0" * 400_000 + assert dt.estimate_document_tokens("pdf", raw) == len(raw) // dt.CHARS_PER_TOKEN + + @pytest.mark.parametrize("fmt", ["docx", "txt", "md", "html", "", None]) + def test_non_pdf_keeps_bytes_over_four(self, fmt): + """Bedrock extracts these to text — no image channel to miss.""" + raw = b"x" * 40_000 + assert dt.estimate_document_tokens(fmt, raw) == 10_000 + + def test_pdf_format_match_is_case_insensitive(self): + raw = build_pdf(["a", "b"]) + assert dt.estimate_document_tokens("PDF", raw) == dt.estimate_document_tokens("pdf", raw) + + def test_unparseable_pdf_falls_back_to_bytes(self): + raw = b"not a pdf at all" * 100 + assert dt.estimate_document_tokens("pdf", raw) == len(raw) // dt.CHARS_PER_TOKEN + + @pytest.mark.parametrize("raw", [None, "", 0, [], {}]) + def test_non_bytes_is_zero(self, raw): + assert dt.estimate_document_tokens("pdf", raw) == 0 + + def test_env_tunable(self, monkeypatch): + """The constant is re-fittable from measured rows without a deploy.""" + monkeypatch.setattr(dt, "PDF_PAGE_TOKEN_ESTIMATE", 2_000) + assert dt.estimate_document_tokens("pdf", build_pdf(["a", "b", "c"])) == 6_000 + + +class TestAgainstTheDevMeasurement: + def test_sixty_page_pdf_lands_near_the_measured_write(self): + """Dev session 61de2256: 60 pages measured ~94K tokens. The estimate + must be the same order of magnitude, and deliberately a little under + (it gates an eviction).""" + raw = build_pdf([f"SECTION {i} clause text" for i in range(1, 61)]) + tokens = dt.estimate_document_tokens("pdf", raw) + measured = 94_000 + assert 0.7 * measured <= tokens <= measured, tokens diff --git a/backend/tests/shared/test_feedback_eval_sampler.py b/backend/tests/shared/test_feedback_eval_sampler.py new file mode 100644 index 000000000..2b06d0ce6 --- /dev/null +++ b/backend/tests/shared/test_feedback_eval_sampler.py @@ -0,0 +1,193 @@ +"""Eval sampling (response-feedback spec §11 PR-4): routing by reason, the +content-free verdict, tool-failure corroboration, and the batch over a real +(moto) table with a fake judge — the AgentCore adapter is a seam, not a +dependency of these tests. +""" + +from __future__ import annotations + +from typing import Any, Dict, List, Sequence + +import boto3 +import pytest +from boto3.dynamodb.conditions import Key + +from apis.shared.feedback_eval import sampler +from apis.shared.sessions import feedback as fb + +OWNER = "user-owner" + + +def _table(): + return boto3.resource("dynamodb", region_name="us-east-1").Table("test-sessions-metadata") + + +def _seed_session(session_id, user_id=OWNER): + _table().put_item(Item={ + "PK": f"USER#{user_id}", "SK": f"S#{session_id}", "GSI_PK": f"SESSION#{session_id}", "GSI_SK": "META", + "sessionId": session_id, "userId": user_id, "status": "active", + }) + + +def _row(session_id, message_id, user_id=OWNER): + return _table().get_item(Key={"PK": f"USER#{user_id}", "SK": f"F#{session_id}#{message_id}"}).get("Item") + + +class FakeJudge: + def __init__(self, results: List[Dict[str, Any]] | None = None, fail_for: str | None = None): + self.results = results or [] + self.calls: List[tuple] = [] + self.fail_for = fail_for + + def judge(self, session_id: str, evaluator_ids: Sequence[str]) -> List[Dict[str, Any]]: + self.calls.append((session_id, tuple(evaluator_ids))) + if session_id == self.fail_for: + raise RuntimeError("judge exploded") + return self.results + + +RAW = [ + {"evaluatorId": "Builtin.Correctness", "value": 1.0, "label": "Perfectly Correct", + "explanation": "The agent correctly reported that SECRET tool call 502'd", "tokenUsage": {"totalTokens": 1728}, + "context": {"spanContext": {"sessionId": "sid-x", "traceId": "t1"}}}, + {"evaluatorId": "Builtin.Correctness", "value": 0.0, "label": "Incorrect", "explanation": "SECRET", "tokenUsage": {"totalTokens": 900}}, + {"evaluatorId": "Builtin.Faithfulness", "value": 0.5, "label": "Partly", "explanation": "SECRET"}, + {"evaluatorId": "Builtin.Faithfulness", "errorCode": "ThrottlingException", "errorMessage": "slow down"}, +] + + +def test_routing_follows_the_spec_buckets(): + assert sampler.evaluators_for("wrong") == ("Builtin.Correctness", "Builtin.Faithfulness") + assert sampler.evaluators_for("instructions") == ("Builtin.InstructionFollowing",) + assert sampler.evaluators_for("length") == ("Builtin.Conciseness",) + assert sampler.evaluators_for("tool_failed") == () + assert sampler.evaluators_for("outdated") == () + assert sampler.evaluators_for("other") == ("Builtin.Helpfulness",) + assert sampler.evaluators_for(None) == ("Builtin.Helpfulness",) + assert sampler.evaluators_for("not-a-code") == ("Builtin.Helpfulness",) + + +def test_summarize_keeps_numbers_and_labels_and_drops_the_explanation(): + scores = sampler.summarize(RAW) + assert scores == { + "Builtin.Correctness": {"value": 0.5, "n": 2, "tokens": 2628, "rating": "Incorrect"}, + "Builtin.Faithfulness": {"value": 0.5, "n": 1, "tokens": 0, "rating": "Partly"}, + } + assert "SECRET" not in repr(scores) and "explanation" not in repr(scores) + + +def test_corroboration_reads_the_call_census(): + assert sampler.corroborate_tool_failure(None) is None + assert sampler.corroborate_tool_failure({"cost": {"total": 1}}) is None + assert sampler.corroborate_tool_failure({"toolCalls": {"web_search": {"calls": 2, "errors": 0}}}) is False + assert sampler.corroborate_tool_failure({"toolCalls": {"web_search": {"calls": 2, "errors": 1}}}) is True + + +def test_build_verdict_shapes(): + v = sampler.build_verdict("wrong", RAW, None) + assert v["reason"] == "wrong" and v["evaluators"] == ["Builtin.Correctness", "Builtin.Faithfulness"] + assert set(v["scores"]) == {"Builtin.Correctness", "Builtin.Faithfulness"} + assert "toolFailureCorroborated" not in v + v = sampler.build_verdict("tool_failed", [], {"toolCalls": {"x": {"calls": 1, "errors": 1}}}) + assert v == {"reason": "tool_failed", "evaluators": [], "toolFailureCorroborated": True} + assert sampler.build_verdict(None, [], None) == {"reason": "none", "evaluators": ["Builtin.Helpfulness"]} + + +@pytest.fixture() +def table(sessions_metadata_table, monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + return _table() + + +@pytest.mark.asyncio +async def test_down_thumbs_are_queued_on_the_timestamp_index_and_up_thumbs_leave_it(table): + _seed_session("s1") + await fb.put_message_feedback("s1", OWNER, 1, -1, reason="wrong") + await fb.put_message_feedback("s1", OWNER, 3, -1) + await fb.put_message_feedback("s1", OWNER, 5, 1) + queue = fb.list_recent_down_thumbs(table, limit=10) + assert sorted(int(r["messageId"]) for r in queue) == [1, 3] + assert all(r["GSI1PK"] == "FEEDBACK#down" and r["GSI1SK"] == r["updatedAt"] for r in queue) + # Flipping to up moves the row out of the down queue. + await fb.put_message_feedback("s1", OWNER, 1, 1) + assert [int(r["messageId"]) for r in fb.list_recent_down_thumbs(table, limit=10)] == [3] + + +@pytest.mark.asyncio +async def test_store_evaluation_refuses_prose_and_requires_the_row(table): + _seed_session("s1") + await fb.put_message_feedback("s1", OWNER, 1, -1, reason="wrong") + with pytest.raises(ValueError): + fb.store_evaluation(table, OWNER, "s1", 1, {"scores": {"x": {"value": 1, "explanation": "SECRET"}}}) + fb.store_evaluation(table, OWNER, "s1", 1, {"reason": "wrong", "scores": {"Builtin.Correctness": {"value": 1, "n": 1}}}) + row = _row("s1", 1) + assert row["evaluatedAt"] and row["evaluation"]["reason"] == "wrong" + from botocore.exceptions import ClientError + with pytest.raises(ClientError): + fb.store_evaluation(table, OWNER, "s1", 99, {"reason": "wrong"}) + + +@pytest.mark.asyncio +async def test_batch_judges_by_reason_skips_judged_and_survives_one_failure(table): + for sid in ("s-wrong", "s-tool", "s-boom", "s-done"): + _seed_session(sid) + await fb.put_message_feedback("s-wrong", OWNER, 1, -1, reason="wrong") + await fb.put_message_feedback("s-tool", OWNER, 2, -1, reason="tool_failed") + await fb.put_message_feedback("s-boom", OWNER, 3, -1, reason="instructions") + await fb.put_message_feedback("s-done", OWNER, 4, -1, reason="length") + fb.store_evaluation(table, OWNER, "s-done", 4, {"reason": "length"}) + judge = FakeJudge(results=RAW, fail_for="s-boom") + + def lookup(session_id, message_id): + return {"toolCalls": {"web_search": {"calls": 1, "errors": 1}}} if session_id == "s-tool" else None + + report = await sampler.run_sampling_batch(table, judge, limit=10, cost_row_lookup=lookup) + + assert (report.candidates, report.judged, report.skipped_no_judge, report.skipped_already, report.failed) == (3, 1, 1, 1, 1) + assert report.corroborated == 1 and report.errors == ["RuntimeError"] + assert report.tokens == 2628 + assert sorted(judge.calls) == [("s-boom", ("Builtin.InstructionFollowing",)), ("s-wrong", ("Builtin.Correctness", "Builtin.Faithfulness"))] + + wrong = _row("s-wrong", 1) + assert wrong["evaluation"]["scores"]["Builtin.Correctness"]["rating"] == "Incorrect" + assert "SECRET" not in repr(wrong) + tool = _row("s-tool", 2) + assert tool["evaluation"] == {"reason": "tool_failed", "evaluators": [], "toolFailureCorroborated": True} + assert "evaluation" not in _row("s-boom", 3) + # The judged row was left alone (its evaluatedAt is the earlier one). + assert _row("s-done", 4)["evaluation"] == {"reason": "length"} + + # A second batch finds nothing new to judge except the failed one. + report = await sampler.run_sampling_batch(table, FakeJudge(results=RAW), limit=10) + assert (report.candidates, report.judged, report.skipped_already) == (1, 1, 3) + + +def test_agentcore_judge_requires_the_log_group(monkeypatch): + monkeypatch.delenv(sampler.RUNTIME_LOG_GROUP_ENV, raising=False) + with pytest.raises(RuntimeError): + sampler.AgentCoreJudge().judge("s1", ["Builtin.Helpfulness"]) + + +def test_agentcore_judge_sends_the_runtime_session_id(monkeypatch): + calls = {} + + class FakeClient: + def __init__(self, region_name=None): + calls["region"] = region_name + + def run(self, **kwargs): + calls.update(kwargs) + return [{"evaluatorId": "Builtin.Helpfulness", "value": 1.0}] + + import types, sys + fake_mod = types.ModuleType("bedrock_agentcore.evaluation") + fake_mod.EvaluationClient = FakeClient + monkeypatch.setitem(sys.modules, "bedrock_agentcore.evaluation", fake_mod) + from apis.shared.harness.runner import runtime_session_id_for + + judge = sampler.AgentCoreJudge(log_group_name="/aws/bedrock-agentcore/runtimes/rt-DEFAULT", region_name="us-west-2") + out = judge.judge("sess-1", ["Builtin.Helpfulness"]) + assert out[0]["value"] == 1.0 + assert calls["session_id"] == runtime_session_id_for("sess-1") + assert calls["log_group_name"] == "/aws/bedrock-agentcore/runtimes/rt-DEFAULT" and calls["region"] == "us-west-2" + assert calls["evaluator_ids"] == ["Builtin.Helpfulness"] diff --git a/backend/tests/shared/test_files.py b/backend/tests/shared/test_files.py index 1a5dddc73..e6f02c4bd 100644 --- a/backend/tests/shared/test_files.py +++ b/backend/tests/shared/test_files.py @@ -44,6 +44,19 @@ async def test_update_status(self, file_repository): assert updated is not None assert updated.status == "ready" + @pytest.mark.asyncio + async def test_update_digest_round_trips_and_needs_the_row(self, file_repository): + await file_repository.create_file(_make_file()) + digest = {"version": 1, "status": "ready", "format": "pdf", "count": 3, + "sections": [{"start": 1, "title": "Intro"}], "tokens": 120, "abstract": "x"} + updated = await file_repository.update_file_digest("u1", "f1", digest) + assert updated is not None and updated.digest["count"] == 3 + assert (await file_repository.get_file("u1", "f1")).digest["sections"] == [{"start": 1, "title": "Intro"}] + assert await file_repository.update_file_digest("u1", "missing", digest) is None + # Rows without a digest read as None, never as an empty map. + await file_repository.create_file(_make_file("f2")) + assert (await file_repository.get_file("u1", "f2")).digest is None + @pytest.mark.asyncio async def test_delete_file(self, file_repository): await file_repository.create_file(_make_file()) @@ -132,3 +145,18 @@ async def test_resolve_max_files(self, file_repository, s3_bucket, aws): resolver._file_repository = file_repository files = await resolver.resolve_files("u1", [f"f{i}" for i in range(10)], max_files=3) assert len(files) == 3 + + @pytest.mark.asyncio + async def test_resolve_files_no_cap_when_max_files_is_none(self, file_repository, s3_bucket, aws): + # The chat route applies the per-message cap itself (so it can tell + # the user which files were left out) and passes None here. + import boto3 + from apis.shared.files.file_resolver import FileResolver + s3 = boto3.client("s3", region_name="us-east-1") + for i in range(7): + s3.put_object(Bucket=s3_bucket, Key=f"uploads/u1/f{i}", Body=b"x") + await file_repository.create_file(_make_file(f"f{i}")) + resolver = FileResolver(s3_client=s3) + resolver._file_repository = file_repository + files = await resolver.resolve_files("u1", [f"f{i}" for i in range(7)], max_files=None) + assert len(files) == 7 diff --git a/backend/tests/shared/test_injected_tool_cache_eligibility.py b/backend/tests/shared/test_injected_tool_cache_eligibility.py index 8c4a05ac9..e04016c8f 100644 --- a/backend/tests/shared/test_injected_tool_cache_eligibility.py +++ b/backend/tests/shared/test_injected_tool_cache_eligibility.py @@ -1,9 +1,9 @@ """`injected_tools_are_key_described` — which turns may reuse a cached Agent. -The agent cache key carries session, user and a hash of enabled_tools. A -per-request tool builder that closes over only those produces tools equivalent -to freshly-built ones, so the cached agent is safe to reuse. One that captures -anything else (an assistant id, a memory binding) is not. +The agent cache key carries session, user, assistant and a hash of +enabled_tools. A per-request tool builder that closes over only those produces +tools equivalent to freshly-built ones, so the cached agent is safe to reuse. +One that captures anything else (a memory binding) is not. Getting this predicate wrong in the permissive direction is a correctness bug — an agent answering against the wrong assistant's corpus or the wrong memory @@ -12,9 +12,13 @@ from apis.shared.tools.injected import ( ARTIFACT_TOOL_IDS, + EXCEL_SPREADSHEET_TOOL_IDS, INJECTED_TOOL_IDS, KEY_DESCRIBED_INJECTED_TOOL_IDS, + POWERPOINT_PRESENTATION_TOOL_IDS, SPREADSHEET_TOOL_IDS, + WORD_DOCUMENT_TOOL_IDS, + WORKSPACE_TOOL_IDS, injected_tools_are_key_described, ) @@ -30,17 +34,20 @@ def test_a_turn_with_no_injected_tools_is_trivially_key_described(self): assert injected_tools_are_key_described(None, has_memory_binding=False) assert injected_tools_are_key_described([], has_memory_binding=False) - def test_spreadsheet_tools_are_not_key_described(self): - # They close over `assistant_id`, which the key does not carry. + def test_spreadsheet_tools_are_key_described(self): + # They close over `assistant_id`, which `_create_cache_key` now carries + # (and `PausedTurnSnapshot` replays on resume). for tool_id in SPREADSHEET_TOOL_IDS: - assert not injected_tools_are_key_described( + assert injected_tools_are_key_described( [tool_id], has_memory_binding=False ) def test_one_unkeyed_builder_disqualifies_the_whole_turn(self): - # The agent is one object; a single unkeyed capture taints it. + # The agent is one object; a single unkeyed capture taints it. With + # every enabled_tools-gated family now described, the only unkeyed + # capture left is the memory binding, which the caller flags. assert not injected_tools_are_key_described( - ["create_artifact", "analyze_spreadsheet"], has_memory_binding=False + ["create_artifact", "analyze_spreadsheet"], has_memory_binding=True ) def test_a_memory_binding_vetoes_regardless_of_enabled_tools(self): @@ -51,27 +58,50 @@ def test_a_memory_binding_vetoes_regardless_of_enabled_tools(self): ) assert not injected_tools_are_key_described([], has_memory_binding=True) - def test_every_family_still_awaiting_promotion_bypasses(self): - """The experiment measures one variable (spec §6). - - Word/Excel/PowerPoint/workspace capture only session+user, so they are - eligible on the same reasoning as artifacts — but they stay out until the - artifact arm reads clean. This fails the day someone widens the set - without revisiting the experiment. - """ - for tool_id in INJECTED_TOOL_IDS - KEY_DESCRIBED_INJECTED_TOOL_IDS: - assert not injected_tools_are_key_described( + def test_session_only_families_are_key_described(self): + """Word/Excel/PowerPoint/workspace close over `(session_id, user_id)` + only — both key elements — so they are eligible on exactly the artifact + reasoning. Promoted once the artifact arm read clean (spec §8).""" + for tool_id in ( + WORD_DOCUMENT_TOOL_IDS + | EXCEL_SPREADSHEET_TOOL_IDS + | POWERPOINT_PRESENTATION_TOOL_IDS + | WORKSPACE_TOOL_IDS + ): + assert injected_tools_are_key_described( [tool_id], has_memory_binding=False - ), f"{tool_id} was promoted without updating the experiment" + ), f"{tool_id} closes over session+user only and must be cacheable" + + def test_a_turn_mixing_every_promoted_family_is_key_described(self): + assert injected_tools_are_key_described( + [ + "create_artifact", + "create_word_document", + "create_excel_spreadsheet", + "create_powerpoint_presentation", + "workspace_files", + "web_search", + ], + has_memory_binding=False, + ) + + def test_every_enabled_tools_gated_family_is_key_described(self): + """Fails the day a new injected family is added without deciding its + cache eligibility: a new builder that closes over something the key + does not carry must be left OUT of the described set until it does.""" + assert INJECTED_TOOL_IDS - KEY_DESCRIBED_INJECTED_TOOL_IDS == frozenset() - def test_the_experiment_arm_is_artifacts_only(self): - assert KEY_DESCRIBED_INJECTED_TOOL_IDS == ARTIFACT_TOOL_IDS + def test_artifacts_remain_in_the_set(self): + assert ARTIFACT_TOOL_IDS <= KEY_DESCRIBED_INJECTED_TOOL_IDS def test_accepts_a_set_as_well_as_a_list(self): # Callers pass whatever `effective_enabled_tools` happens to be. assert injected_tools_are_key_described( {"create_artifact"}, has_memory_binding=False ) - assert not injected_tools_are_key_described( + assert injected_tools_are_key_described( frozenset({"analyze_spreadsheet"}), has_memory_binding=False ) + assert not injected_tools_are_key_described( + frozenset({"analyze_spreadsheet"}), has_memory_binding=True + ) diff --git a/backend/tests/shared/test_kb_backend_parity.py b/backend/tests/shared/test_kb_backend_parity.py index 5f60be192..6d7f5c59a 100644 --- a/backend/tests/shared/test_kb_backend_parity.py +++ b/backend/tests/shared/test_kb_backend_parity.py @@ -58,6 +58,21 @@ CITATION_EXCERPT_CHARS = 500 +@pytest.fixture(autouse=True) +def _no_live_cloudwatch(): + """Never publish metrics from a test. + + ``kb_backend.metrics._publish`` builds ``boto3.client("cloudwatch")`` and + its ``except`` is documented as "observability must not break retrieval" — + so a live call failed silently and the test still passed. See the off-box + socket guard in conftest. + """ + from unittest.mock import patch as _patch + + with _patch("apis.shared.kb_backend.metrics._publish"): + yield + + class FakeManagedBackend: """Protocol-conforming stand-in for the managed backend (task 8.3). diff --git a/backend/tests/shared/test_message_feedback.py b/backend/tests/shared/test_message_feedback.py new file mode 100644 index 000000000..2cd9cad30 --- /dev/null +++ b/backend/tests/shared/test_message_feedback.py @@ -0,0 +1,257 @@ +"""The ``F#`` message-feedback row family (``apis.shared.sessions.feedback``) +against a real (moto) sessions-metadata table. + +Pins the key shape that makes the admin join a one-key lookup, replace-not- +append semantics, the session rollups' deltas, ownership, the content-free +guard, and the read-side merge into the messages-list metadata index. +""" + +from __future__ import annotations + +import boto3 +import pytest +from boto3.dynamodb.conditions import Key + +from apis.shared.sessions import feedback as fb +from apis.shared.sessions import metadata as md + +SESSION = "sess-fb-1" +OWNER = "user-owner" +OTHER = "user-other" + + +def _table(): + return boto3.resource("dynamodb", region_name="us-east-1").Table("test-sessions-metadata") + + +def _seed_session(user_id=OWNER, session_id=SESSION): + _table().put_item(Item={ + "PK": f"USER#{user_id}", + "SK": f"S#{session_id}", + "GSI_PK": f"SESSION#{session_id}", + "GSI_SK": "META", + "sessionId": session_id, + "userId": user_id, + "status": "active", + "messageCount": 4, + }) + + +def _session_row(user_id=OWNER, session_id=SESSION): + return _table().get_item(Key={"PK": f"USER#{user_id}", "SK": f"S#{session_id}"}).get("Item") or {} + + +def _feedback_items(session_id=SESSION): + resp = _table().query( + IndexName="SessionLookupIndex", + KeyConditionExpression=Key("GSI_PK").eq(f"SESSION#{session_id}") & Key("GSI_SK").begins_with("F#"), + ) + return resp.get("Items", []) + + +@pytest.fixture() +def table(sessions_metadata_table, monkeypatch): + monkeypatch.delenv("COST_DIAGNOSTICS_ENABLED", raising=False) + _seed_session() + return _table() + + +@pytest.mark.asyncio +async def test_put_writes_one_row_keyed_beside_the_cost_row(table): + result = await fb.put_message_feedback(SESSION, OWNER, 3, 1) + assert result.value == 1 and result.reason is None and result.updated_at + + items = _feedback_items() + assert len(items) == 1 + row = items[0] + assert row["PK"] == f"USER#{OWNER}" + assert row["SK"] == f"F#{SESSION}#3" + assert row["GSI_PK"] == f"SESSION#{SESSION}" and row["GSI_SK"] == "F#3" + assert int(row["messageId"]) == 3 and row["sessionId"] == SESSION + assert int(row["value"]) == 1 + assert "reason" not in row + assert "ttl" in row + # Content-free: nothing on the row but ids, a number, a timestamp and keys. + assert row["signal"] == "explicit" + assert set(row) <= {"PK", "SK", "GSI_PK", "GSI_SK", "GSI1PK", "GSI1SK", "sessionId", "messageId", "userId", "value", "signal", "retryMessageId", "updatedAt", "ttl"} + + +@pytest.mark.asyncio +async def test_second_thumb_replaces_the_first_and_rollups_follow(table): + await fb.put_message_feedback(SESSION, OWNER, 3, 1) + assert _session_row()["thumbsUp"] == 1 and _session_row()["thumbsDown"] == 0 + + result = await fb.put_message_feedback(SESSION, OWNER, 3, -1, reason="instructions") + assert result.value == -1 and result.reason == "instructions" + + items = _feedback_items() + assert len(items) == 1, "a second click replaces, never appends" + assert int(items[0]["value"]) == -1 and items[0]["reason"] == "instructions" + row = _session_row() + assert row["thumbsUp"] == 0 and row["thumbsDown"] == 1 + + # Same value again: no rollup movement, reason updates in place. + await fb.put_message_feedback(SESSION, OWNER, 3, -1, reason="wrong") + row = _session_row() + assert row["thumbsUp"] == 0 and row["thumbsDown"] == 1 + assert _feedback_items()[0]["reason"] == "wrong" + + +@pytest.mark.asyncio +async def test_retry_link_is_set_once_and_kept_across_later_thumbs(table): + """Retry-with-correction (response-feedback §7): the row links the user + message the correction was sent as, and a later re-thumb without the + field must not drop it.""" + first = await fb.put_message_feedback(SESSION, OWNER, 3, -1, reason="wrong") + assert first.retry_message_id is None + linked = await fb.put_message_feedback(SESSION, OWNER, 3, -1, reason="wrong", retry_message_id=4) + assert linked.retry_message_id == 4 + assert int(_feedback_items()[0]["retryMessageId"]) == 4 + + # Re-thumb (say, the retry was better and they flip to up): link stays. + again = await fb.put_message_feedback(SESSION, OWNER, 3, 1) + assert again.value == 1 and again.retry_message_id == 4 + row = _feedback_items()[0] + assert int(row["retryMessageId"]) == 4 and "reason" not in row + assert _session_row()["thumbsUp"] == 1 and _session_row()["thumbsDown"] == 0 + + with pytest.raises(ValueError): + await fb.put_message_feedback(SESSION, OWNER, 3, -1, retry_message_id=-1) + + +@pytest.mark.asyncio +async def test_delete_removes_the_row_and_decrements(table): + await fb.put_message_feedback(SESSION, OWNER, 1, -1, reason="tool_failed") + await fb.put_message_feedback(SESSION, OWNER, 3, 1) + assert await fb.delete_message_feedback(SESSION, OWNER, 1) is True + assert [int(i["messageId"]) for i in _feedback_items()] == [3] + row = _session_row() + assert row["thumbsUp"] == 1 and row["thumbsDown"] == 0 + # Deleting what isn't there is a no-op, not an error. + assert await fb.delete_message_feedback(SESSION, OWNER, 1) is False + assert _session_row()["thumbsDown"] == 0 + + +@pytest.mark.asyncio +async def test_rollups_are_not_written_while_diagnostics_are_off(table, monkeypatch): + monkeypatch.setenv("COST_DIAGNOSTICS_ENABLED", "false") + await fb.put_message_feedback(SESSION, OWNER, 3, 1) + row = _session_row() + assert "thumbsUp" not in row and "thumbsDown" not in row, "absent reads 'not tracked', never 0" + assert len(_feedback_items()) == 1, "the row itself is still written" + + +@pytest.mark.asyncio +async def test_another_users_session_is_not_found(table): + with pytest.raises(fb.SessionNotOwned): + await fb.put_message_feedback(SESSION, OTHER, 3, 1) + with pytest.raises(fb.SessionNotOwned): + await fb.delete_message_feedback(SESSION, OTHER, 3) + assert _feedback_items() == [] + + +@pytest.mark.asyncio +async def test_storage_refuses_free_text_and_bad_values(table): + with pytest.raises(ValueError): + await fb.put_message_feedback(SESSION, OWNER, 3, 1, reason="it was rude to me") + with pytest.raises(ValueError): + await fb.put_message_feedback(SESSION, OWNER, 3, 2) + assert _feedback_items() == [] + + +@pytest.mark.asyncio +async def test_preview_sessions_echo_without_persisting(table): + result = await fb.put_message_feedback("preview-abc", OWNER, 0, 1) + assert result.value == 1 + assert _feedback_items("preview-abc") == [] + + +@pytest.mark.asyncio +async def test_query_filters_by_user_unless_admin(table): + _seed_session(OTHER, "sess-shared") + _seed_session(OWNER, "sess-shared") + await fb.put_message_feedback("sess-shared", OWNER, 1, 1) + await fb.put_message_feedback("sess-shared", OTHER, 1, -1) + + mine = fb.query_session_feedback(table, "sess-shared", OWNER) + assert {k: v.value for k, v in mine.items()} == {"1": 1} + everyone = fb.query_session_feedback(table, "sess-shared", None) + assert len(everyone) == 1 # keyed by message id; the admin reader returns rows, this map is per message + + +@pytest.mark.asyncio +async def test_implicit_signal_rows_share_the_family_but_never_read_as_thumbs(table): + """Spec §10: implicit signals land in the same F# family under + signal="implicit"; every thumb reader skips them and never sums the two.""" + await fb.put_message_feedback(SESSION, OWNER, 3, 1) + table.put_item(Item={ + "PK": f"USER#{OWNER}", "SK": f"F#{SESSION}#5", "GSI_PK": f"SESSION#{SESSION}", "GSI_SK": "F#5", + "sessionId": SESSION, "messageId": 5, "userId": OWNER, "value": 1, "signal": "implicit", + "updatedAt": "2026-09-16T00:00:00Z", + }) + assert set(fb.query_session_feedback(table, SESSION, OWNER)) == {"3"} + index = await md.get_all_message_metadata(SESSION, OWNER) + assert "5" not in index + # A row written before the discriminator existed is explicit. + assert fb.is_explicit({"value": 1}) and not fb.is_explicit({"value": 1, "signal": "implicit"}) + + +@pytest.mark.asyncio +async def test_implicit_signals_count_under_their_own_key_beside_the_thumb(table): + """Spec §10: copy / continue rows share the F# family, keyed per kind so + they never collide with the thumb, ADD a count, and are invisible to + every thumb reader.""" + await fb.put_message_feedback(SESSION, OWNER, 3, -1, reason="wrong") + await fb.record_implicit_signal(SESSION, OWNER, 3, "copy") + await fb.record_implicit_signal(SESSION, OWNER, 3, "copy") + await fb.record_implicit_signal(SESSION, OWNER, 3, "continue") + + items = {i["SK"]: i for i in _feedback_items()} + assert set(items) == {f"F#{SESSION}#3", f"F#{SESSION}#3#copy", f"F#{SESSION}#3#continue"} + copy = items[f"F#{SESSION}#3#copy"] + assert copy["signal"] == "implicit" and copy["kind"] == "copy" and int(copy["count"]) == 2 + assert copy["GSI_SK"] == "F#3#copy" and int(copy["messageId"]) == 3 and "value" not in copy + assert int(items[f"F#{SESSION}#3"]["value"]) == -1, "the thumb is untouched" + + # Thumb readers see only the thumb; the session rollups did not move. + assert {k: v.value for k, v in fb.query_session_feedback(table, SESSION, OWNER).items()} == {"3": -1} + assert _session_row()["thumbsDown"] == 1 and _session_row()["thumbsUp"] == 0 + + with pytest.raises(ValueError): + await fb.record_implicit_signal(SESSION, OWNER, 3, "abandon") + with pytest.raises(fb.SessionNotOwned): + await fb.record_implicit_signal(SESSION, OTHER, 3, "copy") + await fb.record_implicit_signal("preview-x", OWNER, 0, "copy") + assert _feedback_items("preview-x") == [] + + +@pytest.mark.asyncio +async def test_messages_list_metadata_index_carries_feedback(table, monkeypatch): + """The read path the SPA uses on reload: F# rows merge into the metadata + index by message id, beside (or in place of) the cost record.""" + table.put_item(Item={ + "PK": f"USER#{OWNER}", "SK": "C#2026-09-16T00:00:00Z#u1", + "GSI_PK": f"SESSION#{SESSION}", "GSI_SK": "C#2026-09-16T00:00:00Z", + "sessionId": SESSION, "messageId": 3, "userId": OWNER, + "timestamp": "2026-09-16T00:00:00Z", "cost": {"total": 1}, + }) + await fb.put_message_feedback(SESSION, OWNER, 3, -1, reason="wrong") + await fb.put_message_feedback(SESSION, OWNER, 5, 1) + + index = await md.get_all_message_metadata(SESSION, OWNER) + assert index["3"]["cost"] == {"total": 1} + assert index["3"]["feedback"] == {"value": -1, "reason": "wrong", "updatedAt": index["3"]["feedback"]["updatedAt"]} + assert index["5"] == {"feedback": {"value": 1, "updatedAt": index["5"]["feedback"]["updatedAt"]}} + + monkeypatch.setenv("RESPONSE_FEEDBACK_ENABLED", "false") + index = await md.get_all_message_metadata(SESSION, OWNER) + assert "feedback" not in index["3"] and "5" not in index + + +@pytest.mark.asyncio +async def test_message_metadata_model_round_trips_feedback(table): + from apis.shared.sessions.models import MessageMetadata + + meta = MessageMetadata(**{"cost": 0.1, "feedback": {"value": 1, "updatedAt": "2026-09-16T00:00:00Z"}}) + assert meta.feedback is not None and meta.feedback.value == 1 + assert meta.model_dump(by_alias=True, exclude_none=True)["feedback"] == {"value": 1, "updatedAt": "2026-09-16T00:00:00Z"} diff --git a/backend/tests/shared/test_rbac_cache_service.py b/backend/tests/shared/test_rbac_cache_service.py index 7444aa09e..b71ecdabd 100644 --- a/backend/tests/shared/test_rbac_cache_service.py +++ b/backend/tests/shared/test_rbac_cache_service.py @@ -5,6 +5,27 @@ from unittest.mock import AsyncMock, MagicMock +@pytest.fixture(autouse=True) +def _no_live_tool_catalog(): + """Stub the tool catalog the RBAC service consults. + + ``can_access_tool`` falls through to ``tools.freshness._get_snapshot`` -> + ``get_tool_catalog_repository().list_tools()``, which builds a real + DynamoDB client. These tests mock the *role* repository only, so that call + used to leave the box; its ``except`` swallowed the failure and the + assertions passed anyway. See the off-box socket guard in conftest. + """ + from unittest.mock import AsyncMock as _AsyncMock, MagicMock as _MagicMock, patch as _patch + + repo = _MagicMock() + repo.list_tools = _AsyncMock(return_value=[]) + with _patch( + "apis.shared.tools.repository.get_tool_catalog_repository", + return_value=repo, + ): + yield + + class TestAppRoleCache: @pytest.fixture(autouse=True) def _setup(self): diff --git a/backend/tests/shared/test_session_meta_snapshot.py b/backend/tests/shared/test_session_meta_snapshot.py new file mode 100644 index 000000000..8c4a45c36 --- /dev/null +++ b/backend/tests/shared/test_session_meta_snapshot.py @@ -0,0 +1,350 @@ +"""One read of the session META row, shared across the preamble (PR-2). + +Measured on dev 2026-09-19 (docs/specs/turn-latency-preamble.md): a warm turn +read this one item EIGHT times before the first model call, and a GSI query +from an AgentCore Runtime container costs ~53ms — so those reads were ~445ms +of a ~455ms stage. + +What these tests pin is not the speed (a moto table is not us-west-2) but the +two things that make the speed safe to take: + +1. the snapshot answers exactly what the separate reads answered, including + the cross-user fork case the ownership probe exists for, and +2. `snapshot=None` leaves every other caller on its original per-call read, + because an implicit cache of session state is a bug this repo has shipped + twice (#741, #751). +""" + +import pytest + +SESSION = "sess-1" +OWNER = "owner-1" +OTHER = "other-1" + + +def _put_meta(table, session_id=SESSION, user_id=OWNER, **extra): + item = { + "PK": f"USER#{user_id}", + "SK": f"S#{session_id}", + "GSI_PK": f"SESSION#{session_id}", + "GSI_SK": "META", + "sessionId": session_id, + "userId": user_id, + "title": "Test Session", + "status": "active", + "createdAt": "2026-01-01T00:00:00Z", + "lastMessageAt": "2026-01-01T00:00:00Z", + "messageCount": 0, + "starred": False, + "tags": [], + } + item.update(extra) + table.put_item(Item=item) + return item + + +class TestLoadSessionMeta: + @pytest.mark.asyncio + async def test_returns_the_row_for_its_owner(self, sessions_metadata_table): + from apis.shared.sessions.metadata import load_session_meta + + _put_meta(sessions_metadata_table, title="Mine") + + snap = await load_session_meta(SESSION, OWNER) + + assert snap.owned_by_other is False + assert snap.row is not None + assert snap.row["title"] == "Mine" + + @pytest.mark.asyncio + async def test_absent_session_is_row_none_and_not_owned_by_other( + self, sessions_metadata_table + ): + """`row is None` must mean "no row", distinctly from "someone else's" — + collapsing the two is the ambiguity `session_owned_by_other_user` was + written to resolve.""" + from apis.shared.sessions.metadata import load_session_meta + + snap = await load_session_meta("nope", OWNER) + + assert snap.row is None + assert snap.owned_by_other is False + + @pytest.mark.asyncio + async def test_someone_elses_session_is_owned_by_other(self, sessions_metadata_table): + from apis.shared.sessions.metadata import load_session_meta + + _put_meta(sessions_metadata_table, user_id=OTHER) + + snap = await load_session_meta(SESSION, OWNER) + + assert snap.owned_by_other is True + assert snap.row is None + + @pytest.mark.asyncio + async def test_a_forked_session_still_resolves_to_the_caller_s_own_row( + self, sessions_metadata_table + ): + """Prod 2026-08-31 left two META rows sharing GSI_PK/GSI_SK, returned in + an unspecified order. The owner must still get THEIR row, not a coin + flip — the same item-scan `_get_session_by_gsi` does.""" + from apis.shared.sessions.metadata import load_session_meta + + _put_meta(sessions_metadata_table, user_id=OTHER, title="Fork") + _put_meta(sessions_metadata_table, user_id=OWNER, title="Mine") + + snap = await load_session_meta(SESSION, OWNER) + + assert snap.owned_by_other is False + assert snap.row["title"] == "Mine" + + @pytest.mark.asyncio + async def test_agrees_with_the_two_reads_it_replaces(self, sessions_metadata_table): + """The consolidation must be behaviour-preserving, so assert it against + the originals rather than against a remembered description of them.""" + from apis.shared.sessions.metadata import ( + _get_session_by_gsi, + load_session_meta, + session_owned_by_other_user, + ) + import boto3 + + _put_meta(sessions_metadata_table, user_id=OTHER) + _put_meta(sessions_metadata_table, user_id=OWNER) + table = boto3.resource("dynamodb").Table("test-sessions-metadata") + + snap = await load_session_meta(SESSION, OWNER) + + assert snap.owned_by_other == await session_owned_by_other_user(SESSION, OWNER) + assert snap.row == await _get_session_by_gsi(SESSION, OWNER, table) + + @pytest.mark.asyncio + async def test_a_read_failure_degrades_to_nothing_known_nothing_blocked( + self, monkeypatch, sessions_metadata_table + ): + """Fail-open in both fields, matching what the two reads already did: + the ownership probe returned False and the row read returned None.""" + import boto3 + + from apis.shared.sessions import metadata + + def _boom(*a, **kw): + raise RuntimeError("dynamodb is down") + + # `boto3` is imported inside the function, so the module attribute is + # what the call resolves against. + monkeypatch.setattr(boto3, "resource", _boom) + + snap = await metadata.load_session_meta(SESSION, OWNER) + + assert snap.row is None + assert snap.owned_by_other is False + + +class TestSnapshotThreading: + """`snapshot=` must be the ONLY thing that skips the read.""" + + @pytest.mark.asyncio + async def test_helpers_use_the_snapshot_instead_of_reading( + self, monkeypatch, sessions_metadata_table + ): + from apis.shared.sessions import metadata + + _put_meta(sessions_metadata_table, pausedTurn={"foo": "bar"}) + snap = await metadata.load_session_meta(SESSION, OWNER) + + reads = [] + + async def _counted(session_id, user_id, table): + reads.append(session_id) + return None + + monkeypatch.setattr(metadata, "_get_session_by_gsi", _counted) + + await metadata.clear_paused_turn(SESSION, OWNER, snapshot=snap) + + assert reads == [], "a supplied snapshot must not be re-read" + + @pytest.mark.asyncio + async def test_omitting_the_snapshot_still_reads( + self, monkeypatch, sessions_metadata_table + ): + """The default path is what every non-preamble caller gets, and it must + keep reading — a snapshot that leaked into those callers would be the + stale-session-state bug CLAUDE.md names.""" + from apis.shared.sessions import metadata + + _put_meta(sessions_metadata_table) + reads = [] + original = metadata._get_session_by_gsi + + async def _counted(session_id, user_id, table): + reads.append(session_id) + return await original(session_id, user_id, table) + + monkeypatch.setattr(metadata, "_get_session_by_gsi", _counted) + + await metadata.clear_paused_turn(SESSION, OWNER) + + assert reads == [SESSION] + + @pytest.mark.asyncio + async def test_a_snapshot_with_no_row_is_an_answer_not_a_cache_miss( + self, monkeypatch, sessions_metadata_table + ): + """A brand-new session has `row=None`. If that were treated as "nothing + prefetched" the helper would fall back to a read, and the first turn of + every conversation would silently keep the old cost.""" + from apis.shared.sessions import metadata + + snap = await metadata.load_session_meta("brand-new", OWNER) + assert snap.row is None + + reads = [] + + async def _counted(session_id, user_id, table): + reads.append(session_id) + return None + + monkeypatch.setattr(metadata, "_get_session_by_gsi", _counted) + + await metadata.clear_paused_turn("brand-new", OWNER, snapshot=snap) + + assert reads == [] + + +class TestBehaviourPreserved: + """The clears must still clear, and still no-op when there is nothing to do.""" + + @pytest.mark.asyncio + async def test_clear_paused_turn_removes_the_snapshot_attribute( + self, sessions_metadata_table + ): + from apis.shared.sessions import metadata + + _put_meta(sessions_metadata_table, pausedTurn={"interruptId": "i-1"}) + snap = await metadata.load_session_meta(SESSION, OWNER) + + await metadata.clear_paused_turn(SESSION, OWNER, snapshot=snap) + + row = sessions_metadata_table.get_item( + Key={"PK": f"USER#{OWNER}", "SK": f"S#{SESSION}"} + )["Item"] + assert "pausedTurn" not in row + + @pytest.mark.asyncio + async def test_clear_interrupted_turn_still_pops_the_reason( + self, sessions_metadata_table + ): + """The return value drives the one-turn interruption note in the prompt, + so a snapshot that broke it would silently change what the model sees.""" + from apis.shared.sessions import metadata + + _put_meta( + sessions_metadata_table, + lastTurnInterrupted=True, + lastTurnInterruptReason="user_stopped", + ) + snap = await metadata.load_session_meta(SESSION, OWNER) + + reason = await metadata.clear_interrupted_turn(SESSION, OWNER, snapshot=snap) + + assert reason == "user_stopped" + row = sessions_metadata_table.get_item( + Key={"PK": f"USER#{OWNER}", "SK": f"S#{SESSION}"} + )["Item"] + assert "lastTurnInterrupted" not in row + + @pytest.mark.asyncio + async def test_pop_pending_attachments_still_returns_and_clears( + self, sessions_metadata_table + ): + from datetime import datetime, timezone + + from apis.shared.sessions import metadata + + _put_meta( + sessions_metadata_table, + pendingAttachmentUploadIds=["u-1", "u-2"], + pendingAttachmentsAt=datetime.now(timezone.utc).isoformat(), + ) + snap = await metadata.load_session_meta(SESSION, OWNER) + + got = await metadata.pop_pending_attachments(SESSION, OWNER, snapshot=snap) + + assert got == ["u-1", "u-2"] + row = sessions_metadata_table.get_item( + Key={"PK": f"USER#{OWNER}", "SK": f"S#{SESSION}"} + )["Item"] + assert "pendingAttachmentUploadIds" not in row + + @pytest.mark.asyncio + async def test_a_clean_row_writes_nothing(self, sessions_metadata_table): + """The short-circuit is what makes the warm path cheap: four of the five + clears find nothing and must not spend a write proving it.""" + from apis.shared.sessions import metadata + + _put_meta(sessions_metadata_table) + snap = await metadata.load_session_meta(SESSION, OWNER) + before = sessions_metadata_table.get_item( + Key={"PK": f"USER#{OWNER}", "SK": f"S#{SESSION}"} + )["Item"] + + await metadata.clear_paused_turn(SESSION, OWNER, snapshot=snap) + await metadata.clear_pending_interrupts(SESSION, OWNER, snapshot=snap) + await metadata.clear_truncated_turn(SESSION, OWNER, snapshot=snap) + await metadata.clear_interrupted_turn(SESSION, OWNER, snapshot=snap) + + after = sessions_metadata_table.get_item( + Key={"PK": f"USER#{OWNER}", "SK": f"S#{SESSION}"} + )["Item"] + assert before == after + + @pytest.mark.asyncio + async def test_ensure_metadata_skips_creation_when_the_snapshot_has_a_row( + self, sessions_metadata_table + ): + from apis.shared.sessions import metadata + + _put_meta(sessions_metadata_table) + snap = await metadata.load_session_meta(SESSION, OWNER) + + assert await metadata.ensure_session_metadata_exists( + SESSION, OWNER, snapshot=snap + ) is False + + @pytest.mark.asyncio + async def test_ensure_metadata_creates_for_a_genuinely_new_session( + self, sessions_metadata_table + ): + """`True` is the first-turn signal that fires title generation, so the + snapshot path has to keep returning it.""" + from apis.shared.sessions import metadata + + snap = await metadata.load_session_meta("fresh", OWNER) + + assert await metadata.ensure_session_metadata_exists( + "fresh", OWNER, snapshot=snap + ) is True + + @pytest.mark.asyncio + async def test_ensure_metadata_refuses_to_fork_a_session_it_does_not_own( + self, sessions_metadata_table + ): + """The backstop against the prod 2026-08-31 fork. It previously cost a + SECOND query; the snapshot already carries the answer, and losing the + guard while making it faster would be the worst possible trade.""" + from apis.shared.sessions import metadata + + _put_meta(sessions_metadata_table, user_id=OTHER) + snap = await metadata.load_session_meta(SESSION, OWNER) + + created = await metadata.ensure_session_metadata_exists( + SESSION, OWNER, snapshot=snap + ) + + assert created is False + resp = sessions_metadata_table.get_item( + Key={"PK": f"USER#{OWNER}", "SK": f"S#{SESSION}"} + ) + assert "Item" not in resp, "a second row would fork the session id" diff --git a/backend/tests/shared/test_skills_access.py b/backend/tests/shared/test_skills_access.py index 390e54b70..761d0c450 100644 --- a/backend/tests/shared/test_skills_access.py +++ b/backend/tests/shared/test_skills_access.py @@ -24,6 +24,12 @@ def _user() -> User: class TestResolveAccessibleSkillIds: + """Every case here also enters ``_owned()``: ``resolve_accessible_skill_ids`` + calls ``resolve_owned_skill_ids`` after the RBAC lookup, and that builds a + real skill-catalog repository. Its ``except`` swallowed the resulting AWS + failure, so these assertions passed while the call went out to real + DynamoDB — see the off-box socket guard in ``tests/conftest.py``.""" + async def test_plain_grants_pass_through(self): role_service = MagicMock() role_service.get_accessible_skills = AsyncMock( @@ -32,7 +38,7 @@ async def test_plain_grants_pass_through(self): with patch( "apis.shared.rbac.service.get_app_role_service", return_value=role_service, - ): + ), _owned(): result = await resolve_accessible_skill_ids(_user()) assert result == ["web_research", "pdf_workflows"] @@ -45,7 +51,7 @@ async def test_wildcard_expands_to_all_known_skills_sorted(self): ), patch( "apis.shared.skills.freshness.get_all_skill_ids", AsyncMock(return_value=frozenset({"zeta", "alpha"})), - ): + ), _owned(): result = await resolve_accessible_skill_ids(_user()) assert result == ["alpha", "zeta"] @@ -53,7 +59,7 @@ async def test_failure_degrades_to_no_skills(self): with patch( "apis.shared.rbac.service.get_app_role_service", side_effect=RuntimeError("rbac unavailable"), - ): + ), _owned(): result = await resolve_accessible_skill_ids(_user()) assert result == [] diff --git a/backend/tests/supply_chain/test_image_filesystem_hardening.py b/backend/tests/supply_chain/test_image_filesystem_hardening.py new file mode 100644 index 000000000..878bed909 --- /dev/null +++ b/backend/tests/supply_chain/test_image_filesystem_hardening.py @@ -0,0 +1,98 @@ +"""The service images must ship their code read-only to the runtime user. + +Two properties, both easy to lose in a one-line Dockerfile edit: + +1. No ``chown -R`` over /app. Beyond handing the runtime user write access to + the code it executes, a recursive chown rewrites every file it touches into + a fresh layer — /app ends up shipped twice. On app-api that was a 454MB + duplicate. + +2. No `COPY --chown` of the code either. The runtime user reads and executes + /app/src and /app/.venv; it does not own them, so a process compromised at + runtime cannot rewrite its own source or drop a backdoor into site-packages. + +There is deliberately no writable directory under /app at all. The services +write nothing to local disk — generated output goes to S3 — so the image does +not need one. backend/tests/architecture/test_source_tree_not_written.py is the +code-side half of the same guarantee. +""" + +import re +from pathlib import Path + +import pytest + +_BACKEND = Path(__file__).resolve().parent.parent.parent + +# The two long-lived service images. The Lambda images are a different shape: +# they have no venv copy and AWS controls /var/task ownership. +_SERVICE_DOCKERFILES = ["Dockerfile.app-api", "Dockerfile.inference-api"] + + +def _dockerfile(name: str) -> str: + return (_BACKEND / name).read_text(encoding="utf-8") + + +def _instructions(text: str): + """Yield logical Dockerfile instructions, joining backslash continuations + and dropping comments.""" + joined = re.sub(r"\\\s*\n", " ", text) + for line in joined.splitlines(): + stripped = line.strip() + if stripped and not stripped.startswith("#"): + yield stripped + + +@pytest.mark.parametrize("name", _SERVICE_DOCKERFILES) +def test_no_recursive_chown_of_app(name): + """A `chown -R` over /app duplicates the whole tree into a new layer.""" + offenders = [ + instr + for instr in _instructions(_dockerfile(name)) + if re.search(r"\bchown\b", instr) + and re.search(r"-R|--recursive", instr) + and "/app" in instr + ] + assert not offenders, ( + f"{name} recursively chowns /app, which ships the tree twice and hands the " + f"runtime user write access to its own code:\n " + "\n ".join(offenders) + ) + + +@pytest.mark.parametrize("name", _SERVICE_DOCKERFILES) +def test_code_is_not_copied_with_chown(name): + """/app/src and /app/.venv must land root-owned. + + `COPY --chown=` is cheaper than a recursive chown but still + grants write access to the code, and nothing needs it: the services write + nothing to local disk. + """ + offenders = [ + instr + for instr in _instructions(_dockerfile(name)) + if instr.upper().startswith("COPY") + and "--chown" in instr + and ("/app/src" in instr or "/app/.venv" in instr) + ] + assert not offenders, ( + f"{name} copies code with --chown, so the runtime user can rewrite it:\n " + + "\n ".join(offenders) + ) + + +@pytest.mark.parametrize("name", _SERVICE_DOCKERFILES) +def test_no_dead_toplevel_asset_dirs(name): + """/app/output, /app/uploads, /app/generated_images were never used. + + base_dir has always resolved under /app/src, so nothing read or wrote + these. They are gone; this keeps them gone. + """ + offenders = [ + instr + for instr in _instructions(_dockerfile(name)) + if re.search(r"/app/(output|uploads|generated_images)\b", instr) + ] + assert not offenders, ( + f"{name} recreates the dead top-level asset dirs; runtime paths resolve under " + f"and nothing writes to local disk any more:\n " + "\n ".join(offenders) + ) diff --git a/backend/tests/supply_chain/test_platform_deploy_triggers.py b/backend/tests/supply_chain/test_platform_deploy_triggers.py new file mode 100644 index 000000000..3ce1954e8 --- /dev/null +++ b/backend/tests/supply_chain/test_platform_deploy_triggers.py @@ -0,0 +1,98 @@ +"""Every input the Platform Stack deploy consumes must also trigger it. + +A path the workflow reads but does not watch is the worst kind of deploy bug: +the push is green, no workflow fails, and the old artifact silently stays live. +That happened with the browser sign-in viewer — `infrastructure/assets/**` is +staged into the mcp-sandbox bucket at synth, but nothing in `paths:` matched +it, so a fix to it deployed nowhere while CI reported success. + +These tests derive the expectation from what the workflow and the CDK tree +actually do, rather than restating the `paths:` list back at itself. +""" + +from pathlib import Path + +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[3] +PLATFORM_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "platform.yml" + + +def _trigger_paths() -> list[str]: + # `on` is parsed by PyYAML as the boolean True (YAML 1.1), so accept either. + config = yaml.safe_load(PLATFORM_WORKFLOW.read_text()) + triggers = config.get("on", config.get(True)) + return list(triggers["push"]["paths"]) + + +def _covers(patterns: list[str], repo_relative: str) -> bool: + """True when some glob in `patterns` would match `repo_relative`. + + `repo_relative` may name a file or a directory. A `dir/**` pattern covers + the directory itself as well as anything beneath it, which `Path.match` + alone does not give us. + """ + for pattern in patterns: + if Path(repo_relative).match(pattern): + return True + prefix = pattern.removesuffix("**").rstrip("/") + if prefix and (repo_relative == prefix or repo_relative.startswith(prefix + "/")): + return True + return False + + +def test_scripts_the_workflow_runs_also_trigger_it() -> None: + """A script invoked by the deploy job must be watched by it. + + `fetch-dcv-sdk.sh` pins a version, a SHA256 and a GPG fingerprint. Bumping + any of them is precisely a change that must redeploy. + """ + body = PLATFORM_WORKFLOW.read_text() + patterns = _trigger_paths() + + invoked = { + line.split("bash ", 1)[1].strip() + for line in body.splitlines() + if "bash scripts/" in line + } + assert invoked, "expected the deploy job to invoke at least one script" + + unwatched = sorted(s for s in invoked if not _covers(patterns, s)) + assert not unwatched, ( + "platform.yml runs these scripts but would not redeploy when they " + f"change: {unwatched}" + ) + + +def test_cdk_staged_asset_directories_trigger_a_deploy() -> None: + """Directories CDK stages at synth must be watched. + + Found by walking the constructs for staged directories rather than naming + them here, so a new staged directory is covered the day it is added. + """ + patterns = _trigger_paths() + constructs = REPO_ROOT / "infrastructure" / "lib" / "constructs" + + staged: set[str] = set() + for source in constructs.rglob("*.ts"): + text = source.read_text() + for marker in ("'assets'", '"assets"'): + if marker in text: + staged.add("infrastructure/assets") + + assert staged, "expected to find at least one staged asset directory" + + unwatched = sorted(d for d in staged if not _covers(patterns, d)) + assert not unwatched, ( + "CDK stages these directories at synth, but a change to them would " + f"not trigger a Platform Stack deploy: {unwatched}" + ) + + +def test_the_viewer_asset_specifically_triggers_a_deploy() -> None: + """Regression guard for the exact file that was stranded.""" + viewer = "infrastructure/assets/mcp-sandbox/live-view.js" + assert (REPO_ROOT / viewer).exists(), f"{viewer} moved; update this guard" + assert _covers(_trigger_paths(), viewer), ( + f"{viewer} is deployed by platform.yml but would not trigger it" + ) diff --git a/backend/tests/test_seed_matches_tool_catalog.py b/backend/tests/test_seed_matches_tool_catalog.py new file mode 100644 index 000000000..05a0376a4 --- /dev/null +++ b/backend/tests/test_seed_matches_tool_catalog.py @@ -0,0 +1,76 @@ +"""The bootstrap seeder and the Python tool catalog must describe the same tools. + +There are two independent lists of tools in this repo and nothing syncs them: + + * ``TOOL_CATALOG`` in ``agents/main_agent/tools/tool_catalog.py`` — metadata the + agent side reads. + * ``DEFAULT_TOOLS`` in ``scripts/seed_bootstrap_data.py`` — the ONLY writer of + the ``TOOL#`` rows in DynamoDB that RBAC grants and the tool picker read. + +A tool in the catalog but not the seeder has no row on a fresh install, so no role +can be granted it and no user can enable it — silently, because an absent row reads +as "not in the catalog", not as an error. That is exactly what happened to +``list_spreadsheets`` / ``analyze_spreadsheet``: they were catalogued but never +seeded, and the live environments only have them because the rows were created by +hand. A fork got no spreadsheet analysis at all. + +The reverse direction is legitimate and is NOT asserted: several seeded ids +(``create_artifact``, the document-authoring tools, ``workspace_files``) are +context-bound tools built per request, so they have a catalog ROW without a +``TOOL_CATALOG`` entry. Only catalog-without-row is a defect. + +Deliberately excluded: ids in ``DOCUMENT_TOOL_IDS``. ``document_read`` is gated on +the session having a readable attachment rather than on ``enabled_tools``, so it +has no catalog entry, no RBAC grant and no picker toggle by design +(``apis/shared/tools/injected.py``). +""" + +import os +import re +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "scripts")) + +from agents.main_agent.tools.tool_catalog import TOOL_CATALOG # noqa: E402 +from apis.shared.tools.injected import DOCUMENT_TOOL_IDS # noqa: E402 + +SEED_SCRIPT = os.path.join( + os.path.dirname(__file__), "..", "scripts", "seed_bootstrap_data.py" +) + + +def _seeded_tool_ids() -> set[str]: + """Every ``toolId`` the seeder writes, read from source. + + Parsed rather than imported: the module builds boto3 sessions at import in + some paths, and the list is a plain literal, so a regex is both cheaper and + less likely to break for an unrelated reason. + """ + with open(SEED_SCRIPT, encoding="utf-8") as fh: + source = fh.read() + + return set(re.findall(r'"toolId":\s*"([A-Za-z0-9_]+)"', source)) + + +def test_every_catalogued_tool_has_a_seed_row() -> None: + catalogued = set(TOOL_CATALOG.keys()) - set(DOCUMENT_TOOL_IDS) + missing = catalogued - _seeded_tool_ids() + + assert not missing, ( + f"Tools in TOOL_CATALOG with no row in seed_bootstrap_data.py: " + f"{sorted(missing)}. A fresh deployment would list them nowhere, so no " + f"role could be granted them. Add a DEFAULT_TOOLS entry (and if the tool " + f"is context-bound, check apis/shared/tools/injected.py first — some ids " + f"are deliberately ungated and belong in DOCUMENT_TOOL_IDS instead)." + ) + + +def test_the_seeder_parses_at_all() -> None: + """Guard the regex above: if it stops matching, the test above passes vacuously.""" + seeded = _seeded_tool_ids() + + assert len(seeded) >= 10, ( + f"Only {len(seeded)} toolIds parsed out of the seeder — the literal's " + f"shape probably changed and the parity check above is now vacuous." + ) + assert "calculator" in seeded diff --git a/backend/tests/test_seed_system_admin_jwt.py b/backend/tests/test_seed_system_admin_jwt.py index c40d928b9..b4372e59d 100644 --- a/backend/tests/test_seed_system_admin_jwt.py +++ b/backend/tests/test_seed_system_admin_jwt.py @@ -246,6 +246,25 @@ def test_creates_default_tools(self, dynamodb_table): assert item["GSI1PK"] == "CATEGORY#browser" assert item["GSI1SK"] == "TOOL#browse_web" + # Verify request_user_login. enabledByDefault MUST stay False, and + # this is the strictest default in the seed: while a takeover is live + # the user has a fully interactive Chromium inside our AWS account with + # our egress. It is a separate tool_id precisely so it can be granted + # to named roles — RBAC has no sub-tool gate, so an action on + # browse_web would have shipped it to every user who can browse. + resp = dynamodb_table.get_item( + Key={"PK": "TOOL#request_user_login", "SK": "METADATA"} + ) + item = resp["Item"] + assert item["toolId"] == "request_user_login" + assert item["displayName"] == "Browser Sign-In" + assert item["category"] == "browser" + assert item["protocol"] == "local" + assert item["enabledByDefault"] is False + assert item["isPublic"] is False + assert item["GSI1PK"] == "CATEGORY#browser" + assert item["GSI1SK"] == "TOOL#request_user_login" + # Verify create_excel_spreadsheet (single toggle for the whole Excel toolset) resp = dynamodb_table.get_item( Key={"PK": "TOOL#create_excel_spreadsheet", "SK": "METADATA"} diff --git a/backend/uv.lock b/backend/uv.lock index a0c57cb55..4548c156d 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -15,7 +15,7 @@ constraints = [{ name = "mcp", specifier = "<2" }] [[package]] name = "agentcore-stack" -version = "1.22.0" +version = "1.23.0" source = { editable = "." } dependencies = [ { name = "aiofiles" }, @@ -28,6 +28,7 @@ dependencies = [ { name = "fastapi" }, { name = "httpx" }, { name = "idna" }, + { name = "openpyxl" }, { name = "pillow" }, { name = "pyasn1" }, { name = "pyjwt", extra = ["crypto"] }, @@ -115,6 +116,7 @@ requires-dist = [ { name = "mypy", marker = "extra == 'dev'", specifier = "==1.20.2" }, { name = "numpy", marker = "extra == 'dev'", specifier = "==2.2.6" }, { name = "openai", marker = "extra == 'agentcore'", specifier = "==2.32.0" }, + { name = "openpyxl", specifier = "==3.1.5" }, { name = "pandas", marker = "extra == 'dev'", specifier = "==2.3.3" }, { name = "pillow", specifier = "==12.3.0" }, { name = "pyasn1", specifier = "==0.6.4" }, @@ -1168,6 +1170,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/55/e2/2537ebcff11c1ee1ff17d8d0b6f4db75873e3b0fb32c2d4a2ee31ecb310a/docstring_parser-0.17.0-py3-none-any.whl", hash = "sha256:cf2569abd23dce8099b300f9b4fa8191e9582dda731fd533daf54c4551658708", size = 36896, upload-time = "2025-07-21T07:35:00.684Z" }, ] +[[package]] +name = "et-xmlfile" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d3/38/af70d7ab1ae9d4da450eeec1fa3918940a5fafb9055e934af8d6eb0c2313/et_xmlfile-2.0.0.tar.gz", hash = "sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54", size = 17234, upload-time = "2024-10-25T17:25:40.039Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/8b/5fe2cc11fee489817272089c4203e679c63b570a5aaeb18d852ae3cbba6a/et_xmlfile-2.0.0-py3-none-any.whl", hash = "sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa", size = 18059, upload-time = "2024-10-25T17:25:39.051Z" }, +] + [[package]] name = "exceptiongroup" version = "1.3.1" @@ -2522,6 +2533,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1e/c1/d6e64ccd0536bf616556f0cad2b6d94a8125f508d25cfd814b1d2db4e2f1/openai-2.32.0-py3-none-any.whl", hash = "sha256:4dcc9badeb4bf54ad0d187453742f290226d30150890b7890711bda4f32f192f", size = 1162570, upload-time = "2026-04-15T22:28:17.714Z" }, ] +[[package]] +name = "openpyxl" +version = "3.1.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "et-xmlfile" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3d/f9/88d94a75de065ea32619465d2f77b29a0469500e99012523b91cc4141cd1/openpyxl-3.1.5.tar.gz", hash = "sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050", size = 186464, upload-time = "2024-06-28T14:03:44.161Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" }, +] + [[package]] name = "opentelemetry-api" version = "1.44.0" diff --git a/docs/feature-summaries/FILE_UPLOAD_IMPLEMENTATION.md b/docs/feature-summaries/FILE_UPLOAD_IMPLEMENTATION.md index 0758fa432..a52ca56fc 100644 --- a/docs/feature-summaries/FILE_UPLOAD_IMPLEMENTATION.md +++ b/docs/feature-summaries/FILE_UPLOAD_IMPLEMENTATION.md @@ -206,8 +206,11 @@ PK: USER#{userId}, SK: QUOTA | `S3_USER_FILES_BUCKET_NAME` | S3 bucket name | | `DYNAMODB_USER_FILES_TABLE_NAME` | DynamoDB table name | | `FILE_UPLOAD_MAX_SIZE_BYTES` | Max file size (default 4MB) | -| `FILE_UPLOAD_MAX_FILES_PER_MESSAGE` | Max files per message (default 5) | +| `FILE_UPLOAD_MAX_FILES_PER_MESSAGE` | Max files per message (default 5). Enforced server-side by the inference API across both `files` and `file_upload_ids`; files beyond the cap are reported to the user in the attachment note, not silently dropped. `0` disables. | | `FILE_UPLOAD_USER_QUOTA_BYTES` | User quota (default 1GB) | +| `INLINE_DOCUMENT_MAX_BYTES` | Per-file inline document cap (default 4MB); larger non-tabular files are skipped with a note | +| `INLINE_ATTACHMENTS_MAX_TOTAL_BYTES` | Per-turn budget for all inline attachments, images included (default 7.5MB = the 10MB AgentCore Memory event quota ÷ base64's 4/3). Files that would push the turn over it are skipped with a note instead of failing the history write. `0` disables. | +| `ATTACHMENT_TURN_GUARD_ENABLED` | Kill switch for the two caps above (default on; only `false` disables) | ## Quota Management diff --git a/docs/kaizen/research/2026-09-18.md b/docs/kaizen/research/2026-09-18.md new file mode 100644 index 000000000..3cd304a0d --- /dev/null +++ b/docs/kaizen/research/2026-09-18.md @@ -0,0 +1,372 @@ +# Kaizen Research — Friday, September 18, 2026 + +> Scan window: **September 11 – September 18, 2026 (7 days)**. +> Web budget: **20 / 50** used. Nine of eleven subagents ran almost entirely through the authenticated `gh` CLI and `curl`, which cost nothing against the budget. + +## TL;DR + +**Last week's #1 shipped and the gate it defended is green.** PR CI now runs the frontend suite a second time under the nightly's exact `--coverage` invocation (`.github/workflows/ci.yml:34-38`, `run_frontend_coverage: true`), and `Nightly Build & Test` has passed **seven consecutive nights**, 2026-09-12 → 2026-09-18, after seven consecutive failures. Both halves landed. That is the loop closing, and it is worth saying out loud before anything else in this doc. + +**The most consequential finding is a reversal of our own recommendation from last week.** The GPT-6 Astra model card's tier headings settle the blocking question we left open: *"short context (**272K input tokens or fewer**)"* / *"long context (**more than 272K input tokens**)"*. The price tier is selected by **actual input tokens on the request**, not by the declared context window. Last week's Proposal #3 recommended registering `maxInputTokens: 272000` to avoid a 2× cost; on this evidence that would throw away a genuine 1M-token window to solve a problem pricing does not have. Two further corrections fall out of the same card: the long tier is 2× on input, cache-write and cache-read but only **1.5× on output** ($55.00 → $82.50), so a `long = short × 2` helper would overstate output by 33%; and Astra is **not** a third `cacheRead = input × 0.1` counterexample — it sits exactly on the default. + +**And there is a live deploy trap sitting one dependency bump away.** Strands `python/v1.56.0` (2026-09-15) deleted three `Bidi*` hook events from `strands.experimental.hooks.events` with no alias. `bedrock_agentcore.memory.integrations.strands.session_manager` imports all three at module load, so **every `bedrock-agentcore` release from 1.12.0 through 1.23.0 fails to import against Strands 1.56.0** — and on AgentCore Runtime that surfaces as *"Runtime initialization time exceeded (30s)"* then a 502, which reads as a cold-start problem rather than an ImportError. We import that module at three sites. Our current pins are safe; the fix is that **any bump to Strands 1.56.0 must carry `bedrock-agentcore>=1.23.1` in the same PR**. + +Recommended **#1** is the Astra correction — it is cheap, it reverses a decision that is currently queued as "ship", and it is the one item where acting on last week's doc would make things worse. + +## External Scan + +### What's moving this week + +The week's shape is *the ecosystem independently arriving at conclusions we had already written down, and one of our own conclusions turning out to be wrong.* + +The convergence is striking and it runs three ways. **Skills over MCP (SEP-2640) went Final on 2026-09-13** — an official extension letting an MCP server *serve* Agent Skills, with a rule set that reads like our own cost tenet transcribed: hosts **MUST NOT** retrieve skill files ahead of need, approval binds to a SHA-256 manifest without file retrieval, and only frontmatter enters the prompt. Our Skills v2 epic built exactly that primitive, locally, and closed it in August. **opencode** landed a rule that attachments over a threshold are delivered *by path* with a read tool rather than inlined — which is our document-offload epic, and which structurally fixes the `DOCUMENT_READ_ENABLED`-not-coupled-to-offload gap we flagged in validation, because in their model path delivery is meaningless without the read tool. And **LibreChat** shipped "carry ask-user answers past compaction" the same week our clarifying-questions feature reached prod. + +The second theme is **prompt-cache invalidation as a recognized bug class, not a local curiosity**. Claude Code shipped *four* fixes for it in one week (2.1.269 → 2.1.275), every one the same shape: a non-content side event silently mutating the cacheable prefix — a restored memory file's **age note** changing between requests after a compaction, `/login` discarding earlier thinking, a resume-after-interrupt re-sending earlier context differently. That is #741 and #751 with different names. The age-note variant is the one we have **not** audited for; our own day-stable date fix (`9f246cb7`) removed 23 of 24 daily prefix rewrites but not the 24th. + +The SDK layer moved decisively: the first-party Strands context-manager and offloading stack is **no longer main-only** — #4118/#4146/#4187/#4231/#4254/#4282 all ship in 1.56.0, and `ContextOffloader` manual offloading was closed as released. That is a real future collision with ours, now pinnable. Against that, the standing prompt-caching watch paid out **nothing** this week: no new `CacheConfig` field, our PR #4193 still open and unmerged, issue #3546 still open, so neither half of `usage_normalization.py` can retire. + +The protocol layer held still where we host it (MCP spec revision unchanged at 2026-07-28, `ext-apps` zero commits in-window) and moved where we don't: **FastMCP's repo relocated to `PrefectHQ/fastmcp`**, shipped 4.0.4 and 4.0.5, and carries an open issue — `Context.session_id` changing on every tool call in 4.x — that would break any Gateway-backed server keying state off it. + +One negative result is worth as much as the positives: **frontier-model week was empty.** Anthropic, Google and Meta produced no capability delta in-window. The one plausible item, an OpenAI "Agents API" with automatic context compaction and on-demand tool-definition loading, could not be date-verified because openai.com returned **403** to two fetch attempts. It is reported below as unverified rather than dropped. + +### Notable items by source + +> **Annotation conventions:** `*relevance*:` — impact on existing code. `*unlocks*:` — capability-unlock lens, for new platform primitives, SDK hooks, spec capabilities and UX patterns. + +#### AWS Bedrock / AgentCore + +- **Defense-in-depth authorization for MCP tools via an AgentCore Gateway interceptor** (2026-09-17) — an ML-blog walkthrough wiring Entra ID group/claims JWTs through a **Gateway interceptor** to apply per-user, per-tool RBAC/ABAC server-side, with a CloudTrail audit trail — https://aws.amazon.com/blogs/machine-learning/implementing-defense-in-depth-authorization-for-mcp-tools-on-amazon-quick/ — *relevance*: our MCP+SigV4 Lambda tools sit behind AgentCore Gateway and are gated **only in-process** by the AppRole `grantedTools` check, via `ToolRegistry` filtering — *unlocks*: net-new — per-tool authorization enforced **outside** our process, so a leaked Gateway credential or a tool invoked off our path is still scoped to the end user. Lines up exactly with the "governance = Entra claims" direction. +- **AgentCore Identity Consent Portal — implementation walkthrough** (2026-09-14) — provisioning a Consent Portal, configuring GitHub and Slack authorization-code-grant targets, the end-user consent flow, CloudTrail review — https://aws.amazon.com/blogs/machine-learning/manage-end-user-oauth-consent-for-ai-agents-with-amazon-bedrock-agentcore/ — *relevance*: directly on `OAuthConsentHook` / the `oauth_required` SSE path and the token vault. ⚠️ **Constraint that kills the drop-in read**: a Consent Portal requires a Gateway with **JWT inbound auth** as its source and an IdP whose permitted scopes include `openid`. **Our Gateway targets are SigV4.** *(Last week's flagged items resolve as out-of-window: the Consent Portal devguide entry is dated 2026-09-03 and "Evaluations: TypeScript agent framework support" 2026-09-02 — this blog is the only new material on either.)* +- **AgentCore system-prompt optimization — technical companion** (2026-09-16) — how the optimizer's "reflector engine" turns production traces into proposed configuration changes, validated before promotion — https://aws.amazon.com/blogs/machine-learning/optimizing-agent-system-prompts-with-amazon-bedrock-agentcore/ — *relevance*: same trace substrate as the AgentCore Evaluations wiring in flight on PR #1148, one stage further on. ⚠️ **Cost caveat**: any accepted system-prompt edit re-writes the whole cacheable prefix at the write premium, so a promotion gate must be version-aware and turn-boundary-aligned, never continuous. +- **Optimizing cost and latency with Amazon Bedrock prompt caching** (2026-09-15) — six Converse-API caching scenarios: message content, system prompt, tool definitions, **mixed TTL**, **tenant isolation**, LangChain integration — https://aws.amazon.com/blogs/machine-learning/optimizing-cost-and-latency-with-amazon-bedrock-prompt-caching/ — *relevance*: reference material against our three-cachePoint layout and the `partial_miss` / `cacheStatus` work. **Mixed-TTL is the one scenario we have not modeled** — our 1h-TTL note in `CLAUDE.md` assumes a single TTL, and PR #1132 shipped selective 1h TTL behind a flag that is still off. +- **Managed Knowledge Base: multimodal embeddings with TwelveLabs Marengo 3.0** (2026-09-11) — encodes visual scenes, speech and video cues directly to 512-dim vectors rather than transcribing to text; results carry segment start/end times — https://aws.amazon.com/about-aws/whats-new/2026/09/amazon-bedrock-managed-knowledge-base-multimodal-embeddings-twelvelabs-marengo/ — *relevance*: Managed-only, so it is an argument **for** the Classic→Managed migration rather than a risk to it — *unlocks*: audio/video retrieval with timestamped segments, which our text-only ingestion cannot produce. +- **Selecting a vector store for Bedrock Knowledge Bases** (2026-09-17) — https://aws.amazon.com/blogs/machine-learning/selecting-a-vector-store-for-amazon-bedrock-knowledge-bases/ — *relevance*: decision input for the same migration, before the store choice freezes. ⚠️ Feed summary only; article body unread. +- Also in-window, noted not expanded: *a shared agentic platform for Wood Mackenzie on AgentCore* (09-17, peer-architecture comparison read); *improving HCLS AI reasoning with open-source agent skills* (09-16, 38 skills + a 410-prompt eval — tangential to Skills v2). +- ⚠️ **Coverage caveat**: the ML-blog RSS feed carries 20 items reaching back only to 2026-09-14, so **ML-blog posts dated 09-11 → 09-13 were not visible**. The AgentCore devguide release-notes feed had nothing dated inside the window at all (newest entry 2026-09-03). + +#### Strands Agents + +- **`python/v1.56.0` (2026-09-15) — the only Python release in-window; we are now two behind at 1.55.0** — https://github.com/strands-agents/harness-sdk/releases/tag/python/v1.56.0 — *relevance*: `backend/pyproject.toml:66,81`. +- **⛔ BREAKING — 1.56.0 deleted three Bidi hook events with no alias, and it breaks `bedrock-agentcore` at import time.** `BidiAfterInvocationEvent`, `BidiAgentInitializedEvent` and `BidiMessageAddedEvent` are gone from `strands.experimental.hooks.events`, now a shim that only aliases the four `*Invocation*`→`*Call*` renames. `bedrock_agentcore.memory.integrations.strands.session_manager` imports all three at module load → `ImportError` — for **every** `bedrock-agentcore` release 1.12.0 → 1.23.0. Fixed downstream in **1.23.1** (2026-09-16, PR #664) — https://github.com/strands-agents/harness-sdk/issues/4362 · https://github.com/aws/bedrock-agentcore-sdk-python/issues/665 — *relevance*: we import that module at **three** sites — `apis/shared/sessions/messages.py:20`, `apis/app_api/shares/service.py:392`, `agents/main_agent/session/turn_based_session_manager.py:40`. ⚠️ **On AgentCore Runtime this presents as "Runtime initialization time exceeded (30s)" then a 502** — it looks like a cold-start problem, not an import failure. Current pins (1.55.0 + 1.21.0) are safe. **The bump is a paired pin, not a bump.** +- **`SessionManager` became `Generic[_SessionAgentT]` and dropped its three Bidi callback registrations** (replaced by a single `BidiAgentStopEvent → sync_agent`); `RepositorySessionManager` is now `SessionManager[LocalAgent]`; `SessionAgent.from_bidi_agent` folded into `from_agent`; abstract signatures moved from `agent: "Agent"` to `agent: _SessionAgentT` — *relevance*: runtime-compatible for a subclass that doesn't override the Bidi methods, but it changes what the base registers, and our `TurnBasedSessionManager` sits two levels down. Read before bumping. +- **`Model.stream()` / `_format_request()` gained `agent_metadata`, passed positionally** (now 6 positionals) — *relevance*: our `BedrockRuntimeResponsesModel._format_request` override (`apis/shared/models/bedrock_responses.py:313`) already absorbs extras via `*args, **kwargs`, so **it survives as written** — the defensive signature pays off. `CountTokensBedrockModel` doesn't override `stream` and is unaffected. +- **`InterruptException` now bubbles out of `event_loop_cycle` unwrapped** instead of being caught into `EventLoopException` — *relevance*: this is the class behind `_stop_for_interrupts` / `PausedTurnSnapshot`, i.e. OAuth consent, tool approval and `ask_user_question`. Anything of ours catching `EventLoopException` to detect a pause stops matching on the bump. +- **Open bug #4338 — an `AfterToolsEvent` callback that raises drops the completed `toolResult` before append, and a resume re-runs the already-executed tool** — https://github.com/strands-agents/harness-sdk/issues/4338 — *relevance*: our mid-turn-steering hook peeks at `AfterToolsEvent` and commits on `MessageAddedEvent`, so we do not *cause* it — but any exception escaping that callback would trigger it, and a duplicated tool execution is a direct token-cost bug. Worth one defensive `try/except` audit. +- **Open #4371 / #4330** — `InterventionRegistry._dispatch` wraps handlers in a bare `except Exception`, so an `event.interrupt()` inside a handler's `before_tool_call` is treated as a crash and `on_error='proceed'` **runs the tool unapproved** (we don't use `InterventionRegistry` — a reason *not* to migrate `OAuthConsentHook` onto interventions); and Python retry backoff ignores agent cancellation until the delay completes, which matches our `model_retry` timing caveat — https://github.com/strands-agents/harness-sdk/issues/4371 · https://github.com/strands-agents/harness-sdk/issues/4330 +- **The first-party context-manager / offloading stack is fully released in 1.56.0** — #4118, #4146, #4187, #4231, #4254, #4282; #4101 (manual offloading via the `ContextOffloader` plugin) closed as released. Follow-on **#4367** is open, where the maintainer's 2026-09-16 comment says a `NAMESPACED` sentinel should let a custom stash namespace **share offloaded context across sessions** — https://github.com/strands-agents/harness-sdk/issues/4367 — *relevance*: we subclass the vended `ContextOffloader` at `agents/main_agent/core/tool_result_offload.py:156` — *unlocks*: a cross-session offload namespace, directly evaluable against the document-offload epic. ⚠️ The standing decisions-log entry stands: a bare "adopt the built-in, delete ours" proposal is **out of scope** without a migration design covering tool-content truncation, LTM summary retrieval, DynamoDB checkpoint persistence and the `compaction` SSE-once invariant. +- **No upstream drift on our MCP monkeypatch** — no file under `strands-py/src/strands/tools/mcp/` appears in the `1.55.0...1.56.0` diff. + +**Caching watch — answers (all five)** + +1. **No new `CacheConfig` field.** On both `python/v1.56.0` and `main` the dataclass is exactly `strategy`, `ttl`, `system_prompt_ttl`, `cache_key`, `tools_ttl` — byte-identical between refs. One semantics change: `cache_key` widened from `str | None` to `str | Literal[False] | None`, where `False` is an explicit opt-out and `None` means "derive". +2. **`_openai_cache.py::apply_cache_config` still present, changed in 1.56.0** — gained an `agent_metadata` parameter and a `_resolve_cache_key()` helper (configured string wins → `False` opts out → else `strands-`). Still maps onto `request["prompt_cache_key"]`, still guarded by `if "prompt_cache_key" not in request`. +3. **PR #4193 (ours) still OPEN, unmerged**, no reviews; one in-window comment (2026-09-16) saying it closes the Responses half of #3546 and opening **#4361** for the same gap in the Chat Completions adapter. **Issue #3546 still OPEN.** → **Neither half of `usage_normalization.py` can retire.** +4. **`cache_key` auto-derivation (#4083) shipped in 1.56.0 — and it does not affect us today**, because `apply_cache_config` returns immediately when `cache_config is None`, and only `to_bedrock_config` (`model_config.py:514`) sets one; the Responses/Mantle configs set none, and Converse never reads `prompt_cache_key`. ⚠️ **Latent trap**: the moment anyone adds a `cache_config` to the Responses/Mantle path, upstream's `strands-` lands in `request` first inside `super()._format_request`, and our `request.setdefault("prompt_cache_key", build_prompt_cache_key(...))` at `bedrock_responses.py:259` becomes a **silent no-op** — swapping our prefix-derived, cross-session-shared key for a per-session key that fragments the cache. Cheap insurance: pass `cache_key=False` (or our own key) on any `CacheConfig` ever given an OpenAI-wire model. +5. **The context-manager stack is all released** (see above) — nothing main-only. + +No new Bedrock model family gained prompt caching this week, so the coverage half of the watch produced nothing and `probe_bedrock_cache_point_support.py` was not run. + +#### Reference repo (aws-samples/sample-strands-agent-with-agentcore) + +Low-volume, high-density week: **12 in-window commits, one author, 2 merge commits → 10 substantive commits across exactly 2 merged PRs (#294, #296). Zero Dependabot commits landed** (six Dependabot PRs opened, all still unmerged). PR #294 alone touches 113 files. + +- **[PR #294] System prompt rewritten, plus a real-model A/B eval harness** — `prompt_builder.py` replaces the `` / `` blocks wholesale, and the *concise* variant now **replaces** rather than appends, because appending a "be brief" block left both instruction sets in the prompt and the model split the difference — "the toggle looked broken even though the flag was arriving correctly". Ships with `tests/evals/{compare_response_style.py,exercise_response_style_agent.py,response_style_cases.json}` — https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/294 — *applicability*: **the harness is the portable part, not the prose.** We have no equivalent for our prompt builders, and "two instruction sets fought and the flag looked broken" is a failure we could hit with any additive prompt mode. Their README discipline is worth stealing verbatim: *"Do not infer naturalness from response length, passing keyword checks, or HTTP success… preserve unfavorable examples and repeat suspected regressions before claiming improvement"* — that is our own n<16-is-noise rule, written down. ⚠️ The harness pins the date constant across arms ("changing date is not part of the treatment"), which **confirms their daily-rotating date is still in the system prompt**. Do not port that. +- **[PR #294] Remote MCP tools now connect lazily, only when their skill activates** — `skill_registry.add_tool_loader()` + `_load_deferred_tools()`, RLock-guarded, loader-identity deduped, failed discovery deliberately not cached as empty so a later activation retries; `skill_dispatcher` moves to `@tool(context=True)` and resolves the registry off `tool_context.agent.__dict__` instead of a module global. Their measured result: new-conversation "Starting" **10.417s → 4.081s (−60.8%)**, follow-up **7.377s → 1.117s (−84.9%)**, n=2/3 per arm — https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/294 — *applicability*: **ours is eager** — `base_agent.py:478` `_build_filtered_tools()` loads every enabled external MCP tool at agent build, before the first model call. ⚠️ **The trap in copying it verbatim is severe for us**: their new base prompt now states *"available tools may change between turns within the same conversation"* — they accept a `toolConfig` that varies turn to turn. For us that is a per-turn prefix rewrite on `toolConfigHash`, paid at 1.25× base input over 30k–150k tokens. The shape to steal is **deferring discovery for tools not in this session's `enabled_tools`** — a stable, session-scoped set — never "connect on first use mid-conversation". Separately, their module-global→per-agent registry fix is the same bug class as our "never cache session state on an agent instance" rule, one layer down. +- **[PR #294] Cancelled turns persist a real message instead of the SDK placeholder** — new `cancellation_history.py` wired into `append_message()` via `prepare_message()`, swapping the SDK's `[{"text": "Cancelled by user"}]` for the actually-streamed prose plus a stop marker; `routers/chat.py` settles as `ExecutionStatus.STOPPED` rather than unconditionally `COMPLETED` — *applicability*: **we solve this differently and, for our cost model, better.** Our interrupted-turn state is metadata on the session row (`last_turn_interrupted*`, written by `set_interrupted_turn`) and never enters the Converse payload. **Do not port their approach** — `prepare_message` mutates message content on the append path, which is exactly the "display string in the cacheable prefix, paid at model rates every subsequent turn" anti-pattern `CLAUDE.md` already forbids for `tool_group_summary`. The portable bit is the **`STOPPED` vs `COMPLETED` execution-status distinction**, relevant to the false-"interrupted"-turn marker (#988). +- **[PR #294] A2A client-side cancellation hardening + an a2a-sdk signature change** — `client.cancel_task(TaskIdParams(id=...))` instead of a bare string; server-side `cancel()` now disconnects the client after `interrupt()` because unread terminal messages "could be mistaken for the next task's result"; an empty `final_result` reports `status: "error"` rather than `completed` — *applicability*: we are A2A **client-only** and call `cancel_task` nowhere (zero hits across `backend/src`), so the signature change is a no-op today — but it is a breaking-shape data point for whenever `A2AAgentConfig` grows cancellation. ✅ Re-verified: their `AgentCard` still carries `capabilities=AgentCapabilities(streaming=True)`, so the `CLAUDE.md` note about the ~40-minute client hang remains accurate. +- **[PR #294] SSE reconnect + stopped-turn client reconciliation** — the execution-status route now returns **503** when status is `unavailable`; it previously returned **200 with an `unavailable` body**, so the client treated "can't tell" as authoritative. Also a `sessionStorage` stopped-turn snapshot, and `SESSION_TIMEOUT_MINUTES` **20 → 0** ("a conversation is durable; runtime idleness must not discard the open chat") — *applicability*: the 200-with-`unavailable`-body → 503 fix is the sharpest idea here, and I checked for it: **we have no equivalent endpoint** (`apis/app_api/sessions/routes.py` exposes only list / metadata / messages; turn state rides SSE and the lease row), so there is nothing to port. The `sessionStorage` snapshot is client-invented state reconciled against server history — our contract elsewhere (deliberately not persisting `agent_status` durations) argues against it. +- **[PR #296] Sidebar conversation-title search** — a labelled "Search chat titles" input with a live `N result(s)` count, date grouping suppressed while filtering, "No matching chats" distinguished from "No conversations yet", clears on Escape and on New chat, query preserved across list refreshes; `ui/sidebar.tsx` also sets `inert` on the off-canvas panel while closed — https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/296 — *applicability*: **we have no equivalent** — `components/sidenav/components/session-list/session-list.ts` groups by date with no search at all. Cheap, self-contained, zero backend, **zero token cost** (a pure client-side filter over already-loaded titles). The `inert`-while-closed detail is a real a11y bug worth checking in our own off-canvas sidenav. +- **Negative results (signal too):** pins **unchanged** — still `strands-agents[a2a,otel]==1.54.0` and `bedrock-agentcore==1.22.0`, so we remain ahead on Strands and behind on `bedrock-agentcore`, the same one-minor lag now two weeks old. **Model catalog untouched** — zero commits against `model_catalog.py`, so no new rows to evaluate. **No follow-up to #290's `ensure_unique_document_names`** — the in-place mutation of prior messages we flagged as a cache regression stands unaddressed upstream, so our decision not to port it verbatim needs no revisiting; stop re-checking weekly unless they touch document handling again. No A2A server-side capability change; no Gateway/Identity/Memory wiring changes. + +#### MCP ecosystem + +- **SEP-2640 "Skills over MCP" went Final (merged 2026-09-13; conformance scenarios 09-11; docs 09-13/09-18)** — defines `io.modelcontextprotocol/skills`: servers declare the extension alongside `resources` in `server/discover` and MUST implement `skills/list` + `skills/get`; each entry is `{uri, frontmatter, resources}` where `resources` is a complete manifest of every file with a SHA-256 `digest` and byte `size` (or a `"dynamic"` sentinel). Content is read lazily through ordinary `resources/read` under a `skill://` URI convention. Hard rules: **"Hosts MUST NOT retrieve files ahead of need, including on connection, listing, or approval"**; **"approval binds to the manifest without requiring file retrieval"**; identity is `(server identity, URI)` — names are labels, and a host MUST NOT classify a resource as a skill from the `skill://` scheme alone. `skills/list` carries `ttlMs` + `cacheScope`; `resources/directory/read` is gated behind an optional `directoryRead: true` — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2640 · https://github.com/modelcontextprotocol/ext-skills — *relevance*: our Skills v2 agent primitive (#685) is registry-local with its own RBAC (`grantedSkills`); this is the first standard way a **remote** server can ship skills to us, i.e. our Gateway MCP servers could advertise skills the way they advertise tools — *unlocks*: third-party/served skills with no bespoke ingestion path, and a plausible export shape for the Skill Creator. ⚠️ **Two hard prerequisites**: it requires protocol revision **2026-07-28** and `server/discover`, which the `mcp` 1.x line we are pinned to does not speak; and SDK support is still an **open** PR (python-sdk#3485) across Go/Python/TS/C#. Spec-Final ≠ shippable. +- **Spec revision unchanged at 2026-07-28.** No new revision, no RC. Everything merged into the spec repo in-window was Skills documentation, a WG-lead addition, and Dependabot. SEP-2567 / 2549 / 2243 unchanged since July. +- **An `mcp` Python SDK 2.x line exists and is two minors deep — but nothing landed in-window.** v2.0.0 (07-28), v2.1.x (08-24/25), v2.0.1 (08-26), **v2.2.0 (09-07)**; one commit on `main` since 09-11 (a test-harness change). **v1.x is in maintenance mode, security fixes only**, on a `v1.x` branch — https://github.com/modelcontextprotocol/python-sdk/releases/tag/v2.2.0 — *relevance*: `integrations/mcp_apps.py` substitutes `strands.tools.mcp.mcp_client.ClientSession` with `_UIExtensionClientSession`. **What breaks under 2.x**: (1) v2 removes the transport + `ClientSession` + `initialize()` layering in favour of a single `Client`, so the class our patch subclasses is no longer what Strands would construct; (2) protocol types moved to a separate `mcp-types` distribution; (3) transport dependency `httpx` → `httpx2`; (4) **MCP Apps is a built-in extension API in v2**, so the patch's reason for existing may be obsolete rather than merely broken. v2.2.0 adds hazards on the external-OAuth path: HTTP redirects followed **only within the endpoint's origin**, OAuth AS-metadata `issuer` validated on the legacy discovery path, idle Streamable-HTTP sessions expiring at 30 min with a 10,000-session cap. ✅ **Our guard rail is intact and documented**: `backend/pyproject.toml:120-132` carries a `[tool.uv] constraint mcp<2` with this exact rationale written out — including the note that the only real thing holding us on 1.x is the `idna==3.15` pin, so **a routine `idna` security bump would silently unblock the major crossing with nothing in the diff saying "MCP"**. +- **`ext-apps`: zero commits or releases in-window.** v2.0.0 (2026-09-08) remains latest, last commit 09-09. Re-confirmed from the commit log rather than assumed: v2.0.0 was a migration to MCP SDK 2.0 with 1.x handler-registration forms kept as deprecated overloads and explicit cross-version interop plus captured-1.x-traffic wire-compat tests — **wire protocol unchanged, confirmed by test**. Our `ui_resource` / `ui_tool_input_partial` host path needs nothing. +- **Three in-flight SEPs actively revised in-window, all still OPEN (tracking only)** — **SEP-2643 Structured Authorization Denials** (updated 09-17): a transport-agnostic denial envelope carrying failure classification, remediation posture, a retry-correlation handle and structured remediation hints, including the "user completes an approval at a URL" case composed with URL-mode elicitation; **SEP-2636 Tool Manifests for Incremental Catalog Synchronization** (09-14); **SEP-2631 File Objects and Transfer** (09-17) — https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2643 — *relevance*: **2643 is a standardized version of exactly what our pre-flight `oauth_required` flavor hand-rolls** — the case where a server refuses `tools/list`, the `BeforeToolCall` consent hook cannot fire, and we emit a synthetic-id-less event so the SPA shows Connect without resuming. 2636 speaks to tool-catalog token bloat; 2631 to presigned-URL-in-tool-result breakage and document offload. + +#### FastMCP + +- **⚠️ The repo moved: `jlowin/fastmcp` now redirects to `PrefectHQ/fastmcp`.** `gh` follows it transparently, but any hardcoded `jlowin/` URL in our kaizen skill or server-repo docs is now a redirect — https://github.com/PrefectHQ/fastmcp +- **v4.0.5 "No Country for Loose Ints" (2026-09-17)** — tool parameters declared strict (`Field(strict=True)`, `StrictInt`, strict model config) are honored again on direct calls and task submission; **since the SDK v2 migration the server's lax default silently overrode them and coerced values** — https://github.com/PrefectHQ/fastmcp/releases/tag/v4.0.5 — *relevance*: FastMCP 4.x is built on **MCP Python SDK v2**, so our Lambda-backed Gateway servers are already on the 2.x line even though this repo pins `mcp` 1.x. If any of those servers declared strict numeric params, 4.0.0–4.0.4 was silently coercing them — a wrong-value-**accepted** bug, not an error. +- **v4.0.4 "Here Be No Dragons" (2026-09-16)** — OpenAPI: multipart string arrays as repeated fields, whole-body arguments no longer clobbering same-named HTTP parameters, dictionary bodies and raw content types preserved, JSON scalar bodies encoded, query `explode` defaults corrected. Auth: `OAuthProxy` **rejects self-contained ID-JAG tokens unless identity assertion is configured**, and refuses non-positive upstream token expiries. Protocol: clients follow empty pagination cursors; servers reject malformed cursor offsets. `meta.fastmcp.versions` ordering made deterministic — https://github.com/PrefectHQ/fastmcp/releases/tag/v4.0.4 — *relevance*: the OpenAPI body fixes hit any server we generate from an OpenAPI spec behind Gateway; ⚠️ the `OAuthProxy` change is a **behavior tightening that can break a working deployment** relying on ID-JAG without `identity_assertion` configured. The deterministic `meta.fastmcp.versions` ordering is a small prompt-cache win under our ordering contract if that metadata ever reaches `toolConfig`. +- **Open issue #5134: `Context.session_id` changes on every tool call in 4.x, breaking session state** (09-16, open) — https://github.com/PrefectHQ/fastmcp/issues/5134 — *relevance*: **the highest-risk open FastMCP item for us.** Any Gateway-backed server keying per-session state off `Context.session_id` loses it between calls on 4.x. Worth a direct grep of the server repos before anyone upgrades past 3.4.x. Pairs with MCP SDK v2.2.0's new 30-minute idle Streamable-HTTP expiry — two independent session-identity regressions in the same dependency line. +- **Two open auth-path issues worth watching** (both 09-15/09-17, open) — **#5145**: `GoogleTokenVerifier` puts the access token in the `tokeninfo` **URL**, so the HTTP client's INFO logs expose it; **#5120**: `OAuthProxy` sends its own refresh token to the *upstream* revocation endpoint — https://github.com/PrefectHQ/fastmcp/issues/5145 — *relevance*: #5145 is a credential-in-URL/log-surface pattern, precisely what `scrub_log()` exists to catch on our side — but it happens **inside the MCP server process, outside our sanitizer's reach**. Operational/logging concern, not a security advisory (those are out of scope). + +#### Agentic UI/UX patterns + +- **Skills over MCP (SEP-2640)** — see the MCP ecosystem section for the spec detail — *fit*: **direct port**, and the closest external match to a primitive we already own. The lazy-read + digest-bound-approval design is an exact restatement of our cost tenet: bounded per-turn payloads, only `SKILL.md` frontmatter entering the prefix, bodies on demand — *where it'd land*: the MCP client in `apis/shared/tools/` (capability negotiation + a `skills/list` path feeding the registry), the skill-selection surface under `/customize`, and the consent path — manifest-scoped approval maps onto `tool_approval_required` rather than a per-file prompt. +- **assistant-ui 0.15.20 — nested-conversation tool calls become first-class tasks** (2026-09-15, with #7356) — `thread.tasks` is now **derived** from tool calls carrying a nested conversation and exposed under its own task scope; #7356 keeps the parent message `running` while that nested conversation streams, reporting progress on the nested thread rather than the parent. Subagent transcripts stop being opaque tool results and become an inspectable child thread — https://github.com/assistant-ui/assistant-ui/pull/7359 — *fit*: **pattern-only** (Angular equivalent: a `computed()` task list derived from the message signal keyed by `toolUseId`, plus a nested-thread child component; **no new SSE event needed** — the data is already on the wire) — *where it'd land*: the tool rail / message renderer, fed by `agent_status` (`cycle`, `tool_start`/`tool_end`, `ok`, `durationMs`) plus `tool_group_summary`. This is the missing attribution layer for `@agent` mentions — today a mention-turn's work is flattened into the parent thread and `turnAgentId` is visible only in the data. +- **assistant-ui — per-status counts on grouped parts + name-routed tool rendering** (2026-09-15) — `groupPartByType` exposes per-status counts on a group (render "3 running · 1 failed" without re-scanning) and routes tool calls to a renderer **by tool name**, enabling bespoke per-tool result UIs inside a generic group — https://github.com/assistant-ui/assistant-ui/pull/7354 — *fit*: **pattern-only** (Angular: a `computed()` rollup over the batch signal + a renderer map keyed by tool name with a default fallback) — *where it'd land*: the client-side tool-batch grouping that renders `tool_group_summary`. We already carry the exact inputs (`agent_status.ok=false` marks failures, `durationMs` per tool), and a status rollup **degrades gracefully when `TOOL_SUMMARIES_ENABLED=false`**, where today the deterministic formatter gives "Listed 4 assignments" with no failure signal. +- **[Conformance gap — ours] MCP Apps header should use tool display-name precedence `title` → `annotations.title` → `name`** (PR opened 2026-09-18, fixing ext-apps issue #767) — the reference host rendered the raw `tool.name` in the App frame header; the base MCP schema specifies that precedence, and for `Tool` specifically `annotations.title` outranks `name`. Claude Desktop shows the same "ServerName raw_tool_name" header while the tool row beside it shows the friendly `title` — https://github.com/modelcontextprotocol/ext-apps/pull/778 — *fit*: **direct port, and we have the same bug — verified.** `backend/src/apis/shared/mcp_apps/` contains **no `title` or `annotations` handling at all**; `ui_resource_store.py:215` and `card_store.py:160` write `"toolName": tool_name` straight through — *where it'd land*: the `toolName` derivation feeding the `ui_resource` payload (both the header-only shell at `content_block_start` and the full html-bearing emit), plus the App frame header component. Small, self-contained, and **the wrong string persists with the resource, so it survives reload**. +- **[Cross-ecosystem theme] Tool-approval durability across steps and reloads** — `ai@7.0.106` *"preserve tool calls required by retained pending approvals"* (2026-09-18) and assistant-ui #7310 *"preserve cloud message parts and track tool approvals"* (2026-09-15): two independent frameworks shipped fixes the same week for approval state being lost or desynced from the tool call it gates — https://github.com/assistant-ui/assistant-ui/pull/7310 — *fit*: **pattern-only, but it is our known failure mode** (a stale/abandoned consent bricking later turns; a 424 destroying a consent 409) — *where it'd land*: the interrupt-resume path — `PausedTurnSnapshot` / `PendingInterrupt` breadcrumbs and the resume guard in `inference_api/chat/routes.py`. ⚠️ **Worth a targeted test that an approval interrupt survives a history mutation between pause and resume** — compaction and document offload both now rewrite retained history underneath a pending interrupt. +- **Checked, nothing in window**: `blog.modelcontextprotocol.io` (latest 2026-08-22); Anthropic news (two in-window posts, both policy/life-sciences); Linear blog (latest 2026-09-10); `ext-apps` zero merged PRs or commits. NN/g fired once — ["Test Complex Interactions Earlier with AI Prototyping"](https://www.nngroup.com/articles/test-earlier-with-ai/) (2026-09-11) — but it is design-process guidance about prototyping workflow, not an agentic-app interaction convention, so it is deliberately not written up as an item. + +#### Frontier model announcements + +**A near-empty week for capability deltas — and that is the finding, not a gap in coverage.** + +- **Anthropic — no model or API news in window.** The only in-window post is "Measurements for understanding the pace of AI development inside frontier labs" (2026-09-17), policy/measurement with no capability, pricing or Bedrock content — https://www.anthropic.com/institute/measuring-pace-of-ai-development — *relevance*: none; Haiku 4.5 / Sonnet 4.6 rows unchanged. +- **Google DeepMind — nothing in window.** ⚠️ Reported as **not verified** rather than confirmed-empty: the blog index is JS-paginated and only the top card was visible. +- **Meta / Llama — nothing in window.** No September 2026 Llama release found; we carry no Llama rows. +- **⚠️ OpenAI "Agents API" public beta — exists, but could NOT be date-checked.** openai.com returns **HTTP 403** to WebFetch on both `/news/` and the article, so the only evidence is a search-result summary. Claimed contents: a public-beta Agents API on the Codex harness with (a) **automatic context compaction** as the session nears its limit, and (b) **tool search** loading tool definitions on demand "to reduce token usage and cost while preserving the model's cache" — https://openai.com/index/introducing-the-agents-api/ — *relevance*: if real and recent, both are direct external validation of two backlog items — our compaction stack and the tool-search token-bloat strategy — and "preserving the model's cache" is the same exact-prefix constraint our contract encodes. **Do not cite this as an in-window announcement.** Re-check next week via a non-WebFetch path. + +#### Agent harness patterns + +- **Claude Code shipped four prompt-cache-invalidation fixes in one week** (2.1.269 → 2.1.276 in-window) — 2.1.275 fixed *"a restored memory file's age note changing between requests after a compaction or resume, which caused prompt cache misses"*; 2.1.273 fixed `/login` / `/upgrade` / `/extra-usage` discarding earlier thinking and forcing a full prefix rewrite; 2.1.269 fixed partial invalidation on the turn after an output-token cutoff, and a resume-after-mid-thought-interrupt changing how earlier context was re-sent — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md — *relevance*: `session/turn_based_session_manager.py` (truncation anchor) and `session/compaction_policy.py`; identical shape to #741 / #751 — *unlocks*: **the age-note variant is one we have not audited for.** A sweep for *derived or relative* values — relative timestamps, "N days ago", recomputed ages — anywhere in restored history or the system prompt, plus the same check on the interrupt-resume path. Our own `9f246cb7` removed the hourly tick from the system-prompt date line; the **daily** boundary remains, so a session open across Pacific midnight still re-pays the full prefix once. +- **Claude Code 2.1.273 — opt-in LLM-gateway hint headers** — `x-claude-code-request-class`, `-agent-type`, `-prev-tool-durations`, `-compaction`, `-context-compacted`, behind `CLAUDE_CODE_GATEWAY_HINT_HEADERS=1`: per-request routing and attribution metadata carried out-of-band at **zero prompt-token cost** — same URL — *relevance*: `apis/shared/costs/` and `GET /admin/costs/sessions/{id}/calls`; the same side-channel discipline as `agent_status` / `tool_group_summary` — *unlocks*: tagging `C#` cost rows with request class / `turnAgentId` / compaction state at the transport layer instead of deriving them after the fact. +- **Claude Code 2.1.273 — the context meter double-counted advisor-tool turns** at ~2× real size, so auto-compact fired at about **half** the real window — same URL — *relevance*: `session/compaction_policy.py` model-relative thresholds; premature compaction is the front half of the compaction-death-spiral pattern — *unlocks*: an assertion that our measured per-turn context size for tool-heavy turns tracks CountTokens. Over-counting is a pure cost bug and **it fails silently**. +- **Claude Code 2.1.271/2.1.274 — MCP fixes matching three open traps of ours** — Streamable-HTTP tool calls timing out at ~5 min *even with a longer per-server `timeout`*; a 403 `insufficient_scope` misreported as an expired sign-in (now names the missing permissions); tool search missing an MCP tool when the model selects it by bare name instead of `mcp__server__tool`. Plus `CLAUDE_CODE_MCP_STARTUP_WAIT_MS` to bound the first turn's wait on connecting servers — same URL — *relevance*: our MCP Apps proxy-call 504, the scope-change re-consent path, and the tool-search bloat strategy — three recorded traps upstream just fixed independently. +- **pydantic-ai v2.44.0/v2.45.0 — per-run usage attribution** — *"Report a run's own usage on its span, not the conversation total"* (#8417, #8456), plus `DynamicToolset` resolved once per durable run and one MCP session per durable run rather than per unit — https://github.com/pydantic/pydantic-ai/releases/tag/v2.44.0 — *relevance*: the usage-rollup bug shape behind AdminUsageAggregates triple-counting; per-run MCP session reuse maps to our Gateway/MCP client lifecycle. **For ideas, not adoption.** (LangChain 1.4.1 in-window was a two-line patch; LlamaIndex has shipped nothing since Aug 19.) +- **Anthropic engineering blog: no in-window posts** (most recent 2026-04-23). + +#### opencode (anomalyco/opencode) + +Only **v1.18.31 (2026-09-14)** landed in-window and it was thin (ACP session-option restore, Copilot adaptive thinking) — releases lag development here, so the findings below come from the **229 PRs merged 09-11 → 09-18**, read through the three lenses we track. + +- **[Context engineering] Attachments delivered by path, not inlined** (#49682, plus "attach any file", "stream large attachments", a new `fs.write` endpoint) — an explicit rule: `files[]` means "put these bytes in the model's context"; when that cannot work (text, binary, or >20MB) the model gets the **path** and opens it with tools. Native-readable media under the cap inlines; everything else goes by path — https://github.com/anomalyco/opencode/pull/49682 — *relevance*: `session/document_offload.py`, `core/tool_result_offload.py`, and the "attachments = 31% of prod spend" finding — *unlocks*: a cleaner decision rule than our current size-gate, and it **structurally fixes the `DOCUMENT_READ_ENABLED`-not-coupled-to-rehydrate/offload gap** flagged in offload validation: in their model path delivery is *meaningless* without the read tool, so the coupling is structural rather than a config invariant someone has to remember. +- **[Context engineering] Native compaction moved behind a plugin boundary** (#49575) — core retains only what must stay invariant (policy, provenance, retries, overflow fallback, persistence); the *mechanism* — trigger-vs-endpoint, checkpoint window assembly — becomes a pluggable per-provider strategy, and a `mode: "provider"` policy with no registered strategy **fails loudly** — https://github.com/anomalyco/opencode/pull/49575 — *relevance*: `session/compaction_policy.py` + `compaction_summary.py` — *unlocks*: the seam we would want before adding provider-native compaction (GPT-5.6 vs Claude) without forking policy. The fail-loud-on-missing-strategy detail is the opposite of our silent-fallback default. +- **[Cost] `/btw` side-question command** (#49646) — calls `session.generate` to answer from the current session's context and model **without adding anything to the conversation**; the answer renders in a dialog, never in the transcript — https://github.com/anomalyco/opencode/pull/49646 — *relevance*: structurally identical to `apis/shared/tool_summaries/summarizer.py` and the `session_title` side channel — but **user-facing** — *unlocks*: an "ask about this thread without paying for it on every subsequent turn" affordance. Our side-channel plumbing already exists; only the surface is missing. +- **[Tooling] Permission policy as a config layer users cannot override** (#49578) — resources take the form `:` (e.g. `shell:*systemctl*`); org policy is evaluated *after* user plugin hooks so saved approvals cannot override it, and critically **policy allows are pass-through decisions that do not grant permissions** — https://github.com/anomalyco/opencode/pull/49578 — *relevance*: `apis/shared/rbac/`, `agents/main_agent/tools/tool_filter.py` — *unlocks*: that last clause is our own RBAC invariant stated as a type-level rule (only `granted*` grants; `allowedAppRoles` is display-only). A same-shaped **deny** layer sitting above role grants is something we do not have. +- **[Tooling / cost] Three cheap wins** — `reconcile stale tools at settlement` (#49672) refetches the message window when a tool is still locally "running" after durable state completed; `skip session warming for subagents` (#49387) drops a keep-alive **generation call per subagent**; `reuse tool discovery catalogs` (#49276) caches one rendered catalog per location keyed on registry revision, building the search index on demand so known tool calls skip discovery — https://github.com/anomalyco/opencode/pull/49672 — *relevance*: the dropped-SSE/leaked-lease and false-"interrupted"-turn issues; `tools/tool_catalog.py` (60s cache); and **worth confirming we make no equivalent warm-up call when an `@`-mention builds a second `Agent`**. + +#### LibreChat + +One release in-window: **`v0.8.8-rc3`, pre-release, 2026-09-15** (previous `v0.8.8-rc2` was 09-03, outside the window). Read from the release-notes PR list — titles and highlights only, so treat mechanism claims as inferred. + +- **"Carry Ask User Answers Past Compaction" (#15944) + "Compact Context in a Summarize-Only Turn" (#15690)** — they hit the same interaction we shipped: structured user answers are durable state a summarizer can silently eat, so answers are now carried across the compaction boundary; separately, compaction runs as its own summarize-only turn rather than riding a normal one — https://github.com/danny-avila/LibreChat/releases/tag/v0.8.8-rc3 — *relevance*: maps directly onto `ask_user_question` / `user_question_required` and `TurnBasedSessionManager` compaction. Our answers re-enter as a one-line formatted block in history — exactly the content a checkpoint summary can drop. **Worth acting on as a cheap verification**: confirm an `ask_user_question` answer survives a compaction checkpoint in a long session. Their summarize-only-turn shape is also a parallel to our byte-stability work — a dedicated turn is easier to keep append-only against the cached prefix. +- **MCP OAuth/credential reliability cluster (~10 fixes)** — "Attach Ready OAuth MCP Servers With Empty Catalogs" (#15727), "Land MCP OAuth Callbacks on Waiting Connections" (#15876), "Back Off MCP OAuth Discovery" (#15783), "Retire Cancelled MCP OAuth State" (#15702), "Close MCP OAuth Teardown Races" (#15716), "Stop an MCP Credential Refresh From Fencing Its Own Connection" (#15863) / "…Its Own Catalog Recovery" (#15865), "Support Renewable MCP Credentials for Schedules" (#15908), "Preflight MCP Readiness for Scheduled Chats" (#15782) — same URL — *relevance*: this is our `oauth_required` pre-flight path almost item for item. **#15727 is the same failure our pre-flight event exists to paper over** (a server that authenticates but returns an empty `tools/list` should not be treated as absent), and #15782/#15908 map to the scheduled-runs-ungated gap. **Worth a design read**: independent convergence on "a refresh must not fence its own recovery" suggests our `oauth_token_cache` warm-from-vault retry deserves a look for the same re-entrancy trap. +- **"Conversation Trace Viewer" (#15869) + "Surface Retained Tool-Message Context Usage" (#15796)** — a user-facing trace view of a conversation, plus a UI surface showing how much context retained tool messages actually occupy — same URL — *relevance*: maps to `GET /admin/costs/sessions/{id}/calls` and the parked context-attribution prototype. The notable choice is **placement**: they put retained-tool-context in the *product* UI where we keep the equivalent admin-side. Given tool-call condensing is built-not-committed and attachments are 31% of prod spend, this is a **signal worth noting, not an action** — a per-conversation "what's holding your context" affordance is a plausible successor to that prototype. +- **"Unify Artifact Triggers as Tool Rows" (#15915), "Unify Tool Activity, Reasoning, Search, and Agent Workflows" (#14546), "Preserve Queued Follow-Ups Through Handoff" (#15942)** — artifact creation renders as an ordinary tool row instead of a bespoke trigger; tool activity / reasoning / search / sub-agent workflows collapse into one activity surface; queued follow-up messages survive an agent handoff — same URL — *relevance*: the unification maps to our tool rail + `tool_group_summary` + `agent_status` rendering — pattern-only, but it validates one activity component over per-feature renderers. **#15942 is the closer parallel**: our `steerQueue` lives on the session lease row while an `@`-mention builds a *second* `Agent`, so "queued follow-up lost at handoff" is a plausible shape for us. **Worth a targeted check** on the steering queue across an `@`-mention turn. + +#### Pricing / quota + +- **✅ BLOCKING QUESTION ANSWERED — GPT-6 Astra's price tier is selected by ACTUAL INPUT TOKENS PER REQUEST, not by the declared context window.** The model card's own table headings are the mechanism: *"Commercial Regions — short context (**272K input tokens or fewer**)"* and *"Commercial Regions — long context (**more than 272K input tokens**)"*. One model ID, one 1,050,000-token window, two per-request rate tables — https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-astra.html — *relevance*: **this inverts last week's Proposal #3.** Registering `maxInputTokens: 272000` is now the wrong call — it would throw away a genuine 1M window to solve a pricing problem pricing does not have. The right shape is the one that proposal only reached as a generalization: keep `maxInputTokens: 1050000` as the **capability** bound and add a separate **cost-tier boundary** field at 272,000 — *unlocks*: the full 1M frontier tier with no capability haircut; a 272K cap becomes an optional **policy** lever (quota/budget), not a correctness requirement. +- **⚠️ The long tier is 2× on input but only 1.5× on OUTPUT — do not extrapolate the multiplier across the row.** Verbatim Geo CRIS: short = $11.00 in / $13.75 cache-write / $1.10 cache-read / **$55.00 out**; long = $22.00 / $27.50 / $2.20 / **$82.50 out**. Input, cache-write and cache-read all double; output does not — same card — *relevance*: a `long = short × 2` helper in `curated-models.ts` would overstate long-context output by 33%. **Encode all four long-tier numbers literally.** +- **Astra is NOT a third `cacheRead = input × 0.1` counterexample — it confirms the default.** $1.10/$11.00 = 0.100× and $2.20/$22.00 = 0.100×; cache-write is 1.25× in both tiers. **Fable 5.1 (0.025×) and Grok 4.6 (0.25×) remain the only two exceptions** — same card — *relevance*: Astra rides the existing `cacheReadMultiplier` default with no override. The **1.100× Regional premium re-confirms** here and the card states it outright: *"Commercial In-Region prices include a 10% fee over OpenAI rates"* (Global $10.00 vs Geo $11.00). +- **⚠️ Astra supports Converse on `bedrock-runtime` — but the caching splits the other way.** The card's API matrix shows Converse **supported** on `bedrock-runtime` (plus Responses and Chat Completions; not Messages or Invoke), while `bedrock-mantle` supports only Responses/Chat Completions. **Prompt caching (implicit and explicit) is listed ONLY under `bedrock-mantle`, Responses API only** — the `bedrock-runtime` feature table does not list it, and also rules out server-side tool use, CountTokens and structured outputs. Cache write is a **30-minute** TTL — same card — *relevance*: a real fork in the road. Our Converse path would *reach* Astra on `bedrock-runtime` but, per this card, **without prompt caching** — which for our harness means re-paying a 30k–150k-token prefix at $11/MTok every turn. The Mantle Responses path (us-west-2 only, matches us) is the one that caches. **Treat "Converse works" as a trap, not a shortcut.** The 30m TTL must not be folded into `cache_ttl_seconds_for()`'s 5m/1h assumptions. +- **The aggregate Bedrock pricing page is stale and unusable for our fleet — model cards remain the only source.** It lists Claude 3.5 Sonnet at $6.00/$30.00 and has **no entries at all** for Haiku 4.5, Sonnet 4.6 or GPT-6 Astra; no Astra tier language, no dated September 2026 changelog — https://aws.amazon.com/bedrock/pricing/ — *relevance*: the same failure mode as the Price List API's 10-stale-Claude-SKUs result, now reproduced on the marketing page. `CLAUDE.md`'s provenance rule should be read as covering this page too. **No quota change verified this week**; Astra's 10× output-token TPM burndown is confirmed unchanged. +- **Carried forward unresolved**: the reference repo's `rejectsTemperature: true` for Astra is **still unverified** — the model card says nothing about temperature or unsupported inference parameters. Flagged last week, still open, **do not copy**. + +#### Community + GitHub issues + +- **AgentCore SDK v1.23.0 (2026-09-11) + v1.23.1 (2026-09-16) shipped in-window; we pin 1.21.0, now two minors behind.** Deltas since our pin: v1.22.0 = payments (MPP/x402); v1.23.0 = `WebSearchClient` + `create_web_search_target()` Gateway helper, a RAGAS eval adapter, and a **runtime shell-session wire-protocol fix** (PR #642); v1.23.1 = the memory bidi-hook fix above. Starter toolkit v0.3.13 (09-15) is a single fix: `aws-opentelemetry-distro>=0.18.0` so unified-telemetry traces render in CloudWatch — https://github.com/aws/bedrock-agentcore-sdk-python/releases — *relevance*: `backend/pyproject.toml:70`; the otel bump touches how our Runtime traces land. +- **Both standing AgentCore items are unchanged in-window.** **#564** (metadata-filtered `ListEvents` eventual consistency silently dropping restored history) still OPEN, last activity **2026-07-20**, zero comments, no linked PR, nothing in the 1.22/1.23.x notes touching the `read_agent()`/`read_session()` gate. **#659** still OPEN, last activity 2026-09-04 (its open date), zero comments, no triage change — https://github.com/aws/bedrock-agentcore-sdk-python/issues/564 · https://github.com/aws/bedrock-agentcore-sdk-python/issues/659 — *relevance*: #564 is our session-restore guard item; #659 would hit us on `reasoningContent.redactedContent` from GPT-family models. **Last week's call that #659 did not survive verification stands** — nothing has landed. +- **Minor AgentCore activity, noted only**: #663 (09-15, open) span helpers / `WaitConfig` / batch insights not reachable from a public import; #661 (09-11, open) port `AgentCoreMemoryStore` from the TS SDK; #402 (bumped 09-17) still no SDK API to **list active runtime sessions**, mirrored by toolkit #498 ("runaway agents burn uncapped costs") — *relevance*: #402/#498 are the upstream gap behind our own lease-row single-flight guard. +- **"Cache-Tax" (HN, 2026-09-18)** — a third-party Claude Code plugin that prices a cold prompt cache and refuses the send once, then arms a `keepwarm` timer sending a tool-less fork every ~50 idle minutes so the server answers from cache. Its stated break-even: a ping ≈ a cache read (~$0.05 on 200k) against ~$4.00 for the cold re-write, and it self-stops when a ping writes ≥1/10 of what it read — https://github.com/karanb192/claude-code-mods/tree/main/plugins/cache-tax — *relevance*: the "keep the prefix warm vs pay the write premium" trade is exactly our `AgentCoreStack/PromptCache` EMF surface; we have **detection** (`cacheStatus`, `partial_miss`) but no **warm-hold** mechanism. ⚠️ **Read for the idea, not for installation** — its own README declares "reach L2, drives Claude" and that it sees every prompt typed. +- **AgentCore/Strands signal on HN was otherwise near-zero.** 504 `agentcore` hits are almost all false positives ("agent core", AgentConnect). The only substantive mention: a commenter noting AgentCore Runtime has no built-in egress filter and that VPC deployment is how traffic rules get applied (https://news.ycombinator.com/item?id=49719728). No reliability or cost regressions being reported publicly. +- **An HN comment (09-18) reports a 12-system agent-memory benchmark scored on accuracy *and token cost per 1,000 questions*** ($256 vs $503 for two systems tied at 97.1/100, the cheaper one ~5× slower) — https://news.ycombinator.com/item?id=49753998 — *relevance*: cost-per-outcome as the benchmark axis matches our cost-effectiveness tenet. ⚠️ **The parent story was not retrievable from the Algolia items API**, so the methodology is unverified — treat the numbers as a claim, not a finding. +- **r/LocalLLaMA and r/MachineLearning: not scanned — Reddit is blocked at the domain level in this environment** (see the 2026-05-18 decisions-log entry). No requests spent. + +#### Cookbook / courses + +- **Only two commits in the window, and they are one change**: the Slack and Sentry `managed_agents` apps moved out of the cookbook into `claude-quickstarts` (`b18fd7f0` → `c245d39f`, 2026-09-16) — https://github.com/anthropics/anthropic-cookbook/commit/b18fd7f0 — *relevance*: any link we hold to those two examples is now stale; the agent-loop patterns **moved repos, they were not deleted**. +- **The `cost_optimization` cookbook did NOT change in-window** — its only commit remains `fa3aa15e` (2026-08-12) — https://github.com/anthropics/anthropic-cookbook/tree/main/cost_optimization — *relevance*: our standing queued audit against the prompt-cache contract is still runnable against exactly the content it was queued for. No re-baseline needed, no new upstream guidance to reconcile. + +#### Seasonal + +Out of window — none scanned this week. (re:Invent is late Nov / early Dec; no conference proceedings due.) + +### Patterns worth considering + +- **Lazy tool/skill discovery, scoped to the session rather than the turn.** Both the reference repo (#294) and opencode (#49276) shipped catalog laziness this week, and the reference repo measured a 60–85% cut in first-token latency from it. Our `_build_filtered_tools()` is eager. + - **Where**: ref-repo `skill_registry.add_tool_loader()`; opencode's revision-keyed catalog reuse. + - **Fit**: the latency win is real and we have the same eager cost. **The cost trap is the difference between session-scoped and turn-scoped.** Their base prompt now tells the model tools may change *between turns in one conversation*, which for us is a `toolConfigHash` rewrite at 1.25× base input over a 30k–150k prefix. Defer discovery for tools **not in this session's `enabled_tools`** and nothing else. + - **Verdict**: **Worth trying**, strictly session-scoped. + +- **Side channels as a user-facing surface, not just an internal one.** opencode's `/btw` and LibreChat's trace viewer both take machinery we already have (a concurrent model call on its own messages; per-call context accounting) and put it in front of the user. + - **Where**: opencode #49646; LibreChat #15796/#15869. + - **Fit**: our `tool_summaries/summarizer.py` and `session_title` are exactly this shape and already prove the pattern is free against the prefix. We have the plumbing and none of the surface. + - **Verdict**: **Monitor** — a good candidate once the document-offload measurement window closes, not while three epics are mid-flight. + +- **Prompt-cache invalidation as an auditable bug class.** Four Claude Code fixes in one week, all "a non-content side event mutated the prefix". We have the detection layer (`systemPromptHash` / `toolConfigHash` / `historyHash`, `partial_miss`) that most of the ecosystem lacks — what we lack is a *sweep*. + - **Where**: Claude Code 2.1.269–2.1.275. + - **Fit**: the age-note variant is a category we have never checked. Our own daily date boundary is the known remaining instance. + - **Verdict**: **Worth trying** — it is a grep plus a test, and the instrument to prove it already exists. + +## Internal Audit + +### Activity (last 7 days) + +- **Commits on develop**: **146** non-merge commits across **91 merge commits** — **33 PRs merged** into `develop` (#1104 → #1149). This is one of the busiest weeks on record. +- **By area** (files touched): `frontend/ai.client` 861, `backend/src` 345, `backend/tests` 173, `docs/specs` 49, `backend/scripts` 25, `infrastructure/{lib,test}` 30, `.github/workflows` 11, `.claude/skills` 11. +- **PRs opened and still open**: 3 — [#1148](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1148) (feedback eval sampling, 0 comments), [#1147](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1147) (PDF-aware `documentTokens`, 1 comment), [#1146](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1146) (implicit feedback signals, 0 comments). **Reverts: 0.** +- **Issues opened: 0. Issues closed: 0.** The team is shipping through PRs, not tracking through issues — the review queue is doing that job instead (see below). +- **CI (workflow → count)**: `CI` ran 60 times in-window — **54 success, 5 failure, 1 cancelled (90% green)**. `Nightly Build & Test`: **7 runs, 7 green** (2026-09-12 → 09-18). + +### Repeated friction signals + +- **✅ RESOLVED — last week's #1 shipped, both halves.** `.github/workflows/ci.yml:34-38` now runs the frontend suite a second time under `run_frontend_coverage: true` — "the nightly's invocation", per the comment in the file — and the nightly has been green for seven consecutive nights after seven consecutive failures. The class of failure that PR CI was *structurally incapable* of catching is now caught on PRs. Nothing further needed; recorded so review-prep can resolve the queue entry. + +- **The new coverage-build job is now the most frequent CI failure — and mostly for the right reason** (4 of 5 in-window failures). Evidence: runs [35168191840](https://github.com/Boise-State-Development/agentcore-public-stack/actions/runs/35168191840), [35148364997](https://github.com/Boise-State-Development/agentcore-public-stack/actions/runs/35148364997), [35058101120](https://github.com/Boise-State-Development/agentcore-public-stack/actions/runs/35058101120), [35046168467](https://github.com/Boise-State-Development/agentcore-public-stack/actions/runs/35046168467). + - **Hypothesis**: three of the four are **real**, not coverage-specific — the regular `Test frontend` job failed identically alongside it with the same assertion (`file-preview-panel.component.spec.ts > keeps download reachable when the document cannot be rendered`, `expected ' plan.docx Word document…' to contain "couldn't be read as a Word document"`). Only **one** run (35046168467) failed in the coverage job alone. + - **Fix candidate**: none needed for the gate itself — it is working. The signal to record is that the gate **doubles frontend CI wall-clock on every PR** and its unique catch rate so far is 1 in 5. Worth revisiting the cost/benefit once it has a month of data, not now. + +- **⚠️ A forward-risk the new coverage job surfaced and nothing tracks**: every frontend CI run now prints *"Promise returned by `expect(actual).rejects.toThrow(expected)` was not awaited. Vitest currently auto-awaits hanging assertions at the end of the test, but **this will cause the test to fail in the next Vitest major**."* There are **6 unawaited `.rejects.*` assertions** across the SPA suite (of 53 total uses) — `fine-tuning-upload.service.spec.ts:104,113,123`, `file-upload.service.spec.ts:277`, `chat-http.service.spec.ts:183,208`. Each is a spec that will silently stop asserting, or start failing, on a vitest 4→5 bump. **A one-line `await` per site removes a migration blocker before the migration exists.** + +- **The review queue is growing faster than it is being resolved, and it now contains items that already shipped.** 53 entries under `## Open` against 42 under `## Resolved`; the oldest open entry dates to **2026-05-10 (131 days)**. Verified concrete example: **[2026-05-15] "Wire per-tool `duration_ms` into `tool_result` SSE"** is still marked `open`, but it **shipped** — `session/hooks/agent_status.py:266,279` emits `durationMs` on `tool_end`, the SPA types it at `stream-parser-types.ts:363`, and `tool-rail.component.ts:100` sums it. Six entries were added on 2026-09-16 and six on 2026-09-11; at that rate the queue's signal decays into a backlog. This is a **subtraction** opportunity for review-prep, which owns the `## Resolved` move. + +- **`DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` defaults to 100, which is the evaluation spec's own "Ship" state — while that spec is still Draft with no harness and its quality gate written as a veto.** Raised by the author on [#1147](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1147) and explicitly flagged as *time-sensitive against the release, not against Friday*. The `crc32(session_id) % 100` bucket exists precisely to give that evaluation concurrent arms; at 100% in prod there is no control arm and the only comparison left is before/after across a release boundary confounded by everything else in the same release. **Both choices are defensible; defaulting into one is not.** Carried here so it is visible in the Friday agenda, but the decision belongs to the release, not to kaizen. + +### Version-pin lag + +> `pydantic` and `botocore` are transitive-only (no pin in `pyproject.toml`); versions shown are the `backend/uv.lock` resolutions. There is **no `@analogjs/platform` pin** — Analog is present as `@analogjs/vite-plugin-angular` and `@analogjs/vitest-angular`, both on the `3.0.0-alpha` train. + +| Dep | Pinned | Latest | Lag | Notes | +|---|---|---|---|---| +| `strands-agents` | 1.55.0 | 1.56.0 | 2 releases / 7d | ⛔ **Not a routine minor — see Risks.** 1.56.0 deletes three `Bidi*` hook events with no alias and breaks `bedrock-agentcore` ≤1.23.0 at import. **Paired pin with `bedrock-agentcore>=1.23.1` or don't bump.** Upside once paired: the first-party context-manager/offloading stack, OpenAI prompt-cache key auto-derivation, and 1.55.1's `fix(models): resolve nested Bedrock model prefixes` (relevant to our `us.*`/`global.*` CRIS ids). | +| `strands-agents-tools` | 0.8.8 | 0.8.9 | 1 / 11d | ⛔ **Do not bump blind.** 0.8.9 is `fix: update mcp_client tool to use MCP 2.x client` — it assumes `mcp` 2.x, which our `constraint-dependencies = ["mcp<2"]` forbids. A one-line version bump becomes the MCP v2 migration. Also deprecates `http_request`, `tavily`, `exa`, `video`, `bright_data` and `journal` (extends the known `memory`-deprecated finding). | +| `boto3` | 1.43.68 | 1.43.97 | 29 / 38d | Patch-only; daily API-model updates, no breaking changes by policy. Routine. | +| `botocore` | 1.43.68 *(transitive)* | 1.43.97 | 29 / 38d | Lockstep with boto3. Nothing to decide separately. | +| `fastapi` | 0.136.1 | 0.141.1 | 26 / 97d *(pin is 148d old)* | ⚠️ **The riskiest unforced gap.** Pre-1.0, so "minor" can break: **0.137.0 is a declared breaking change** — internals refactored to preserve `APIRouter`/`APIRoute` instances instead of cloning them on `include_router`, which is the exact pattern `app_api` is built from (`/admin//`, per-domain routers). Gets harder the longer it sits. Upside: 0.140.0 is `⚡️ Reduce memory usage in dependencies` — a real win for a per-request-DI service. 0.138/0.141's `app.frontend()` is irrelevant (SPA ships via S3 + CloudFront). | +| `bedrock-agentcore` | 1.21.0 | 1.23.1 | 3 / 41d | Minor, additive, **and the required half of the Strands 1.56.0 pair.** 1.23.0 adds a RAGAS eval adapter, `create_web_search_target()` and a `WebSearchClient` (both usable by our Gateway/tool layer) plus `fix(runtime): update shell session wire protocol`. 1.22.0 is payments (MPP/x402) — not ours. Low-risk with real feature pickup. | +| `pydantic` | 2.12.5 *(transitive)* | 2.13.5 | 6 / 275d | Minor. 2.13 updates the `pydantic.v1` namespace and bumps jiter to 0.14.0 for a **musl-Linux segfault fix** — worth knowing which base image the inference-api container uses. Moves on a `uv lock` refresh, not a manifest edit. | +| `mcp` | 1.28.1 *(transitive, `mcp<2` constraint)* | 2.2.0 *(1.x tail: 1.30.0)* | 8 / 73d overall; 3 inside `<2` | ⚠️ **2.x is now the default `pip install mcp`; 1.x is maintenance-only, security fixes.** Our constraint is the only thing holding the line — and note the headroom: Strands declares `mcp>=1.23.0,<2.2`, so **2.0.0–2.1.1 sit inside Strands' range** and only our own constraint blocks them. Even inside `<2`, 1.30.0 changes behavior on our path: redirects followed only within the endpoint origin, and idle Streamable-HTTP sessions expiring at 30 min with a 10k cap. | +| `openpyxl` | 3.1.5 | 3.1.5 | 0 | **Current.** Latest stable is still the 2024-06-28 release; only a `3.2.0b1` prerelease exists. | +| `@angular/core` | 21.2.19 | 22.1.7 *(v21 LTS tail: 21.2.23)* | 21 stable / 49d | ⚠️ **Angular 22 is a migration, not a bump** — `@angular/compiler-cli@22` requires `typescript >=6.0 <6.1`, forcing the TS major, *and* our Analog pin caps `@angular-devkit/architect` at `<0.2200.0` (Angular 21 only). The free move today is **21.2.19 → 21.2.23**, four patches on the LTS line. | +| `vitest` | 4.1.5 | 5.0.1 *(v4 tail: 4.1.11)* | 8 stable / 147d | ⛔ **Major = migration, and currently blocked**: `@analogjs/vitest-angular` peers `vitest ^1 \|\| ^2 \|\| ^3 \|\| ^4` even at alpha.87, so vitest 5 has no Analog support yet. When it lands: 30+ breaking changes incl. **Node 22 + Vite 6.4 minimum**, **mocks cleared by default before each test**, `@vitest/runner`/`expect` inlined and unpublished, config no longer resolved from ancestor directories (worktree-sensitive), inline projects extending the root config. Several intersect the known SPA flake cluster — could fix it or reshuffle it. **And see the 6 unawaited `.rejects.*` assertions in Repeated friction signals — those fail on this major.** Reachable move: **4.1.11**. | +| `typescript` | 5.9.3 | 7.0.2 | 3 stable / 281d | **TS 7 is unreachable regardless of appetite** — Angular 22's compiler-cli peers `>=6.0 <6.1`, so no Angular release supports TS 7 yet. **TS 6.0.3 is already in range for our Angular 21** (`>=5.9 <6.1`), making it the only real candidate and a standalone major migration under strict mode. | +| `tailwindcss` | 4.2.4 | 4.3.3 | 4 / 86d | **Lowest-risk bump on the board.** Minor, additive, no breaking changes. Genuinely useful here: `scrollbar-*` / `scrollbar-gutter-*`, `@variant` with stacked and compound variants, `@container-size`, `tab-*`, `zoom-*`. Also fixes relative `@import`/`@plugin` resolution under `@tailwindcss/vite` and ensures `@variant`-containing CSS is processed — both plausible latent bugs in our setup. | +| `@analogjs/vite-plugin-angular` | 3.0.0-alpha.53 | 3.0.0-alpha.87 | 34 alphas / 134d | We are on a **pre-release train** with no stable target — `latest` (2.7.2) is *behind* our alpha line. alpha.87 is what unlocks Angular 22 (`@angular/build ^22.0.0`). Pinning an alpha is a standing risk: no semver guarantees across those 34 builds. | +| `@analogjs/vitest-angular` | 3.0.0-alpha.30 | 3.0.0-alpha.87 | 57 alphas / 150d | ⚠️ **The largest relative gap in the repo, and the structural one.** This single package gates **both** Angular 22 (`architect <0.2200.0` at our pin vs `<0.2300.0` at alpha.87) and vitest 5 (peer still caps at `^4`). The frontend's entire major-version runway sits behind one unpinned-upstream alpha we don't control. | +| `aws-cdk-lib` | 2.265.0 | 2.270.0 | 5 / 35d | **The cheapest high-value bump available**, and verified safe: 2.268.0 removes the `AWS::BedrockAgentCore::Browser`, `::CodeInterpreter` and `::TokenVault` L1s, and we use **none** of them (only `CfnBrowserCustom`, `CfnCodeInterpreterCustom`, `CfnMemory`, `CfnRuntime`, `CfnWorkloadIdentity`). Gains: 2.269.0 graduates the **bedrockagentcore Policy submodule to stable** and fixes "removing the last target group from a service is not deployed" (ECS); 2.267.0 adds `manageDeliveryResourcePolicy` opt-out for runtime observability and fixes `TableV2.grants.*Data` omitting index resources (**DynamoDB GSI grants** — pointed, given our GSI history); 2.270.0 adds Lambda `DirectS3Read`. Since it bundles its deps, the lag is missing surface, not transitive risk. | +| `constructs` | 10.6.0 | 10.8.1 | 5 / 133d | Minor within the CDK-v2-compatible 10.x line. Low risk, but move it **together with `aws-cdk-lib`** — they are a matched pair. | +| `aws-cdk` (CLI) | 2.1128.0 | 2.1142.0 | 17 / 91d | Minor, CLI-only — no synth-output change by itself. ⚠️ **npm lists a `3.0.0`; it is a deprecated accidental publish** ("This version was published accidentally. Please use 2.x.x instead.", 2025-04-09). There is no CDK CLI v3 — worth knowing before an automated tool "helpfully" offers it. | + +**Where the lag actually costs something.** Two pins: **`fastapi`** (26 releases, a 148-day-old pin, with a declared breaking change at 0.137.0 landing on the exact router-nesting pattern `app_api` is built from — the longer it sits the worse the crossing gets), and **`@analogjs/vitest-angular`** (57 alphas behind, single-handedly gating both Angular 22 and vitest 5). Two need a *decision* rather than a bump: **`strands-agents` 1.56.0** (paired pin — see Risks) and **`strands-agents-tools` 0.8.9** (silently an MCP v2 migration). Four are genuinely cheap: **`aws-cdk-lib` 2.270.0**, **`tailwindcss` 4.3.3**, and the free tails on **`@angular/core` 21.2.23** and **`vitest` 4.1.11**. + +**⚠️ Incidental, and worth fixing today: local `infrastructure/node_modules` is stale.** `infrastructure/node_modules/aws-cdk-lib/package.json` reports **2.251.0** while both `infrastructure/package.json` and `infrastructure/package-lock.json` say **2.265.0** — verified. CI runs `npm ci` fresh so the pipeline is unaffected, but **local `cdk synth` and `npx jest` are not testing the pinned version**, which is exactly where a CDK change gets validated before it is pushed. One `npm ci` in `infrastructure/` fixes it. + +### Retirement candidates + +- **`.claude/skills/angualar-best-practices/`** — last commit **2026-04-27 (144 days)**. The directory name is **misspelled** ("angualar") while the frontmatter inside correctly says `name: angular-best-practices`, so the two disagree. This is the skill most likely to be wanted on a repo that touched 861 frontend files this week. ⚠️ **This has now been raised in five consecutive research docs** (2026-06-12, 06-19, 08-14, 09-11, and here) and never actioned — which makes the *recommendation* itself a retirement candidate. Either rename it in a two-minute PR or log a Decline in `decisions.md` so it stops costing a paragraph every Friday. +- **`.claude/skills/frontend-design/`** and **`.claude/skills/cors-deployment/`** — both last committed **2026-04-27 (144 days)**, neither referenced in any PR in the window. `frontend-design` additionally duplicates a plugin skill of the same name available in-session. Same five-week history; same recommendation — decide, don't re-raise. +- **`.claude/skills/kaizen-review-prep/SKILL.md`** — last commit **2026-05-10 (131 days)**, while `kaizen-research` was updated 2026-09-11. The two are a contract; only one side has moved in four months. Worth a read-through for drift rather than retirement. +- **Stray root `package-lock.json`** — untracked, 6 lines, `"packages": {}`. Someone ran `npm install` at the repo root, where no npm project lives (they are in `frontend/ai.client` and `infrastructure`). Delete it or `.gitignore` it; it will otherwise get committed by accident. +- **Stale queue entries that already shipped** — see the friction signal above; `[2026-05-15] duration_ms` is verified-delivered and should move to `## Resolved`. + +### Risks introduced this week + +- **⛔ A Strands 1.56.0 bump without `bedrock-agentcore>=1.23.1` breaks the inference-api container at import, and it will look like a cold-start timeout.** — https://github.com/aws/bedrock-agentcore-sdk-python/issues/665 — *what breaks if we ignore this*: `bedrock_agentcore.memory.integrations.strands.session_manager` raises `ImportError` at module load; on AgentCore Runtime that surfaces as **"Runtime initialization time exceeded (30s)" → 502**, so the first hour of debugging goes to cold-start tuning rather than the pin. Three import sites: `apis/shared/sessions/messages.py:20`, `apis/app_api/shares/service.py:392`, `agents/main_agent/session/turn_based_session_manager.py:40`. +- **⚠️ A routine `idna` security bump silently unblocks the `mcp` 1.x → 2.x major crossing.** — `backend/pyproject.toml:120-132` — *what breaks if we ignore this*: `mcp` 2.x removes the `ClientSession` layering our `mcp_apps.py` monkeypatch subclasses, moves protocol types to a separate `mcp-types` distribution, and switches the transport to `httpx2`. The `[tool.uv] constraint mcp<2` guard rail holds — but the comment itself notes that the *effective* block is `idna==3.15`, and **nothing in an `idna` Dependabot diff would say "MCP"**. Worth a CI assertion that the resolved `mcp` version stays on the 1.x line, so the guard is enforced rather than documented. +- **⚠️ FastMCP 4.x changes `Context.session_id` on every tool call**, breaking any Gateway-backed server keying per-session state off it — https://github.com/PrefectHQ/fastmcp/issues/5134 — *what breaks if we ignore this*: a server repo upgraded past 3.4.x silently loses session state between calls. Pairs with MCP SDK v2.2.0's new 30-minute idle Streamable-HTTP expiry — two independent session-identity regressions in one dependency line. **Grep the server repos before anyone upgrades them.** +- **⚠️ FastMCP 4.0.4's `OAuthProxy` now rejects self-contained ID-JAG tokens unless identity assertion is configured** — https://github.com/PrefectHQ/fastmcp/releases/tag/v4.0.4 — *what breaks if we ignore this*: a working deployment relying on ID-JAG without `identity_assertion` starts refusing tokens on upgrade. A behavior tightening, not a bug fix, and it is not called out as breaking. +- **⚠️ Adding a `cache_config` to the Responses/Mantle path would silently disable our prompt-cache key.** Upstream's `strands-` lands in `request` first inside `super()._format_request`, making our `request.setdefault("prompt_cache_key", ...)` at `bedrock_responses.py:259` a no-op — swapping a prefix-derived, cross-session-shared key for a per-session key that **fragments the cache**. *What breaks if we ignore this*: nothing today (we set no `cache_config` there); the day someone adds one, cache hit rate drops and no test fails. +- **⚠️ An exception escaping our `AfterToolsEvent` steering callback would drop the completed `toolResult` and re-run the tool on resume** — https://github.com/strands-agents/harness-sdk/issues/4338 — *what breaks if we ignore this*: a duplicated tool execution, which is a direct token-cost bug on top of a correctness one. Our commit-on-append design means we do not *cause* it; one defensive `try/except` audit closes the last path. + +## Ideas — Top 5 (ranked) + +| # | Idea | Surface | Effort | Impact | Subtracts? | Unlocks? | +|---|---|---|---|---|---|---| +| 1 | Reverse the Astra decision — register at the **full 1M window** and make the price tier its own catalog field | frontend / backend | **L** | **H** | yes — retires a queued recommendation that would have cost us a capability, and the `long = short × 2` assumption | a genuine 1M frontier tier with no haircut; a tier-aware catalog | +| 2 | Make the Strands 1.56.0 bump a **paired pin** with `bedrock-agentcore>=1.23.1`, and add a CI guard that the resolved `mcp` stays on 1.x | backend / CI | **L–M** | **H** | yes — converts two documented-only guard rails into enforced ones | the released first-party context-manager stack becomes pinnable | +| 3 | Sweep the cacheable prefix for **derived and relative values**, and close the last daily boundary | backend | **L–M** | **M–H** | yes — removes the final scheduled prefix rewrite and a whole undetected category | a bug class our fingerprint hashes can now *prove* absent | +| 4 | Fix the MCP Apps header's tool display name (`title` → `annotations.title` → `name`) | backend / frontend | **L** | **M** | no — addition, justified because it corrects a persisted wrong string | spec-conformant App headers that match the tool rail beside them | +| 5 | Defer external MCP tool discovery to **session-scoped** laziness — and add the sidebar conversation search while in the SPA | backend / frontend | **M** | **M–H** | yes — retires the eager load of every enabled external MCP client at agent build | a 60–85% first-token latency cut (their measurement), and a search affordance we lack entirely | + +### 1. Reverse the Astra decision — register at the full 1M window and make the price tier its own catalog field + +- **Source**: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-astra.html — **directly supersedes the open `[2026-09-11] Add GPT-6 Astra at the 272K tier` queue entry**, whose blocking question this answers in the opposite direction. Merge them at review. +- **Surface area**: `frontend/ai.client/src/app/admin/manage-models/models/curated-models.ts`, the Mantle Responses path (`_create_mantle_model` / `bedrock_mantle_config`), `apis/app_api/admin/services/model_access.py` for the role grant, and the model picker. +- **Change**: (a) register `us.openai.gpt-6-astra` at `maxInputTokens: 1050000` — the capability bound — and add a **separate cost-tier boundary field** at 272,000. (b) Encode **all four** long-tier rates literally ($22.00 / $27.50 / $2.20 / **$82.50**), because output is 1.5×, not 2×, and a `× 2` helper would overstate it by 33%. (c) Take the **Mantle Responses** path, not Converse: per the card, prompt caching is listed only under `bedrock-mantle`/Responses, and `bedrock-runtime` Converse would re-pay the whole prefix at $11/MTok every turn. (d) Leave `cacheReadMultiplier` at the default — Astra is 0.100× in both tiers and is *not* a third counterexample. +- **Subtracts**: a queued recommendation that would have shipped a capability regression; the `long = short × 2` mental model; and the lingering ambiguity about which API surface to reach Astra through. +- **Unlocks**: a genuine **1M-token tier** — the first model in the catalog where a whole corpus fits in one turn — with no artificial cap; and a **tier-aware catalog**, so the admin page can show the boundary, cost projections can respect it, and Astra's 10× output-token quota burn can be accounted for against the known campus TPM ceiling. The 272K cap survives as an optional *policy* lever. +- **Effort × Impact**: **Low × High** +- **Verdict**: **Worth trying — recommended #1.** It is the cheapest item in the scan and the only one where acting on last week's doc would actively make things worse. ⚠️ Two constraints carry forward unresolved: the ref repo's `rejectsTemperature: true` is **still not on the model card** — verify, don't copy; and Astra's cache-write SKU is a **30-minute** TTL, which must not be folded into `cache_ttl_seconds_for()`'s 5m/1h assumptions. ⚠️ `bedrock-mantle` is us-west-2 only (matches us), and on `bedrock-runtime` Astra supports no server-side tool use, no structured output and no CountTokens. + +### 2. Make the Strands 1.56.0 bump a paired pin — and enforce the two guard rails we currently only document + +- **Source**: https://github.com/aws/bedrock-agentcore-sdk-python/issues/665 · https://github.com/strands-agents/harness-sdk/issues/4362 · `backend/pyproject.toml:120-132` +- **Surface area**: `backend/pyproject.toml:66,70,81`, `backend/uv.lock`, a new supply-chain test alongside `tests/supply_chain/test_nightly_ref_allowlist.py`, and the three `bedrock_agentcore...strands.session_manager` import sites. +- **Change**: (a) when we bump, bump **both** — `strands-agents==1.56.0` **and** `bedrock-agentcore>=1.23.1` in one PR; neither alone resolves. (b) Add a test asserting the resolved `mcp` version stays on the 1.x line, so the `[tool.uv] constraint mcp<2` guard rail is *enforced* rather than described — today the effective block is the incidental `idna==3.15` pin, and an `idna` Dependabot bump would cross the major with nothing in the diff naming MCP. (c) Before bumping, read three 1.56.0 deltas against our code: `SessionManager` becoming `Generic[_SessionAgentT]` and dropping its Bidi registrations; `InterruptException` now bubbling **unwrapped** out of `event_loop_cycle` (our OAuth-consent / tool-approval / `ask_user_question` pause paths); and `agent_metadata` added positionally to `_format_request` — which our `*args, **kwargs` override already absorbs. +- **Subtracts**: converts two guard rails that exist only as prose into assertions CI can fail on; and closes the two-release lag on a pin whose gap now contains a runtime-breaking interaction. +- **Unlocks**: the first-party Strands **context-manager / offloading stack is fully released in 1.56.0** (#4118/#4146/#4187/#4231/#4254/#4282, and `ContextOffloader` manual offloading closed as released) — it becomes pinnable and evaluable against ours, with #4367's `NAMESPACED` stash sentinel offering **cross-session offload sharing** for the document-offload work. +- **Effort × Impact**: **Low–Medium × High** +- **Verdict**: **Worth trying**, second only to #1. The evidence is complete and reproduced by two independent sources this week. ⚠️ **Do not treat this as a routine dep bump** — the failure mode on Runtime is a 502 that reads as a cold-start timeout, so a bad bump costs a debugging session before it costs a rollback. ⚠️ The standing decisions-log entry still applies: adopting the upstream context manager is **out of scope** without a migration design covering tool-content truncation, LTM summary retrieval, DynamoDB checkpoint persistence and the `compaction` SSE-once invariant. This idea makes it *evaluable*, not adopted. + +### 3. Sweep the cacheable prefix for derived and relative values — and close the last daily boundary + +- **Source**: Claude Code 2.1.275 (*"a restored memory file's age note changing between requests after a compaction or resume, which caused prompt cache misses"*) and 2.1.269/2.1.273 — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md — plus our own `9f246cb7`. +- **Surface area**: `agents/main_agent/utils/` (`get_current_date_pacific()` and the `SystemPromptBuilder` tail), `session/turn_based_session_manager.py` (the truncation anchor), `session/document_rehydration.py`, `apis/shared/sessions/messages.py`, and the interrupt-resume path in `inference_api/chat/routes.py`. +- **Change**: (a) a **grep-plus-test sweep** for any *derived or relative* value that can reach the system prompt or restored history — relative timestamps, "N days ago", recomputed ages, sizes recomputed at read time. I checked the obvious paths and found none today; the point is to make that a test rather than an observation, because it is exactly the kind of thing a future feature adds without noticing. (b) Extend the sweep to the resume path specifically, where Claude Code found two of its four bugs. (c) Decide the remaining **daily** boundary: `9f246cb7` took the date line from 24 rewrites a day to 1, but a session open across Pacific midnight still re-pays the whole prefix once — either drop the date from the cacheable segment entirely, or place it after the last cachePoint. +- **Subtracts**: the final *scheduled* prefix rewrite, and an entire category of invalidation that currently has no detector even though we already own the instrument to detect it. +- **Effort × Impact**: **Low–Medium × Medium–High** — a grep, a test, and one placement decision. +- **Verdict**: **Worth trying.** The unusual thing here is that we are ahead of the ecosystem on *detection* (`systemPromptHash` / `toolConfigHash` / `historyHash` on every `C#` row, plus `partial_miss`) and behind on *sweeping*. ⚠️ Pair it with Claude Code's other finding — the context meter double-counting advisor-tool turns, which fired auto-compact at half the real window — by asserting our per-turn context size for tool-heavy turns tracks CountTokens. **Over-counting is a pure cost bug and it fails silently.** ⚠️ CountTokens rejects `us.*` ids, so that assertion needs a `global.*` id or an alternative estimator. + +### 4. Fix the MCP Apps header's tool display name — `title` → `annotations.title` → `name` + +- **Source**: https://github.com/modelcontextprotocol/ext-apps/pull/778 (opened 2026-09-18, fixing ext-apps issue #767) — **narrows the open `[2026-09-11] Retire the ClientSession monkeypatch via _meta serverInfo` entry**; this is an independent, shippable piece of the same header. +- **Surface area**: `backend/src/apis/shared/mcp_apps/ui_resource_store.py:215`, `card_store.py:160`, the `toolName` derivation feeding both the header-only shell at `content_block_start` and the full html-bearing `ui_resource` emit, and the App frame header component in the SPA. +- **Change**: derive the displayed tool name by the base MCP schema's precedence — `title`, then `annotations.title`, then `name` (for `Tool` specifically, `annotations.title` outranks `name`). Today we write the raw agent-facing `tool_name` straight through; **verified: `apis/shared/mcp_apps/` contains no `title` or `annotations` handling at all.** +- **Subtracts**: no — addition. Justified because the wrong string is **persisted with the resource**, so it survives reload: the App frame header shows `raw_tool_name` while the tool rail immediately beside it shows the friendly title, and the two disagree permanently. +- **Unlocks**: spec-conformant App headers, on the same precedence every other MCP host is converging to — and it is a prerequisite for the eventual `_meta serverInfo` migration rather than competing with it. +- **Effort × Impact**: **Low × Medium** +- **Verdict**: **Worth trying.** Small, self-contained, and confirmed against our tree rather than inferred. ⚠️ The upstream PR is **open, not merged**, so treat it as evidence of the correct precedence (which comes from the base schema, not the PR) rather than as a shipped standard. ⚠️ While in this file, note that `ext-apps` v2.0.0 changed host-facing error codes (`-32002` → `-32602`, `-32603` → `-32602`) and removed the `MCP error N:` message prefix — the App-error chain from #1009/#1013 was built substantially on message text and deserves a grep in the same PR. + +### 5. Defer external MCP tool discovery to session scope — and add the sidebar conversation search while in the SPA + +- **Source**: https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/294 (lazy loaders; measured "Starting" 10.417s → 4.081s, follow-up 7.377s → 1.117s) and https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/296 (sidebar search); corroborated by opencode #49276 (revision-keyed catalog reuse) and #49387 (skip warming for subagents). +- **Surface area**: `backend/src/agents/main_agent/base_agent.py:478` (`_build_filtered_tools`), `integrations/external_integration.py` and `gateway_integration.py`; and separately `frontend/ai.client/src/app/components/sidenav/components/session-list/session-list.{ts,html}`. +- **Change**: (a) stop loading every enabled external MCP client at agent build. Defer discovery for tools **not in this session's `enabled_tools`** — a set that is stable for the session — and leave the in-session set eager. (b) Confirm we make no equivalent keep-alive/warm-up call when an `@`-mention builds a second `Agent` (opencode #49387 found exactly that cost per subagent). (c) Independently, add a client-side "Search chat titles" filter to the session list: live result count, date grouping suppressed while filtering, "No matching chats" distinguished from "No conversations yet", Escape to clear, query preserved across refreshes — plus `inert` on the off-canvas panel while closed. +- **Subtracts**: the eager load of every enabled external MCP client at agent build, on every agent construction. +- **Unlocks**: their measured **60–85% first-token latency cut** on new and follow-up conversations; and a conversation-search affordance we have **none** of today — `session-list.ts` groups by date with no filter at all, and it costs **zero tokens** (a pure client-side filter over already-loaded titles). +- **Effort × Impact**: **Medium × Medium–High** +- **Verdict**: **Worth trying**, with one hard boundary. ⚠️ **Session-scoped only — never turn-scoped.** Their implementation goes further than this proposal: their new base prompt tells the model *"available tools may change between turns within the same conversation"*, accepting a `toolConfig` that varies per turn. For us that is a `toolConfigHash` rewrite at 1.25× base input over a 30k–150k-token prefix, every turn — the latency win would be paid for many times over in cache writes. ⚠️ Their registry fix (module global → per-agent, resolved off `tool_context.agent`) is the same bug class as our "never cache session state on an agent instance" rule; worth reading before touching this code. ⚠️ The two halves are independent — the SPA search can ship alone if the backend half needs more design. + +## Take + +The system is trending **toward** the ecosystem, and this week it is the ecosystem arriving at conclusions we had already written down: Skills over MCP went Final with a lazy-read, manifest-bound-approval design that restates our own cost tenet almost clause for clause; opencode shipped attachments-by-path, which is our document-offload epic with the `DOCUMENT_READ_ENABLED` coupling made structural instead of remembered; and Claude Code spent a week fixing prompt-cache invalidation bugs that are #741 and #751 under other names. The uncomfortable finding is that our own recommendation from last week was wrong — the Astra price tier is per-request, not per-declared-window, so the `maxInputTokens: 272000` cap queued as "ship" would have thrown away a 1M-token capability to solve a problem that does not exist. Fix that first; it is the cheapest item here and the only one where last week's doc is actively misleading. What Phil would notice first if shipped: GPT-6 Astra in the model picker with its real window, and — on the next cost-anatomy read — a `systemPromptHash` that finally holds steady across a midnight boundary. + +--- + +## Sources Scanned + +| # | Source | URL | Accessed | Items | +|---|---|---|---|---| +| 1 | AWS What's New (RSS) | https://aws.amazon.com/about-aws/whats-new/recent/feed/ | 2026-09-18 | 1 | +| 2 | AWS ML blog (RSS) | https://aws.amazon.com/blogs/machine-learning/feed/ | 2026-09-18 | 6 | +| 3 | AgentCore devguide release notes | https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/release-notes.html | 2026-09-18 | 0 (newest entry 2026-09-03) | +| 4 | Strands `harness-sdk` releases / commits / issues / source | https://github.com/strands-agents/harness-sdk | 2026-09-18 | 9 | +| 5 | Reference repo commits + PRs #294, #296 | https://github.com/aws-samples/sample-strands-agent-with-agentcore | 2026-09-18 | 6 | +| 6 | MCP spec repo (SEP-2640, 2643, 2636, 2631) | https://github.com/modelcontextprotocol/modelcontextprotocol | 2026-09-18 | 5 | +| 7 | MCP `ext-apps` + `ext-skills` | https://github.com/modelcontextprotocol/ext-apps | 2026-09-18 | 2 | +| 8 | MCP Python SDK releases | https://github.com/modelcontextprotocol/python-sdk/releases/tag/v2.2.0 | 2026-09-18 | 1 | +| 9 | MCP blog | https://blog.modelcontextprotocol.io | 2026-09-18 | 0 (latest 2026-08-22) | +| 10 | FastMCP (**moved to PrefectHQ**) | https://github.com/PrefectHQ/fastmcp | 2026-09-18 | 5 | +| 11 | assistant-ui releases | https://github.com/assistant-ui/assistant-ui | 2026-09-18 | 3 | +| 12 | Vercel AI SDK releases | https://github.com/vercel/ai | 2026-09-18 | 1 | +| 13 | Nielsen Norman Group — AI | https://www.nngroup.com/topic/artificial-intelligence/ | 2026-09-18 | 1 (not written up) | +| 14 | Linear blog | https://linear.app/blog | 2026-09-18 | 0 (latest 2026-09-10) | +| 15 | Anthropic news | https://www.anthropic.com/news | 2026-09-18 | 1 (no capability delta) | +| 16 | Anthropic engineering | https://www.anthropic.com/engineering | 2026-09-18 | 0 (latest 2026-04-23) | +| 17 | OpenAI news | https://openai.com/news/ | 2026-09-18 | **HTTP 403 — not retrieved** | +| 18 | Google DeepMind blog | https://blog.google/technology/google-deepmind/ | 2026-09-18 | 0 (index incomplete — not verified) | +| 19 | Meta AI blog | https://ai.meta.com/blog/ | 2026-09-18 | 0 | +| 20 | GPT-6 Astra model card | https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-astra.html | 2026-09-18 | 4 | +| 21 | Bedrock pricing page | https://aws.amazon.com/bedrock/pricing/ | 2026-09-18 | 1 (stale — recorded as such) | +| 22 | Claude Code CHANGELOG | https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md | 2026-09-18 | 4 | +| 23 | pydantic-ai / LangChain / LlamaIndex releases | https://github.com/pydantic/pydantic-ai/releases/tag/v2.44.0 | 2026-09-18 | 1 | +| 24 | opencode releases + merged PRs | https://github.com/anomalyco/opencode | 2026-09-18 | 5 | +| 25 | LibreChat releases | https://github.com/danny-avila/LibreChat/releases/tag/v0.8.8-rc3 | 2026-09-18 | 4 | +| 26 | AgentCore SDK + starter-toolkit issues/releases | https://github.com/aws/bedrock-agentcore-sdk-python | 2026-09-18 | 4 | +| 27 | HN via Algolia (`agentcore`, `bedrock`, `strands agents`, `"claude code"`) | https://hn.algolia.com | 2026-09-18 | 3 | +| 28 | Anthropic cookbook commits | https://github.com/anthropics/anthropic-cookbook | 2026-09-18 | 2 | +| 29 | Reddit (r/LocalLLaMA, r/MachineLearning) | — | — | **not scanned — blocked at domain level** | +| 30 | Seasonal (re:Invent / conference proceedings) | — | — | **out of window** | + +## Web Budget + +Used: **20 / 50** requests (target). + +Skipped (unreachable / rate-limited): +- **openai.com** — HTTP 403 on both `/news/` and `/index/introducing-the-agents-api/`. Two requests spent, zero content retrieved. The Agents API item is reported as **unverified**. +- **Reddit** — blocked at the domain level via WebFetch; per the 2026-05-18 decisions-log entry, no requests were spent. +- **blog.google DeepMind index** — retrieved but JS-paginated; only the top card was visible, so "nothing in window" is **not verified**. +- **HN story 49753997** — returns empty from the Algolia items API, so the agent-memory benchmark's methodology is unverified. + +Skipped (other): +- **AWS ML blog 2026-09-11 → 09-13** — the 20-item RSS feed reaches back only to 09-14, so three days of that source were not visible. A dated-archive fetch would have cost 1–2 more requests; recorded rather than spent. +- **`probe_bedrock_cache_point_support.py`** — not run: no new Bedrock model family gained prompt caching this week, so the coverage half of the caching watch had nothing to measure. + +Notes: the budget came in well under target because nine of eleven subagents ran almost entirely through the authenticated `gh` CLI and `curl` against JSON APIs, which cost nothing. The two web-heavy categories were pricing/model-cards (8) and agentic UI/UX (6). diff --git a/docs/kaizen/review-queue.md b/docs/kaizen/review-queue.md index bbadbe4b7..c1cd3dd22 100644 --- a/docs/kaizen/review-queue.md +++ b/docs/kaizen/review-queue.md @@ -5,6 +5,172 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Open +### [2026-09-18] ✅ **CLOSED** — tests made real authenticated AWS calls; fail-open hid it; suite also halved +- **Source**: measured. A full-suite run stalled **72 minutes on 4m26s of CPU**, 0% CPU, holding four ESTABLISHED connections to `ec2-3-218-*.compute-1.amazonaws.com:443`. Root cause then found with a socket guard (an autouse patch of `socket.socket.connect` failing any off-box destination — moto never opens a real socket, so this detects escapes precisely, where hooking botocore is ambiguous because moto itself intercepts `before-send`). One instrumented run: **25 off-box connection attempts across 6 test files**, every one to real us-east-1 AWS. +- **The mechanism, in one example.** `tests/shared/test_skills_access.py::test_plain_grants_pass_through` patches the *role service*. But `resolve_accessible_skill_ids` (`apis/shared/skills/access.py:76`) then calls `resolve_owned_skill_ids`, which builds a **real** `get_skill_catalog_repository()` → DynamoDB client → live call. Its `except Exception` returns `[]`, `catalog + [] == catalog`, and **the assertion passes either way**. That is the archetype: *fail-open production code plus a partially-mocked test equals a silently real AWS call that never fails the suite.* +- **Why it matters more than a flake.** `~/.aws/config` on a dev machine carries a **`[default]` profile** (alongside `dev-ai` / `prod-ai`), so these are not merely failed calls — they resolve **real credentials** and are authenticated requests against real AWS from a unit-test run. The 72-minute hang is just the tail: normally the connection fails fast and is swallowed; occasionally TLS stalls and botocore's 60 s connect timeout × retries compounds under load. +- **Offending files** (25 cases): `tests/routes/test_pbt_model_access_preservation.py`, `tests/routes/test_role_agent_pins.py`, `tests/routes/test_sessions.py`, `tests/shared/test_kb_backend_parity.py`, `tests/shared/test_rbac_cache_service.py`, `tests/shared/test_skills_access.py`. +- **Surface**: `backend/tests/conftest.py` (the guard's home) plus the six files above. +- **Effort × Impact**: M × H. +- **Subtracts**: yes — it removes a whole class of silent test dishonesty, and the CI hang risk with it. +- **Status**: **closed** (`76bedf01`, `f15b2370`). The guard in `tests/conftest.py` is **unconditional** — any test opening a non-localhost socket fails — and the quarantine list is deleted. 17 files fixed; **9196 passed, 0 errors**. + - **The suite now runs in 8:02, down from ~14–16 min.** That gap *was* the bug: connection attempts to real AWS, each with botocore's connect timeout and retries, on every run. Roughly half the wall clock was spent waiting on calls whose results a fail-open `except` then discarded. + - Most of the 51 route cases shared a handful of seams, so the fixes are one autouse fixture in `tests/routes/conftest.py`: inference path (session metadata, the `document_read` gate), converse path (managed-model routing, rate-limit window, three quota lookups), RBAC role resolution, agent-detail labels, and the artifact share cascade on session delete. + - ⚠️ **Three traps, all of which made these look unreproducible or mis-targeted:** (1) patching a *repository accessor* is too late when the code receives an already-constructed service — patch the method on the service class; (2) `tools/freshness` memoizes on a module-level TTL, so which test reaches AWS depends on which warmed it first; (3) the artifact share cascade only fires once an earlier test has left artifacts "configured". Several cases passed in isolation and failed only in full-suite order. + - ⚠️ **`test_converse_cost_accounting`'s `mock_pricing` fixture already existed** and its docstring already said *"without this mock the call hits DynamoDB and fails"* — it was merely opt-in, and the tests that did not request it called the real table. Now autouse. Worth a glance for other opt-in fixtures with that shape. + +### [2026-09-18] Document offload — rollout percent: what it actually gates (less than it looks), and why 100% is the reasonable call +- **Source**: validation sweep of the merged epic (#1137–#1143, #1145) on develop @ 8911d85f, 2026-09-16/17 — static review, 41 probes, and a live dev clickthrough (session `61de2256`). `docs/specs/document-offload-evaluation.md` is still **`Status: Draft`, no harness**, and its §2 quality gate is written as a **veto** ("no cost result, however good, ships a confirmed quality regression"). Meanwhile `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` **defaults to 100**, which is that same spec's §4.2 **"Ship"** state. +- **Surface**: backend — `session/document_offload.py` (`rollout_percent`, `offload_enabled_for`, bucket = `crc32(session_id) % 100`); the eval harness per evaluation spec §2/§4.2; no infra wiring exists for any of the epic's flags, so this is a runtime env var on inference-api, not a CDK change. +- **Effort × Impact**: S × H — the decision is one env var; the cost of not making it is that the causal question can never be answered on prod data. +- **Subtracts**: no — it is the condition under which the epic's defaults may stay on in prod. +- **Status**: open, but **smaller than first written — corrected 2026-09-18 after reading the gate.** `offload_enabled_for` (kill switch + bucket) is consulted in exactly two places: the head-of-turn live offload (`turn_based_session_manager.py:1337`) and restore-path slice ageing (`:293`). **PR-3 — the digest-on-restore that is the epic's actual behaviour change — is NOT bucketed**; it rides `DOCUMENT_REHYDRATE_ENABLED`, a plain global flag. So do PR-1 (`DOCUMENT_READ_ENABLED`), PR-2 (`DOCUMENT_DIGEST_ENABLED`) and PR-6 (`ATTACHMENT_TURN_GUARD_ENABLED`). The percent therefore splits only the *least* consequential half of the epic, and it was never a control arm for PRs 1–3 — those can only ever be measured before/after a deploy, whatever the percent says. + - **Recommendation: ship at 100%.** The evaluation spec's B-vs-C arm measures PR-4's incremental effect, and readout item 1 predicts PR-4 fires ≈0 times (restore reaches the document first — `AGENT_CACHE_BYPASS`). Splitting the fleet to A/B a mechanism that may not run buys little; **counting `document_offload` events at 100% answers the same question more cheaply.** If the count turns out to be high rather than ~0, the events carry their own `documentTokens` and `cacheGapSeconds`, so PR-4's share is still attributable from the ledger without arms. + - **What 100% genuinely gives up**: clean causal separation of PR-4's cost effect from PR-3's, *if* PR-4 turns out to fire often. Accept it as a recorded decision rather than a default nobody chose. + - **The real rollback lever is not the percent.** For the change users would actually notice, it is `DOCUMENT_REHYDRATE_ENABLED=false` (back to the pre-epic contentless placeholder); `DOCUMENT_READ_ENABLED=false` now also stops the live offload (#1147), and `DOCUMENT_OFFLOAD_ENABLED=false` stops only PR-4. + - **Still required either way**: the §2 quality veto has not run. Note the epic's §5 rule that PRs 1–3 are a correctness fix (quality should go **up**) while only PR-4 is the cost trade, so A→B and B→C must be scored separately or a strip-fix win masks an offload regression. + +### [2026-09-18] Document offload — the prod readout: five numbers, what each decides, and the one that is probably zero +- **Source**: same sweep. Live dev run confirmed the full lifecycle (turn 1 **full** → turn 2 **digest-only** → turn 3 **retrieved** via `document_read(page_range="44")`, 170 ms → slice aged on schedule at `DOCUMENT_SLICE_MAX_TURNS=2`), and measured the prefix collapsing **109.1K → ~15K** on one session. +- **Surface**: backend — `C#` rows (`hasDocuments` / `documentCount` / `documentTokens` / `documentDigests` / `documentSlices` / `documentReads`), `compactionEvents[].kind` ∈ {`document_stripped`, `document_rehydrated`, `document_offload`}, `S#` rollups (`fullDocumentCalls`, `digestOnlyCalls`, `documentReadCalls`, `documentReadPages`), `F#` feedback rows (#1142), EMF in `AgentCoreStack/Compaction`, `GET /admin/costs/sessions/{id}/calls`. +- **Effort × Impact**: S × H — every field already ships; this is a query session, not a build. +- **Subtracts**: no. +- **Status**: open — run after the epic reaches prod and has a week of attachment volume (~50 attachment sessions/week at the audited rate, so plan 2–4 weeks for anything cost-shaped). + 1. **Is PR-4 doing anything at all?** Count `document_offload` events. **Predicted ≈ 0** — measured in dev: `document_read` is itself an `extra_tool`, so the agent is rebuilt every turn, restore runs every turn, and the *restore* path (PR-3) reaches the document before the live offload ever can. The admin anatomy page diagnoses this itself (`AGENT_CACHE_BYPASS`). If it is ~0, then **all** the benefit is PR-3 and `DOCUMENT_OFFLOAD_PIN_TURNS` is inert code — which changes what to keep, and per evaluation spec §4.2 "a win with zero offload events is a confound, not a result." + 2. **Did the recovery path reach users?** `document_stripped` should fall to the unmatched residue (direct base64 attachments, deleted files) and `document_rehydrated` should carry the rest. Re-upload rate (same filename ≥2× in a session) **14% → target <3%**; this is the one metric PRs 1–3 move on their own. + 3. **Is the digest the right size?** `documentReadCalls` per attachment session against the health band **≈0.3–3**. Near zero means the digest is answering unaided (suspicious — or the model is guessing); >3/turn means it is too thin. + 4. **Was the offload ever wrong?** `document_offload` events with `cacheGapSeconds` under the cache TTL: **target zero**; any non-`over_ceiling` / non-`prefix_changed` one is the paid-when-free guard breaking. + 5. **The outcome signal, for the first time.** Down-thumb rate split by turn class (**full** / **digest-only** / **retrieved**) from the `F#` rows joined on `(sessionId, messageId)`. This is what PR-7 was built for and what every previous cost change on this platform shipped without. Report with `n` per class; per response-feedback spec §9 it is a *comparison between arms*, never a quality score. + +### [2026-09-18] Document offload — recalibrate `PDF_PAGE_TOKEN_ESTIMATE` from prod rows (the six findings are all fixed) +- **Source**: same sweep. #1147 fixed two of six findings — `documentTokens` was counting bytes, not PDF pages (**13.7× low**: 6,894 reported against ~94K measured, because Bedrock dual-encodes each page as an image), and `DOCUMENT_READ_ENABLED` did not reach the restore or offload paths. +- **Surface**: backend — `apis/shared/files/document_tokens.py` (`PDF_PAGE_TOKEN_ESTIMATE`, env-tunable); `apis/shared/files/document_read.py` (`_compile`/`_grep_lines`); `apis/shared/files/models.py` (`INLINE_ATTACHMENTS_MAX_TOTAL_BYTES`); `apis/shared/files/document_digest.py` (`render_digest`); `apis/app_api/admin/costs/` (`prefixTokens`). +- **Effort × Impact**: S × M each. +- **Subtracts**: no. +- **Status**: **all six findings fixed** (`82e0587d`, `10e511e5`, `71b0fdfb`, `e196a00e` — all on PR #1147). What remains is the one thing that needs prod data: + - **✅ FIXED 2026-09-18 (`71b0fdfb`)** — ReDoS in `document_read` pattern mode. `catastrophic_pattern` refuses the nested-quantifier family up front (searched literally instead, with a `pattern_note` on the payload; 7/7 dangerous shapes flagged, 0/13 false positives), and `DOCUMENT_READ_PATTERN_BUDGET_SECONDS` (default 2.0) bounds the walk for shapes it does not catch. **Residual, by construction:** a single `re.search` still cannot be interrupted — CPython exposes no cancel and the scan runs in a worker thread where signals are unavailable — so the budget is a backstop, not a guarantee. The complete fix is the `regex` module's native `timeout=`; it is installed transitively but **not declared**, so adopting it is a dependency decision needing approval. Re-open only if `timed_out` payloads show up in prod. + - **Recalibrate the per-page constant.** The 1,500 default was fitted to a single dev PDF (1,500 vs 1,573 actual/page, deliberately ~5% low because it gates an eviction). Once prod has attach turns, fit it against `cacheWriteInputTokens` minus `prefixTokens.system + prefixTokens.tools` on rows where `documentsAttached = 1`. It is an env var — no deploy. + - **✅ FIXED 2026-09-18 (`e196a00e`)** — `INLINE_ATTACHMENTS_MAX_TOTAL_BYTES` default 7,500,000 → **7,000,000** (encodes to 9,333,336, ~666 KB of headroom under the 10 MB event quota). Still far above prod's p90 attachment cluster (2.58 MB), so no real turn is newly trimmed. Watch `AttachmentTurnOverQuota` in prod anyway — a `SessionException` hole in history would mean the quota is stricter than the decimal reading. + - **✅ FIXED 2026-09-18 (`e196a00e`)** — the digest cap is now hard: escape first, cut on an entity-safe boundary (`&` never left as `&am`), and drop the abstract entirely rather than overshoot. Only the opening tag — the `document_read` handle — may outlive a budget too small for anything else. + - **✅ DIAGNOSED + GUARDED 2026-09-18 (`e196a00e`)** — not an off-by-one. `toolTokens` is a **residual** between two independently sourced counts, `full` (Strands' projection) minus `no_tools` (our own CountTokens call), so any disagreement about how a content block is counted lands wholly in it. The arithmetic is conclusive: 106,756 − 12,516 = **94,240**, against a document measured at ~94,485 — the entire document attributed to tools, because Bedrock counts a PDF page as an image *and* a text layer and the two sources did not agree. The counting disagreement is Bedrock/Strands behaviour we do not control, so the split is simply **not computed while inline document or image bytes are in context** and is taken on a later clean turn; absent reads "not tracked", which a wrong number does not. A digest is text, so attachment sessions still get the field from turn 2. **Residual question for the prod readout:** whether `full` and `no_tools` disagree on *images* too — if `prefixTokens` goes missing on image-only sessions for more than the attach turn, that is the tell. + - **`compaction_policy._block_tokens` has the same PDF blind spot** #1147 fixed elsewhere, and was deliberately left alone: it rescales to measured history and drives cut thresholds, so changing it moves when compaction fires. Its own decision. + +### [2026-09-18] Reverse the Astra decision — register at the full 1M window and make the price tier its own catalog field +- **Source**: research/2026-09-18.md ▸ Idea #1 — https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-astra.html. ⚠️ **Directly supersedes the open `[2026-09-11] Add GPT-6 Astra at the 272K tier` entry**, whose blocking question this answers in the *opposite* direction: the model card's own headings ("short context (272K input tokens or fewer)" / "long context (more than 272K input tokens)") show the tier is selected by **actual input tokens per request**, not by the declared window. Merge the two at review. +- **Surface**: frontend (`admin/manage-models/models/curated-models.ts`, model picker) + backend (Mantle Responses path, `admin/services/model_access.py`) +- **Effort × Impact**: L × H +- **Subtracts**: yes — a queued recommendation (`maxInputTokens: 272000`) that would have shipped a capability regression; the `long = short × 2` assumption (output is **1.5×**: $55.00 → $82.50, not $110.00); and the ambiguity about which API surface reaches Astra +- **Unlocks**: + - A genuine **1M-token frontier tier** — the first catalog model where a whole corpus fits in one turn — with no artificial cap + - A **tier-aware catalog**: the admin page can show the boundary, cost projections can respect it, and Astra's 10× output-token quota burn can be accounted for against the known campus TPM ceiling. The 272K cap survives as an optional *policy* lever +- **Notes**: ⚠️ Take the **Mantle Responses** path, not Converse — the card lists prompt caching only under `bedrock-mantle`/Responses, so `bedrock-runtime` Converse would re-pay a 30k–150k prefix at $11/MTok every turn. ⚠️ Astra is **not** a third `cacheRead × 0.1` counterexample (it is exactly 0.100× in both tiers); Fable 5.1 and Grok 4.6 remain the only two. ⚠️ The ref repo's `rejectsTemperature: true` is **still not on the model card** — verify, don't copy. ⚠️ Astra's cache-write SKU is a **30-minute** TTL — do not fold into `cache_ttl_seconds_for()`'s 5m/1h assumptions. +- **Status**: open + +### [2026-09-18] Make the Strands 1.56.0 bump a paired pin with `bedrock-agentcore>=1.23.1` — and enforce the two guard rails we only document +- **Source**: research/2026-09-18.md ▸ Idea #2 — https://github.com/aws/bedrock-agentcore-sdk-python/issues/665 · https://github.com/strands-agents/harness-sdk/issues/4362 · `backend/pyproject.toml:120-132` +- **Surface**: backend (`pyproject.toml:66,70,81`, `uv.lock`) + CI (a supply-chain test beside `tests/supply_chain/test_nightly_ref_allowlist.py`) +- **Effort × Impact**: L–M × H +- **Subtracts**: yes — converts two guard rails that exist only as prose into assertions CI can fail on, and closes a two-release lag whose gap contains a runtime-breaking interaction +- **Unlocks**: the first-party Strands **context-manager / offloading stack is fully released in 1.56.0** (#4118/#4146/#4187/#4231/#4254/#4282; `ContextOffloader` manual offloading closed as released) — pinnable and evaluable against ours, with #4367's `NAMESPACED` stash sentinel offering **cross-session offload sharing** for the document-offload work +- **Notes**: ⛔ Strands 1.56.0 deleted `BidiAfterInvocationEvent` / `BidiAgentInitializedEvent` / `BidiMessageAddedEvent` with no alias; `bedrock_agentcore.memory.integrations.strands.session_manager` imports all three at module load, so **every `bedrock-agentcore` 1.12.0 → 1.23.0 fails to import**. We import it at three sites (`apis/shared/sessions/messages.py:20`, `apis/app_api/shares/service.py:392`, `agents/main_agent/session/turn_based_session_manager.py:40`). ⚠️ **On AgentCore Runtime this presents as "Runtime initialization time exceeded (30s)" → 502** — it reads as cold-start, not ImportError. ⚠️ Second guard rail: the `[tool.uv] constraint mcp<2` is effectively held by the incidental `idna==3.15` pin, so **a routine `idna` Dependabot bump would cross the MCP major with nothing in the diff naming MCP**. ⚠️ Read three 1.56.0 deltas first: `SessionManager` → `Generic[_SessionAgentT]`, `InterruptException` now bubbling **unwrapped** out of `event_loop_cycle` (our pause paths), and `agent_metadata` added positionally to `_format_request` (our override already absorbs it). ⚠️ The standing decisions-log entry stands — adopting the upstream context manager is out of scope without a migration design; this makes it *evaluable*, not adopted. +- **Status**: open + +### [2026-09-18] Sweep the cacheable prefix for derived and relative values — and close the last daily boundary +- **Source**: research/2026-09-18.md ▸ Idea #3 — Claude Code 2.1.275 (*"a restored memory file's age note changing between requests after a compaction or resume, which caused prompt cache misses"*), 2.1.269, 2.1.273 — https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md — plus our own `9f246cb7` +- **Surface**: backend — `agents/main_agent/utils/` (`get_current_date_pacific()` + the `SystemPromptBuilder` tail), `session/turn_based_session_manager.py` (truncation anchor), `session/document_rehydration.py`, `apis/shared/sessions/messages.py`, and the resume path in `inference_api/chat/routes.py` +- **Effort × Impact**: L–M × M–H +- **Subtracts**: yes — the final *scheduled* prefix rewrite, and an entire category of cache invalidation that currently has no detector even though we already own the instrument +- **Notes**: Claude Code shipped **four** fixes for this bug class in one week, all the same shape: a non-content side event silently mutating the cacheable prefix. I checked the obvious paths and found no live instance today — the point is to make it a **test** rather than an observation, because it is exactly what a future feature adds without noticing. ⚠️ `9f246cb7` took the date line from 24 rewrites/day to 1; a session open across **Pacific midnight** still re-pays the whole prefix once. Either drop the date from the cacheable segment or place it after the last cachePoint. ⚠️ Pair with Claude Code's other finding — a context meter double-counting advisor-tool turns fired auto-compact at **half** the real window — by asserting our per-turn context size for tool-heavy turns tracks CountTokens. Over-counting is a pure cost bug and it fails silently. ⚠️ CountTokens rejects `us.*` ids, so that assertion needs a `global.*` id or an alternative estimator. +- **Status**: open + +### [2026-09-18] Fix the MCP Apps header's tool display name — `title` → `annotations.title` → `name` +- **Source**: research/2026-09-18.md ▸ Idea #4 — https://github.com/modelcontextprotocol/ext-apps/pull/778 (opened 2026-09-18, fixing ext-apps #767). **Narrows the open `[2026-09-11] Retire the ClientSession monkeypatch via _meta serverInfo` entry** — this is an independent, shippable piece of the same header +- **Surface**: backend (`apis/shared/mcp_apps/ui_resource_store.py:215`, `card_store.py:160`, the `toolName` derivation feeding both the header-only shell and the full `ui_resource` emit) + frontend (App frame header component) +- **Effort × Impact**: L × M +- **Subtracts**: no — addition, justified because the wrong string is **persisted with the resource** and survives reload: the App frame header shows `raw_tool_name` while the tool rail immediately beside it shows the friendly title, and the two disagree permanently +- **Unlocks**: spec-conformant App headers on the precedence every other MCP host is converging to — a prerequisite for the eventual `_meta serverInfo` migration rather than a competitor to it +- **Notes**: ✅ **Verified against our tree** — `apis/shared/mcp_apps/` contains no `title` or `annotations` handling at all. ⚠️ The upstream PR is **open, not merged**; the precedence comes from the base MCP schema, not the PR. ⚠️ While in this file: `ext-apps` v2.0.0 changed host-facing error codes (`-32002` → `-32602`, `-32603` → `-32602`) and removed the `MCP error N:` message prefix — the App-error chain from #1009/#1013 was built substantially on message text and deserves a grep in the same PR. +- **Status**: open + +### [2026-09-18] Defer external MCP tool discovery to session scope — and add the sidebar conversation search while in the SPA +- **Source**: research/2026-09-18.md ▸ Idea #5 — https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/294 (lazy loaders; measured "Starting" 10.417s → 4.081s, follow-up 7.377s → 1.117s) and PR #296 (sidebar search); corroborated by opencode #49276 and #49387 +- **Surface**: backend (`agents/main_agent/base_agent.py:478` `_build_filtered_tools`, `integrations/external_integration.py`, `gateway_integration.py`) + frontend (`components/sidenav/components/session-list/session-list.{ts,html}`) +- **Effort × Impact**: M × M–H +- **Subtracts**: yes — the eager load of every enabled external MCP client at agent build, on every agent construction +- **Unlocks**: + - Their measured **60–85% first-token latency cut** on new and follow-up conversations + - A conversation-search affordance we have **none** of today — `session-list.ts` groups by date with no filter at all — at **zero token cost** (a pure client-side filter over already-loaded titles) +- **Notes**: ⚠️ **Session-scoped only — never turn-scoped.** Their implementation goes further: their new base prompt tells the model *"available tools may change between turns within the same conversation"*, accepting a `toolConfig` that varies per turn. For us that is a `toolConfigHash` rewrite at 1.25× base input over a 30k–150k prefix **every turn** — the latency win would be paid back many times in cache writes. Defer discovery only for tools **not in this session's `enabled_tools`**, a set that is stable for the session. ⚠️ Also confirm we make no equivalent keep-alive/warm-up call when an `@`-mention builds a second `Agent` (opencode #49387 found exactly that cost per subagent). ⚠️ Their module-global → per-agent registry fix is the same bug class as our "never cache session state on an agent instance" rule — read it before touching this code. ⚠️ The two halves are independent; the SPA search can ship alone. +- **Status**: open + +### [2026-09-18] Merge #1147 — it carries two fixes AND the offload epic's own measurement plan +- **Source**: direct observation by `kaizen-review-prep` 2026-09-18 — [#1147](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1147), open since 2026-09-17, `MERGEABLE`, CI green, **unreviewed and unmerged for a day**. Produced by a validation sweep over the shipped epic (#1137–#1143, #1145): static review, 41 purpose-written probes, and a live dev clickthrough (session `61de2256`). +- **Surface**: backend — `apis/shared/files/document_tokens.py` (new), `session/document_offload.py`, `session/document_context.py`, `session/document_rehydration.py`, `apis/shared/feature_flags.py`; **plus `docs/kaizen/review-queue.md`** — see the conflict note below. +- **Effort × Impact**: S × H — the code is written, tested (31 new tests) and green on a 9,050-pass suite. The remaining work is a review and a merge button. +- **Subtracts**: yes — two defects, one of which made a kill switch *worse than the feature's absence*. +- **Status**: open — **recommended #1 in reviews/2026-09-18.md.** **(1) `documentTokens` counted bytes, not PDF pages.** Bedrock dual-encodes each PDF page as an image *and* a text layer, so `bytes/4` cannot see the image channel: measured on dev session `61de2256` at **6,894 against ~94,485 actual — 13.7× low**, reporting documents as 6.3% of the prefix where truth was ~87%. That matters because §6.1 is the instrument the evaluation spec's §4.2 ship/abandon rule reads off: at 14× error the rows argue *abandon* for a change that in the same session cut the prefix **109.1K → ~15K**. Fix is `max(pages × PDF_PAGE_TOKEN_ESTIMATE, bytes/4)` at 1,500/page, wired into **every** consumer because `Candidate.tokens` feeds the eviction floor. ⚠️ Real behaviour change: PDFs previously stuck below the 5,000-token floor forever become offload candidates. `compaction_policy._block_tokens` shares the blind spot and is deliberately untouched — it drives cut thresholds and needs its own decision. **(2) `DOCUMENT_READ_ENABLED` did not reach the paths that promise the tool.** Verified independently 2026-09-18: the flag is read in exactly one place (`inference_api/chat/routes.py:821`) and **zero** times under `session/`. So pulling the one kill switch an operator would reach for left restore advertising a retrieval handle for a tool that would not be injected, and the live offload evicting bytes with the only recovery path off — **strictly worse than the pre-offload world**. Spec §5: *"a digest that points at a tool nobody has is no better than today's placeholder."* +- **The merge also lands the epic's measurement plan.** #1147 adds three `[2026-09-18]` entries to this queue — rollout-percent decision, the five-number prod readout, and recalibration + the four unfixed findings (ReDoS first). Those are the authoritative versions; **this review deliberately does not duplicate them.** ⚠️ **Queue conflict**: #1147 and the review PR both insert at the top of `## Open`, so whichever merges second needs a trivial conflict resolution in this file. Merging #1147 first is the cleaner order, since its entries are the ones later work reads. +- ⚠️ **A correction this forum owes #1147.** The first draft of reviews/2026-09-18.md proposed moving `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` off its default of 100 to "restore the control arm." **That was wrong, and #1147's own entry had already refuted it**: `offload_enabled_for` gates only two call sites — the head-of-turn live offload and restore-path slice ageing — while **PR-3, the digest-on-restore that is the epic's actual behaviour change, is not bucketed at all** (it rides `DOCUMENT_REHYDRATE_ENABLED`, a plain global flag), as do PR-1, PR-2 and PR-6. The percent therefore splits only the least consequential half and was **never** a control arm for PRs 1–3, which can only be measured before/after a deploy whatever it says. #1147's recommendation — **ship at 100% and count `document_offload` events instead**, since PR-4 is predicted to fire ≈0 times because restore reaches the document first (`AGENT_CACHE_BYPASS`) — is the better-reasoned call and is adopted. What survives from the original framing is narrower and is filed separately: the **§2 quality veto still has not run**, which both documents agree on. + +### [2026-09-18] Two quality vetoes are unrun while both stacks' defaults are live — run them, or record the waiver in the spec +- **Source**: direct observation by `kaizen-review-prep` 2026-09-18, across two specs that independently wrote their gate as a **veto**: `docs/specs/compaction-model-relative-thresholds.md` §5 (*"Veto before default change in prod"*, the spiral spec §4.3 eval with the fixed-threshold arm as control) and `docs/specs/document-offload-evaluation.md` §2 (*"no cost result, however good, ships a confirmed quality regression"*, still `Status: Draft`). Verified: `find backend -name "*eval*"` returns only vendored site-packages — **neither harness exists on disk**. +- **Surface**: backend — the §4.3 eval harness (constraint retention / revision continuity / reference lookup) with AgentCore Evaluations judges for the three families and a blinded holistic judge outside the managed path; `session/compaction_policy.py`, `session/compaction_summary.py`. Record results in the thresholds spec §5 and the evaluation spec §4.2. +- **Effort × Impact**: M × H +- **Subtracts**: no — it is the condition under which both stacks' defaults may stay on. Justified because without it the deepest, earliest cut the cost arc has ever shipped is defended by **dollars alone** ($73.70 vs $102.64 on a 20-session replay), with quality asserted rather than measured. +- **Status**: open — **this is the pattern, not either stack individually.** Two specs, written weeks apart, each chose the strongest possible gate wording and each shipped with defaults live and the harness unbuilt: `COMPACTION_CEILING_RATIO = 0.5` / `HARD_CEILING_RATIO = 0.7` are live in dev now, and the offload epic is default-on. `develop` auto-deploys to dev, so "before it reaches prod" is the only window still open, and it closes at the next release. **The asymmetry is the point**: `CLAUDE.md`'s cost tenet says plainly that *"when cost and answer quality genuinely conflict, quality wins"* — which is unfalsifiable while every cost change is measured and no quality change is. ⚠️ Score **A→B and B→C separately** (evaluation spec §4.2 / thresholds §5): PRs 1–3 of the offload epic are a *correctness* fix where quality should go **up**, and only PR-4 is the cost trade, so a strip-fix win can mask an offload regression. ⚠️ The **honest fallback is recording the waiver**, not silence: a gate documented as waived tells the next reader what happened; a gate merely unrun reads as passed. Note #1142/#1145 just shipped the `F#` down-thumb rows, so the cheap first version of this is a **comparison between turn classes**, not a new harness — per response-feedback spec §9 a comparison between arms, never a quality score. + +### [2026-09-18] Fix the kaizen pipeline spacing — this week's review ran without a research doc +- **Source**: direct observation by `kaizen-review-prep` 2026-09-18, from the scheduled-task run records. +- **Surface**: scheduled tasks (outside the repo) — `kaizen-research` (`0 6 * * 5`, jitter 164s) and `kaizen-review-prep` (`0 8 * * 5`, jitter 467s). +- **Effort × Impact**: S × M +- **Subtracts**: yes — a load-bearing timing assumption ("~2 hours later") that lives only in two skill descriptions and is enforced nowhere. +- **Status**: open — `kaizen-research` started **2026-09-18T13:50Z** against **12:09Z** the week before, and was still `running` 30 minutes into this prep; last week's run took **2h18m**. `kaizen-review-prep` started 14:08Z, an **18-minute** gap instead of two hours, so this review had no current research doc and was built queue-first and outcome-first instead. That is the skill's designed fallback (the queue's `## Open` entries are the primary source and research is one input) and it degraded well — but by accident of design, not by intent. Fix: move review-prep to ~10am MT, or add a preflight that checks for the current week's research doc and states its absence up front. ⚠️ Both schedules are Phil's and live outside the repo, so this is a recommendation, not a repo change. + + + +### [2026-09-16] Compaction stack (#1125 → #1128 → #1129 → #1131 → #1132) — run the quality-veto eval BEFORE the new defaults reach prod +- **Source**: Phil-initiated — `docs/specs/compaction-model-relative-thresholds.md` §5 (the gate the spec sets for itself) + `docs/specs/compaction-over-threshold-cache-spiral.md` §4.3 (the veto eval) + `docs/specs/agentcore-evaluations-spike-findings.md` (what the managed evaluation service already supplies). Stack built 2026-09-15/16; the 2026-09-15 replay of 20 heavy Sonnet 5 sessions priced 100k/25k at **$73.70 vs $102.64 actual** input-side. +- **Surface**: backend — `session/compaction_policy.py` (floor-seeking cut), `session/compaction_summary.py` (8k summary via Nova Micro), `session/turn_based_session_manager.py` (`apply_pending_compaction`); eval harness per the spiral spec §4.3 (constraint retention / revision continuity / reference lookup), fixed-threshold arm as control. +- **Effort × Impact**: M × H — the stack cuts deeper and earlier than v1 ever did and rewrites the summary with a cheap model; that is a bigger model-visible context change than anything the cost arc has shipped. Cost is measured (replay + PR-2/3 metrics); quality is not yet. +- **Subtracts**: no — it is the condition under which the stack's defaults may stay on in prod. Without it we are optimizing dollars against an asserted quality constant (the counterweight the roadmap has never had). +- **Status**: open — **blocks calling the feature done.** Run once the stack is merged to develop (dev auto-deploys): paired arms, same corpus, blinded holistic judge outside the managed path (spike §2), plus the AgentCore Evaluations judges for the three families. Record the result in the thresholds spec §5 and move `FLOOR_RATIO` / `CEILING_CAP_TOKENS` only on that evidence. Merge mechanics: **in order, parents first, never `--delete-branch` on a parent (kills the child PR), and inner stack PRs get zero CI — let each re-targeted PR run before merging the next** (memory: stacked-PR merge traps). + +### [2026-09-16] Decide PR-5 (selective 1h TTL on the static prefix, #1132, flag OFF) — a dev week AFTER the hourly system-prompt tick fix +- **Source**: measured — `backend/scripts/probe_static_prefix_ttl.py` in dev-ai, 2026-09-16, Haiku 4.5. **420 s gap:** Bedrock honors `ttl: "1h"` on the tools+system points — 1h arm second call read 5,924 / wrote 327 (message segment only) vs the 5m arm re-writing all 6,251; pair **12% cheaper**. **60 s gap:** both arms warm; 1h arm **+$0.005157**, exactly the 0.75×-base premium on the first write with nothing to recover. Both recorded in the thresholds spec §6 PR-5 and the probe docstring. +- **Surface**: backend — `core/model_config.py` (`static_prefix_cache_ttl`, `CacheConfig(system_prompt_ttl="1h", tools_ttl="1h")`), `apis/shared/costs/calculator.py` (1h writes billed at 2× base from the context-breakdown static size), the `staticPrefixTtl: "1h"` marker on `C#` rows; the hourly `%H:00` tick in `utils/timezone.py` (separate task chip, in progress). +- **Effort × Impact**: S × M — the code is done; the decision is a flag flip. Arithmetic: surcharge ≈ $0.023 per static write on a 28k prod prefix at Haiku rates ($0.042 Sonnet 5); saving ≈ $0.034 / $0.061 per return landing 5–60 min later; pays when such returns outnumber static writes by > ~0.7 : 1. The audit's 36% cold-re-write share says it might; the probe cannot tell. +- **Subtracts**: no — an experiment arm. Deliberately default-off against the flags-default-on house style (#954/#956: never adopt a caching default on inspection). +- **Status**: open — **do not flip until the hourly tick fix has landed** (it would bust the 1h entry every hour and make the week meaningless). Then `AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h` in dev for a week; compare cache-write dollars per session split on `staticPrefixTtl`, and the write:read ratio. Ship / Decline on that number. + +### [2026-09-16] The 28% "neither" cohort — content-class ordering inside the cut is the next compaction lever, if the metrics say the floor is still missed +- **Source**: 2026-09-15 prod cost audit (content-free): of 95 September sessions over 100k, **52 (55%) had a ≥4k-token tool result in the last 3 turns** (→ #1131 intake offload), **25 (26%) an attachment there** (→ document offload, its own item below), **27 (28%) neither** — long sessions (a3b884b1 46 turns, 4cb93e6e 45 turns) whose bulk is old history plus a 23–40k summary. +- **Surface**: backend — `compaction_policy.choose_checkpoint` (today: oldest tool-pair-safe cut under the floor, whole turns), `CompactionFloorUnreachable` / `CompactionRetainedTokens` on the `AgentCoreStack/Compaction` EMF record, `compactionEvents` on `C#` rows once #1130 lands. +- **Effort × Impact**: M × M +- **Subtracts**: no +- **Status**: open — **watch, don't build yet.** PR-1's cut + PR-2's 8k cap should reach this cohort. If `CompactionFloorUnreachable` stays high after a dev week, the lever is evicting old tool results before old prose inside the retained window (content-class ordering), never a deeper protected-tail cut. Also still unbuilt from the spec's §7.2 data points: **retained-estimate vs next turn's measured `contextBreakdown.messages`** (estimator calibration, one number per cut) and **down-thumb rate keyed to turns-since-last-cut** (the first outcome signal joined to compaction; kaizen has listed this counterweight before). + +### [2026-09-16] Small follow-ups from the compaction stack (fold into any PR, no new epic) +- **Source**: Phil-initiated — thresholds spec §3.6 / §7 "not in PR-N" notes. +- **Surface**: backend — `apis/app_api/admin/costs/diagnoses.py` (`OVER_COMPACTION_THRESHOLD` still compares against the fixed 100k default; should read the persisted `compaction.policy` map's `ceiling`), `core/model_config.py` (`context_window_limit` is a valid `BedrockConfig` key in 1.55 but needs the catalog window at agent construction — `ModelConfig` has no field for it; only affects Strands' own `estimate_utilization` warning today). +- **Effort × Impact**: S × L +- **Subtracts**: no +- **Status**: open — housekeeping; do when touching those files. + +### [2026-09-16] Compaction v2 (#835) go/no-go — re-evaluate once the model-relative stack has a week of data +- **Source**: `docs/specs/compaction-v2-versioned-prefix.md` §7 (gate criteria) vs the stack just built, which implements v2's I3 (hysteresis/floor) and I4 (paid-when-free) as policy on the v1 machinery. +- **Surface**: backend — `TurnBasedSessionManager` vs Strands `SummarizingConversationManager` + `compression/` helpers (the engine swap v2 proposes); upstream `main` is also growing a first-party context-manager/offloading stack (#4254 et al., noted 2026-09-11) that would collide with a hand-built engine. +- **Effort × Impact**: L × M (if go) — the policy layer maps onto v2 unchanged, so the remaining value of v2 is the frozen-segment engine (I1/I2/I6), i.e. removing custody of slice/summary/restore-derivation code. +- **Subtracts**: yes, if go — `_apply_compaction`'s slicing, `_retrieve_session_summaries`, `_prepend_summary_to_first_message`. +- **Status**: open — **defer until the stack's metrics exist.** v2's own go criteria (over-threshold sessions > ~3% of active or > ~15% of spend; or `partial_miss` waste from compaction-state mutation > ~$50/month post-fixes) are now measurable from `CompactionForced`, `CompactionApplied` and the ledger. Decide with numbers, and weigh against adopting upstream's context manager when it ships rather than v2's hand-built engine. + +### [2026-09-16] Document offload epic — attachments are 31% of spend and 26% of the over-100k tail; spawned as a task, with actionable analytics as a first-class deliverable +- **Source**: `docs/specs/document-context-offload.md` (Draft, no branch; 7 PRs, PRs 1–3 a unit) + the 2026-09-15 audit cohort split + `docs/specs/document-conversations-cost.md` (31%). The compaction stack deliberately does not touch attachments — a cut cannot fix them, a digest + page-range read can. +- **Surface**: backend — `document_read` tool gated on the session having an attachment (ids kept out of `INJECTED_TOOL_IDS`), `DocumentDigest` at upload on `FileMetadata`, `_strip_document_bytes` → digest + live handle on restore; analytics: `hasDocuments` / `documentTokens` on `MessageMetadata` (the spec's PR-7) plus whatever else makes the cost-effectiveness of the next change *decidable* — per-session attachment tokens vs. answer quality signal, `document_read` call counts and page volumes, digest-vs-full-document turn shares. +- **Effort × Impact**: L × H +- **Subtracts**: partially — retires the contentless restore placeholder (a correctness bug: a returning user's document is silently lost) and the 14% re-upload rate. +- **Status**: **BUILT 2026-09-16** — shipped as PRs #1137–#1143 (+#1145); analytics were pulled forward into PR-1 as asked, and the per-call fields do align with the compaction ledger and `AgentCoreStack/Compaction`. Validated 2026-09-16/17 (41 probes + a live dev clickthrough); two defects found and fixed in #1147. What remains is **measurement, not build** — see the three [2026-09-18] entries at the top of this queue. Original ask, for the record: task chip spawned 2026-09-16 ("Explore document offloading with actionable cost analytics"). Ask of the exploration: sequence PRs 1–3 together as the correctness fix, pull the analytics (PR-7) forward so the cost work (PR-4) is measured from day one, and align the per-call fields with the compaction ledger (#1130) and `AgentCoreStack/Compaction` so one anatomy page explains a session's cost. + + ### [2026-09-11] ✅ ANSWERED: `bedrock_cache_points_supported()` STAYS — and it is wrong in the opposite direction from the one we suspected - **Source**: measured, not read — `backend/scripts/probe_bedrock_cache_point_support.py` (added by this entry) against the pinned strands-agents 1.55.0 and live `bedrock-runtime` in dev-ai/us-west-2, 2026-09-11. **Answers check (d)** of the [2026-09-11] "Retire the hand-placed system `cachePoint`" entry below, and **narrows (c)** of the [2026-09-11] `ClientSession` monkeypatch entry. Does **not** close either — (a), (b) and the `tools_ttl` mixed-TTL evaluation are untouched. - **Surface**: backend — `core/model_config.py` (`bedrock_cache_points_supported`, the cachePoint-budget comment), `core/agent_factory.py` (the hand-placed system block), `backend/scripts/probe_bedrock_cache_point_support.py`. @@ -21,53 +187,16 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **The one genuine (small) opportunity this surfaced:** the API-key `/chat/api-converse` path bills full input on every turn, forever, because it hand-rolls boto3 and places no cache points. 1.49M tokens in six weeks from **three** sessions. Negligible today, but it scales linearly with API-key adoption and the handler already moved to app-api in #621. File as its own item if API-key traffic grows. -### [2026-09-11] Fix the nightly, both halves — anchor the branding specs' paths, and put `--coverage` where PR CI can see it -- **Source**: research/2026-09-11.md ▸ Top 5 #1 — internal. Runs [33957119927](https://github.com/Boise-State-Development/agentcore-public-stack/actions/runs/33957119927) (2026-09-05, first red) → [34582692457](https://github.com/Boise-State-Development/agentcore-public-stack/actions/runs/34582692457) (2026-09-11). Specs landed with PR #933. -- **Surface**: CI / frontend — `.github/workflows/tests.yml` (`test-frontend`), `frontend/ai.client/package.json` (`test:ci`), `scripts/frontend/test.sh`, and six specs: `src/branding/rebranding-guide.doc-presence.spec.ts`, `src/branding/prestart-generator-parity.spec.ts`, `src/branding/brand-theme-golden.spec.ts`, `src/branding/surface-theme-golden.spec.ts`, `src/app/surface-literal-guard.spec.ts`, `src/app/tailwind-theme-import-guard.spec.ts`. Plus one CloudFormation stack in dev-ai. -- **Effort × Impact**: L–M × H -- **Subtracts**: yes — a gate that has produced only false signal for a week; a `DELETE_FAILED` stack whose retained resources are still billing; and the *category* of repo-hygiene spec that silently depends on how it was invoked. -- **Status**: open — **recommended #1, cheapest item in the scan, evidence complete.** **Seven consecutive nightly failures** (green through 2026-09-04), **two independent root causes, neither a flake.** **(a)** `Test Frontend with Coverage` fails while `deploy / [deploy-develop] Test frontend` **succeeds in the same run on the same ref** — the jobs differ in exactly one thing: nightly runs `ng test --no-watch --coverage`, PR CI and the deploy track run `npm run test:ci` = `ng test --watch=false`. Under `--coverage`, `import.meta.url` / the `__dirname` shim resolves **two levels shallower**. Three converging proofs: `rebranding-guide.doc-presence.spec.ts` read `'# AiClient\n\nThis project was genera…'` (= `frontend/ai.client/README.md`) when `resolve(SPEC_DIR, './README.md')` should reach `src/branding/README.md`, which begins `# Rebranding Guide` on both branches; `prestart-generator-parity.spec.ts` fails at **collection** on line 31 = `readdirSync(resolve(SPEC_DIR, '../../scripts/branding'))`, an ENOENT; and both guards *find violations* rather than nothing (`surface-literal-guard.spec.ts:44` does `path.join(__dirname)`, `tailwind-theme-import-guard.spec.ts:30` does `path.resolve(__dirname, '..')` — a shallower base walks the whole package, incl. generated Tailwind CSS full of literal hex and a direct `@import "tailwindcss"`). **PR CI structurally cannot catch this class** — it never passes `--coverage`. Fix = anchor the roots to a stable constant *and* put the coverage invocation somewhere a PR can see. **(b)** `nightly-develop-PlatformStack` is wedged in `DELETE_FAILED`: `Stack:arn:aws:cloudformation:us-west-2:490617140655:stack/nightly-develop-PlatformStack/… is in DELETE_FAILED state and can not be updated.` → `exit 1`. Nothing self-heals. ⚠️ **Reproduce (a) before fixing it** — `cd frontend/ai.client && ./node_modules/.bin/ng test --no-watch --coverage` on `develop`; the mechanism is inferred from logs, not executed. ⚠️ ⚠️ **single-source** on (b): this session's AWS creds are account 029812070295, not dev-ai 490617140655, so `StackStatusReason` and the retained-resource list could not be read — read them before deleting, since a stuck ephemeral stack usually means a retained resource that is **still billing**. ⚠️ Do **not** combine with the vitest 4→5 major — the suite is not a trustworthy migration baseline until green. Mirror-image of the [2026-09-04] "never resolve a flaky entry on a green streak" rule: this is a deterministic red that read as noise for a week, with **0 issues filed**. - ### [2026-09-11] Retire the hand-placed system `cachePoint` — `CacheConfig.system_prompt_ttl` / `tools_ttl` are already in our pin +- **✅ PARTIALLY SHIPPED 2026-09-18** — check **(b) is DONE**: the false 1.51-era invariant at `agent_factory.py` is rewritten and now states the real mechanism (upstream `_should_cache_system()` stands down on `not any("cachePoint" in block …)`; measured on the pinned 1.55.0 to yield exactly ONE system cachePoint with our block present and one without it), and the old claim is explicitly marked *"do not restore it"*. Checks (a) and (d) were already answered. **Still open: (c)** — the `tools_ttl` mixed-TTL evaluation against the 54%-cheaper layered technique, and the **1.55.0 → 1.55.1 patch bump**, which did NOT ship (`pyproject.toml` still pins `strands-agents==1.55.0`). The bump carries #4199 (malformed snapshot IDs) and #4228 (context-manager parity). - **Source**: research/2026-09-11.md ▸ Top 5 #2 — https://github.com/strands-agents/harness-sdk/blob/python/v1.55.1/strands-py/src/strands/models/model.py + https://github.com/strands-agents/harness-sdk/issues/4168 (**closed 2026-09-04**). **Supersedes the [2026-09-04] "Gate the Strands 1.51 → 1.54 bump on the system-cachePoint collision" entry** — we are already on 1.55.0, so the bump is done and only the collision question remains. Merge them at review. - **Surface**: backend — `core/agent_factory.py:213-224` (the hand-placed system `cachePoint`; the now-false comment at :222), `core/model_config.py:375-400` (`strategy="auto"`, the `bedrock_cache_points_supported()` gate, the ~40-line cachePoint-budget comment at :380), `backend/pyproject.toml:59,74`. - **Effort × Impact**: M × H - **Subtracts**: yes — the hand-placed `SystemContentBlock`, the ~40-line comment defending it, a comment asserting a false invariant, and possibly `bedrock_cache_points_supported()`. The library-native subtraction this skill weights for, finally payable. - **Status**: open — **the standing §2a caching watch finally paid out, and the payout is already inside the version we pinned three days ago.** `CacheConfig` at 1.55.0/1.55.1 carries `strategy`, `ttl`, **`system_prompt_ttl: bool | str = True`**, `cache_key`, **`tools_ttl: bool | str | None`**, and `strands/models/_openai_cache.py` exists. Four checks: (a) does `strategy="auto"` + `system_prompt_ttl` place the point our block places — delete ours if so; (b) rewrite `agent_factory.py:222`, which asserts *"auto strategy strips only message-level cachePoints, never system ones"*, a 1.51-era fact; (c) evaluate `tools_ttl` against the layered mixed-TTL technique the [2026-08-14] cookbook entry measured at **54% cheaper** upstream — previously blocked, now reachable; (d) ~~re-test whether `bedrock_cache_points_supported()` is still load-bearing now #4168 is fixed in our line~~ — **✅ ANSWERED 2026-09-11, measured: KEEP IT.** See the entry at the top of this queue. The tools half is redundant with upstream; the **system** half is not retirable by any upstream fix, because Bedrock — not the SDK — rejects the point with `AccessDeniedException`. Do not re-propose deleting it. **(a) is also now partly answered**: upstream's auto-injected system point only arms on `_cache_strategy == "anthropic"`, so deleting our block would silently drop the system cachePoint on any non-Claude Bedrock model we ever add. Also bump 1.55.0 → **1.55.1** (patch-only) for #4199 (malformed snapshot IDs) and #4228 (context-manager parity). **De-risking fact:** upstream's docstring says a hand-placed system cache point is *"honored rather than doubled"*, so an incremental migration will not double-write. ⚠️ **Never adopt a caching default on inspection alone** — #954 shipped on a wrong premise and measured **57% more expensive** live before #956 reverted it. `backend/scripts/probe_gpt56_cache_rates.py --mode both --grow-history` is the gate. ⚠️ The reference repo runs 1.54 green and **does not hand-place a system cachePoint at all** — their green build is not evidence for us; they have nothing to collide. ⚠️ On upstream `main` but unreleased: `cache_key` widens to `str | Literal[False] | None` and **auto-derives `strands-`** when a session manager is present — that is `build_prompt_cache_key()` retiring on the next minor, not this one. ⚠️ Also on `main`: a first-party context-manager/offloading stack (#4254, #4187, #4146, #4118, #4231) that will collide with `ContextOffloader` — the decisions log forbids a bare "delete ours" proposal, so watch it, don't adopt it piecemeal. -### [2026-09-11] Add GPT-6 Astra at the 272K tier — and make the context-tier boundary a catalog field -- **Source**: research/2026-09-11.md ▸ Top 5 #3 — https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-astra.html + https://aws.amazon.com/about-aws/whats-new/2026/09/openai-gpt-6-astra-on-amazon-bedrock/ (GA 2026-09-08) + https://github.com/aws-samples/sample-strands-agent-with-agentcore/pull/291 — **tier question resolved 2026-09-11** against https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-56-terra.html + the `AmazonBedrock` Price List offer `20260911124408` + our own `docs/specs/gpt-5-6-prompt-caching.md` billing observation. -- **Surface**: backend / frontend — `curated-models.ts`, the Mantle Responses path (`_create_mantle_model` / `bedrock_mantle_config`, per-model region routing), `admin/services/model_access.py` for the role grant, the model picker. -- **Effort × Impact**: L–M × H — the Mantle Responses path already exists; this rides it. -- **Subtracts**: no — addition. Justified because it *prevents* a regression we would otherwise import. -- **Unlocks**: - - **A genuine 1M-token frontier tier** (1,050,000 context / 128,000 max output) alongside Fable 5.1 — the first catalog model where a whole corpus fits in one turn. - - **A tier-aware catalog.** Today `maxInputTokens` doubles as both a capability bound and a pricing cap; Astra is the second model where those differ. Making the boundary explicit lets the admin page show it, lets cost projections respect it, and lets quota math account for Astra's **10× output-token quota burn** — which matters directly against 1.20.0's finding that the campus ceiling is the Bedrock TPM quota, not compute. -- **Status**: open — **✅ BLOCKING QUESTION ANSWERED: the tier is selected by the ACTUAL token count of the request, not by the declared window. `272000` is therefore a *policy* choice, not a correctness requirement — but it should still ship, for a corrected reason (below).** AWS states the rule in prose **nowhere**; the answer rests on three independent strands that converge, two of them external. - - **(1) Price List API — SKU shape.** In offer `20260911124408`, GPT-5.6 Terra/Luna carry `-long-ctx` as a value of the **`tokenType` attribute** under an *identical* `model` attribute, orthogonally across all four token buckets and all three service tiers: `input_tokens_mantle` $2.64 vs `input-tokens-long-ctx` $5.28; `cache-write-tokens-30m` $3.30 vs `…-30m-long-ctx` $6.60; `Cache Read Input Tokens` $0.264 vs `cache-read-tokens-long-ctx` $0.528; `output_tokens_mantle` $15.84 vs `output-tokens-long-ctx` $23.76. A `tokenType` is a **per-token billing meter**, not a configuration or a profile — no `model`, `inferenceType` or profile attribute distinguishes the tiers. ⚠️ These are GovCloud (`UGW1-`) rows, the only region where the OpenAI family has long-ctx SKUs published, and **GPT-6 Astra has ZERO SKUs in the entire 11,657-SKU offer file** — so the Price List speaks to the *family convention*, not to Astra directly. - - **(2) The GPT-5.6 Terra model card refutes the "declared window" reading outright.** Its card states **Context window: 1M tokens** and lists exactly **one model ID per endpoint** (`openai.gpt-5.6-terra` / `us.` / `global.`) — there is no long-context variant ID or inference profile to declare. Yet it publishes *both* a Short Context (272K) and a Long Context (1M) price table. If the tier followed the declared window, every Terra call would bill Long and the Short table would be unreachable. Its GovCloud tables match the Price List SKUs **to the cent** ($2.64 / $3.30 / $0.264 / $15.84 short; $5.28 / $6.60 / $0.528 / $23.76 long) — two independent sources agreeing, which also proves `-long-ctx` *is* the "Long Context Window" table. - - **(3) Our own live billing corroborates it.** `docs/specs/gpt-5-6-prompt-caching.md` records that in dev-ai Cost Explorer *"every row observed is `_standard` and no `-long-ctx` usage type has ever appeared in this account"* — while we call the **1M-window** `us.openai.gpt-5.6-*` IDs with requests held under 272K. A 1M-declared model billing at short-context rates is only possible if the tier follows actual request size. - - **(4) There is no declaration channel at all.** `maxInputTokens` is *ours*, not AWS's: in `backend/src/` it is read only to populate `final_metadata["contextWindow"]` and the compaction threshold (`stream_coordinator.py:710-712`, `:2808-2818`) and to serialize the catalog. **It is never placed in a Bedrock request.** Bedrock cannot see it, so it cannot price on it. -- **Consequence — our own framing was wrong, and the correction matters.** "Copying their row opts the whole fleet into the expensive tier" is **false**. Registering Astra at `maxInputTokens: 1_000_000` does not double our input cost; it opens the 2× card only on turns that *actually* exceed 272K. **But ship `272_000` anyway**, for the reason already written at `curated-models.ts:363`: `CuratedModel` holds **one flat rate per bucket**, and the cap is what keeps that single rate arithmetically correct. Raising it does not over-charge the fleet — it silently **under-charges** every turn that crosses the boundary (input 2×, output 1.5×, and per the SKU shape cache-read/write 2× as well). Same action, different reason, and the difference is the whole point of the generalization: **the tier boundary is a *billing-model* field, not a capability field.** ⚠️ Also note the ref repo sets `maxInputTokens: 1000000` on Sol/Terra/Luna too — their Astra row is consistency with their own family rows, **not** an independent judgment about Astra's tiering, so it is not evidence either way. -- **Residual unknown (honest, and only matters if we ever raise the cap).** AWS nowhere publishes the **predicate**: whether the 272K test is on input tokens alone, input+output, or total including cache reads; nor whether crossing it reprices the whole request or only the excess. The SKU shape — all four buckets have long-ctx twins — implies **whole-request repricing**, but that is inference, not a published rule. What would settle it: a single measured call above 272K in dev-ai with a same-day Cost Explorer read (the `probe_gpt56_cache_rates.py` harness already does this shape), or an AWS support case. Not worth doing while the cap holds. -- **Rates (unchanged, re-verified on the card).** Geo CRIS Short (≤272K) **$11.00 in / $13.75 cache-write / $1.10 cache-read / $55.00 out** per MTok; Long (1.05M) **$22.00 / $27.50 / $2.20 / $82.50**. Global CRIS a flat 10% under Geo ($10.00/$12.50/$1.00/$50.00 and $20.00/$25.00/$2.00/$75.00). In-Region is priced identically to Geo but is **not offered** on `bedrock-runtime`. -- **Also confirmed while there:** - - **`rejectsTemperature: true` is REAL and family-wide — not an Astra-specific invention.** In the ref repo's catalog (`revision 2026-09-09.1`) it is `true` for Astra, **all three GPT-5.6 models**, Claude Opus 5, Sonnet 5 and Grok 4.6 — it tracks reasoning models generally, and we already run the GPT-5.6 family. **Neither Astra's nor Terra's card has a supported-inference-parameters section at all**, so AWS does not contradict it, merely omits it. ⚠️ **single-source** for Astra specifically (ref repo + PR #291's claimed live `direct Bedrock Responses invocation`). Per **#915**, ship it as an explicit **non-empty** `supportedParams` that omits temperature — **never an empty `supportedParams`**, which the guard reads as a bypass. - - **Harness gaps — all CONFIRMED verbatim on the Astra card.** `bedrock-runtime` Not Supported: **server-side tool use, intelligent prompt routing, count tokens, structured outputs**. **CRIS-only** — Programmatic Access lists In-Region "Not supported", leaving `us.openai.gpt-6-astra` / `global.openai.gpt-6-astra`. The endpoint split is real: Guardrails and application inference profiles are supported on `bedrock-runtime` **Converse API only**, and application inference profiles are **Not Supported** on `bedrock-mantle`. `bedrock-mantle` is **us-west-2 only**. - - ⚠️ **NEW, and not previously in this entry: prompt caching is NOT listed for Astra on `bedrock-runtime`.** Terra's card lists Implicit **and** Explicit Prompt Caching under *both* endpoints; **Astra's lists them under `bedrock-mantle` only (Responses API only)**, and caching appears in **neither column** of its `bedrock-runtime` feature table. The plan is `us.openai.gpt-6-astra` on `bedrock-runtime`, so **resolve this before banking any cache savings** — our entire prompt-cache contract assumes explicit cache points, and a 1M-window model with no caching is a very different cost profile. - - **Cache ratios 0.1× read / 1.25× write — confirmed twice.** Arithmetic on the card holds in both tiers ($11.00→$1.10/$13.75; $22.00→$2.20/$27.50), and AWS states the rule in prose: *"Cached input is billed at a 90% discount compared to uncached input tokens, and tokens written to cache are billed at 1.25 times the uncached input rate."* - - **The 30-minute TTL is a real SKU dimension, not card prose** — the Price List `tokenType` is literally `cache-write-tokens-30m`. **Do not route it through `cache_ttl_seconds_for()`** (5m/1h, Anthropic). - - **10× output TPM burn — CONFIRMED verbatim**, and note the scoping: *"On the `bedrock-runtime` endpoint, limits are managed as tokens per minute (TPM) with a 10x burndown rate, where 1 output token consumes 10 tokens."* Identical sentence on Terra's card. ⚠️ It is stated **for `bedrock-runtime`**; the cards say nothing about burndown on `bedrock-mantle`. This lands directly against 1.20.0's finding that the campus ceiling is the Bedrock TPM quota, not compute — at 128,000 max output, one saturating Astra turn burns **1.28M TPM**. - - ⚠️ Dev/prod prefix divergence still applies (`global.*` is SCP-denied in dev, fine in prod). - -### [2026-09-11] Correct the rate provenance in `CLAUDE.md` and make cache-read a per-model input — Grok 4.6 is the second ratio-breaker -- **Source**: research/2026-09-11.md ▸ Top 5 #4 — a full diff of the republished offer files (`AmazonBedrock` `20260804165549` → `20260911124408`) + https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-xai-grok-4-6.html. **This closes the ⚠️ on the open [2026-09-04] "Finish the #914 rate correction" entry** — its provenance question is now answered; merge and ship the two together at review. -- **Surface**: docs / frontend — the prompt-cache contract bullet in `CLAUDE.md`; `curated-models.ts:96-97` (the `cacheRead = input × 0.1` helper); and the six stale `$2.50/MTok` sites the [2026-09-04] entry enumerates, headed by `model_config.py:380`. -- **Effort × Impact**: L × M–H — cheapest high-leverage item after #1. -- **Subtracts**: yes — a false sourcing instruction in the file every contributor reads before touching cost code; a hardcoded ratio now falsified by **two** live models; six duplicated wrong constants. -- **Status**: open — three verified facts. **(1) The provenance claim reproduces as FALSE for the second consecutive week.** `CLAUDE.md` says rates "come from the AWS Price List API, not the pricing page"; the entire `AmazonBedrock` offer file contains **10 Claude SKUs, none newer than Claude 3**, and Haiku 4.5 / Sonnet 4.6 / Fable 5.1 / GPT-5.4 / Nova Micro have **no us-west-2 SKUs at all**. Correct text: **per-model AWS model cards are authoritative for Claude, Nova and the OpenAI/Mantle family; the Price List API is authoritative for xAI, Google and AgentCore.** Reproducible commands are in research/2026-09-11.md ▸ Sources. **(2) The `0.1×` cache-read ratio now has two counterexamples.** Fable 5.1 reads at **0.025×**; **Grok 4.6 reads at 0.25×** ($0.55 against $2.20 input) — and Grok is worse than a wrong number, because it has **no cache-write SKU at all** and its card lists only *implicit* caching, so there is nothing for our explicit-`cachePoint` contract to place. Fix = accept cache-read as an **input** on the model row with `0.1×` as a documented default. **(3) The 1.100× Regional premium survives independently** — re-confirmed on the 18 newly-published us-west-2 Grok SKUs ($2.20 Geo vs $2.00 Global) and on both Astra tiers — so #914's *substance* is safe even though its *provenance* was not. ⚠️ Treat Grok as a **non-cached** model in cost projections until implicit caching is measured against `cacheStatus` on a real warm turn. ⚠️ Also noted: all 48 price changes in the republish were **GovCloud-only** GPT-5.6 moves (Luna $1.32 → $0.264/MTok, Terra −20%) — no us-west-2 change, but AWS is repricing that family somewhere, so re-check Oregon next week. - ### [2026-09-11] Retire the `ClientSession` monkeypatch via `_meta` `serverInfo` — without waiting on `server/discover` +- **✅ PARTIALLY SHIPPED 2026-09-18** — the urgent half is closed. `backend/pyproject.toml` now carries `constraint-dependencies = ["mcp<2"]` with a comment recording the whole mechanism (Strands 1.55.0 declares `mcp>=1.23.0,<2.2`, so the resolver was free to cross a major under a symbol patch never tested against it; the failure mode is silent — App-frame headers go generic, not an ImportError). **Still open: (b)/(c)** — migrating the header onto `ext-apps` v2.0.0's documented `io.modelcontextprotocol/serverInfo` in result `_meta` and deleting the patch. Deferred in reviews/2026-09-11.md until **2026-09-25** — not yet due. ⚠️ Still unverified: whether our Gateway targets and Lambda FastMCP servers actually populate that field. - **Source**: research/2026-09-11.md ▸ Top 5 #5 — https://github.com/modelcontextprotocol/ext-apps/releases/tag/v2.0.0 (2026-09-08) + the MCP-client changes in https://github.com/strands-agents/harness-sdk/releases/tag/python%2Fv1.55.1. **Narrows the open [2026-09-04] "Migrate the MCP Apps host off `initialize` to `server/discover`" entry** — this is the half available now; keep that entry for the rest. - **Surface**: backend — `integrations/mcp_apps.py:23-32` (the `ClientSession` symbol patch) and `:673` (the `serverInfo` capture); the App-frame header fields (`serverName`, `icon`, `toolName`) on the `ui_resource` event. - **Effort × Impact**: M × M @@ -182,6 +311,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Status**: open — cheapest high-leverage item after #1. Labels available essentially for free: `tools_changed`, `system_prompt_changed`, `history_changed`, `agent_switched` (the `agentSwitched` flag already exists), `ttl_expired`, `cold_start`. Claude Code shipped exactly this in 2.1.260 ("a likely cause for prompt-cache misses"), which is corroboration the ergonomics are worth it. ### [2026-09-04] Close the `ActiveSessionCount` alarm item — the metric now exists — and scope instance-based Runtime against W5 +- **✅ PARTIALLY SHIPPED 2026-09-18** — part **(1) is DONE**: [#1051](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1051) wires the concurrent-Runtime-session alarm through `AlarmFactory`, and it was sized against evidence rather than a guess — `config.ts:481` records it as *"Measured against 7 days of real prod `ActiveSessionCount` (2026-09-04..11)"*. **Still open: part (2)**, the instance-based Runtime arithmetic against W5 — 889 `Runtime:Instance-based:*:Management-Hours` SKUs vs consumption rates, crossover at our measured 18–50 min microVM lifetimes. **Fifth month with no proposal against the largest share of the bill** (Runtime memory, 73% of the AICC bill); reviews/2026-09-18 recommends doing the paragraph or declining W5 as a kaizen lane. - **Source**: research/2026-09-04.md ▸ Top 5 #5 — AWS Price List API (`AmazonBedrockAgentCore` us-west-2: **889 new `Runtime:Instance-based::Management-Hours` SKUs** in the 2026-09-01 republish) + https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/release-notes.html. **Unblocks the [2026-07-10] `ActiveSessionCount` entry** — merge them at review. - **Surface**: infrastructure — `lib/constructs/observability/` (the `AlarmFactory` that #910 made the only sanctioned path), the AgentCore Runtime construct; plus W5 in `project_cost_effectiveness_roadmap.md` - **Effort × Impact**: L (alarm) + M (W5 scope) × M–H @@ -210,7 +340,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Surface**: backend (`core/model_config.py` cache-point placement + `CacheConfig(strategy="auto")` at L389; `session/turn_based_session_manager.py` truncation anchor + repeated-compaction path; `session/compaction_models.py` `cache_ttl_seconds`; system-prompt assembly) - **Effort × Impact**: L × M–H - **Subtracts**: yes — every item is "find waste and delete it"; no new abstraction, no new dependency -- **Status**: open — **highest confidence-per-hour item in the scan.** Four checks: (1) **byte-stable prefix** — grep system-prompt assembly for time/random/env-derived values (the cookbook measured a **44% swing from one `datetime.now()`**; this also tests whether we're *reading* `systemPromptHash`); (2) **layered mixed-TTL breakpoints** — 54% cheaper upstream; ✅ **the "blocked by Strands #3758" caveat is STRUCK as of 2026-08-28** — the Python-side fix shipped in **1.53.0 via #3858**, so the technique becomes available on the [2026-08-28] Strands 1.51 → 1.54 bump. Sequence it *behind* that bump, since #3681 changes the cache-point placement this would tune; (3) **repeated-compaction audit** — does pass 2 preserve tool pairing and avoid re-writing the prefix? (our death-spiral incident says this is where the money went); (4) **retry cap with jitter** — an uncapped retry on a 424/throttle re-writes the whole prefix per attempt. +- **Status**: open — **(2) layered mixed-TTL is now BUILT behind a flag (2026-09-16, compaction PR-5: `AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h`, default off; gate = `scripts/probe_static_prefix_ttl.py` + a dev week of cost rows, which bill the 1h premium honestly). (3) repeated-compaction audit answered by the compaction stack (#1125→#1131: hysteresis, bounded summary, paid-when-free apply). (1) and (4) still open. Original: **highest confidence-per-hour item in the scan.** Four checks: (1) **byte-stable prefix** — grep system-prompt assembly for time/random/env-derived values (the cookbook measured a **44% swing from one `datetime.now()`**; this also tests whether we're *reading* `systemPromptHash`); (2) **layered mixed-TTL breakpoints** — 54% cheaper upstream; ✅ **the "blocked by Strands #3758" caveat is STRUCK as of 2026-08-28** — the Python-side fix shipped in **1.53.0 via #3858**, so the technique becomes available on the [2026-08-28] Strands 1.51 → 1.54 bump. Sequence it *behind* that bump, since #3681 changes the cache-point placement this would tune; (3) **repeated-compaction audit** — does pass 2 preserve tool pairing and avoid re-writing the prefix? (our death-spiral incident says this is where the money went); (4) **retry cap with jitter** — an uncapped retry on a 424/throttle re-writes the whole prefix per attempt. ### [2026-08-14] Guard against `bedrock-agentcore` #564 — the one failure class the 1.21.0 bump did NOT close - **Source**: research/2026-08-14.md ▸ Community + GitHub issues; the carried-forward residue of the four now-resolved `bedrock-agentcore` bump entries — https://github.com/aws/bedrock-agentcore-sdk-python/issues/564 (**still open**; #482 and #571 closed, #564 did not) @@ -247,7 +377,7 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. - **Surface**: backend (`agent_factory.py` — `plugins=` already plumbed; a custom S3 `Storage` backend is the real build — the plugin ships InMemory/File only, and references must survive AgentCore Runtime restores across turns) - **Effort × Impact**: M–H × M–H (payoff directly measurable by the PR #697 cache/cost metrics) - **Subtracts**: partial — bounds MCP/tool payload growth at the source instead of relying solely on reactive below-anchor truncation in `TurnBasedSessionManager` (which stays for legacy history) -- **Status**: open — spike before commitment; four known gotchas: (1) `evict_after_cycles=20` runs on `BeforeModelCallEvent` and touches *prior* messages — potential byte-stability cache-buster, verify semantics or set `None` and expire via S3 lifecycle; (2) `model.count_tokens` per tool result adds latency, and Bedrock CountTokens rejects `us.*` inference-profile ids (de-prefix precedent in context attribution); (3) adoption flips `toolConfigHash` once (expected; the new tool must land in the deterministically-ordered tool list); (4) check SPA tool-result rendering against placeholder content. +- **Status**: **ADOPTED 2026-09-15** as compaction PR-4 (branch `feature/compaction-offload-escalation`, stacked on #1129; spec `compaction-model-relative-thresholds.md` §3.6). Gotchas as resolved: (1) `evict_after_cycles=None`, expiry via a 90-day S3 lifecycle rule on `compaction-offload/`; (2) a chars/4 pre-filter skips CountTokens for results under half the gate, and `CountTokensBedrockModel` already de-prefixes `us.*`; (3) `retrieve_offloaded_content` lands once in the tool list; (4) the preview is plain text in the tool result. Original entry: spike before commitment; four known gotchas: (1) `evict_after_cycles=20` runs on `BeforeModelCallEvent` and touches *prior* messages — potential byte-stability cache-buster, verify semantics or set `None` and expire via S3 lifecycle; (2) `model.count_tokens` per tool result adds latency, and Bedrock CountTokens rejects `us.*` inference-profile ids (de-prefix precedent in context attribution); (3) adoption flips `toolConfigHash` once (expected; the new tool must land in the deterministically-ordered tool list); (4) check SPA tool-result rendering against placeholder content. ### [2026-07-17] Adopt Strands `Limits` on the unattended Scheduled Runs / headless lane - **Source**: research/2026-07-17.md ▸ Top 5 #3 — convergent harness rail (Claude Code 2.1.212 spawn cap + opencode 1.18.2 `subagent_depth`); Strands `Limits` now available (we're on 1.47). @@ -427,6 +557,22 @@ Items added by `kaizen-research`, consumed by `kaizen-review-prep`. ## Resolved +### [2026-09-11] Fix the nightly, both halves — anchor the branding specs' paths, and put `--coverage` where PR CI can see it → RESOLVED — **BOTH HALVES SHIPPED**; seven consecutive nightly greens +- **Decision**: resolved by outcome (a merged PR or a superseding entry), not by a review mark. +- **Reasoning**: Last review's Proposal #1, and it converted. `Nightly Build & Test` on `main` has run **seven consecutive successes** (2026-09-11T17:50Z, 2026-09-11T20:50Z, then 09-12 → 09-18 daily) after seven consecutive failures (09-05 → 09-11). Both root causes are closed: the `--coverage` path-resolution class and the wedged `nightly-develop-PlatformStack` `DELETE_FAILED` stack. ⚠️ **The structural half is worth re-checking once**: the review's fix specified putting the coverage invocation *somewhere a PR can see*, because PR CI never passes `--coverage` and so structurally cannot catch that class again. Seven greens prove the specs pass; they do not prove a PR would now catch a regression in them. If `test-frontend` still runs `--coverage` only on the nightly, the gap the entry named is still open — re-queue that half alone. Per the [2026-09-04] rule (c), this is resolved on a **root-cause fix**, not on the green count. +- **Reviewed in**: reviews/2026-09-18.md ▸ What Shipped + +### [2026-09-11] Correct the rate provenance in `CLAUDE.md` and make cache-read a per-model input — Grok 4.6 is the second ratio-breaker → RESOLVED — **SHIPPED**, all three parts verified on disk +- **Decision**: resolved by outcome (a merged PR or a superseding entry), not by a review mark. +- **Reasoning**: Last review's Proposal #2, converted **standalone** — no carrier, which is the observation that falsified reviews/2026-09-11.md ▸ Take. Verified 2026-09-18: **(1)** `CLAUDE.md` now reads *"per-model AWS **model cards** are authoritative for Claude, Nova and the OpenAI/Mantle family; the **Price List API** is authoritative for xAI, Google and AgentCore"*, with the 10-Claude-SKU / none-newer-than-Claude-3 enumeration recorded as the reason and the 1.100× Regional premium preserved as still reproducing. **(2)** Zero remaining `$2.50/MTok` or `$2.5/MTok` sites across `backend/src`, `infrastructure/lib` and `frontend/ai.client/src` (was six). **(3)** Cache-read is now a per-model **input**: `ratesWithDerivedCache(input, output, cacheReadMultiplier = DEFAULT_CACHE_READ_MULTIPLIER)` with the docstring *"Pass `cacheReadMultiplier` when a model's card says otherwise"* — the hardcoded `input × 0.1` is gone. **Residue carried, not dropped**: Grok 4.6 should still be treated as **non-cached** in projections (0.25× read, no cache-write SKU, implicit-only caching) until measured against `cacheStatus` on a real warm turn, and AWS was repricing the GPT-5.6 family in GovCloud — re-check us-west-2. +- **Reviewed in**: reviews/2026-09-18.md ▸ What Shipped + +### [2026-09-11] Add GPT-6 Astra at the 272K tier — and make the context-tier boundary a catalog field → RESOLVED — **SHIPPED** at the 272K tier, with the caching caveat carried into the code +- **Decision**: resolved by outcome (a merged PR or a superseding entry), not by a review mark. +- **Reasoning**: Last review's Proposal #4, converted standalone. `CURATED_BEDROCK_RESPONSES_MODELS` now carries `key: 'gpt-6-astra'` / `modelId: 'us.openai.gpt-6-astra'` at `ratesWithDerivedCache(11.0, 55.0)` (Geo CRIS Short), `maxOutputTokens: 128_000`, `knowledgeCutoffDate: '2026-04-30'`, and a non-empty `supportedParams` via `openaiResponsesParams(128_000)` that omits temperature — per **#915**, never an empty `supportedParams`, which the guard reads as a bypass. `model-catalog.page.spec.ts` pins the id and the $11.00 input rate. **The entry's own ⚠️ made it into the source rather than being lost at merge**: an eleven-line comment records that Astra's card lists prompt caching under `bedrock-mantle` **only** while this row routes over `bedrock-runtime`, and explains why `supportsCaching` still inherits `true` (a `false` would zero the cache-rate fields and price cached tokens at $0.00 while AWS bills them in full — a stale capability chip is the cheaper error). **Residue carried**: that verification is still owed before anyone banks cache savings on this row. The **context-tier-boundary catalog field** — the generalization half of the entry — did **not** ship; `maxInputTokens` still doubles as capability bound and pricing cap, correct only because the cap is pinned at 272K. Re-queue that half if a second tiered model lands. +- **Reviewed in**: reviews/2026-09-18.md ▸ What Shipped + + ### [2026-09-11] ✅ VERIFIED NOT APPLICABLE — AgentCore SDK #659 (silent SSE corruption on unserialisable events) → RESOLVED — **premise verified NOT APPLICABLE** - **Decision**: resolved by outcome (a merged PR or a superseding entry), not by a review mark. - **Reasoning**: Closed on inspection in research/2026-09-11.md and recorded so it is not re-proposed. Verified three ways: we never import `BedrockAgentCoreApp` (zero grep hits — we run our own FastAPI app in the Runtime container and hand-roll SSE); `stream_processor.py:117-121` base64-encodes `bytes` explicitly; and `_create_event()` at `:157` runs `_serialize_object(data)` on every emitted event, with `redactedContent` routed through it at `:910`. The scanning subagent ranked this "highest of the week for us" and it is not a finding — the negative result is. diff --git a/docs/kaizen/reviews/2026-09-18.md b/docs/kaizen/reviews/2026-09-18.md new file mode 100644 index 000000000..aad7e4818 --- /dev/null +++ b/docs/kaizen/reviews/2026-09-18.md @@ -0,0 +1,223 @@ +# Kaizen Review — Friday, September 18, 2026 + +> Prepared 08:20am MT. Review window: [September 11 – 18] (7 days). +> Source: review-queue.md (52 open items) + merged-PR outcomes against reviews/2026-09-11.md. +> ⚠️ **This morning's research doc does not exist yet.** `kaizen-research` fired at +> 07:50 MT (vs 06:09 last week) and was still running 30 minutes into this prep; +> last week's run took 2h18m. Per the skill's own priority the queue's `## Open` +> entries are the primary source and research is one input — so this review is +> built queue-first and outcome-first, which is also the correction +> reviews/2026-09-11.md ▸ Take made about itself. The pipeline gap is Proposal #5. + +## Week in Review + +The biggest week in the repo's history by volume — **114 PRs merged, 148 non-merge commits, three releases (1.20.0, 1.21.0, 1.22.0), zero reverts** — and three whole stacks landed: compaction model-relative thresholds (#1125–#1132), document offload (#1137–#1143), and response feedback (#1142, #1145, with #1146/#1148 still open). **Last week's forum converted 6 of its 9 proposals**, which falsifies last week's central claim that the ranking was sound and the shipping model broken: #2, #3, #4 and #7 all shipped *standalone*, without the "named carrier" that Take insisted they needed, and the nightly (#1) has run **seven consecutive greens** after seven consecutive failures. + +The debt this week left is not shipping velocity — it is that **two specs which each wrote their quality gate as a veto both shipped with defaults live and the harness unbuilt.** The compaction spec's §5 "veto before default change" eval does not exist on disk, and the document-offload epic's companion evaluation spec is still `Status: Draft`. `develop` auto-deploys to dev, so the window named in both specs — before prod — closes at the next release. Against that, the week's best artifact is a validation sweep of the epic that just shipped: **#1147 found a 13.7× measurement error in the very instrument the ship/abandon rule reads off**, plus a kill switch that made things strictly worse than the feature's absence. It has been mergeable and unreviewed for a day. + +The one class that has still never converted, for the **fifth consecutive cycle**, is the process item: `decisions.md` untouched since 2026-05-18, the POC-comment loop still described in `kaizen-research/SKILL.md`, and `kaizen-review-prep/SKILL.md` unmodified since 2026-05-10. Last week said it would ride the review PR. It didn't. + +## Friction — the week's signal + +### Repeated patterns (≥2 occurrences) + +- **A quality gate written as a veto, shipped unrun, with defaults already live** (2 times) — `compaction-model-relative-thresholds.md` §5 says *"Veto before default change in prod"*; `document-offload-evaluation.md` §2 says *"no cost result, however good, ships a confirmed quality regression"* and is still `Status: Draft`. Verified: `find backend -name "*eval*"` returns only vendored site-packages — **neither harness exists**. Meanwhile `COMPACTION_CEILING_RATIO = 0.5` / `HARD_CEILING_RATIO = 0.7` are live in dev and the offload epic is default-on. + - *Hypothesis*: not carelessness, and not a flag-default problem (see the correction below — the first draft of this document got that wrong). Both specs chose the strongest available gate wording and neither had an owner for the harness, because a harness is the one deliverable in an epic that ships no user-visible behaviour. The asymmetry is structural: `CLAUDE.md`'s cost tenet says *"when cost and answer quality genuinely conflict, quality wins"*, which cannot be tested while every cost change is measured and no quality change is. + - *Candidate fix*: run the §4.3 eval, or **record the waiver in the spec** — a gate documented as waived tells the next reader what happened, where a gate merely unrun reads as passed. The cheap first version is not a new harness at all: #1142/#1145 just shipped `F#` down-thumb rows, so a comparison across turn classes is available now (Proposal #3). + +- **Real defects have no trackable home** (2+ cycles) — **0 issues created in 8 days** against 114 merged PRs; **1 open issue in the whole repo**. Last week recorded the same in a 110-PR window. #1147 carries six findings, four still unfixed, and they live in a PR body and this queue — nowhere else. + - *Hypothesis*: the queue and PR bodies have absorbed the tracker's job. That works while the author is the reader and fails when a finding outlives its PR. Verified 2026-09-04 that no `kaizen` label exists, so the previously-proposed ✅→issue layer has no substrate. + - *Candidate fix*: none as a build — and note the mitigation already happened without this forum: **#1147 queued its own residue** into `review-queue.md` as three well-scoped entries. That is the working pattern; the right move is to merge it, not to re-file it (Proposal #1). + +- **Process/meta items never convert** (5th cycle) — the four standing declines, the POC-loop retirement, and the review-prep skill edit. Recommended Ship four times, "ship in this PR" once, ridden once. Zero keystrokes. + - *Hypothesis*: not priority — **authorship**. Six of nine engineering proposals converted in seven days, including three ten-minute comment fixes. The difference is that nobody's in-flight branch ever passes through `decisions.md`. + - *Candidate fix*: last week committed to the answer — *"if it does not land this week, the honest move next week is the Decline branch, permanently."* Proposal #4 takes that branch. + +### One-offs worth watching + +- **A `develop`-push deploy failure recurred** (1 occurrence) — `Frontend Deploy` failed on `develop` 2026-09-16 on the #1121 merge. Different workflow from the 2026-09-01 `Backend Deploy` case, same class: fails on a push, self-heals on the next. Gives the [2026-09-04] platform-before-backend entry its second data point (Proposal #7). +- **Three PR-CI failures on feature branches** — #1149, #1142, #1137 (plus #1125, #1126 pre-merge), all resolved before merge. Normal signal; noted because last week's count was also three. +- **Two kaizen PRs will conflict in one file.** #1147 and this review PR both insert at the top of `review-queue.md` ▸ `## Open`. Trivial to resolve, but worth knowing before merging: **#1147 first is the cleaner order**, since its entries are the ones later work reads. + +### Silence that matters + +- **⭐ Zero comments on every kaizen PR, ever.** #1045 (last week's research), #1046 (last week's review) and #926 (two weeks ago) each have **0 comments and 0 reviews**. Seven cycles for the POC-comment loop the skills describe as their feedback channel, with the "tested outranks untested" tiebreak never once firing — while 6 of 9 proposals converted through merged PRs, the channel the skills do not describe. The loop is not underused; it has never been used. **And the substitute is visibly better**: the deepest analysis this week (#1147's rollout-percent entry) arrived as a *queue write on a PR branch*, and it is what corrected this document. +- **`kaizen-review-prep/SKILL.md` unmodified since 2026-05-10** — five reviews have proposed editing it, including one that specified it must ride the review PR. +- **`docs/kaizen/decisions.md` four months stale** (last write 2026-05-18, #341) while the queue carries items on their 6th, 8th and 11th cycles. The file exists to stop research re-proposing; it is not doing that job. +- **`cors-deployment` and `frontend-design` skills at 144 days** — sixth listing. reviews/2026-09-04 settled this as **keep, stop flagging**; the verdict was never written down, so it keeps resurfacing. A symptom of the bullet above, not an independent finding. + +## Proposals — ranked + +### 1. Merge #1147 — two fixes, and the offload epic's own measurement plan +- **Source**: direct observation — [#1147](https://github.com/Boise-State-Development/agentcore-public-stack/pull/1147), open since 2026-09-17, `MERGEABLE`, CI green, **unreviewed** +- **Surface area**: backend — `apis/shared/files/document_tokens.py` (new), `session/document_offload.py`, `session/document_context.py`, `session/document_rehydration.py`, `apis/shared/feature_flags.py`; plus `docs/kaizen/review-queue.md` +- **Change**: merge as-is. **(1)** `documentTokens` counted bytes, not PDF pages — Bedrock dual-encodes each page as an image *and* a text layer, so `bytes/4` missed the image channel: **6,894 against ~94,485 measured, 13.7× low**, reporting documents at 6.3% of the prefix where truth was ~87%. **(2)** `DOCUMENT_READ_ENABLED` was read in exactly one place (`routes.py:821`) and **zero** times under `session/` — verified independently today — so pulling the one kill switch an operator would reach for left restore advertising a handle for an uninjected tool and the live offload evicting bytes with the only recovery path off. +- **Subtracts**: yes — a 14×-wrong instrument, and a kill switch that made things **strictly worse than the feature's absence**. +- **Effort**: Low — it is a review and a merge button; 31 new tests, 9,050-pass suite +- **Impact**: High — §6.1 is explicitly the instrument the evaluation spec's §4.2 ship/abandon rule reads off. At 14× error those rows argue *abandon* for a change that in the same session cut the prefix **109.1K → ~15K**. A wrong instrument is worse than no instrument. +- **POC findings**: fully POCed — static review, 41 purpose-written probes, and a live dev clickthrough of the whole lifecycle (turn 1 full → turn 2 digest → turn 3 `document_read(page_range="44")`, 170 ms → slice aged on schedule). +- **Ship means**: review and merge #1147, **before** this review PR, so its queue entries land first. +- **Decline means**: the epic's own analytics keep reporting documents at ~6% of a prefix they dominate at ~87%, and its kill switch stays worse than useless. +- **Recommendation**: **Ship.** Cheapest high-impact item on the board by a wide margin, and it is the release gate for everything the epic is supposed to measure. ⚠️ Real behaviour change it flags itself: PDFs previously stuck below the 5,000-token floor forever become offload candidates — the bug working correctly, but it changes what gets evicted. `compaction_policy._block_tokens` shares the blind spot and is deliberately untouched; it drives cut thresholds and needs its own decision. + +### 2. Bound the `document_read` pattern regex — measured exponential, model-supplied, hangs the turn +- **Source**: #1147's own residue entry, which ranks it **first** among the four unfixed findings; re-verified on disk today at `apis/shared/files/document_read.py:528` +- **Surface area**: backend — `document_read.py` `_compile()` / `_grep_lines()` and the three `_*_pattern` callers +- **Change**: bound the input (pattern length, line length, match budget) and/or run the scan under a hard timeout, instead of handing a raw model-supplied string to `re.compile`. +- **Subtracts**: yes — an unbounded input path. `_compile` correctly degrades an *invalid* regex to a literal via `re.escape`, which catches malformed patterns but not expensive ones; there is no length cap, no complexity check, and Python's `re` has **no match timeout**. +- **Effort**: Low +- **Impact**: Med–High — measured at **0.87 s at 24 characters, 3.5 s at 26, 14 s at 28**, exponential, against real PDF lines of **60–100**. `pattern` is model-supplied and the search runs in `asyncio.to_thread`, so **the turn hangs to the 600 s SSE timeout** — a user-visible dead conversation, not just a busy worker. +- **POC findings**: timings measured in the sweep. Not reproduced as a fleet-level availability event; the per-turn hang is the established consequence. +- **Ship means**: a PR bounding `_compile`/`_grep_lines`, with the exponential case as a test. +- **Decline means**: accept a model-reachable 600-second hang, and record it. +- **Recommendation**: **Ship.** The only item here with a measured exponential and no owner, and it is a small local fix in one function. Both #1147's author and this review independently ranked it top of the residue. + +### 3. Run the two quality vetoes — or record the waiver in the spec +- **Source**: review-queue.md ▸ [2026-09-18] (new) and [2026-09-16] (the queue's #1 item for the second week); `compaction-model-relative-thresholds.md` §5; `compaction-over-threshold-cache-spiral.md` §4.3; `document-offload-evaluation.md` §2 +- **Surface area**: backend — the §4.3 eval harness (constraint retention / revision continuity / reference lookup), fixed-threshold arm as control, AgentCore Evaluations judges for the three families plus a blinded holistic judge outside the managed path; `session/compaction_policy.py`, `session/compaction_summary.py` +- **Change**: run paired arms on the same corpus and record the result in the thresholds spec §5 and the evaluation spec §4.2. Move `FLOOR_RATIO` / `CEILING_CAP_TOKENS` only on that evidence. +- **Subtracts**: no — it is the condition under which both stacks' defaults may stay on. Justified because otherwise the deepest, earliest cut the cost arc has ever shipped is defended by **dollars alone** ($73.70 vs $102.64 on a 20-session replay) with quality asserted. +- **Effort**: Med — or **Low** for the cheap first version, which is not a harness at all (see below) +- **Impact**: High — this is the counterweight the cost roadmap has never had, and the window both specs named ("before prod") closes at the next release. +- **POC findings**: cost side measured (2026-09-15 replay + PR-2/3 metrics). Quality side: no harness on disk, for either spec. +- **Ship means**: either build the §4.3 harness, **or** take the cheap version — #1142/#1145 just shipped `F#` down-thumb rows, so a **comparison across turn classes** (full / digest-only / retrieved) is available today with a query, not a build. Per response-feedback spec §9 that is a comparison between arms, never a quality score. +- **Decline means**: write the waiver into both specs so the next reader knows the gate was waived rather than passed. +- **Recommendation**: **Ship the cheap version this week; Defer the full harness 2 weeks (revisit 2026-10-02).** ⚠️ Score **A→B and B→C separately** — PRs 1–3 of the offload epic are a *correctness* fix where quality should go **up** and only PR-4 is the cost trade, so a strip-fix win can mask an offload regression. ⚠️ Second consecutive week as the queue's #1; if it slips again, take the Decline branch explicitly, because a recorded waiver beats a fourth deferral. + +### 4. The process writes — fifth cycle, and last week named the consequence +- **Source**: review-queue.md ▸ [2026-09-04] POC-retirement entry; reviews/2026-09-11.md ▸ Proposal #8 and its Take +- **Surface area**: skills + docs — `.claude/skills/kaizen-research/SKILL.md`, `.claude/skills/kaizen-review-prep/SKILL.md`, `docs/kaizen/decisions.md` +- **Change**: (a) retire the POC-comment loop — the `POC findings` field, the "tested outranks untested" tiebreak, the one-week-lag philosophy — and replace it with the signal that demonstrably works: review-prep reads **merged PRs against the prior review's proposals**. (b) write the four standing declines and the dormant-skill *keep* verdict into `decisions.md`. +- **Subtracts**: yes — the largest subtraction available, and it simplifies the forum rather than the code. Seven cycles, **zero comments across three PRs**, tiebreak never fired. +- **Effort**: Low +- **Impact**: Med–High — it stops the queue re-proposing settled items; the dormant-skill flag is on its sixth listing purely because a verdict was never logged. +- **POC findings**: not applicable — the evidence *is* the seven-cycle silence. +- **Ship means**: the edits ride **this** PR, the only channel by which a process item has ever reached the repo. +- **Decline means**: per reviews/2026-09-11.md ▸ #8 verbatim — *"the honest move next week is the Decline branch, permanently."* One line in `decisions.md` declining the class, and both skills keep describing a loop that does not exist. +- **Recommendation**: **Decline the POC-comment loop permanently and log it** — with the replacement named, because it already works: this review's outcome scorecard and its own correction both came from reading PR branches, which no skill instruction asked for. Note that *I cannot make these edits* — the skill forbids touching skill files or `CLAUDE.md` unilaterally — which is the fifth-cycle mechanism in one sentence. + +### 5. Fix the kaizen pipeline spacing — this review ran without a research doc +- **Source**: direct observation this morning, from the scheduled-task run records +- **Surface area**: scheduled tasks, outside the repo — `kaizen-research` (`0 6 * * 5`, jitter 164s) and `kaizen-review-prep` (`0 8 * * 5`, jitter 467s) +- **Change**: widen the gap, or give review-prep a preflight that checks for the current week's research doc and states its absence up front. +- **Subtracts**: yes — a load-bearing timing assumption ("~2 hours later") that lives only in two skill descriptions and is enforced nowhere. +- **Effort**: Low +- **Impact**: Med — the two skills are designed as a pipeline and today the second half ran on a week-old first half. It degraded gracefully because the queue is the primary source, which was design; that it degraded *silently* was not. +- **POC findings**: observed live today — research started 13:50Z against 12:09Z the week before, still `running` at prep time, against a 2h18m previous run. +- **Ship means**: move review-prep to ~10am MT, or add the preflight. +- **Decline means**: accept that roughly one review a quarter runs research-blind, which is survivable given the queue-first design. +- **Recommendation**: **Ship** — it is a cron edit. ⚠️ Both schedules are Phil's and live outside the repo, so this is a recommendation, not a repo change. + +### 6. Derive `missCause` from the fingerprint that flipped +- **Source**: review-queue.md ▸ [2026-09-04], open 14 days +- **Surface area**: backend — `apis/shared/observability/prompt_cache.py`, `session/hooks/prefix_fingerprint.py`, `GET /admin/costs/sessions/{id}/calls`, the admin cost-anatomy page +- **Change**: label each cache miss with the hash that changed — `tools_changed`, `system_prompt_changed`, `history_changed`, `agent_switched`, `ttl_expired`, `cold_start`. +- **Subtracts**: yes — the manual hash-diff ritual `CLAUDE.md` currently instructs a human to perform on every cost-spike investigation. Everything needed is already stored; nobody has written the comparison. +- **Effort**: Low–Med +- **Impact**: High — and higher this week than when queued: the per-call context ledger (#1130) and the compaction ledger just shipped, so the rows this would label now exist per call. +- **POC findings**: not POCed. Verified previously that `toolConfigHash` hashes full specs including `inputSchema`, so the external-MCP schema-drift class is already **detected** — this is what would let us **attribute** it. +- **Ship means**: a PR deriving the label in `prompt_cache.py` and surfacing it on the anatomy page. +- **Decline means**: keep diffing hashes by hand. +- **Recommendation**: **Ship if a fourth slot exists; otherwise Defer 1 week.** It has been "the cheapest high-leverage item after #1" for three consecutive reviews without ever being in the top three — which is itself a signal that it should either move up or be declined outright. + +### 7. Enforce platform-before-backend on a develop push — second occurrence +- **Source**: review-queue.md ▸ [2026-09-04]; new data point 2026-09-16 +- **Surface area**: CI — `.github/workflows/backend.yml`, `frontend-deploy.yml`, `platform.yml` +- **Change**: serialize the deploy workflows or gate them on platform completion, and make the runtime read retry instead of `exit 3`. +- **Subtracts**: yes — an unenforced deploy-order convention that lives only in `CLAUDE.md` prose, plus a class of red `develop` build that presents as a flake and self-heals. +- **Effort**: Low–Med +- **Impact**: Med — two occurrences in 17 days (`Backend Deploy` 09-01, `Frontend Deploy` 09-16), each self-healing, each burning a red build. +- **POC findings**: not POCed. +- **Ship means**: a PR adding the gate plus a retry. +- **Decline means**: accept ~1 self-healing red deploy per fortnight. +- **Recommendation**: **Defer 2 weeks (revisit 2026-10-02).** ⚠️ The known trap first: a shared concurrency group has previously **cancelled** a CDK deploy rather than queueing it, which is worse than what it fixes. + +### 8. Retire the `ClientSession` monkeypatch (b) and (c) +- **Source**: review-queue.md ▸ [2026-09-11]; deferred in reviews/2026-09-11.md ▸ #9 until **2026-09-25** +- **Surface area**: backend — `integrations/mcp_apps.py:23-32`, `:673` +- **Change**: move the App-frame header off the `initialize` symbol patch onto `ext-apps` v2.0.0's documented `io.modelcontextprotocol/serverInfo` in result `_meta`. +- **Subtracts**: yes — a monkeypatch on a Strands internal that upstream changed four times in one week. +- **Effort**: Med +- **Impact**: Med +- **POC findings**: not POCed. **(a) shipped** — `pyproject.toml:146` now carries `constraint-dependencies = ["mcp<2"]` with the full mechanism recorded in a comment. That was the urgent half. +- **Ship means**: verify our Gateway targets and Lambda FastMCP servers actually populate the `_meta` field, then delete the patch. +- **Decline means**: keep the patch with the constraint pin holding the blast radius. +- **Recommendation**: **Not due until 2026-09-25 — leave deferred.** Listed so it does not vanish; the risk it carried was closed by (a). + +## Carried Over From Prior Reviews + +- **`duration_ms` tool-timing** (carried since 2026-05-15; DROP 07-03, Decline 08-14, 08-28, 09-04, 09-11) — **eleventh cycle, five decline recommendations, zero keystrokes.** *Recommendation*: **Decline and log** (Proposal #4). +- **`oauth_required` SSE flow audit** (deferred 2026-05-10 until 2026-05-24) — **eighth surfacing, ~17 weeks overdue.** reviews/2026-08-14 said it *"must not carry a fifth review."* It has carried three more. *Recommendation*: **Decline and log**, or name one entry point and one failure. +- **docling #405 and Guardrails #480** (sixth carry) — reviews/2026-09-04 recommended declining them **as kaizen items** while keeping the issues as product backlog. Never written down. *Recommendation*: **Decline as kaizen items and log.** +- **Audit whether derived execution contexts re-evaluate RBAC** (due 2026-09-11, no decision recorded) — **still due, second time.** Never had a concrete surface named. *Recommendation*: **Decline or re-scope**, unchanged from last week. +- **W5 Runtime Instances, track 2** (due 2026-09-11, no decision recorded) — **fifth month with no proposal against the largest share of the bill** (Runtime memory, 73% of the AICC bill). Last week's recommendation — "hold as arithmetic, not a build: a spreadsheet and a paragraph" — was not taken. *Recommendation*: **do the paragraph, or Decline W5 as a kaizen lane** and leave it to the cost roadmap. Five months of neither is the worst option. Part (1) of its sibling entry did convert this week (#1051), so the entry is not inert — only this half is. +- **`bedrock-agentcore` #629** (dropped end-of-invocation spans → cost telemetry under-reports the last turn of every session) — **fourth cycle unowned.** *Recommendation*: **Accept explicitly in `decisions.md`.** Silence is the option already tried three times. +- **`bedrock-agentcore` #646** (watch extended to 2026-10-09) — not due. +- **Named A2A participants** (deferred 2026-05-15, no revisit date) — still blocked on an A2A server construct. *Recommendation*: **keep deferred, no date**, per last week. +- **`ClientSession` (b)/(c)** (deferred 2026-09-11 until 2026-09-25) — not due. → Proposal #8. + +## Retirement Candidates + +- **⭐ The POC-comment loop in both kaizen skills** — seven cycles, **zero comments across #926, #1045 and #1046**, tiebreak never fired, while the channel that works (merged-PR and PR-branch reads) is described in neither skill. Highest-value subtraction available, and it is not code. Proposal #4. +- **⭐ Four unexecuted declines and a `decisions.md` last written 2026-05-18** — four months stale while the queue carries items on their 6th, 8th and 11th cycles. Proposal #4. +- **The dormant-skill flag itself** — `cors-deployment` and `frontend-design` at **144 days**. reviews/2026-09-04 settled this as **keep, stop flagging**; sixth listing because the verdict was never logged. Log it (Proposal #4); do not re-decide it. +- **`INLINE_ATTACHMENTS_MAX_TOTAL_BYTES` as a protective cap** — 7,500,000 base64-encodes to **exactly** 10,000,000, the quota it exists to stay under, before the JSON envelope. Verified on disk today. One env knob (~7,000,000) fixes it; no code change. Now live-relevant because PR-6 (#1143) just began holding turns to that quota. Filed in #1147's residue entry. +- **`DOCUMENT_OFFLOAD_PIN_TURNS` and the pinning path, pending the prod readout** — #1147's readout predicts `document_offload` fires **≈0** times, because `document_read` is itself an `extra_tool`, so the agent is rebuilt every turn and the *restore* path reaches the document before the live offload can. If the count is ~0, pinning is inert code and PR-4's machinery is a retirement candidate rather than a feature. **Decide on the number, not now.** +- **The `ClientSession` symbol patch** — still listed, de-risked by the `mcp<2` constraint pin that shipped this week. Proposal #8, not due until 09-25. +- **`scripts/sourcedir.tar.gz`** in the fine-tuning S3 prefix — orphaned by 1.20.0's per-family `sourcedir-{text,vision,vlm}.tar.gz`. Third listing; the changelog says it is safe to delete by hand. +- **The `bedrock-agentcore` row in the version-pin lag table** — fourth consecutive no-op scan. It costs a row and informs nothing. + +## Risks Acknowledged But Not Acted On + +- **⭐ Two quality vetoes unrun while both stacks' defaults are live in dev** — *what breaks*: the cost roadmap's only counterweight. Both stacks are defended by measured dollars and asserted quality, which is the asymmetry `CLAUDE.md`'s cost tenet forbids. The window both specs named closes at the next release. — **Address now**, Proposal #3. +- **⚠️ A measured exponential regex on model-supplied input hangs the turn to the 600 s SSE timeout** — 0.87 s at 24 chars, 3.5 s at 26, 14 s at 28, against real lines of 60–100. — *what breaks*: a user-visible dead conversation, with a document's own text layer as a plausible injection surface. — **Address now**, Proposal #2. +- **⚠️ A kill switch that makes things worse than the feature's absence** — fixed by #1147, which is **unmerged**. Until it lands, `DOCUMENT_READ_ENABLED=false` strips document bytes with no recovery path. — *what breaks*: the one lever an operator reaches for in an incident. — **Address now** by merging #1147, Proposal #1. +- **⚠️ `PDF_PAGE_TOKEN_ESTIMATE` was fitted to a single dev PDF** — 1,500/page against 1,573 actual on one document, deliberately ~5% low because it gates an eviction. — *what breaks*: nothing yet, but every document cost figure now rides one sample. — recommendation: **Address after the prod readout** — it is an env var, no deploy, refittable from `cacheWriteInputTokens − (prefixTokens.system + prefixTokens.tools)` on rows where `documentsAttached = 1`. +- **⚠️ Upstream is building a first-party context-manager / offloading stack** (#4254, #4187, #4146, #4118, #4231) — and this week we shipped a *second* hand-built stack into the same territory. — *what breaks*: nothing this week; the collision surface doubled. — recommendation: **Watch until 2026-10-09**, and decide Compaction v2 (#835) against *upstream's* engine rather than ours. +- **Zero issues filed in a 114-PR week, 1 open issue repo-wide** — *what breaks*: nothing mechanical, and #1147 demonstrated the working substitute by queueing its own residue. — recommendation: **Accept**, and stop proposing an issue-tracker layer (verified: no `kaizen` label exists). +- **The #629 consequence is still unowned, fourth cycle** — cost telemetry under-reports the last turn of every session. — recommendation: **Accept explicitly in `decisions.md`.** + +## What Shipped This Week + +*(7-day window — **114 PRs** merged into `develop`, **148** non-merge commits, **three releases** (1.20.0 09-09, 1.21.0 09-13, 1.22.0 09-14). **Zero reverts.** Nightly: **7 consecutive greens** after 7 consecutive failures. One `develop`-push deploy failure.)* + +- **#1125–#1132 — the compaction stack, model-relative thresholds** — *floor-seeking cut with hysteresis, an 8k-bounded summary via Nova Micro, post-turn parked cuts applied in place when the re-write is free, oversized tool results offloaded at intake, and a per-call context ledger. A 20-session replay priced 100k/25k at $73.70 against $102.64 actual.* +- **#1137–#1143 — the document-offload epic, seven PRs** — *`document_read` with page-range and pattern modes, `DocumentDigest` at upload, restore rehydrating stripped documents as digests, live offload when the re-write is free, and the truncation anchor's cache gap on the ledger. Attachments were 31% of prod spend; one dev session's prefix went 109.1K → ~15K.* +- **#1142, #1145 — response feedback, the missing quality axis** — *content-free thumbs on assistant messages joined to cost rows as `F#`, six reason buckets, an explicit/implicit signal discriminator, and retry-with-correction — the consequence that keeps a thumb from decaying to a zero response rate. The first outcome signal this stack has ever had on a cost row, and (per Proposal #3) the cheapest available version of the quality gate.* +- **Last week's forum, 6 of 9 converted** — *#1 the nightly (both halves; 7 greens since), #2 the rate correction (`CLAUDE.md` provenance rewritten, the six `$2.50/MTok` constants gone, `cacheReadMultiplier` now a per-model input with a documented default), #3 the false cachePoint invariant (rewritten with a measurement, old claim marked "do not restore it"), #4 GPT-6 Astra at the 272K tier with tests and its caching caveat carried into the source, #7 the `ActiveSessionCount` alarm (#1051, sized against 7 days of real prod data), #9(a) the `mcp<2` constraint pin. Not converted: #5 workspaces (S3 bridge blocked before any code), #6 Strands Snapshots (not started), #8 the process writes (fifth cycle).* +- **#1134, #1121, #1122 — file previews** — *.docx, .pptx, .csv and .xlsx render in a docked pane, uploads as well as generated files.* +- **#1133 — gzip on app-api JSON, SSE passed through un-buffered** — *plus the follow-up catching that the middleware delayed SSE response headers.* +- **#1135 — KaTeX stopped eating currency in prose**, and the HTML-entity dollar rule that never worked was dropped. +- **#1115 — an `@`-mention binds the conversation** — *reversing decision D11 on prod evidence: 247 of 247 mentions started the conversation, so the borrow bought nothing and cost an invisible tool-loss failure.* +- **#1149 — Agent Templates**: create-form prefill plus an admin-managed backend. + +## Take + +**The forum's shipping model works, and last week's Take was wrong about it.** Six of nine proposals converted in seven days — including the three standalone "ten-minute" comment fixes that last week argued could only land by riding a carrier. That thesis was a fair reading of a two-week sample and it did not survive the third week. What *did* survive is narrower and more specific: **engineering items convert reliably; items requiring an edit to `CLAUDE.md` or a skill file have never converted once.** That is not priority, it is authorship, and it is why Proposal #4's honest answer is now the Decline branch the forum pre-committed to. + +**The debt this week left is a quality gate, not a velocity problem.** Two specs, written weeks apart, each chose the strongest possible gate wording — a veto — and each shipped with defaults live and the harness unbuilt. Neither is carelessness; a harness is the one deliverable in a cost epic that ships no user-visible behaviour, so it has no natural owner. The asymmetry is what matters: `CLAUDE.md` says quality wins when it conflicts with cost, and that is unfalsifiable while every cost change is measured and no quality change is. The good news is that this week also shipped the cheap version of the instrument — `F#` down-thumb rows split by turn class is a query, not a harness. + +**If Phil ships three: #1** (merge #1147 — it is a review and a merge button, it fixes a 14×-wrong instrument and a kill switch that is worse than useless, and it lands the epic's own measurement plan), **#2** (the ReDoS — measured exponential, model-reachable, hangs the turn to the 600 s timeout, small local fix, and independently ranked first by the person who found it), and **#3's cheap half** (the turn-class down-thumb comparison, which closes the veto gap with a query rather than a build). **#4** needs one keystroke or one line of honesty, and after five cycles the line of honesty is worth more. + +**A correction this document owes #1147, stated plainly because it changed the recommendation.** The first draft ranked "an experiment's default must be its control" as Proposal #3 and told Phil to move `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` off its default of 100. **That was wrong**, and the refutation was already sitting in #1147's own queue write, on a branch this review had not yet read: `offload_enabled_for` gates only two call sites, and **PR-3 — the digest-on-restore that is the epic's actual behaviour change — is not bucketed at all**, nor are PR-1, PR-2 or PR-6. The percent splits only the least consequential half and was never a control arm for PRs 1–3, which can only be measured before/after a deploy whatever it says. #1147's call — ship at 100%, count `document_offload` events instead, since PR-4 is predicted to fire ≈0 times because restore reaches the document first — is better reasoned and is adopted here. What survived is the narrower and truer finding: **the vetoes are unrun**, which both documents agree on. The process lesson is the one already in the Silence section: **the best analysis of the week arrived as a queue write on an unmerged branch, and a review that reads only `develop` will keep being corrected by it.** + +--- + +## Review Protocol (for Phil) + +1. Read Friction (2 min). +2. Scan Proposals — mark ✅ Ship / ❌ Decline / ⏸ Defer on each (3-5 min). #1 is a merge button; #2 is a small local fix; #3 has a cheap half and an expensive half. +3. Scan Retirement Candidates — same marks (1-2 min). Two are "decide on the prod number, not now." +4. Resolve Carried Over — **six are due and five need a keystroke in `decisions.md`, not a deliberation** (2-3 min). +5. Resolve the Risks block. +6. Pick 1-3 to ship this week. Decline or defer the rest with a reason. + +Target: 10-15 minutes. + +## Post-review (for Phil — separate PRs) + +- ✅ Ship items → individual feature PRs over the week. **Merge #1147 before this PR** — both edit `review-queue.md` ▸ `## Open` and #1147's entries are the ones later work reads. +- ❌ Decline items → appended to `docs/kaizen/decisions.md`. Five carried-over items and the dormant-skill *keep* verdict belong there this week; on the fifth cycle, writing them is worth more than re-ranking them. +- ⏸ Defer items → kept open in `review-queue.md` with a revisit date; they resurface when due. + +This skill produces the agenda. Implementation never happens here. diff --git a/docs/one-pagers/cost-effectiveness-roadmap.md b/docs/one-pagers/cost-effectiveness-roadmap.md index c9a3450ce..f7a7127dc 100644 --- a/docs/one-pagers/cost-effectiveness-roadmap.md +++ b/docs/one-pagers/cost-effectiveness-roadmap.md @@ -7,7 +7,7 @@ workstream's authority lives in its own spec, and if this page disagrees with a spec, the spec wins and this page gets fixed. *Companion specs:* `compaction-over-threshold-cache-spiral.md` (#833) · -`agent-cache-extra-tools-bypass.md` (#834) · `compaction-v2-versioned-prefix.md` +`agent-cache-extra-tools-bypass.md` (#834) · `compaction-model-relative-thresholds.md` (2026-09-15) · `compaction-v2-versioned-prefix.md` (#835) · `document-context-offload.md` + validation + evaluation (#836) · `quota-cooldown-windows.md` · `tool-search-token-bloat-strategy.md` · `session-workspace-tools.md` · `share-large-conversations-s3-offload.md` · @@ -99,7 +99,7 @@ how the backlog below is ranked: | # | workstream | what it protects | authority | state | |---|---|---|---|---| | W1 | **Measurement** | every other row of this table | #833 PR-1 (`partial_miss`), cohort scan §4.1, fleet anatomy scan, dashboards #699/#700 | **PR-1 merged (#838) and live in dev**; prod awaits a release, and that is when the G0 clock starts. §4.1 cohort scan **run 2026-08-05**; fleet anatomy **run 2026-08-05** (`scan_fleet_prefix_spend.py`, reproducible) | -| W2 | **Prefix stability** | don't rewrite what didn't change — 55.2% of spend | #833 PR-2/3/4 → #835 v2 (gated) | #833 PR-2/3/4 unbuilt. ⚠️ **PR-4 re-ranked 2026-08-05**: the system prompt mutates mid-session in 12.3% of multi-turn conversations, making it the most *general* item here, not a footnote to D2/D3. ⚠️ **#834 has left this row** — G1 disproved its prefix-cost thesis; it is a latency fix and now lives in W6 | +| W2 | **Prefix stability** | don't rewrite what didn't change — 55.2% of spend | #833 PR-2/3/4 → `compaction-model-relative-thresholds.md` (PR-1 in progress 2026-09-15: per-model ceiling/floor/hard ceiling, floor-seeking cut, hysteresis; PR-3 = paid-when-free scheduling) → #835 v2 (gated) | #833 PR-2/3/4 unbuilt. ⚠️ **PR-4 re-ranked 2026-08-05**: the system prompt mutates mid-session in 12.3% of multi-turn conversations, making it the most *general* item here, not a footnote to D2/D3. ⚠️ **#834 has left this row** — G1 disproved its prefix-cost thesis; it is a latency fix and now lives in W6 | | W2b | **Short-conversation cache economics** (new 2026-08-05) | the 43% of spend in sessions of ≤15 calls, which no item in this arc touches | **un-specced — gap** | 700 sessions wrote cache and read none back ($33.92, 7.3% of all write spend). Single-call sessions are 29% of all sessions and spend 67% of their money on writes they can never use. Needs a cachePoint-policy spec for first/short turns | | W6 | **Turn latency** | time-to-answer, not tokens | #834 (bypass narrowing + family promotion) · #841 (runtime session affinity) | **#841 merged and verified in dev** — steady-state turns ~7.6s → ~3.9s. Split: warm container ~7.6→4.8s (all sessions), reused Agent ~4.8→3.9s (cacheable only). #839's treatment arm now equals the ceiling | | W3 | **Payload boundedness** | nothing unbounded enters the prefix | #836 offload · tool-search strategy · workspace tools (PR-1 built) · S3 share-offload | offload PRs unbuilt; citations baseline probe required first | diff --git a/docs/one-pagers/load-test-response-to-aurax-2026-09.md b/docs/one-pagers/load-test-response-to-aurax-2026-09.md new file mode 100644 index 000000000..6a959a625 --- /dev/null +++ b/docs/one-pagers/load-test-response-to-aurax-2026-09.md @@ -0,0 +1,452 @@ +# Response to "Five performance fixes, proven by a controlled A/B" + +**To:** AuraX platform team +**From:** Boise State AI platform team +**Re:** `AgentCoreSolution-PerformanceTesting-ProductionReadyFixes.pdf`, run 2026-09-15 +**Date:** 2026-09-19 +**Working assessment:** [`docs/specs/load-test-assessment-2026-09.md`](../specs/load-test-assessment-2026-09.md) + +--- + +## Short version + +Thank you for this. Four of your five findings describe real defects in our +mainline, the fifth is a deployment-configuration gap we had left as a warning, +and all three of your stated residuals are correctly identified. We validated +each claim against `develop` before touching anything, then shipped. **All five +findings now have code merged to `develop`**, across five pull requests — items +1 through 4 are verified running in dev, the rest deploy on the next rollout. Two of your items led us somewhere you did not claim: a +long-standing bug in our token-attribution telemetry, and a second layer of the +per-turn rebuild that your fix would only have half-addressed. + +Two things in the report do not transfer to our numbers, and one residual is +mis-attributed to the wrong AWS quota. Both are detailed below — neither +changes your conclusion. + +Your methodology is the part we would keep. Building both arms from one source +tree and reverting only the five changes, rather than comparing against +mainline, is exactly right given that mainline carries a load-shedding +subsystem your branch does not. We would not have trusted the deltas otherwise. + +--- + +## Status at a glance + +| # | Your finding | Our verdict | Shipped as | State | +|---|---|---|---|---| +| 1 | Prompt counted 3× per turn; agent rebuilt each turn | **Confirmed**, plus a second cause you did not reach | #1154, #1155 | Merged · verified in dev | +| 2 | Image ships no compiled bytecode | **Confirmed** | #1154 | Merged · verified in dev | +| 3 | One CountTokens in front of every model stream | **Confirmed**; fixed differently, see below | #1154 | Merged · verified in dev | +| 4 | Uploads bucket has no CORS rule | **Environment gap, not a mainline bug** | #1154 | Merged · verified in dev | +| 5 | Assistant refuses an attached spreadsheet | **Confirmed**; narrower scope, see below | #1158 | Merged · deploy pending | +| R1 | 6 conversation-memory lookups per message | **Confirmed mechanism, wrong count** | #1157, #1155 | Merged · deploy pending | +| R2 | Memory quota 30 lookups/s | **Right number, wrong API** | — | Quota request open | +| R3 | Micro-VM cold start ~12 s | **Plausible, unverified by us** | — | AWS thread open | +| — | *(not claimed)* System-prompt token count was never authoritative | Found while verifying your probe-message detail | #1156 | Merged · deploy pending | + +Everything above is merged to `develop`. Items 1–4 are verified running in dev; +5, R1 and the probe fix deploy on the next rollout. Production ships on the +next release; every change carries a kill switch (table at the +end). + +--- + +## 1 · The prompt was counted three times per turn + +**Your finding.** The agent is rebuilt each turn and re-splits its context +attribution, costing two extra Bedrock CountTokens calls on top of the SDK's +own. Fix: memoise the split across the per-turn rebuild. + +**Our verdict: confirmed, and the rebuild itself had a cause worth fixing.** + +We reproduced the mechanism exactly. Our context-attribution hook memoised its +split on the `Agent` *instance*, which is sufficient only while one `Agent` +serves a session for its whole life. It does not, because our agent cache +refused to cache any agent carrying per-request ("injected") tools — a cohort +that was **2,720 of 3,565 prod sessions (76%)** on our last measurement, driven +by the spreadsheet-analysis tools. Those sessions rebuilt the `Agent` every +turn, and with it a fresh session manager, a full `initialize()` and an +AgentCore Memory restore. Your two extra counts rode on top of that. + +**What we changed.** + +- **Memoised the split by session and configuration**, not by instance — keyed + on `(session_id, system-prompt digest, tool-spec digest)` in a bounded + process-level map, so a rebuilt `Agent` adopts its predecessor's measurement + instead of re-counting ([#1154]). +- **Made the rebuild stop happening.** The agent cache refused these sessions + because their tool builders close over request scope the cache key did not + describe. We widened the key rather than the exception: Word, Excel, + PowerPoint and workspace tools were already describable (they capture only + session and user) and were promoted ([#1154]); the spreadsheet family also + captures the assistant id, so we added `assistant_id` to the cache key, the + construction snapshot and the paused-turn snapshot, and made the resume path + replay it ([#1155]). Every picker-gated injected tool family is now + cache-eligible. + +That second half matters for your §5 residual too: a cached agent skips +`initialize()`, which is where five of the memory lookups live. + +**Dev evidence.** Injected-tool sessions now log `agent_cache outcome=hit` from +their second turn onward, where they previously logged `miss` on every turn. + +**What's left.** Confirm the per-turn CountTokens count is ≤ 1 after cold start +under load, in the A/B rerun below. + +--- + +## 2 · Every new chat compiled the app's bytecode + +**Your finding.** The image ships no compiled bytecode (a side effect of the +package installer), so each fresh micro-VM compiles the app — and the symbolic +math library behind the calculator tool — when the first message arrives. Fix: +precompile into the image and warm the first turn's residual work at startup. + +**Our verdict: confirmed, exactly as described.** Our inference-api Dockerfile +runs `uv sync` without bytecode compilation and copies source raw. The +container runs as a non-root user and every micro-VM starts from the image +layer, so nothing written at runtime survives to the next session — the image +is the only place the bytecode can live. + +**What we changed** ([#1154]): + +- `UV_COMPILE_BYTECODE=1` for the virtual environment and `compileall` over our + own source in the final image stage. +- A daemon-thread warm-up at application startup that imports the calculator's + symbolic-math dependency and the agent-construction modules, and builds one + client per AWS service so the service models are parsed and cached. It runs + off the request path; the health endpoint answers immediately, and a request + arriving mid-warm-up waits on Python's import lock rather than redoing the + work. + +**Dev evidence.** Warm-up completes in **400–550 ms per container**, of which +**350–480 ms is the symbolic-math import** — that is the work that used to land +on the first user's first message, and it is now paid while the runtime is +still coming up. + +**What's left.** Nothing. The remaining cold-start time is item R3. + +--- + +## 3 · One token-count call still sat in front of the model + +**Your finding.** A single CountTokens call remained on the critical path ahead +of every model stream; under load AWS throttled it before the model began +writing. This is the call whose removal drives the 541 → 0 throttle result. +Fix: run the count as a background side-request that never mutates model +configuration, counting the system prompt against a probe user message. + +**Our verdict: confirmed mechanism. We bounded the call rather than +backgrounding it, deliberately.** + +The mechanism is exactly as you describe: enabling native token counting makes +the SDK await one CountTokens before every model call, and CountTokens has its +own request-rate quota separate from the model's token quota, so it throttles +first. We would add one detail: the latency is not the call, it is the SDK's +shared client retrying the throttle with backoff before falling back to its +heuristic. The reply waits out that backoff. + +**What we changed** ([#1154]): + +- **A dedicated CountTokens client with one attempt and a 2-second timeout.** A + throttle now costs one failed request before the heuristic estimate, instead + of a retry sequence on the reply's first-token latency. Both values are + environment-tunable. +- **Removed the model-id mutation.** We suspect this is what your "never + mutates model configuration" is pointing at, and we think it is the sharpest + observation in the report. Our subclass swapped `config["model_id"]` to the + base foundation-model id for the duration of the count and restored it + afterward — safe *only* while a count and a stream can never overlap on one + model instance. That invariant dies the moment the count goes concurrent, and + a stream reading the base id fails on-demand invocation. We now pass the base + id to the API explicitly and never touch the config. + +**Why we did not move it fully off the reply path.** Our proactive compaction +reads the same projected token count. Compaction timing is a prompt-cache cost +contract for us: compacting a turn early or late rewrites a 30k–150k-token +cached prefix at the cache-write premium, which is 1.25× the model's own base +input rate. Dropping the authoritative count on checkpoint-deciding turns trades +a latency win for a cost regression we would not see in a latency-only harness. +The bounded client captures most of the throttle-latency win with none of that +exposure. **We would revisit this if your ledgers show the residual mattered** — +specifically, whether the background count changed compaction frequency in your +arm. That is question 2 at the end. + +**What's left.** Measure whether the bound is sufficient under the 35 msg/s +ceiling, or whether the remaining awaited call still shows up in first-token +p95. That is part of the A/B rerun. + +--- + +## 4 · File uploads failed in a real browser + +**Your finding.** The uploads bucket had no cross-origin rule, so the browser +blocked every upload. A load-only program would have shipped this bug. + +**Our verdict: a deployment-configuration gap, not a mainline defect — and the +gap was ours to make it possible.** Our infrastructure code does attach a CORS +rule to the uploads bucket, but only when a domain or an explicit origin list +is configured. With neither set, it printed a warning to synth output and +created the bucket with no rule. Your environment hit exactly that path. + +We want to be clear that this does not diminish the finding. A warning nobody +reads in CloudFormation synth output is not a safeguard, and your point about +load-only testing stands on its own — the load harness uploads outside a +browser, so no amount of load testing would have caught it. + +**What we changed** ([#1154]): missing CORS origins now **fail synth** rather +than warn, with an explicit opt-out environment variable for a deployment that +genuinely has no browser front-end. A deployment that would have shipped this +bug no longer deploys. + +**What's left.** Nothing on our side. Worth confirming your environment now sets +a domain or origin list, since the guard will stop your next synth otherwise. + +--- + +## 5 · The assistant refused to read an attached spreadsheet + +**Your finding.** Attaching a CSV with the default tool set makes the assistant +decline and tell the user to enable a file tool first — a dead end unless they +know the trick. Fix: auto-enable the read-only file tools when a turn carries +an attachment, gated on the caller's existing permissions. + +**Our verdict: confirmed, and the dead end was worse than "unhelpful" — it was +instructed.** Spreadsheets never go inline at any size (they route through the +analysis tools to avoid Bedrock's document-size limit), the analysis tools are +opt-in in the picker, and our system prompt contains an explicit clause telling +the model to name the capability and send the user to the sidebar. So the model +was following instructions into a wall. + +**What we changed** ([#1158]). A session that holds a spreadsheet — this turn's +attachment, or a ready upload from an earlier turn — gets the Spreadsheet +Analysis tools added to that turn's effective tool set, **gated on the caller's +role grant** through the same predicate the picker and agent bindings use. It +enables; it never grants. We also rewrote the fallback message for the +remaining case (a role that genuinely lacks the tool) so it no longer points at +a sidebar toggle the user may not have. + +**Two deliberate narrowings from your version:** + +- **Spreadsheets only, not read-only file tools generally.** PDFs, Word, text + and Markdown attachments already reach the model through a document-read tool + that is gated on the attachment itself rather than on the picker, so they + were never in the dead end. Spreadsheets were the gap. +- **Sticky per session.** The answer feeds our agent-cache key, so a gate that + flipped between the attach turn and the follow-up would bounce the session + between two cache slots and re-write the cached prefix each time. We memoise + the positive answer for the session's life. + +**What's left.** One open review question we flagged on the PR: the auto-enable +currently also applies on top of an explicit Agent tool binding, on the +reasoning that the user's own attachment is the governing intent. Reasonable +people could want bindings to stay closed. Not urgent; no prod Agent binding +excludes these tools today. + +--- + +## R1 · Conversation-memory lookups per message + +**Your finding.** Six conversation-memory lookups per message; platform code; +open, reduce to three. + +**Our verdict: the mechanism is real, but "6 per message" conflates two +different APIs charged against two different quotas.** What we measured: + +| Operation | API | When | Count | +|---|---|---|---| +| Session and agent restore | `ListEvents` | Cold `initialize()` only | 5 | +| Long-term memory retrieval | `RetrieveMemoryRecords` | **Every user message** | up to 3 | + +The five restore calls are per *cold start*, not per message — but for the 76% +cohort in item 1, every turn *was* a cold start, which is very likely how the +two collapsed into one number on your side. Fixing the rebuild ([#1155]) removes +those five from every turn but the first. + +The three per-message calls are one per configured memory strategy. The third +was the current session's own conversation summary: AgentCore's summary of the +conversation the model already has in front of it, retrieved and prepended to +every user message. That is input tokens on every turn for information already +in context, plus a lookup against the quota. + +**What we changed** ([#1157]): + +- **Dropped the summary-namespace retrieval from the per-message path.** The + compaction path still reads summaries where they actually matter, at + checkpoint advance. Restorable by environment variable. Lookups per message: + **3 → 2**. +- **Bounded the retrieval client**, same treatment as CountTokens in item 3 and + for the same reason: the retrieval hook is awaited before the model call, so + a throttled lookup put retry backoff on first-token latency. One attempt, a + 2-second timeout, and a turn that simply runs without long-term context — + which is what a miss already meant. Writes keep their retries; a dropped + message is a corrupted conversation, a missed retrieval is not. + +**What's left.** Your target of three is met and beaten for the steady-state +path. We have not yet reduced the five cold-restore `ListEvents` calls +themselves — they are SDK-internal and would need an upstream change or a +narrower restore. Worth doing only if the A/B shows cold restore still +dominating. + +--- + +## R2 · The memory service quota + +**Your finding.** Conversation-memory service quota of 30 lookups/s, an AWS +limit, remedied by a quota request to 300/s; crossed at roughly 5 messages per +second. + +**Our verdict: the number is right and the remedy is right, but it applies to a +different API than the residual above implies — and there is a second cap that +no quota increase will move.** From the AgentCore quota tables: + +| Quota | Default | Adjustable | +|---|---|---| +| `RetrieveMemoryRecords` per account | 30/s | Yes | +| `ListEvents` per account | 200/s | Yes | +| **`ListEvents` per actor, per session** | **20/s** | **No** | +| `CreateEvent` per account | 200/s | Yes | +| `CreateEvent` per actor/session (conversational) | 5/s | No | + +So the 30/s you crossed is the long-term-memory retrieval quota, and raising it +to 300/s is the correct ask. But the restore calls run against `ListEvents`, +whose per-session cap of 20/s is **fixed**. That cap is generous for one user, +and it means per-session lookup *count* is the only lever there — a quota +increase cannot buy it back. This is an argument for the rebuild fix in item 1 +being the load-bearing change, not the quota request. + +**What's left (ours).** File the `RetrieveMemoryRecords` increase from 30 to +300/s. We deliberately did the code reduction first so the request is sized +against real post-fix traffic rather than pre-fix waste. + +--- + +## R3 · Micro-VM cold start + +**Your finding.** Roughly 12 seconds to boot a fresh machine; AWS platform; no +code lever, support request only. + +**Our verdict: plausible, and we have not independently measured it.** Our own +controlled probe in August measured a cold first turn end-to-end at 7–8 seconds +in dev, which is not the same quantity — that number includes our own +initialization, and it was measured at a load nothing like yours. We have no +reason to dispute 12 seconds at the ceiling. + +We agree there is no code lever, and we agree with your framing of the honest +remedies: an AWS-side warm pool, or a pre-warm-on-new-chat feature that costs +idle capacity. We would add that the warm-up in item 2 removes 400–550 ms that +was previously *inside* this number and attributable to us, so the AWS-side +figure should be measured after that change lands, or it will be overstated. + +**What's left (ours).** Open the support thread, with our post-warm-up cold-turn +breakdown attached so the conversation is about AWS boot time and not our +imports. + +--- + +## Not in your report: your probe-message detail found us a bug + +Item 3 mentions, almost in passing, that you count the system prompt against a +probe user message because "the model rejects an empty conversation." We went to +verify that, because our own context-attribution code counts the system prompt +with an empty message list. It is true: + +``` +CountTokens, empty messages + system → ValidationException: + "A conversation must start with a user message." +``` + +Our SDK swallows that exception into a characters-divided-by-four heuristic. So +the system-prompt partition of our per-turn context breakdown — which we surface +to users in a context badge and persist on every call's cost row — **has been an +estimate since the day it shipped, presented as an authoritative count.** No +alarm fired, because the fallback is silent by design. + +Fixed in [#1156] the way you described: count against a fixed probe user message +and subtract the probe's own weight, which is a per-model constant (24 tokens on +our default model) measured once per process. + +We would not have found this from the report's headline numbers. Thank you. + +--- + +## Where our numbers will not match yours + +Three caveats, none of which undercut your conclusion — we are recording them so +nobody later cites your table as a prediction for our production fleet. + +**Your baseline is worse than our mainline, so the deltas are upper bounds.** +Your branch predates our runtime session-affinity fix from 2026-08-05. Before +that fix, nothing forwarded a runtime session id, AWS assigned a fresh micro-VM +per invocation, and our agent cache therefore never hit *at all* — for any +session, not just the injected-tool cohort. Your baseline arm rebuilt the agent +on every turn for 100% of sessions; our mainline did so for 76%. Expect the +real-world item-1 gain to be large but smaller than 18.2 s → 6.6 s. + +**The cost figure should not be reused.** $0.0025 per message at a 99% +prompt-cache hit rate reflects short, highly repetitive synthetic prompts. Our +production write-to-read ratio runs near 1:2, and document conversations alone +are about 31% of spend. We will publish our own post-fix number from the admin +cost drill-down rather than cite yours. + +**Your load model checks out.** 15 messages per user per day with 10% in the +busiest hour gives 1 message/s ≈ 2,400 daily users, so 17 msg/s ≈ 40,800 is a +fair readiness target. We will reuse this mapping so our rerun is comparable to +yours. + +We also agree with your handling of the two starred success cells. Footnoting +them as ramp-opening cold-start bursts, rather than hiding them or calling them +a load ceiling, is the right call, and the throttle curve carries the +conclusion without them. + +--- + +## Open questions for you + +1. **The upstream contribution.** The report says one of the five is already + contributed back upstream. We searched the Strands SDK for token-counting and + estimation pull requests and found nothing matching the background + side-request or probe-message shape. A link would let us drop our local + version when it lands, rather than carry a divergent fix. +2. **Compaction behaviour in your arm.** When you moved the count off the reply + path, did proactive compaction fire at a different rate or at different + checkpoints? This is the one measurement that would move us from bounding the + call to backgrounding it (see item 3). +3. **The probe string.** Which probe user message did you use? It is a per-model + constant that shifts every system-prompt figure derived from it, so if we are + comparing attribution numbers later we should use the same one. +4. **The diff, the ledgers, and the merge base.** The branch or patch set, the + run ledgers behind the tables, and the commit your branch forked from. The + fastest path for us is reviewing it as a pull request against our `develop`. + +--- + +## What is left to do + +| Item | Owner | State | +|---|---|---| +| Merge all five findings | Us | **Done** — #1154, #1155, #1156, #1157, #1158 on `develop` | +| Rerun the A/B on our harness: 10 and 17 msg/s, 8-minute holds, before/after | Us | Not started; needs the load-test provisioning gate (creates users in the shared pool and suspends cost limits, so it is a deliberate manual step) | +| File `RetrieveMemoryRecords` quota increase, 30 → 300/s | Us | Not filed; sized after the code reduction | +| Open the AWS support thread on micro-VM cold start | Us | Not opened; attach post-warm-up breakdown | +| Production rollout | Us | Next release; all changes carry kill switches | +| Answer the four questions above | AuraX | Open | +| Narrow the picker's tool list through role grants on the chat route | Us | Noticed in passing, not scoped — the schedules and runs routes do this, the chat route does not | +| Make two document-read tests deterministic under parallel workers | Us | In progress — their regex guard is a wall-clock budget that CPU contention trips | + +### Kill switches, for your environment + +| Change | Variable | Effect | +|---|---|---| +| Startup warm-up | `INFERENCE_WARMUP_ENABLED=false` | Disables warm-up; bytecode precompile stays | +| CountTokens bound | `COUNT_TOKENS_MAX_ATTEMPTS`, `COUNT_TOKENS_TIMEOUT_SECONDS` | Retries and timeout for the count | +| Memory retrieval bound | `MEMORY_RETRIEVAL_MAX_ATTEMPTS`, `MEMORY_RETRIEVAL_TIMEOUT_SECONDS` | Same, for long-term memory | +| Summary namespace | `MEMORY_SUMMARY_NAMESPACE_RETRIEVAL_ENABLED=true` | Restores per-message summary retrieval | +| Spreadsheet auto-enable | `ATTACHMENT_TOOL_AUTOENABLE_ENABLED=false` | Back to picker-only | +| CORS synth guard | `CDK_ALLOW_NO_CORS_ORIGINS=true` | Allows a deployment with no browser front-end | + +[#1154]: https://github.com/Boise-State-Development/agentcore-public-stack/pull/1154 +[#1155]: https://github.com/Boise-State-Development/agentcore-public-stack/pull/1155 +[#1156]: https://github.com/Boise-State-Development/agentcore-public-stack/pull/1156 +[#1157]: https://github.com/Boise-State-Development/agentcore-public-stack/pull/1157 +[#1158]: https://github.com/Boise-State-Development/agentcore-public-stack/pull/1158 diff --git a/docs/specs/agent-state-feedback.md b/docs/specs/agent-state-feedback.md new file mode 100644 index 000000000..7e998633a --- /dev/null +++ b/docs/specs/agent-state-feedback.md @@ -0,0 +1,506 @@ +# Agent state feedback + +**Status: CLOSED 2026-09-19.** Every state shipped and observed working on +dev. PRs #1159, #1160, #1163, #1165, #1167, #1170, #1173, #1176, #1178, #1180. +See "Closing summary" at the foot of this document. +**Follow-up to:** `d2ee13e2` (emit agent_status and tool-batch summaries), `9bc9bc6b` / `5f0cd52a` / `67234329` (loading-indicator series) +**Related:** `docs/specs/mid-turn-steering.md` (the other consumer of the drain), CLAUDE.md § SSE Event Types → `agent_status`, +`docs/specs/turn-latency-preamble.md` — which opens the `preamble` stage this spec +measured and then left closed. That stage is the largest remaining pre-stream wait +and is deliberately NOT a status problem: no label makes it shorter. + +## Problem + +The live indicator shipped in `9bc9bc6b` replaced twenty invented phrases with +facts, and that was the right trade. But it settled at two states — `Thinking…` +and `Running …` — and a turn has more shape than that. Time the user +currently cannot account for: + +* **Agent build.** Tool-registry assembly, MCP `tools/list` pre-flight round + trips, session restore from AgentCore Memory, Runtime cold start. On an + agent-cache miss this is the longest unexplained pause in the product. +* **Head-of-turn context work.** `apply_pending_compaction` and + `apply_document_offload` both run before the first model call + (`stream_coordinator.py:283`). The `compaction` SSE that describes them fires + *after* `metadata`, purely retrospectively. +* **Model generation vs. tool execution.** Both read `Thinking…` today unless + the content stream happens to carry an unresolved `toolUse` block. +* **Reasoning.** A reasoning block's header is the static string `Thinking` + with no duration, live or afterwards. +* **Queued behind the single-flight lease.** Indistinguishable from a hang. + +## What already exists — and is thrown away + +Two findings shape everything below. + +**1. `thinking` and `tool_start` cross the wire and are dropped on the floor.** +`ToolInsightService.recordStatus` writes every `agent_status` transition into a +`statusBySession` signal. Nothing reads it — `statusFor()` has zero callers in +the SPA. Only `tool_end` is consumed, for the per-tool durations on the rail. +The loader's label is derived from the *content* stream instead +(`message-list.component.ts:331`). + +**2. That is not a style choice. It is a latency bug.** +`_drain_agent_status_events` runs between yields of the agent stream +(`stream_coordinator.py:1051`). During tool execution Strands yields nothing, so +a `tool_start` sits queued for exactly the silence it exists to explain, and +arrives bundled with its own `tool_end`. The measurement recorded in +`message-list.component.ts:313` — a three-tool browse turn — read `Thinking` for +all 4.5s and never showed a tool name. + +**Every phase added through `AgentStatusHook` inherits this.** Decoupling the +drain is therefore a precondition for PR-3, not a cleanup task. + +## Non-goals + +* **A `responding` phase.** The SPA knows text is streaming from the deltas + first-hand. A backend-derived duplicate would only disagree at the edges. + (Already settled in `agent_status.py`; restated here so it is not re-litigated.) +* **"Almost done thinking…".** Claude can say this because it knows the + thinking budget and how much is spent. Deriving it from elapsed time is an + inference wearing a fact's clothes. The existing stall ladder — `Still + working…` → `…longer than usual` — gives the same reassurance and claims only + what it knows. +* **Per-tool invented phrasing** ("Searching the catalog…", "Browsing…"). + A tool's own name is the most accurate label available and invents nothing. +* **Persisting durations.** Unchanged from the `agent_status` contract: a + reloaded conversation shows summaries without timings rather than a number + the user cannot trust. + +## Cost + +Nothing in this spec reaches the model. No phase, label, or duration is appended +to the conversation, so the cacheable prefix is untouched — the same standing +this feature has had since `d2ee13e2`. PR-2 adds SSE frames but no model tokens. + +## PR-1 — "Thought for 17s" (SHIPPED, #1159) + +**Scope:** SPA only. No backend change, no dependency on the drain fix. + +The reasoning block header (`reasoning-content.component.ts:48`) says `Thinking` +forever, including on a block that finished nine minutes ago. It becomes +`Thought for 17s` the moment the model stops reasoning. + +**Measurement.** Client-observed, first-hand: the span from the first reasoning +delta to the moment the model demonstrably switched to output. This differs from +the tool durations on the rail, which are measured inside Strands' event loop +and shipped over `agent_status`. It is stated as such in the code, because the +two are not the same kind of number and should not be presented as though they +were. + +**Why a client-side span is sound here.** The agent loop starts a new Bedrock +message at every tool round trip (`assistant-message.component.ts:310`), and +`handleReasoning` merges into the first reasoning block of the *current* +message. A reasoning block therefore never spans a tool call, so first-delta to +last-output-switch cannot silently absorb tool time. The undercount is the +reasoning's own time-to-first-token plus the trailing gap before output — both +sub-second, both absorbed by whole-second rounding. + +**End of reasoning** is stamped at the first of: +1. a non-reasoning `content_block_start` in the same message, +2. a non-reasoning `content_block_delta` in the same message, +3. `message_stop`. + +(1) and (2) are the accurate moments; (3) is the backstop for a cycle that +produced reasoning and nothing else. + +**Live-only, by construction.** The duration rides `ContentBlock`, which only +the stream parser populates. `GET /messages` never sets it, so a reloaded +conversation falls back to `Thinking` with no extra code and no persisted +number — the same posture as the tool rail's `totalDurationMs`. + +**Placement on the model.** `reasoningDurationMs` sits at the top level of the +SPA's `ContentBlock`, deliberately *not* inside `reasoningContent`. That nested +object is Bedrock's Converse shape; a display string there would be the mistake +CLAUDE.md calls out for `tool_group_summary`. + +**Formatting.** `<1s`, `3s`, `17s`, `1m 12s`. Never rounds a sub-second block up +to `1s`. + +## PR-2 — decouple the drain, then trust `agent_status` (SHIPPED, #1160) + +**Backend.** `AgentStatusHook` pushes to an `asyncio.Queue` instead of a list, +and the coordinator merges that queue with the agent stream rather than polling +it between yields. The queue caps (`_MAX_QUEUED_STATUSES`) and the +per-turn reset at `BeforeInvocationEvent` carry over unchanged; the reset is +what keeps an interrupted turn's transitions out of the next one. + +**SPA.** Wire the loader's label to the now-live `statusFor()`: + +| Phase | Label | +|-------|-------| +| `thinking` | `Waiting for the model…` | +| `tool_start` | `Running …` | +| batch of >1 | `Running and 2 more…` | + +**Also fixes parallel batches.** `loaderStatusTool` returns the *first* +unresolved `toolUse` on the streaming message, so three tools in flight names +one of them arbitrarily. With a trustworthy `tool_start`/`tool_end` pair the +count is known. + +**Keep the content-stream derivation as the fallback.** It is strictly more +current when it fires, and it is the only source that works if a future SDK +change starves the queue. + +**Found while building it.** The loader stays mounted for the WHOLE turn — +`isChatLoading` clears at stream close, not at the first token — so a label +keyed on the `thinking` phase would contradict text the user can already read. +"Waiting for the model" is therefore gated on the answer still being silent, +falling back to the previous wording once text arrives. The existing +`loaderStatus` docstring asserts the loader "is gone anyway" at that point; +that comment is wrong about the code and was deliberately left alone rather +than widen this PR. Worth revisiting on its own: the right fix is probably that +the loader should stop being mounted under a streaming answer at all, which is +a change to when it renders, not to what it says. + +## Verified on dev (2026-09-19) + +Measured by teeing the SSE body in the browser and timestamping frames from the +click. Recipe: patch `window.fetch`, `res.body.tee()`, split on `\n\n`. + +**The drain works.** `tool_start` landed at 4396ms against `tool_result` at +4769ms — the status frame beat the event it describes by 373ms, which the old +between-yields drain could not do by construction. The 100ms poll cadence is +visible as quantization in the frame timestamps. + +**The batch count was dead code and has been removed.** +`agent_factory.py` pins `tool_executor=SequentialToolExecutor()` so concurrent +browser tools cannot start two Playwright sessions. A three-tool batch therefore +emits strictly interleaved `start,end,start,end,start,end` and more than one +tool is never in flight. PR-2's description claimed "a parallel batch is finally +legible"; that was wrong as shipped. `ToolInsightService.runningTools` keeps the +list shape, which costs nothing and is what a concurrent executor would need. + +**The content-stream fallback still drives the visible label**, because it fires +when the model starts streaming a tool's ARGUMENTS while `tool_start` fires when +the tool starts EXECUTING — a measured ~640ms gap in which the UI named a tool +that was not yet running. Which is preferable is a product call, not a bug. + +**Where the time actually goes** (warm container, artifact turn, from click): + +| From click | Event | +|-----------:|-------| +| 2ms | request dispatched | +| **3750ms** | **first SSE byte** | +| 4757ms | `agent_status thinking` | +| 5483ms | `message_start` | +| 6111→6311ms | `tool_start` → `tool_end` (225ms of actual execution) | +| 7506ms | `done` | + +A cold turn measured **6.7s** before the first status frame. + +## PR-3 — narrate the cold agent build (BUILT) + +The original plan — emit a phase per pre-stream stage from the chat route — +could not be built as written, and the measurement that proved it also made it +unnecessary. What is left is much smaller. + +### Why the original plan was impossible + +FastAPI flushes response headers when the handler returns its +`StreamingResponse`, and `inference_api/chat/routes.py` awaits `get_agent(...)` +— along with model resolution, RAG retrieval and tool building — *before* that +return. During the whole window the plan wanted to narrate, **no SSE channel is +open**. app-api cannot cover it either: `chat/proxy_routes.py` awaits the +upstream response before constructing its own `StreamingResponse`. + +### What the measurement says + +`turn_prelude` on dev, four turns (2026-09-19): + +| Turn | total | preamble | rag | tools | agent_build | +|------|------:|---------:|----:|------:|------------:| +| cold agent cache | 2542 | 802 | 0 | 261 | **1478** | +| warm | 641 | 494 | 0 | 146 | 0 | +| warm | 644 | 460 | 0 | 149 | 34 | +| warm | 678 | 484 | 0 | 154 | 38 | + +Against a client-side click→first-byte of **1156ms** on that last turn, the warm +budget is ~478ms outside the handler (app-api hop, auth, Runtime routing), 484ms +`preamble`, 154ms `tools`, 38ms `agent_build`. + +Two conclusions: + +1. **A warm turn does not need narrating.** ~1.2s to first byte with no stage + dominating, and 41% of it not even in the handler. There is no honest phase + label that helps, and the loading indicator already covers it. +2. **A cold agent build is the whole problem.** 1478ms in one stage, and the + 6.7s cold turn measured earlier is worse. It is the only place in the + prelude where a phase label earns its keep. + +That second conclusion is about *narration*, and it should not be read as a +verdict on `preamble`. On a warm turn `preamble` is the largest stage in the +table — 460–494ms of a 641–678ms total — and nothing here looked inside it. It +is taken up in `docs/specs/turn-latency-preamble.md`, as a latency problem +rather than a status one. + +### The plan + +**Wrap `get_agent` only.** Move that one call inside the stream generator and +emit `Getting ready…` before it. Leave `preamble`, `rag` and `tools` eager where +they are. This sidesteps most of what made a handler restructure frightening: +the quota check, the session-ownership 404 and every other HTTP-error-capable +guard live in `preamble`, which stays ahead of the stream. + +**Feasibility check: PASSED.** The only `raise HTTPException` between +`get_agent` and the `StreamingResponse` return is guarded by `if is_resume:` +(interrupt-id validation). Resume keeps its eager build, so nothing on the +deferred path can need an HTTP status after the first byte. + +**As built**, three things the plan did not anticipate: + +1. **The frame is emitted unconditionally; the SPA decides whether to show it.** + A warm build is 0-38ms, and "Getting ready" for 38ms is a flicker — landing, + worse, *after* the generic "Thinking" the client shows from the moment the + user hits send, which reads as going backwards. + + The first attempt raced the build against a 250ms timer **in the + generator** and emitted only if it was still running. **That cannot work, + and dev proved it:** `create_agent` is synchronous + (`agent_factory.py`), so a cold build occupies the runtime's event loop for + its whole duration and `asyncio.wait` never gets to fire its timeout — it + returned only once the build was already finished, `finished` was non-empty, + and the frame was never sent. A 1548ms build, six times the threshold, + emitted nothing. + + A timer only works where the clock actually runs, which is the client. The + backend now always announces the build; `message-list.component.ts` holds + the phase for 250ms before rendering it, so a warm build is superseded by + `thinking` and never reaches the screen. Cost of the change: one extra SSE + frame per non-resume turn. + + (Peeking `_agent_cache` to predict a miss was considered and rejected both + times: it means rebuilding its key out in the route, and a key that drifts + from the real one is a bug this repo has paid for.) + + **Re-verified end to end (2026-09-19, session `9786b58b`)** on a cold + container, after the turn-duration fix: + + | Measure | Value | + |---------|------:| + | Client stopwatch (click → `done`) | 7821ms | + | Footer / `turnDurationMs` | 6291ms | + | `metrics.latencyMs` (model only) | 2364ms | + | `turn_prelude` handler total | 3862ms — preamble 883 · tools 250 · **agent_build 2728** | + + The label sequence was `Thinking… → Getting ready… → Waiting for the model…` + with no step backwards, and `turnDurationMs` is plainly distinct from + `latencyMs`, which is what proves it is no longer the post-build remainder. + + **The suppression did not work in production, and a third fix was needed + (2026-09-19).** A controlled four-turn session on one container: + + | Turn | `agent_build` | Cache | "Getting ready…" | + |------|--------------:|-------|------------------| + | 1 | 1631ms | miss | correct | + | 2 | **1ms** | hit | **wrongly shown** | + | 3 | **40ms** | hit | **wrongly shown** | + | 4 | **0ms** | hit | **wrongly shown** | + + `preparing` was never a state with an end — it was just the latest event. + The build finished in 1ms, but `thinking` does not arrive until the + head-of-turn work and the event loop's startup have also run, hundreds of + ms later, so the client's settle timer fired on a build that was long over. + The unit tests passed because they fed `thinking` 40ms after `preparing`, + a sequence the backend never emits. + + The backend now emits an explicit **`prepared`** frame when the build ends, + carrying its measured duration. The SPA treats it as the end of the wait. + The lesson generalises past this feature: a phase that only ever means + "most recent event" cannot express a wait that has finished, and a test + that invents the next event will agree with whatever the code does. + + **Verified on dev (2026-09-19), and it found one more bug.** Five turns, + builds of 1504ms / 549ms / ~650ms×3, all narrated correctly — turn 1 showed + "Getting ready…" for 2.1s of a 2.3s wait. But the label stepped *backwards* + on the way out — "Getting ready…" → "Thinking…" (~40ms) → "Waiting for the + model…" — on all five. The settle flag was cleared when the phase left + `preparing`, which opened a propagation window where the label read the old + phase and the new flag. It is now armed on the way IN and never cleared on + the way out; the fast-build case is suppressed by cancelling the pending + timer, which is what was doing that work anyway. + + **Still unverified in a browser:** fast-build suppression. Every turn in the + session missed the agent cache (~650ms builds), so the warm path was never + observed live. The logs prove warm builds exist (`agent_build` of 1ms and + 41ms on other sessions) and the suppression is unit-tested. +2. **A failed build needs its own error path.** The handler has already + returned by then, so neither `except` arm can see it; without an in-generator + catch a build failure is a silent hung stream. It now surfaces as a + conversational error (the house rule) and the `finally` still releases the + lease. +3. **The SPA validator had to be relaxed.** It required + `typeof cycle === 'number'`, and `preparing` precedes the event loop so it + carries no cycle — every frame would have been dropped silently. The + relaxation is scoped to `preparing`; the other phases still require a cycle, + because the SPA uses it to tell event-loop passes apart. + +**Kill switch:** `AGENT_PREPARING_PHASE_ENABLED` (default on). Off restores the +eager build exactly. Its own flag rather than riding `AGENT_STATUS_ENABLED`, +which gates narration — this changes when the agent is built. + +**Cost:** the label is one SSE frame. Nothing reaches the model. + +### Declined + +- **A client-side label for the pre-first-byte window.** Unnecessary once the + narrow fix lands, and it could only name a window, not a phase. +- **Restructuring app-api's relay.** It would cover the ~478ms outside the + handler, which is not where the pain is, and it would force upstream HTTP + errors to become SSE `stream_error` frames. +- **Restructuring the whole inference-api handler.** The measurement says three + of its four stages are not worth narrating, so the risk buys nothing. + +### Original target (superseded, kept for the record) + + +These are not Strands hook events; they happen before the loop exists, so they +need emit points in the invocation path. + +| Phase | Label | Emit point | +|-------|-------|------------| +| Agent build | `Getting ready…` | inference-api chat route, around `_create_agent` | +| MCP pre-flight | `Connecting tools…` | around the pre-flight `tools/list` | +| Head-of-turn context work | `Reorganizing context…` | `stream_coordinator.py:283–300` | +| Queued behind the lease | `Finishing your previous message…` | single-flight acquire | + +`Connecting tools…` is worth its own label rather than folding into `Getting +ready…`: it is the phase that fails (401, timeout) and a user who saw it named +has a chance of connecting the notice that follows to the pause that preceded it. + +**Open question for PR-3.** These precede `message_start`, and the SPA's loader +currently starts on the request's falling edge. Whether they arrive as +`agent_status` frames with new phase values or as a distinct pre-turn event is +not yet decided; `agent_status` is preferred if the drain refactor in PR-2 makes +a pre-loop emitter practical. + +## Turn recap (BUILT) + +When a turn ends, everything describing it disappears: the loading line goes +and takes the elapsed timer with it. The rail keeps per-tool durations, but +nothing said how long the turn took. A finished turn now carries a one-line +footer — `9.6s · 4 tools` — anchored to the END of the turn, because that is +what it describes: a turn spans several assistant messages and the number +covers all of them plus the tools and the agent build between. + +**It needed a new field, and that is the interesting part.** The obvious +source, `latency.endToEndLatency`, is not the number it appears to be: on the +persist path it prefers the provider's own API-call time +(`stream_coordinator._store_message_metadata`), so summing it across a turn +drops tool execution and the pre-stream agent build — a turn the user watched +for 9s reads as 3s. It also disagrees with itself: the LIVE `metadata` event's +`metrics.latencyMs` IS the whole turn, so the same field would show one number +during the turn and a smaller one after a refresh. + +So `turnDurationMs` is explicit: measured server-side from the invocation +arriving to the stream ending, emitted on the live event AND persisted on the +turn's last message. One field, one meaning, identical live and reloaded. + +**Known limits, both deliberate:** + +- It starts when the invocation reaches inference-api, so it **excludes + everything before the container**: the app-api hop, auth, and Runtime + routing. That shortfall is **~1.5s on a cold path**, not the ~478ms first + recorded here — that earlier figure came from a warm turn, and routing to a + cold container costs substantially more. Measured end to end on dev + 2026-09-19: a turn whose client stopwatch read 7821ms reported **6291ms**, + short by 1530ms, or 20% of the turn. + + So the recap is honest about the server's turn and reads visibly under the + wait the user feels. Matching the stopwatch means measuring on the client, + which cannot survive a reload — the trade "always on" already decided. +- **It must be measured from the handler, not from `stream_response`.** Those + were the same thing until PR-3 deferred the agent build into the stream + generator, which runs BEFORE that generator is iterated — so the + coordinator's own `stream_start_time` now excludes the build. Shipped that + way and caught on dev: a turn the user waited 7.8s for reported **2.1s**, + exactly the post-build remainder. The route hands the coordinator + `TurnPrelude.started_at` instead. Note that property is deliberately WALL + clock while the stage marks are `perf_counter`: the coordinator subtracts it + from a `time.time()` reading, and mixing domains gives a meaningless number + rather than a slightly wrong one. The alternative — a client-measured + click-to-`done` — is truer to the felt wait but cannot survive a reload, + which "always on" requires. +- Turns written before the field show **nothing** rather than a zero. Same + rule as the tool-rail durations: no number beats a number nobody measured. + +--- + +## Closing summary + +### What a turn says now + +| When | What the user sees | Source | +|------|--------------------|--------| +| Model call being retried | `The model is busy. Retrying…` (amber) | `model_retry` | +| 30s / 90s of silence | `Still working…` → `…longer than usual.` (amber) | client timer | +| Tool executing | `Running …` | `agent_status` `tool_start`, content stream as fallback | +| Agent being built (>250ms) | `Getting ready…` | `agent_status` `preparing` / `prepared` | +| Model call in flight | `Waiting for the model…` | `agent_status` `thinking` | +| Anything else | `Thinking…` | fallback | +| Throughout | elapsed timer | client | +| After a reasoning block | `Thought for 17s` | client-measured span | +| After a tool batch | per-tool durations + a model-written summary | `tool_end`, `tool_group_summary` | +| After the turn | `9.6s · 4 tools` | `turnDurationMs` | + +### Declined, with the measurement that settled it + +- **`Connecting tools…`** — `tools` measured 250ms. Nothing to explain. +- **`Reorganizing context…`** — `rag` measured 0ms on a plain turn. +- **A full inference-api handler restructure** — three of its four pre-stream + stages do not need narrating, so the risk bought nothing. +- **Restructuring app-api's relay** — would cover the ~1.5s outside the + container, which is real but is a latency problem, not a narration one. +- **`Almost done thinking…`** and a **`responding`** phase — see Non-goals. + +### What this cost, and what it taught + +Four defects shipped to dev and were caught there. **All four passed CI**, and +all four were about timing or interaction, which the unit tests could not see: + +1. **A server-side timer that could not fire** (#1167). `create_agent` is + synchronous, so a cold build owns the event loop and `asyncio.wait` never + reaches its timeout. The feature was inert in production from the moment it + shipped — silently. The test faked the build with `asyncio.sleep`, which + yields; the real one does not. +2. **A label that stepped backwards** (#1170). A flag cleared before the phase + did, so `Getting ready…` → `Thinking…` → `Waiting for the model…` on every + turn, ~40ms of it. +3. **A turn duration that excluded the wait it described** (#1176). Deferring + the agent build moved it outside the window `stream_response` measures. A + 7.8s turn reported 2.1s. +4. **A label that overstayed** (#1180). `preparing` had no end, so the client + inferred one from `thinking` — which arrives hundreds of ms later. Builds of + 0ms, 1ms and 40ms all rendered it. The test sent `thinking` 40ms after + `preparing`, **a sequence the backend never emits**. + +Three of the four share one root cause: **treating "most recent event" as if it +were "current state"**. An event stream says what just happened, not what is +still true, and a phase with no explicit end cannot express a wait that is over. + +The practical lesson for anyone extending this: a test that invents the next +event agrees with whatever the code does. Model the real frame order, including +the gaps — and for anything whose correctness IS its timing, treat green CI as +weak evidence and go watch a turn. + +### One thing measurement killed + +`#1171` was filed claiming the agent cache missed on most consecutive turns. +It does not. That data was gathered while repeatedly testing immediately after +deploys, when containers were cycling. A session sticks to its container and +the cache hits from the second turn on (measured: 1631ms, then 1ms / 40ms / +0ms). Closed as not reproducible. + +The one real residual is that the **first** turn of every session pays a full +build, because `session_id` is part of the cache key. That is by design and is +a different, narrower question than the one that issue asked. + +### Left on the table, deliberately + +- **`preamble` costs 450-900ms on every turn** and nobody has looked inside it. + It is the largest remaining avoidable wait. Not a status gap — narrating it + would explain nothing — but worth its own investigation. +- **The recap reads ~1.5s under the user's stopwatch**, excluding everything + before the container. Fixing that means client-side measurement, which + cannot survive a reload. +- **Conversations from before the recap** show no footer until their next turn. diff --git a/docs/specs/authenticated-web-assessment.md b/docs/specs/authenticated-web-assessment.md index 0b72bf9e8..71a77b4a8 100644 --- a/docs/specs/authenticated-web-assessment.md +++ b/docs/specs/authenticated-web-assessment.md @@ -1,6 +1,6 @@ # Authenticated web assessment (browser takeover, profiles, axe) -**Status:** PROPOSED — no code written. Sized for four PRs. +**Status:** IN PROGRESS — **PRs 1, 2, 3 merged; 2b (the viewer) built**. D3 and D6 rewritten (backend takeover: `request_user_login`, take/release control, the `browser_login_required` interrupt + SSE event, the D4 metadata projection, reaper pinning, the abandonment deadline, and the flag) and **PR 2's backend + SPA half** (the live-view route, its IAM, and the SPA's event/resume wiring). The viewer page itself is deliberately still open — see "Build notes (PR 2)". PRs 3-4 not started. Sized for four PRs. **Driver:** Two user requests, both blocked on the same missing capability: a Library agent that evaluates the accessibility of the databases they renew annually (subscription-gated), and a VPAT-evaluation agent that must test @@ -103,13 +103,13 @@ breadcrumb, and the resume route in `inference_api/chat/routes.py`. | # | Decision | |---|----------| -| D1 | Takeover is a **separately grantable tool** (`request_user_login`), not an action on `browse_web` — RBAC granularity is per `tool_id` | +| D1 | Takeover is a **separately grantable tool** (`request_user_login`), not an action on `browse_web` — RBAC granularity is per `tool_id`. **Kept as a rollout control, not a security boundary** — see D6 and Security 3 | | D1b | Takeover is an **interrupt**, not a new endpoint — reuse `ask_user_question`'s machinery | | D2 | The live-view URL is **never** model-visible; it is minted on demand by app-api | -| D3 | DCV is embedded via a **static viewer page at the sandbox origin**, framed by the SPA — not a React island in the Angular app | +| D3 | DCV is embedded via a **plain-JS viewer page at the sandbox origin**, framed by the SPA. No React and no npm: AWS's `BrowserLiveView` imports `dcv`/`dcv-ui`, which are undeclared and unavailable on npm. The SDK is fetched and signature-verified at build time, never committed — it is EULA-licensed and this repo is public | | D4 | Browser session identity moves to the **DynamoDB session-metadata row** so app-api can act on it | | D5 | Profiles are keyed `user + assessment target`, with an explicit user-facing "forget this login" | -| D6 | A **second** `CfnBrowserCustom` for assessment; never mutate the fleet's browser | +| D6 | **One** browser, with a Chromium `URLBlocklist`. ⚠️ Session-level policies are **RECOMMENDED-only** (measured — MANAGED is rejected), so this constrains the agent but not a human in a takeover; the real control needs `CreateBrowser`. Supersedes the second-resource/`URLAllowlist` draft — the browser resource is immutable (no `UpdateBrowser`) and RBAC cannot express a per-site rule | | D7 | axe-core ships as a **browser extension**, and the full report goes to the workspace, never to the model | | D8 | Everything rides `BROWSER_TAKEOVER_ENABLED` (default on, kill switch) | | D9 | A 40-target sweep is **40 sessions, not one turn** — the cost hazard is session shape, not takeover | @@ -211,31 +211,82 @@ inference-api — the Runtime data plane proxies only `/invocations` and `/ping` so a route added there would 404 in cloud. app-api's task role needs `ConnectBrowserLiveViewStream`, `UpdateBrowserStream` -and `GetBrowserSession` on the assessment browser ARN. +and `GetBrowserSession` on the browser ARN. (Built in PR 2, deliberately +narrower than the Runtime's own grant — no Start/Stop and no +`ConnectBrowserAutomationStream`, because app-api must never drive the +browser.) -### D3 — DCV embeds via a viewer page at the sandbox origin +### D3 — A plain-JS viewer at the sandbox origin, SDK fetched at build time -AWS ships React; we are Angular. Three options were considered: +**This supersedes an earlier draft** that reached for AWS's React +`` component from the `bedrock-agentcore` npm package. That +package is official AWS and Apache-2.0, but the component **cannot be built +from public npm**: -| Option | Verdict | -|---|---| -| Drive the DCV Web Client SDK directly from Angular | Reimplements connection setup, SigV4 handling and frame rendering; carries the maintenance forever | -| Mount a React island inside the Angular SPA | Adds React to the SPA bundle for one component; poor trade | -| **Static viewer page at a separate origin, framed by the SPA** | **Chosen** | - -The third is what MCP Apps already do. A `live-view.html` asset built against -`bedrock-agentcore`'s `BrowserLiveView`, deployed to the same CloudFront/S3 -origin pattern with `frame-ancestors `. React stays entirely -outside the SPA bundle. The SPA frames it and passes the conversation id; the -page calls the D2 route for its own URL and refreshes it on expiry. - -`remoteWidth`/`remoteHeight` **must** match the session viewport or the stream -crops. `DEFAULT_VIEWPORT` is currently `1280x800` (`session_pool.py:45`); the -viewport must be carried on the event rather than duplicated as a constant in -the frontend, so the two cannot drift. +``` +import dcv from 'dcv'; +import { DCVViewer } from 'dcv-ui'; +``` -⚠️ `bedrock-agentcore` (npm) is a **new dependency** and needs explicit -approval before anyone installs it. Exact-pin, no `^`. +Neither is in its `dependencies`, `peerDependencies` or +`optionalDependencies`. `dcv-ui` does not exist on the registry at all, and the +only `dcv` package there is an unrelated third-party Vue component library. The +component silently assumes you have already vendored the Amazon DCV Web Client +SDK and made it resolvable under those names. So the React path is a strict +*superset* of the vendoring work, not an alternative to it — and it would pull +React plus ~189 packages into a surface where a user types a password. + +**So: no React, no npm, no third-party code.** The viewer is +`infrastructure/assets/mcp-sandbox/live-view.{html,js}` — plain ES2017 against +the official Amazon DCV Web Client SDK, which is framework-agnostic. + +**The SDK is fetched at build time, never committed.** It is a EULA-licensed +AWS download and **this repository is public**, so vendoring it would be +redistribution. `scripts/build/fetch-dcv-sdk.sh` downloads it, verifies it, and +extracts it to a gitignored directory; `platform.yml` runs it before the CDK +deploy. This is also AWS's own instruction — "place the extracted directory on +your web server" — and it means the signature is checked on every build rather +than trusted once at commit time. + +What makes that safe is **not** "it comes from AWS". Three pins do: the exact +version, the archive SHA256, and the signing-key fingerprint — the last two +held in the script rather than read from the network, because whoever could +swap the artifact could swap the published checksum and serve a different key. +Importing whatever key the URL offers today and trusting it is +trust-on-first-use on every build. ⚠️ The fingerprint was captured once and +pinned at review time; AWS does not appear to publish it out-of-band. + +**Why a separate origin rather than an Angular component in the SPA.** The +original reason — "AWS ships only React" — is gone. Two reasons remain, and the +first is the real one: + +- This is the one surface where a user types a password, and `dcv.js` forwards + their keystrokes. Keeping it in an origin that holds **no session cookie** + means a compromised SDK cannot also reach their session. +- The SDK must be *served* with its folder structure and license files intact, + which suits a static origin and fights a bundler. + +Note the isolation argument that justifies this origin for MCP Apps does **not** +transfer: DCV streams pixels and forwards input, so the remote page's code runs +in AWS's Chromium, never in our DOM. This is defence in depth, chosen +deliberately rather than inherited. + +**The SPA mints; the viewer never calls app-api.** app-api's CORS is a +credentialed allowlist, so letting the viewer call it would mean allowlisting an +origin that also serves untrusted MCP App HTML. Instead the SPA mints the URL +and `postMessage`s it in, targeted at the sandbox origin explicitly — never +`'*'`, which would post a live credential to whatever happened to be framed. The +viewer accepts messages only from the origin that framed it, and stores nothing. + +**CSP.** The sandbox origin's CloudFront function already composes `connect-src` +from a `?csp=` query parameter, and is attached to the **default** behaviour, so +it covers any path there. The SPA mints first and then names the minted URL's +own origin in that parameter, so the grant is exactly the endpoint in use rather +than a wildcard. This resolves risk 3 below. + +`remoteWidth`/`remoteHeight` come from the event's `viewport` and are applied +via `requestDisplayLayout`; a mismatch crops or letterboxes the stream, which is +why the viewport is carried rather than re-declared in the frontend. ### D4 — Browser session identity in the session-metadata row @@ -274,22 +325,124 @@ Profiles are per-user by default. A shared institutional account (one Library subscription, several evaluators) is a genuinely different model with a different blast radius and is **out of scope** for v1. -### D6 — A second browser resource +### D6 — One browser, a MANAGED URL blocklist applied per session + +**This supersedes an earlier draft** which added a second `CfnBrowserCustom` +carrying a `URLAllowlist` of the targets under evaluation. Three findings +killed that design; the replacement is simpler and is the control that +actually implements the requirement. + +**The requirement, stated from the outcome.** Faculty and staff run +accessibility and VPAT checks behind authenticated pages. Nobody — including a +grad assistant who legitimately holds a staff role — uses the same capability +to have an agent submit their own coursework. That is a constraint on **which +sites**, not on **which people**, and RBAC cannot express it: effective +permissions merge as a **union** across every role a user matches +(`rbac/service.py`, "Tools: Union"), priority decides only the quota tier, and +there is no deny. A user in both the staff and the current-term student Entra +groups gets the union of both roles' tools. At a university that population is +large and permanent. + +**An application-level URL check cannot implement it either.** Takeover hands +the human a fully interactive Chromium. A check inside `request_user_login` +sees only the *starting* page; once the user is driving they can navigate +anywhere. The only thing that constrains them is Chromium itself refusing, +which means an enterprise policy. + +**Blocklist, not allowlist — because the browser resource is immutable.** +There is no `UpdateBrowser` operation (the control plane has only +`CreateBrowser`, `GetBrowser`, `DeleteBrowser`, `ListBrowsers`), and per the +[enterprise-policy docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-enterprise-policies.html) +"policy files are read from Amazon S3 at the time of the API call. Changes to +policy files in Amazon S3 after calling `CreateBrowser` or +`StartBrowserSession` are not reflected." So a policy attached at +`CreateBrowser` is frozen for the life of the resource, and every edit means +replacing it. An allowlist of vendors under evaluation would therefore require +a browser replacement per VPAT review — untenable. A **blocklist of +institutional systems** (the LMS, the SIS, HR, email) is small, changes about +yearly, and fits an immutable resource. It also matches the requirement's +actual shape: everything is open *except* where an agent acting as the user has +academic or administrative consequences. + +**⚠️ MEASURED 2026-09-19, AND THIS IS WHERE THE DESIGN BROKE.** The plan was +to apply the policy at `StartBrowserSession` as MANAGED, because its `type` +enum is `['MANAGED', 'RECOMMENDED']`. **The enum lies.** The service returns: -Add `browser-assessment` (`CfnBrowserCustom`) alongside the existing one: +``` +ValidationException ... Invalid value for parameter 'type'. +MANAGED is not supported for session-level policies. +``` + +And it does not degrade — `StartBrowserSession` fails outright, so **every** +browser session dies and `browse_web` stops working entirely. That regression +reached dev (never prod) and was reverted to `RECOMMENDED`. + +So risk 2b resolved **against** the design. Session-level policies are +RECOMMENDED-only, which Chromium treats as a user-overridable default. What +ships today therefore constrains the **agent** — which drives via CDP and never +opens settings — but **not a human holding the browser during a takeover**, +who is the threat this control exists for. + +**Consequence: `request_user_login` must stay ungranted** until the MANAGED +policy is applied at `CreateBrowser`. That needs a custom resource, since +`CfnBrowserCustom` does not expose `enterprisePolicies`, and it reinstates +replace-per-edit on the browser resource — survivable only because profiles +turned out to be account-level (risk 2). The RBAC grant and +`BROWSER_TAKEOVER_ENABLED` hold the line meanwhile. + +The original reasoning for preferring session level, now moot: +`StartBrowserSession` takes `enterprisePolicies`, and the docs' prose describes +MANAGED as a `CreateBrowser` thing. This matters because the two levels are not +interchangeable: MANAGED is "required and mandated by an administrator… cannot +be overridden" and lands in `/etc/chromium/policies/managed/`; RECOMMENDED is +user-overridable and lower-precedence, which makes it advisory rather than a +boundary against the person driving the browser. + +Applying it per session buys three things a `CreateBrowser` policy cannot: the +list is read fresh from S3 on every session, it needs no `CfnBrowserCustom` +change, and it works **today** — `CfnBrowserCustom` in our pinned aws-cdk-lib +(2.251.0) does not expose `enterprisePolicies` at all (its props are +`browserSigning`, `description`, `executionRoleArn`, `name`, +`networkConfiguration`, `recordingConfig`, `tags`), so the CFN route would need +a custom resource *and* would inherit the replacement-per-edit problem. + +⚠️ **Verify before granting the tool to anyone:** that MANAGED is honoured at +`StartBrowserSession` and not silently downgraded. An enum accepting a value is +not proof the service enforces it. The test is one browser session on dev: +apply the policy, navigate to a blocked host, confirm Chromium refuses. Until +that passes, the blocklist is unproven and the RBAC grant is the only control. + +**Consequences:** + +- **No second browser resource.** One `URLBlocklist` covers `browse_web` and + `request_user_login` alike. The LMS already has a first-class API + integration in this product, so the browser reaching it was the unsanctioned + path regardless. +- **VPC `networkMode`, `certificates` and `recording` are decoupled** from this + and can be revisited on their own merits, for campus-only targets. +- `session_pool._start_remote_session` is the single place a session is + started, so "the policy is always applied" is one assertion in one place. + +**The list.** Seeded with the LMS and extended after the CISO review. Match on +the origin actually navigated to, not a vanity hostname — a CNAME that +redirects to the real origin is not what Chromium's `URLBlocklist` sees: + +```json +{ + "URLBlocklist": [ + "boisestatecanvas.instructure.com" + ] +} +``` -- `networkMode: 'VPC'` with subnets/SGs, for campus-reachable targets; -- `recording: { enabled: true, s3Location: ... }` — for a VPAT, evidence that a - test ran against a specific build is often worth more than the findings; -- `enterprise_policies` carrying a Chromium `URLAllowlist` restricted to the - targets under evaluation (see Security); -- `certificates` for internal CAs. +It lives in CDK config and deploys as the S3 object, rather than being editable +in place. A security control that can be changed out-of-band without review is +worse than one that needs a deploy, and this list changes about yearly. -Never mutate the existing browser. `executionRoleArn` is create-only and the -construct documents the replacement collision; assume `networkConfiguration` is -create-only too rather than discover it during a deploy. `BROWSER_ID` already -selects the browser per environment (`session_pool.py:64`), so routing -assessment sessions to the second resource is a config change. +**Residual risk, stated plainly.** A blocklist is only as good as its entries. +It stops the named threat and whatever else is enumerated; it does not stop a +takeover against some authenticated system nobody listed. Session recording +(Security 2) is the detective complement. ### D7 — axe-core as an extension; report to the workspace @@ -381,13 +534,21 @@ look expensive when the feature itself is nearly free. **An interactive browser inside our AWS account is the headline risk.** During takeover the user has a fully interactive Chromium with our egress. Controls: -1. **`URLAllowlist` Chromium enterprise policy** on the assessment browser, - scoped to the targets under evaluation. This is the primary control. +1. **A MANAGED `URLBlocklist` Chromium enterprise policy**, applied on every + `StartBrowserSession` (D6). This is the primary control, and it is the only + one on this list that constrains **where the human can go** once they hold + the browser. MANAGED specifically: RECOMMENDED is user-overridable and so is + not a boundary against the person driving. 2. **Session recording to S3** — every takeover is recorded. This must be disclosed to users in the UI before control is handed over; silent recording of a session in which someone types a password is not acceptable. -3. **Takeover is gated by the same RBAC as `browse_web`**, and the assessment - browser is a separate resource so it can be granted separately. +3. **RBAC is a rollout control, not a security boundary.** It was drafted as + one, and it cannot be: effective permissions are a **union** across matched + roles (`rbac/service.py`) with no deny, so a grad assistant holding both a + staff and a student role receives the union. Granting `request_user_login` + separately from `browse_web` is still worth keeping — it lets the capability + be piloted with the evaluator group before it is widened — but the thing + that stops coursework submission is control 1, not this. 4. **`frame-ancestors`** on the viewer origin, so only the SPA can frame it. 5. **The agent provably cannot act while the human holds control** — the automation stream is `DISABLED` at the service, not by convention in our code. @@ -407,19 +568,131 @@ For a Library procurement workflow this is fixable at the source — ask for scanning rights in the renewal terms, or request an evaluation tenant. Worth having in writing before forty scans run, not after. +## Build notes (PR 1) + +Two deliberate deviations from the sketch above, both recorded here so PR 2 +does not have to rediscover them: + +1. **`sessionId` on the event is the conversation, not the browser session.** + D1b wrote a single ambiguous `sessionId`. Every other SSE event in this + codebase uses `sessionId` for the conversation, and D2's route is owner- + scoped by conversation, so the event carries both: `sessionId` (conversation) + and `browserSessionId`. Collapsing them would have collided the moment + app-api needed to check ownership. + +2. **The conversation id and the D4 projection are stamped on in the streaming + layer, not the tool.** The tool knows only which browser session it handed + over; `stream_coordinator` is the layer that holds `session_id` / `user_id`. + Keeping identity out of the tool also keeps it a *static* registry tool + rather than an `extra_tools` injection, so it does not touch the injected- + tool agent-cache bypass. + +Three things worth knowing before building PR 2: + +* **Strands re-executes an interrupted tool from the top on resume** + (`strands/types/interrupt.py`), so `request_user_login` runs twice per + takeover. `take_control` is idempotent and a marker on `agent.state` carries + the descriptor across the pause; without the marker a resume that lands in a + container which lost the session would call `acquire` and silently start a + *second, unauthenticated* browser, then report success. +* **The abandonment deadline is enforced by the reaper, not by a timer.** + Nothing runs while a turn is paused. `_reap_idle` exempts a user-controlled + session only until `deadlineAt`, and it runs on any conversation's next + browser call in the same container, with the remote TTL as the backstop. + `BROWSER_TAKEOVER_DEADLINE_SECONDS` defaults to 480, with a 60s grace on the + resume side so a user who finished at 7:59 whose POST lands at 8:01 is not + told their sign-in expired. +* **`live_view` is gone from `browse_web`.** It returned a presigned URL as + tool-result text — the exact failure PR #1101 diagnosed — and its 300-second + cap made it useless for a human anyway. + +⚠️ Still unverified, and still blocking PR 3: whether `profileConfiguration` +survives a browser *resource* replacement (risk 2 below). + +## Build notes (PR 2) + +The route, its IAM and the SPA's handling of `browser_login_required` are +built. The **viewer page is not**, and two findings from building the rest +change its design — which is why it was left open rather than stubbed. + +### D3 correction — the viewer must NOT call app-api itself + +D3 says "the page calls the D2 route for its own URL and refreshes it on +expiry." It must not. app-api's CORS is an allowlist with +`allow_credentials=True`, so for the viewer to call the route from the sandbox +origin, **that origin would have to be added to `CORS_ORIGINS`** — and the +mcp-sandbox origin is where we frame *untrusted MCP App HTML*. Granting it +credentialed access to app-api would hand every App a same-origin-ish path to +the user's session. + +The viewer should instead receive its URL by `postMessage` from the SPA, which +already holds the cookie and is already the trusted origin: + +- the SPA calls `POST /sessions/{id}/browser/live-view` (built, and the SPA + client for it is in `BrowserLoginService.mintLiveView`), +- posts `{url, viewport}` into the frame, +- re-mints before `expiresAt` and posts again. + +The viewer then needs no credentials, no CORS entry, and no knowledge of +app-api at all. This is strictly less surface than D3 as drafted. + +### D3 correction — no new distribution, no new certificate + +D3 implies a viewer origin built like the mcp-sandbox one. A *second* +CloudFront distribution means a second subdomain, a second us-east-1 ACM cert +and a second `CDK_*_CERTIFICATE_ARN` deploy var — and per the scar tissue in +`mcp-sandbox-distribution-construct.ts` (#396) a configured domain with a +missing cert **throws at synth**, so adding one ahead of the cert would break +every deploy until ops caught up. + +The mcp-sandbox origin already is what D3 describes: a static, non-SPA origin +with `frame-ancestors` locked to the SPA. The viewer should be one more static +file served from it. Note its `BucketDeployment` is `prune: true` with no key +prefix, so the page belongs in `infrastructure/assets/mcp-sandbox/` — a second +prefixed deployment to the same bucket would be pruned by the first. + +### What PR 2 ships + +`POST /sessions/{session_id}/browser/live-view` on app-api (404 for a +conversation that is not the caller's or has no browser session, 409 for one +whose session has ended, 404 while the flag is off), a **narrower** IAM +statement than the Runtime's — `ConnectBrowserLiveViewStream`, +`UpdateBrowserStream`, `GetBrowserSession` only, no Start/Stop and no +`ConnectBrowserAutomationStream`, because app-api must never drive the browser +— plus the SPA's validator, `BrowserLoginService`, and the resume path. + +The SPA validator rejects any event carrying a `url`, mirroring the backend's +`assert_no_url`: nothing upstream should ever put one there, so its presence +means a contract regression shipped and refusing to render beats framing a URL +of unknown provenance. + +⚠️ Still unverified, and now the *first* thing the viewer PR must answer: does +DCV work inside a cross-origin iframe under this CSP? It opens a WebSocket to +the AgentCore data plane, which the sandbox origin's `connect-src` does not +currently allow (risk 3 below). + ## PR breakdown | PR | Scope | Notes | |----|-------|-------| | **1** | Backend takeover: `request_user_login`, take/release control, interrupt + `browser_login_required` SSE event, D4 metadata row, reaper pinning, abandonment deadline, flag | Testable end to end with a curl-minted live-view URL — no frontend needed | -| **2** | app-api live-view route + IAM; `live-view.html` viewer at the sandbox origin; SPA framing and resume | Needs the `bedrock-agentcore` npm dependency approved first | -| **3** | Profiles (D5) + the Customize-surface "saved logins" list with forget/expiry | The PR that makes the annual sweep actually repeatable | -| **4** | `browser-assessment` CDK resource (D6) + axe extension and `accessibility_scan` (D7) | D6 needs campus network decisions from whoever owns routing | +| **2** | app-api live-view route + IAM; SPA `browser_login_required` handling and resume | Shipped WITHOUT the viewer — see "Build notes (PR 2)" | +| **2b** | The viewer itself: sign-in prompt component, plain-JS `live-view.{html,js}`, the DCV SDK fetch step, `sandboxOrigin` on the event | The PR that makes the feature usable at all. No npm dependency (D3) | +| **3** | **MANAGED `URLBlocklist` applied per session (D6)** + the policy object and its IAM | Re-sequenced ahead of profiles. This is the control that lets the tool be granted at all, so it gates rollout rather than following it | +| **4** | Profiles (D5) + the Customize-surface "saved logins" list with forget/expiry | The PR that makes the annual sweep actually repeatable | +| **5** | axe extension and `accessibility_scan` (D7) | Turns the output into VPAT-grade evidence | PR 1 alone unblocks a developer-driven demo. PRs 1+2 unblock the Library user for anything login-gated and publicly routable, which is most of the renewal -list. PR 3 is what makes it worth doing annually. PR 4 covers campus-only -targets and turns the output into VPAT-grade evidence. +list. **PR 3 is the gate on granting the tool to real users** — until the +blocklist is in place and verified, RBAC is the only control, and Security 3 +explains why that is not enough on its own. PR 4 is what makes the sweep worth +doing annually. + +Profiles were re-sequenced *after* the blocklist deliberately: a persisted +login is standing authenticated access with no takeover in the path, so +shipping it first would remove the one control the user actually passes +through. ## Testing @@ -438,16 +711,31 @@ targets and turns the output into VPAT-grade evidence. ## Risks and open questions -1. **Is `networkConfiguration` create-only on `CfnBrowserCustom`?** Assumed yes. - Confirm before planning PR 4; if it is not, D6 could collapse to a config - change on the existing resource — though a separate resource is still the - better design for blast radius. -2. **Does `profileConfiguration` survive a browser *resource* replacement?** - If profiles are scoped to the browser id rather than the account, a PR-4 - deploy could invalidate every saved login. Verify before shipping PR 3. -3. **Does the DCV viewer work inside a cross-origin iframe with our CSP?** - MCP Apps proved the pattern for `srcdoc` content; DCV opens a WebSocket and - may need `connect-src` allowances the current policy does not grant. +1. ~~**Is `networkConfiguration` create-only on `CfnBrowserCustom`?**~~ + **Moot, and worse than assumed.** There is no `UpdateBrowser` operation at + all, so *every* property is effectively create-only and any change replaces + the resource. This is what moved the URL policy off the browser resource and + onto `StartBrowserSession` (D6). +2. ~~**Does `profileConfiguration` survive a browser *resource* replacement?**~~ + **Answered: yes.** `CreateBrowserProfile` takes `name`, `description`, + `clientToken` and `tags` — and **no `browserId`**. Profiles are account-level + resources, not scoped to a browser, so replacing the browser resource does + not invalidate saved logins. PR 3 and the policy work can land in either + order. +2b. ~~**Is `type: MANAGED` honoured at `StartBrowserSession`?**~~ + **ANSWERED, AGAINST US (dev, 2026-09-19).** It is rejected: + `Invalid value for parameter 'type'. MANAGED is not supported for + session-level policies.` Not a silent downgrade — `StartBrowserSession` + fails and every session dies. See D6. **Do not grant + `request_user_login` until the MANAGED policy is applied at + `CreateBrowser`.** +3. ~~**Does the DCV viewer work inside a cross-origin iframe with our CSP?**~~ + **Resolved in principle.** The sandbox origin's CloudFront function composes + `connect-src` from a `?csp=` query parameter and is attached to the + *default* behaviour, so it covers `live-view.html` too. The SPA names the + minted URL's own origin there. Still to confirm against a live session: + that DCV needs nothing beyond `connect-src` (its workers and decoders are + same-origin, and `worker-src 'self' blob:` is already granted). 4. **Mobile.** DCV interaction on a phone, for a login form in a 1280x800 remote viewport, is likely poor. May need an explicit "open in a new tab" escape hatch rather than pretending the frame works everywhere. diff --git a/docs/specs/compaction-model-relative-thresholds.md b/docs/specs/compaction-model-relative-thresholds.md new file mode 100644 index 000000000..5a5068003 --- /dev/null +++ b/docs/specs/compaction-model-relative-thresholds.md @@ -0,0 +1,574 @@ +# Compaction relative to the model window — a trigger ceiling, a target floor, and paying the rewrite only when it is free + +**Status:** PR-1 open (#1125, this document rides with it). PR-2 (bounded +summary + compaction metrics, #1128), PR-3 (paid-when-free apply, #1129), +PR-4 (tool-result offload at intake, #1131) and PR-5 (selective 1h TTL, +flag off, built 2026-09-16) stacked on top of it. +**Owner:** Phil Merrell +**Related:** `compaction-over-threshold-cache-spiral.md` (#833 — the incident +and the summary-cap PR this spec depends on) · +`compaction-v2-versioned-prefix.md` (#835 — the frozen-segment redesign; this +spec implements v2's I3/I4 *policy* on the v1 machinery so it survives either +way) · `agent-cache-extra-tools-bypass.md` · `document-context-offload.md` · +`gpt-5-6-prompt-caching.md` (why `maxInputTokens` is a pricing cap) · +`docs/one-pagers/cost-effectiveness-roadmap.md` (W2 row) · the 2026-09-15 prod +cost audit (top-5 September users + 7 largest conversations, via +`/admin/costs`), whose measurements §2 quotes + +--- + +## 1. The question this answers + +Compaction fires at a fixed 100,000 tokens +(`AGENTCORE_MEMORY_COMPACTION_TOKEN_THRESHOLD`), regardless of whether the +model's window is 200k, 272k or 1M. Should the trigger vary with the window? + +Yes — but the trigger is the least important number. Under Bedrock prompt +caching the marginal costs are: + +| operation, at a 200k-token prefix (Sonnet 5, `global.*`, $2.00/MTok input) | cost | +|---|---| +| keep the history for one more turn (cache read, 0.1× input) | ~$0.04 | +| change the history once (cache write, 1.25× input) | ~$0.50 | + +Retention is ~12× cheaper than mutation *per event*. What compaction spends +is **prefix rewrites**, and what it saves is **the size of every read after**. +So the two numbers that matter are (a) how small the conversation is after a +compaction — the *floor* — and (b) whether the rewrite lands on a turn that +was going to rewrite anyway. The July replay in the 2026-07-27 measurement +made this concrete: compacting the ten biggest sessions at 120k down to 25k +cut input-side cost 64%; deeper-and-rarer beat shallower-and-more-often in +every trajectory. + +A model-relative threshold that merely *raises* the ceiling on 1M models +would therefore make things worse: every read is larger, every cache bust is +larger, and long-context quality degrades well before a 1M window fills +(`document-context-offload.md` §"context rot"). The design below scales the +trigger with the window but **caps** it, and puts the engineering weight on +the floor and the scheduling. + +## 2. Current state (verified 2026-09-15 on `develop` @ 222e1d26) + +- `CompactionConfig.token_threshold` = 100,000 (`constants.py` Defaults), + compared in `update_after_turn` against the turn's cache-inclusive input + (`inputTokens + cacheRead + cacheWrite`, the only correct context size under + caching — see the coordinator comment at the call site). +- The catalog already carries the window: `ManagedModel.max_input_tokens` + (`maxInputTokens`) — 200k on the Claude 4.x rows, 1,000,000 on Sonnet 5, + 272,000 on the GPT-5.6 family and GPT-6 Astra, 256,000 on Qwen. The stream + coordinator looks it up every turn for the badge (`final_metadata["contextWindow"]`) + and the storage path. **It is not a capability field on the OpenAI rows: it + is the short-context pricing cap** (`curated-models.ts:373`). Deriving + thresholds from it therefore keeps us inside short-context pricing for free. +- The checkpoint is chosen by *turn count*: `cutoffs[-protected_turns]`, i.e. + "keep the last 3 user turns". Turn count is uncorrelated with tokens — the + byte-stability audit found three cheap turns summarized while a 92k tool + result was kept. +- There is **no hysteresis.** Over threshold, the checkpoint advances on every + turn (each new turn pushes `cutoffs[-3]` forward), the LTM summary join is + re-fetched and re-persisted, and `Threshold exceeded` logs every turn — the + spiral session did this 56 times. +- `update_after_turn` receives the agent's live list, which on a restored + session is *already sliced* at the checkpoint, and compares a slice-relative + index against the persisted absolute checkpoint (spiral spec D3; 199 of + 1,238 rows carry the mismatch). +- **Strands' default 40-message sliding window runs underneath all of this.** + `AgentFactory.create_agent` passes no `conversation_manager`, so the SDK + installs `SlidingWindowConversationManager(window_size=40)` and applies it + after every event-loop cycle (`agent.py:1631`). Past 40 messages the front + of `agent.messages` slides every turn. The 2026-09-15 prod cost audit (20 + sessions, $127, content-free) measured the consequence: fingerprint + `messageCount` pinned at 39–41, every turn start reading only tools+system + and re-writing the whole window (session e7e75953 flips exactly at message + 41), and the checkpoint/anchor coordinate mismatch (`ANCHOR_MISMATCH`) on + 14 of 20 sessions — because the list the checkpoint indexes into is being + mutated by something other than compaction. +- The same audit's cache-write attribution for September (Sonnet 5 = $683 of + $742; cache writes ~54% of it): 36% full re-writes after a >5 min pause + (cost scales with the context size at the pause — 17 of 20 sessions + peaked above 100k), 22% live re-writes inside over-100k sessions (the + spiral, still live), 2.6% the hourly system-prompt tick + (`get_current_date_pacific()` renders `%H:00`, so every Pacific hour + boundary flips `systemPromptHash`), ~2% the 40-message window in sub-100k + sessions. Summaries of 23k–40k tokens were present in 6 of 20 sessions. +- Compaction changes bytes in two places only: the restore-time slice in + `_apply_compaction` (cold starts) and — nowhere on a warm agent. That is the + subject of PR-3, not PR-1. +- Strands 1.55 ships the native form of "trigger as a ratio of the window": + `SummarizingConversationManager(proactive_compression={"compression_threshold": r})` + reads `model.context_window_limit` (a `BedrockConfig` key we do not set). It + has no floor, no cache-aware scheduling and no summary budget, and the + 2026-05-18 decision bars it as a bare swap. Per v2 §4.2 it is the *engine* + we would move onto, not the *policy*. + +## 3. Design + +### 3.0 Prerequisite: one owner of history size + +Compaction cannot express a checkpoint in a list that something else is +trimming. PR-1 sets the conversation manager explicitly: +`SlidingWindowConversationManager(window_size=2000, should_truncate_results=True)` +(`AGENTCORE_CONVERSATION_WINDOW_MESSAGES`; `40` restores the SDK default). +The manager is kept rather than replaced with `NullConversationManager` +because its `reduce_context` is the stack's only +`ContextWindowOverflowException` recovery, and that path is independent of +the window size. Consequence to state plainly: conversations between 40 +messages and the ceiling now go to the model whole — more *read* tokens per +turn (0.1×), far fewer *re-writes* (1.25×), and the model sees the +conversation instead of its last 40 messages. Above the ceiling the +compaction policy bounds it. + +### 3.1 Three numbers per model, derived from `maxInputTokens` + +``` +ceiling = min(window × CEILING_RATIO, CEILING_CAP_TOKENS) # trigger +floor = ceiling × FLOOR_RATIO # target after a cut +hard_ceiling = min(window × HARD_CEILING_RATIO, ceiling × HARD_MULT) # force, even on a warm cache +``` + +Defaults: `CEILING_RATIO 0.5`, `CEILING_CAP_TOKENS 100_000`, `FLOOR_RATIO 0.25`, +`HARD_CEILING_RATIO 0.7`, `HARD_MULT 1.5`. Which gives: + +| `maxInputTokens` | ceiling | floor | hard ceiling | +|---|---|---|---| +| 128,000 (a small-window model, for illustration) | 64,000 | 16,000 | 89,600 | +| 200,000 (Claude 4.x, Haiku 4.5) | 100,000 | 25,000 | 140,000 | +| 256,000 (Qwen) | 100,000 | 25,000 | 150,000 | +| 272,000 (GPT-5.6 / GPT-6 pricing cap) | 100,000 | 25,000 | 150,000 | +| 1,000,000 (Sonnet 5) | 100,000 | 25,000 | 150,000 | +| unknown (catalog miss) | `token_threshold` (100,000) | 25,000 | 150,000 | + +So the window scales the ceiling **down** for small-window models and never +up: for every model in the catalog today the cap binds at 100k. That is the +honest answer to "shouldn't the threshold vary with the window" — under +cache economics, no, not upward. Why: + +- **The cap was drafted at 200k for 1M models and moved to 100k on + evidence.** The 2026-09-15 replay of the 20 audited Sonnet 5 sessions + (input side, this spec's PR-3 scheduling rule applied, priced at the + incident's $2.50/$0.20 per MTok) gave: actual today **$102.64**, no + compaction **$208.81**, 100k/25k **$73.70**, 200k/50k **$105.30**. The 200k + policy was worse on 13 of 20 sessions and never better, and roughly equal + to today — raising the ceiling gives back the whole PR-1 win on that cohort. + Mechanism: 36% of cache-write dollars are cold re-writes after a >5 min + pause, and their size is the context *at the pause*; under 200k/50k most + heavy sessions never reach the ceiling and run 100–190k the whole time, so + each return costs ~$0.375 instead of $0.08–0.12. Caveats: 20 sessions, all + Sonnet 5, all heavy; the "actual" column already benefits from the + 40-message window that PR-1 removes, so "no compaction" is the baseline + PR-1 replaces. Raise the cap only when `compaction_forced` and the cost + anatomy show sessions that need more room; it is one constant. +- **Quality is on the same side.** Every warm turn reads the whole prefix; + the offload spec's evidence on context rot says the model is not better at + 400k of chat history than at 100k plus a good summary. +- **The floor at a quarter of the ceiling** is the "deeper and rarer" result: + a session that compacts to 25k and grows back to 100k pays one rewrite of + ~25k and then ~75k tokens' worth of *reads* before the next cut. A session + that compacts 100k → 80k pays a rewrite of 80k every few turns. +- **The hard ceiling** exists so that PR-3's "wait for a free turn" cannot + wait forever. 70% of the window leaves room for the turn's own output and + for one oversized tool result without an overflow. On a 200k window the 70% + term (140k) binds; on larger windows the 1.5× term (150k) does. +- **These are starting points, not tuned constants.** §5 says how to move + them. Every one is an env override (`AGENTCORE_MEMORY_COMPACTION_*`), and + `AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED=false` reverts to the + fixed 100k threshold and the legacy turn-count cut exactly. + +### 3.2 A floor-seeking, token-aware checkpoint + +When the trigger fires, the cut is chosen to land the retained history **at +or below the floor**, keeping as much as fits: + +1. Estimate tokens per message from the message's serialized size (chars/4; + flat 1,500 for an inline image; document bytes are already stripped at + restore). Calibrate the estimates so they sum to the turn's *history* + tokens — the `messages` partition from the context-attribution breakdown + when the turn has one, otherwise the full input count. Over-attributing + system/tools tokens to history biases the cut slightly deeper, which is + the safe direction. +2. Candidate cuts are the same tool-pair-safe boundaries as today (user + messages that are not tool results), and never newer than + `cutoffs[-protected_turns]` — the last N turns are always kept, as today. +3. Choose the **oldest** candidate whose retained estimate is ≤ floor. If + even the minimum-protection cut exceeds the floor (a giant tool result + inside the protected tail), take the minimum-protection cut and log it; + evicting inside the protected tail is PR-4's escalation, not a deeper cut. + +### 3.3 Hysteresis: a cut disarms the trigger + +`CompactionState.armed` (persisted, legacy rows default `True`): + +- Over the ceiling **and armed** → cut, then `armed = False`. +- Over the ceiling **and disarmed** → do nothing unless input ≥ hard ceiling, + in which case cut anyway and log `compaction_forced`. A forced cut is the + signal that the previous cut did not take (the summary is too large, or the + slice has not landed on this agent yet) — it is a metric, not a code path + we expect to run. +- At or below the ceiling → `armed = True`. + +This makes v2's I3 ("threshold exceeded twice in a row is a bug by +definition") structural on the v1 code: the spiral's 56 consecutive cuts +become one cut plus 55 no-ops, and the LTM summary fetch stops being a +per-turn call. + +### 3.4 One coordinate system + +`update_after_turn` computes cuts over the agent's live list. The live list +starts at the absolute index the restore sliced at (or 0). The manager now +tracks that offset (`_live_offset`, set in `_apply_compaction`) and persists +`checkpoint = _live_offset + relative_cut`. The persisted checkpoint stays +absolute — the coordinate `_apply_compaction` slices with — and the D3 +comparison (`new <= current`) is finally like-for-like. The deeper D3 +question (the restore window itself moving — `original=74` frozen) is +untouched here and stays with spiral-spec PR-3. + +### 3.5 Pay the rewrite when it is free (PR-3) — BUILT + +Everything above decides *what* to cut. PR-3 decides *when the bytes change*. +As built (`apply_pending_compaction` + the `pending*` fields on +`CompactionState`; kill switch +`AGENTCORE_MEMORY_COMPACTION_DEFERRED_APPLY_ENABLED=false` applies cuts +immediately as PR-1/2 did; legacy mode is always immediate): + +- **Post-turn parks, never applies.** `update_after_turn` computes the cut and + the bounded summary and stores them as `pendingCheckpoint` / + `pendingSummary` / `pendingHardCeiling` / `pendingSince`. `checkpoint` stays + the *applied* value — the one `_apply_compaction` slices at on restore. A + second over-ceiling turn while a cut is parked is a no-op + (`compaction_pending_waiting`); it never cuts deeper. +- **Head of turn decides.** The stream coordinator calls + `apply_pending_compaction(agent, prefix_key="|")` before the + first model call of every turn, on cached and freshly restored agents + alike. It applies when, in this order: `cache_expired` (more than + `cache_ttl_seconds` since the previous turn's save — the entry is gone and + the next call re-writes the prefix regardless), `prefix_changed` (the + model|agent key differs from the persisted `lastPrefixKey` — the cached + prefix is already invalid), or `hard_ceiling` (the previous turn's input + reached the hard ceiling the cut was computed under). Otherwise it waits. +- **Applied in place.** `messages[:] = [first_with_summary] + messages[k+1:]` + where `k = pendingCheckpoint − _live_offset`; then `_live_offset` moves to + the checkpoint and the state is promoted (`checkpoint`, `truncation_anchor`, + `summary`) and persisted. The result is byte-identical to what + `_apply_compaction` derives from stored history under the promoted state + (pinned by `test_live_apply_matches_a_cold_restore_of_the_same_state`), so + a cold restore after a live apply reads the same prefix. +- **Aliasing carries the offset.** `_adopt_session_conversation` copies + `_live_offset` when it points a new agent at the live list. +- **Measured.** Each application persists `applied` (the reason), + `cacheGapSeconds` and `pendingSince` on `compaction.policy`, logs + `rewrite_scheduled` vs `rewrite_forced`, and emits `CompactionApplied`, + `CompactionAppliedForced` and `CompactionCacheGapSeconds`. +- **Not done here:** `context_window_limit` on the Strands model config + (needs the window at agent construction; small follow-up). + +The original design sketch, kept for the record: + +- Post-turn computes and persists the pending checkpoint + summary (the + expensive part, off the critical path). Nothing is applied. +- Pre-call on the next turn, the pending cut is applied to the live list + **in place** (`agent.messages[:] = ...`, never rebound — the #741 alias) + when any of: the gap since the last call exceeds the Bedrock TTL + (`cacheGapSeconds` > 300 — the prefix was going to rewrite anyway), the + model or the `@`-mentioned agent changed (prefix already invalid), or the + turn's input is projected at or above the hard ceiling. +- Between the ceiling and the hard ceiling on a warm cache, the cut waits. + `rewrite_scheduled` vs `rewrite_forced` is logged per application so I4's + effectiveness is measurable. +- Aliasing the message list across agent instances must also alias the + offset; `_adopt_session_conversation` syncs `_live_offset` when it adopts. + +### 3.6 The rest of the sequence + +- **Bounded summary** (= spiral spec PR-2, unchanged): an 8k-token budget, + re-summarized once with the cheap model at cut time. Without it the floor + is unreachable — a 40k-token summary is larger than the 25k floor. PR-1's + `compaction_forced` metric will show exactly how often this bites until it + lands. +- **Offload escalation** (PR-4) — BUILT for tool results, at intake. The + case that defeats the floor is a huge tool result inside the protected + tail. Rather than escalate at cut time (which would mutate a protected turn + and need a restore-replay of every edit), oversized tool results are bounded + the moment they are produced, before they enter the prefix: Strands 1.55's + vended `ContextOffloader` on `AfterToolCallEvent`, S3 storage in the + user-files bucket under `compaction-offload/{userId}/{sessionId}/` (one + namespaced storage per agent, so references are session-scoped by + construction and an `@`-mention agent resolves the same ones), no eviction + from the model path (`evict_after_cycles=None`; a 90-day S3 lifecycle rule + expires the objects), gate 4,000 tokens / preview 1,000 (env-backed; + `AGENTCORE_TOOL_RESULT_OFFLOAD_ENABLED=false` removes the plugin). The + persisted message already carries the bounded form, so restore reproduces + it byte-for-byte. Our subclass adds a chars/4 pre-filter so the plugin's + per-result `CountTokens` round trip only runs for results near or over the + gate, and a content-free `ToolResultOffloaded` record per offload. The + model keeps a preview and pulls spans back with + `retrieve_offloaded_content` (pattern / line range / full) — one stable + spec in `toolConfig`, not an RBAC-gated tool, on the `read_skill_file` + precedent (the `workspace_files` catalog key is granted to no prod role, + so escalating into the workspace tools would have shipped dark). **User + attachments are not touched**: the digest + page-range read in + `document-context-offload.md` is the right shape for those and stays that + spec's PRs 1–4. What remains of the floor-unreachable case after this is + measured by `CompactionFloorUnreachable` on the cut record. +- **`context_window_limit` on the model** (PR-3, small): plumb + `maxInputTokens` into `ModelConfig` and set `context_window_limit` in + `to_bedrock_config` (a valid `BedrockConfig` key in 1.55), so Strands' + own `estimate_utilization` and our policy agree on the window, and the + eventual v2 engine swap inherits it. +- **Per-section cache TTL** (PR-5) — BUILT, **default off**. `AGENTCORE_PROMPT_CACHE_STATIC_PREFIX_TTL=1h` + makes `ModelConfig.to_bedrock_config` emit + `CacheConfig(system_prompt_ttl="1h", tools_ttl="1h")`; upstream rewrites + the hand-placed TTL-less system point and gives the tools point its own, + the message point stays at 5m (`cache_config.ttl` unset), so the order is + tools(1h) → system(1h) → messages(5m), which Bedrock requires + non-increasing. Anything but the literal `1h` emits exactly today's bytes. + **Why off by default, against the flags-default-on house style:** the + 2026-07-27 model found a *blanket* 1h TTL a wash — the 2× write premium ate + the saving — and only the selective variant worth testing; and the repo's + standing rule since #954/#956 is that a caching default is never adopted on + inspection alone. The gate is `scripts/probe_static_prefix_ttl.py` (two + arms, same static prefix, a configurable gap) plus a week of cost rows in + dev with the flag on. **The experiment arm's rows are priced honestly:** + `CostCalculator.calculate_message_cost` bills the unread remainder of the + static segment at the 1h premium (2× base) and the rest of the write at the + catalog's 5m rate, using the context-attribution breakdown for the static + size; every such row carries `staticPrefixTtl: "1h"` so the anatomy can + split arms. Economics to expect: +0.75× base on every static write, −1.15× + base on every return inside the hour but past five minutes; it pays when + the second is more frequent than the first (the audit's 36% cold re-writes + after a >5 min pause say it might, on the 28k static prefix; the hourly + system-prompt tick would bust it hourly until that fix lands). + +## 4. Cost model + +Per-turn input cost for a session sitting at `P` prefix tokens on a model with +base input rate `r`: + +``` +warm turn ≈ 0.10 r P (cache read) +cache-busted turn ≈ 1.25 r P (cache write) +compaction turn ≈ 1.25 r F + summarizer call (F = post-cut size) +``` + +For the July cohort (10 sessions, peaks 112k–597k, ~19 cache tokens written +per output token) the replay gave input-side cost **$90.35 → $32.40** when +cutting at 120k to 25k. The same replay with the ceiling at 200k and the floor +at 50k is the number to produce for the Sonnet 5 row before ratifying §3.1 — +`scan_fleet_prefix_spend.py` and the spiral spec's §4.2 harness already +replay real trajectories under a policy. + +## 5. Quality gate and tuning + +- **Veto before default change in prod:** the spiral spec §4.3 long-session + eval (constraint retention / revision continuity / reference lookup) runs + on PR-1 with the fixed-threshold arm as control. A deeper cut is a bigger + context change than the summary cap, so the veto applies with full force. +- **Tuning knobs move on evidence, not taste:** raise `FLOOR_RATIO` if the + eval shows retention loss; lower `CEILING_CAP_TOKENS` if the Sonnet 5 + cohort's write:read ratio stays worse than 1:5 after PR-3; never raise the + cap above 272k while GPT-family rows share the constants (pricing tier). +- **Summarizer prompt:** preserve standing user instructions and constraints + verbatim (kaizen 2026-05-29 item); this is PR-2's prompt, not PR-1's. + +## 6. PR breakdown + +### PR-1 — policy, floor-seeking cut, hysteresis, coordinates (this PR) + +- `AgentFactory.build_conversation_manager()`: the explicit 2000-message + window (§3.0) — the prerequisite for every coordinate claim below. +- `compaction_policy.py`: `CompactionPolicy.resolve(config, context_window)`, + `estimate_message_tokens`, `choose_checkpoint`. +- `CompactionConfig`: `model_relative_enabled` + the five ratio/cap fields, + all env-backed; `CompactionState.armed` + a `policy` snapshot of the cut; + `CompactionResult` gains `context_window`, `ceiling`, `floor`, + `hard_ceiling`, `forced`, `retained_tokens_estimate`. +- `TurnBasedSessionManager`: `_live_offset`; `update_after_turn(..., + context_window=, history_tokens=)` implements §3.2–§3.4. **No change to + when bytes change** — the slice still applies at restore only. +- Stream coordinator passes the catalog window (already looked up for the + badge) and the breakdown's `messages` tokens; the `compaction` SSE payload + carries the policy fields (additive — the SPA validator ignores extras; + the TS interface gains them as optional). +- Kill switch `AGENTCORE_MEMORY_COMPACTION_MODEL_RELATIVE_ENABLED=false` → + fixed threshold, turn-count cut, no arming. Default on (house style). + +**Acceptance:** unit tests for the table in §3.1 (including the unknown-window +and kill-switch rows); a 5-turn conversation with 1,000-token threshold cuts +to the oldest candidate under the floor; a turn over the ceiling on a disarmed +state is a no-op and a turn at the hard ceiling is a forced cut; the existing +byte-stability suite is unchanged; a replayed spiral-shaped sequence (input +constant above ceiling for 10 turns) produces exactly one checkpoint advance. + +### PR-2 — bounded summary (spiral spec PR-2, as written there) — BUILT + +As built (`compaction_summary.py`, stacked on PR-1): `bound_summary()` holds +the persisted summary at `COMPACTION_SUMMARY_TOKEN_BUDGET` (8,000 tokens, +chars/4 — the same estimate the admin `SUMMARY_OVER_BUDGET` diagnosis uses). +Within budget → unchanged. Over budget → one Nova Micro `converse` call +(`AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ID`; kill switch +`AGENTCORE_MEMORY_COMPACTION_SUMMARY_MODEL_ENABLED=false`) with a prompt that +keeps standing instructions, decisions, current state of the work, open items +and exact identifiers, and drops narration and superseded drafts (the kaizen +2026-05-29 item). Model failure, a ceiling-hit generation, or an overshoot → +newest-first truncation (keep the newest records that fit; if none fit, the +tail of the newest). Runs once at cut time and the result is persisted +verbatim, so the byte-stability contract is untouched. Provenance lands in +the persisted `compaction.policy` map (`summarySource`, `summaryOutcome`, +`summaryTokensBefore/After`, `summaryTokenBudget`). + +**Sequenced immediately after PR-1, ahead of PR-3, on evidence.** In the +2026-09-15 audit all 16 over-100k sessions carried `AGENT_CACHE_BYPASS` +(spreadsheet/word/ppt tools enabled), so they rebuild the agent every turn +and the restore slice already runs for them; zero were warm-agent +(`create_artifact`-only) sessions. What bit in every one was the slice +running and *still* not getting under the threshold — 23k–40k-token summaries +and the protected tail. Session 65b6d4ab: 17 live full re-writes from message +15 onward at 40–100k context, before the 40-message window ever engaged — +the compaction slice re-running on a fresh agent each turn with a changing +summary. That is this PR's problem, not PR-3's. + +### PR-3 — paid-when-free scheduling (§3.5) — BUILT; `context_window_limit` deferred + +Value on the audited cohort is the `create_artifact` (warm-agent) sessions and +turn latency, not the over-100k dollars — those are PR-1 + PR-2. Two of the +four window-pinned sessions examined (e7e75953, 0d8ba8f8) never exceeded 100k +and were pure 40-message-window effects, which PR-1's §3.0 change alone +recovers. + +**Acceptance:** on a warm agent held above the ceiling, the live list shrinks +on the first turn after a >300s gap and not before (unless hard ceiling); +`partial_miss` on the cut turn only; `test_second_cache_key_for_a_session_shares_the_conversation` +still passes with the in-place apply. + +### PR-4 — offload escalation (§3.6) — BUILT as tool-result offload at intake + +**Cohort split (2026-09-15 audit, content-free, September prod).** Of the 95 +sessions that peaked over 100k: 52 (55%) had a single tool result ≥4k tokens +in their last three turns (43 only that, 9 also an attachment) — PR-4's +target; 25 (26%) had an attachment there (16 only that) — the document-offload +spec's; 27 (28%) had neither — long sessions whose bulk is old history plus a +23–40k summary, which the cut and the summary cap reach. The biggest single +intra-turn writes are squarely tool results (123k, 149k, 131k in one call); +the attachment-only cases include 1–3-turn sessions at 280–530k that are one +huge upload. Method: turns split at `cacheGapSeconds ≥ 10s`; "big tool result" += an intra-turn call that read the prior prefix and wrote ≥4,000 tokens +(slightly overstated on long `tool_use` blocks); "attachment in the last 3 +turns" = an upload row between the first call of those turns minus 5 min and +the last call. + +### PR-5 — selective 1h TTL experiment (§3.6) — BUILT, flag off; enable per the gate above + +**Probe run 2026-09-16, dev-ai us-west-2, Haiku 4.5, 420 s gap +(`scripts/probe_static_prefix_ttl.py --gap-seconds 420`):** + +| arm | call | read | write | $ (base $1.10/MTok) | +|---|---|---|---|---| +| 5m | first | 0 | 6,251 | 0.008598 | +| 5m | second | 0 | 6,251 | 0.008598 | +| 1h | first | 0 | 6,251 | 0.013756 | +| 1h | second | **5,924** | **327** | 0.001374 | + +Bedrock honors `ttl: "1h"` on the tools and system points: after the 5m +entry expired, the 1h arm **read** the static segment (5,924 tokens) and +re-wrote only the message segment (327), while the 5m arm re-wrote all +6,251. Pair cost $0.01513 vs $0.01720 — the 1h arm was **12% cheaper at this +gap**, having recovered its 60% dearer first write on one return. The +arithmetic that generalizes: the premium is 0.75× base per static write +(~$0.0052 on this prefix), the saving 1.15× base per return inside the hour +but past five minutes (~$0.0075); the arm pays when such returns outnumber +static writes by more than ~0.7 : 1. Static writes also happen on every +prefix *change* (a model or tool-set switch, and today the hourly +system-prompt tick), so the dev-week measurement should run **after** the +tick fix lands. + +**60 s gap, same day:** both arms read the full 6,251 on the second call +(5m: $0.008598 + $0.000691; 1h: $0.013756 + $0.000691). The 1h arm was +**$0.005157 more expensive** — exactly the 0.75× base premium on the first +write, with nothing to recover inside five minutes. So the arm is a fixed +surcharge per static write, ~$0.0052 on this 6.3k prefix (≈$0.023 on a 28k +prod prefix at Haiku rates, ≈$0.042 on Sonnet 5), recovered at ~$0.0075 / +$0.034 / $0.061 per return that lands between five and sixty minutes. The +dev week decides whether prod sessions return in that window often enough; +the probe cannot. + +## 7. Observability + +### 7.1 What PR-2 adds — one content-free record per cut + +`AgentCoreStack/Compaction` EMF namespace (silenced with the rest of the cost +observability layer by `PROMPT_CACHE_OBSERVABILITY_ENABLED=false`): + +| metric | answers | +|---|---| +| `CompactionCut` | cadence — cuts per session-day is v2 §8's spiral detector (alarm at >2/day) | +| `CompactionForced` | how often a cut ran while disarmed = how often the previous cut did not take | +| `CompactionInputTokens`, `CompactionRetainedTokens` | how far above the ceiling cuts fire and how deep they land (is the floor being reached?) | +| `CompactionSummaryTokens`, `CompactionSummaryOverBudget` | is the summary the reason cuts miss the floor; how often the model vs truncation path runs | +| `CompactionFloorUnreachable` (PR-4) | how often the protected tail alone still exceeds the floor after intake offload — the residual document/attachment case | +| `ToolResultOffloaded`, `ToolResultOffloadedTokens` (PR-4) | how much tool payload was kept out of the prefix, per tool (`toolName` property) | + +Properties (queryable in Logs Insights, not dimensions): `policySource`, +`contextWindow`, `ceiling`, `floor`, `summaryOutcome`, `summaryTokensBefore`. +No text from the conversation or the summary is ever emitted. + +### 7.2 Data points still worth collecting (not built) + +The question behind all of them is *what does a turn cost because of history, +and what did compaction do to it* — today we can see the first half (cost +rows) and, after PR-2, the second, but not joined: + +- **Per-call `checkpoint` / `armed` / `liveOffset` on the `C#` cost row** — + lets the anatomy page show the context trajectory against the cuts without + correlating timestamps by hand. Additive fields on an existing write. +- ~~**`cacheGapSeconds` next to every cut**~~ — done in PR-3: each application + records the reason and the gap. +- **Retained-vs-actual calibration** — the next turn's measured + `contextBreakdown.messages` against the cut's `retainedTokensEstimate`. One + number per cut, and the only way to know whether the estimator is off by 5% + or 50%. +- **Turn shape** — messages per turn and tool-result bytes per turn (the + `ToolCensusHook` has the count; bytes are one more `ADD`). Partly covered + by PR-4's `ToolResultOffloadedTokens`; the sub-gate long tail is not. +- **Outcome signal joined to compaction** — a down-thumb rate keyed to + "turns since last cut" is the first evidence that a cut costs anything + besides dollars (the kaizen review-queue already lists this as the + counterweight the cost roadmap lacks). Without it the §5 quality gate stays + a one-off eval rather than a standing measurement. +- **Warm-vs-cold split per cut** — whether the cut's session was on the + agent-cache bypass path (restore every turn) or a warm agent; the 2026-09-15 + audit had to infer this from `enabledTools`. It decides how much PR-3 is + worth. + +- Log lines: `compaction_cut` (window, ceiling, floor, hard, relative cut, + absolute checkpoint, retained estimate), `compaction_disarmed_noop`, + `compaction_forced`, `compaction_rearmed`. +- The persisted `compaction.policy` map on the session row records what the + last cut believed the window and thresholds were, so + `GET /admin/costs/sessions/{id}/profile` can show it. The + `OVER_COMPACTION_THRESHOLD` diagnosis should read that map instead of the + fixed default once PR-1 is live (small follow-up, not in PR-1). +- **Measurement confounder:** the hourly system-prompt tick (`%H:00` in + `get_current_date_pacific()`) re-writes every session's prefix once an hour + regardless of compaction. Fix it (render the date only, or the hour in a + turn-scoped message) before attributing write:read movement to this spec's + PRs; filed separately, not in PR-1. +- `compactionCount` (already emitted) divided by session-days is the + cadence metric v2 §8 asks for; an alarm on >2/day is the spiral detector. + +## 8. Non-goals + +- Changing the cachePoint layout (tools/system/auto) — v2 §9. +- Replacing the mutation engine with Strands' conversation manager — that is + v2 and stays gated on its §7 criteria; every policy field here maps onto + v2's thin policy layer unchanged. +- Cross-session memory quality; the quota system; model routing. + +## 9. Open questions + +- Should an explicit `AGENTCORE_MEMORY_COMPACTION_TOKEN_THRESHOLD` in an + environment override the model-relative ceiling? PR-1 says no: it is the + fallback for an unknown window only. Flip this if an operator needs a + per-environment ceiling. +- Whether Bedrock prices Claude's 1M window in tiers the way it prices + GPT-6 Astra. The model cards say no long-context premium for the Claude + rows today; if that changes, `CEILING_CAP_TOKENS` is the one constant to + move. diff --git a/docs/specs/compaction-over-threshold-cache-spiral.md b/docs/specs/compaction-over-threshold-cache-spiral.md index 799d83068..3f1a0e1f2 100644 --- a/docs/specs/compaction-over-threshold-cache-spiral.md +++ b/docs/specs/compaction-over-threshold-cache-spiral.md @@ -1,8 +1,8 @@ # Compaction over-threshold cache spiral — stop paying a full prefix re-write on every turn **Status:** PR-1 shipped (#838). PR-5 shipped (#845 — see "As shipped" under -§3 PR-5; its acceptance replay moved one of §3's own numbers). PR-2 through -PR-4 unbuilt. **§4.1's adversarial re-scan is run** (2026-08-05, results +§3 PR-5; its acceptance replay moved one of §3's own numbers). PR-2 built +2026-09-15 (stacked on #1125, see below). PR-3 and PR-4 unbuilt. **§4.1's adversarial re-scan is run** (2026-08-05, results inline below): D2 and D3 both reproduce on sessions other than the incident's, and the harm is already multi-user. **Motivating incident:** prod, 2026-08-05 analysis. One faculty user exhausted @@ -277,7 +277,9 @@ regression case works forward from the deploy, not backward, and the fleet baseline in the roadmap's metric 1 starts at the first prod release — dev deployment alone does not start that clock. -### PR-2 — bound the compaction summary +### PR-2 — bound the compaction summary — BUILT 2026-09-15 + +*Built as `compaction_summary.py` on the model-relative-thresholds branch (stacked on #1125); see `compaction-model-relative-thresholds.md` §6 PR-2 for the as-built notes (budget constant, Nova Micro side-channel, newest-first fallback, provenance fields, EMF).* - Add `COMPACTION_SUMMARY_TOKEN_BUDGET` (default **8_000** tokens ≈ 32k chars) to [constants.py](../../backend/src/agents/main_agent/config/constants.py). diff --git a/docs/specs/document-context-offload.md b/docs/specs/document-context-offload.md index 1dc6cf656..7273a138d 100644 --- a/docs/specs/document-context-offload.md +++ b/docs/specs/document-context-offload.md @@ -1,8 +1,16 @@ # Document context offload — bound the cost of conversations with attachments -**Status:** Draft (no branch yet) +**Status:** PR-1 built — `feature/document-offload-pr1`, off `develop` @ +`c28ccdbd` (after the compaction stack #1125 → #1128 → #1129 → #1131 → +#1132 merged). Revised 2026-09-16: analytics pulled forward from PR-7 into +PR-1 (§5, §6.1), decision log in §8. **Motivating measurement:** prod scan 2026-08-03, all 18,942 `C#` cost rows -joined to `boisestateai-v2-user-file-uploads` on `GSI1PK = CONV#{sessionId}` +joined to `boisestateai-v2-user-file-uploads` on `GSI1PK = CONV#{sessionId}`; +re-confirmed by the 2026-09-15 prod cost audit — of 95 September sessions +that peaked over 100k tokens, 25 (26%) had an attachment in their last three +turns, including 1–3-turn sessions at 280k–530k that are a single huge +upload. A compaction cut cannot touch those (thresholds spec §3.6 keeps +attachments out of scope on purpose); a digest plus a page-range read can. **Related:** [[project-prod-cache-write-premium]] (the compaction root cause this depends on), `docs/specs/session-workspace-tools.md` (the retrieval primitive this extends), `docs/specs/tool-search-token-bloat-strategy.md` (same tenet, @@ -383,15 +391,23 @@ Re-sequenced around defect 4: the loss fires on turn 2 for ~80% of attachment sessions, so the recovery path and the digest are the urgent half, and the turn-level offload — the cost work — comes after. +**Revised 2026-09-16.** The analytics that were PR-7 now ship *in* PR-1, so +the cost work is measured from the first day the recovery path exists and the +ship / widen-pinning / abandon decision in the evaluation spec (§4.2 there) +is decidable from stored rows rather than a one-off scan. The field design is +§6.1 below; the kaizen finding that the compaction work has no quality signal +joined to its cost data is addressed there too — as far as the platform +currently allows (see the outcome-signal row). + | PR | Scope | Gate | |----|-------|------| -| 1 | `document_read` (page-range + pattern), native `document` block reassembly, **gated on the session having an attachment**; ids kept out of `INJECTED_TOOL_IDS` | 47-page PDF: `page_range={4,7}` returns 4 pages, ≤6k tok; `max_pages` cap holds; tool present for a `student`-role session with no grants | -| 2 | `DocumentDigest` model + Haiku extractor + persist on `FileMetadata`, generated at upload; **not yet used in context** | digest ≤1,500 tok for a 200-page PDF; extractor p95 < 8s; no chat-path change | -| 3 | `_strip_document_bytes` → digest + live `document_read` handle instead of the placeholder (restore path only) | a session with `analyze_spreadsheet` enabled answers a document question correctly on **turn 2**; re-upload rate starts falling | -| 4 | Offload trigger in `update_after_turn` + pinning, behind `DOCUMENT_OFFLOAD_ENABLED` | slice-assignment guard test passes; offload fires at most once per document; never fires while cache is live | -| 5 | Fix the cache-live guard on the *existing* truncation deferral (same predicate as PR-4) | truncation events while cache live: 72% → ~0 | +| 1 — **built** | `document_read` (page-range + pattern + bounded text; native `document` block reassembly), **gated on the session having a readable attachment**, id kept out of `INJECTED_TOOL_IDS`, presence carried in the agent-cache key; **analytics**: per-call document context on `C#` rows, the `documentReads` ledger entry, the `document_stripped` ledger event, session-row rollups, anatomy + profile surfaces, EMF; `document_read` results exempt from the tool-result offloader | 12-page PDF: `page_range="4-7"` returns 4 pages as one native block whose page 1 is original page 4; `max_pages` and the hard cap hold; tool built for a session with no grants and no picker toggle; content-policy test walks the new fields; full backend suite green | +| 2 — **built** (`feature/document-offload-pr2`, stacked on PR-1) | `DocumentDigest` (`apis/shared/files/document_digest.py`): a deterministic outline (headings with page / paragraph / line anchors, table and figure mentions, counts) plus a 3–5-sentence abstract from the cheap text model (Nova Micro, `DOCUMENT_DIGEST_MODEL_ID`), built off the request path when a document upload completes and persisted as `FileMetadata.digest`; `render_digest` produces the `` block under a hard 1,500-token budget (sections dropped first, then the abstract) and stores the estimate as `digest.tokens`; `digest.abstract` / `digest.sections` denylisted, coverage on the attachment profile; **not yet used in context** | 200-page PDF digest ≤1,500 tokens (tested); extraction and abstract each fail open; kill switch `DOCUMENT_DIGEST_ENABLED`; no chat-path change; p95 latency to be read from `DocumentDigestMs` in dev | +| 3 — **built** (`feature/document-offload-pr3`, stacked on PR-2) | `_strip_document_bytes` → `session/document_rehydration.py`: each inline document block is matched to its upload row (sanitized filename incl. PromptBuilder's `_2` suffix, byte-size tiebreak, newest first, no double-claiming) and replaced by the rendered `` block; rows without a digest get an **outline-only digest built from the bytes already in the restored message** (no S3, no model call) and persisted; unmatched blocks keep the placeholder byte-for-byte; ledger records `document_rehydrated` and `document_stripped` separately; kill switch `DOCUMENT_REHYDRATE_ENABLED` | restore output is stable across restores (tested); a matched block carries the abstract, outline and handle; unmatched blocks are unchanged from today; `document_stripped` per session-day should fall to the unmatched residue (direct base64 attachments, deleted files) — read it from the ledger; re-upload rate (byte-identical) starts falling | +| 4 — **built** (`feature/document-offload-pr4`, stacked on PR-3) | Offload at **head-of-turn**, in the slot right after `apply_pending_compaction` (not post-turn: the compaction stack's §3.5 "decide post-turn, apply when free" is the same rule) — `session/document_offload.py` + `TurnBasedSessionManager.apply_document_offload`: pinning (attach turn + next, prompt names it, recent `document_read` result), `DOCUMENT_OFFLOAD_MIN_TOKENS` (5,000), and the cache-gap decision (`cache_expired` / `prefix_changed` / `over_ceiling`, else wait); the replacement is PR-3's restore transformation so the live block equals a cold restore's; aged `document_read` slices stubbed on both paths; `DOCUMENT_OFFLOAD_ENABLED` kill switch + `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` bucket on `crc32(session_id)`; records `document_offload` with `cacheGapSeconds` | in-place mutation only (no rebinding); a pinned document never moves; nothing moves while the cache is live (tested); **zero** `document_offload` events with `cacheGapSeconds` under the TTL except `over_ceiling` / `prefix_changed` ones, which carry their reason | +| 5 — **verified closed + measured** (`feature/document-offload-pr5`, stacked on PR-4) | The defect this row was written against (July 2026: 34 of 47 truncation events fired inside the 5-minute TTL) was closed by the compaction stack before this spec's PRs began: the truncation anchor now moves only with a cut (applied paid-when-free, thresholds spec §3.5) or in `_maybe_advance_truncation_anchor`, which checks `_cache_window_expired` first — the same predicate PR-4 reuses — and the stability suite already pins "a warm cache never advances the anchor". What was missing was the proof from rows: the advance was a log line. PR-5 records a `truncation_anchor` ledger event (`anchorFrom`, `anchorTo`, `cacheGapSeconds` measured before the save re-stamps `updated_at`) plus `TruncationAnchorAdvanced` / `TruncationAnchorCacheGapSeconds` EMF, and puts `cacheGapSeconds` on the restore-time `applied` event so a slice or truncation landing inside the TTL is countable as what it is — a rebuild while the cache was live | `truncation_anchor` events with `cacheGapSeconds` under the TTL: **zero** (any one is a regression of the guard); restore `applied` events with `truncatedToolResults > 0` and a short gap measure the rebuild-while-live cohort the `extra_tools` bypass spec owns | | 6 | Backend guard on a turn's aggregate inline attachment bytes (~7.5 MB), mirroring the SPA's `MAX_FILES_PER_MESSAGE` | oversized turn degrades to the `oversized_inline` guidance path, never to `SessionException` | -| 7 | `hasDocuments` / `documentTokens` on `MessageMetadata`; admin cost anatomy shows document share | the two-table join this spec required becomes a dashboard column | +| 7 | **Outcome signal**: a content-free thumbs up/down on assistant messages, persisted keyed on `(sessionId, messageId)` so it joins the `C#` row's `hasDocuments` / `documentDigests` / `documentReads` in one key; fleet-level document-share column on the admin dashboard | down-thumb rate reported by turn class (full / digest-only / retrieved) with n per class | **PRs 1–3 are the correctness fix and should ship together as a unit.** None carries cost-regression risk: the digest is strictly smaller than the document, @@ -400,7 +416,9 @@ must precede PR-3 — a digest that points at a tool nobody has is no better tha today's placeholder. PR-4 is the cost work and is independently revertible. PR-6 is unrelated to both -and can go whenever. +and can go whenever. PR-7 exists because **no feedback surface exists today** +(`MessageMetadata` carries only a `# feedback: …` placeholder comment); the +rows PR-1 writes make the join a one-key lookup the moment one does. **Out of scope, but surfaced by this work:** the `extra_tools` agent-cache bypass makes ~76% of *all* sessions rebuild their Agent every turn. Fixing that @@ -431,6 +449,90 @@ quality regression**, measured as: digest is too good to be true and the model is answering without the source; if it is >3/turn the digest is too thin +### 6.1 Analytics field design (PR-1, built) + +**Rule: content-free by construction.** Every field below is an id, a count, +a byte size, a token estimate or an enum key. No document text, title or +filename is ever persisted in a metric, a cost row or an EMF record — the +`document_read` tool result carries filenames (that is conversation content, +where the model needs them); the ledger reads only the result's numbers. The +content-policy test walks the new projections and response models. + +**Alignment.** Same mechanism as the compaction ledger (#1130): per-call +facts ride the `C#` cost row as flat extra fields next to `prefixTokens` / +`windowRemovedMessages` / `compactionEvents`; lifecycle events use +`record_compaction_event` and the `compactionEvents` list; session rollups are +`ADD`ed on the `S#` row beside `compactionAppliedCount`; EMF goes to +`AgentCoreStack/Compaction`; the anatomy (`GET /admin/costs/sessions/{id}/calls`) +and the profile read them. One namespace, one ledger, one anatomy view. + +**Per model call (`C#` row).** Written by the stream coordinator from +`agent.messages` at turn end (`session/document_context.py`); gated by +`COST_DIAGNOSTICS_ENABLED` like the rest of the ledger, so an absent field +reads "not tracked", never 0. + +| field | meaning | +|---|---| +| `hasDocuments` | ≥1 inline attachment block (document or image bytes) is in the live context | +| `documentCount` | inline attachment blocks on user prompts | +| `documentTokens` | their estimated weight — the compaction estimator's bytes/4 per document, flat per image. Heuristic, comparable across rows; the measured total it is a share of is `tokenUsage` (input + cacheRead + cacheWrite) minus `prefixTokens.system + prefixTokens.tools` | +| `documentDigests` | digest / placeholder stand-ins in context (`[Document placeholder:` today, ` 0` and not `hasDocuments`), *retrieved* +(`documentReads.pages > 0`), or *none*. This is the digest-vs-full share. + +**Lifecycle events** (`compactionEvents[].kind`, numbers only): + +| kind | when | fields | +|---|---|---| +| `document_stripped` | restore replaced inline documents with contentless placeholders (the defect; recorded from PR-1 so its reach is measured before and after PR-3) | `documents`, `documentTokens` | +| `document_rehydrated` | PR-3: restore replaced them with a digest + live handle | `documents`, `documentTokens` | +| `document_offload` | PR-4: the post-turn trigger swapped a document for its digest | `documents`, `documentTokens`, **`cacheGapSeconds`** at the moment it fired | + +**Session rollups (`S#` row, `ADD`):** `fullDocumentCalls`, `digestOnlyCalls`, +`documentReadCalls`, `documentReadPages` — so the profile can answer without +the rows once they expire, and so a fleet query needs one table. + +**EMF (`AgentCoreStack/Compaction`):** `DocumentRead` / `DocumentReadPages` / +`DocumentReadBytes` with properties `mode` (`list` / `index` / `pages` / +`pattern` / `text`) and `format`. PR-4 adds `DocumentOffloaded` / +`DocumentOffloadedTokens` beside `ToolResultOffloaded`. + +**Admin surfaces:** anatomy rows carry every per-call field (`documentReads` +as a map); the profile carries `fullDocumentCalls`, `digestOnlyCalls`, +`peakDocumentTokens`, `documentReadCalls`, `documentReadPages` and +`dataCoverage.documents`; the SPA anatomy page shows a per-row `doc` / +`digest` / `+Np` badge, a Documents line in the expanded row, and the +consumption summary under the Attachments card. + +**What is decidable from the rows alone** (the point of pulling this forward): + +- *Document share of the prefix*, per call: `documentTokens / (context − + prefixTokens.system − prefixTokens.tools)`. Summed over cold rows + (`cacheStatus ∉ {hit}`), `cacheWriteInputTokens × share` is the document + part of every re-write — the recoverable envelope the evaluation spec's §4.1 + says was unmeasured, and the number that turns "−15%" into a derived target. +- *Was the offload ever wrong*: `document_offload` events whose + `cacheGapSeconds` is under the cache TTL. Target zero; any other value is + the compaction PR-3 rule being broken. +- *Did the recovery path reach users*: `document_stripped` per session-day + before PR-3, `document_rehydrated` after; `documentReadCalls` per attachment + session against the evaluation's health band (≈0.3–3; ≈0 means the digest is + answering unaided, >3/turn means it is too thin). +- *Digest-vs-full shares and the B/C arms*: `fullDocumentCalls : + digestOnlyCalls` per session, split by the PR-4 rollout bucket. +- *The outcome signal* — **not buildable today.** There is no feedback + surface (`MessageMetadata` has a `# feedback:` placeholder only). The rows + are keyed so that a thumbs row on `(sessionId, messageId)` joins the turn + class in one lookup; PR-7 adds that surface. Until then the quality gate + stays the evaluation spec's offline harness, and this remains the kaizen + gap it has been for the compaction work too. + ### Quality gate — this must not ship on cost numbers alone **Full evaluation design: `docs/specs/document-offload-evaluation.md`** (three @@ -498,6 +600,178 @@ visual fidelity before any digest comparison is scored. --- +## 8. Decision log — PR-1 (2026-09-16) + +1. **Analytics ship first, not last.** PR-7's fields moved into PR-1 (§6.1) so + the cost work is measured from day one and the evaluation's stopping rule + is decidable from stored rows. The `document_stripped` event is recorded + *before* the fix that removes it, so PR-3's effect is a before/after on one + counter rather than a scan. +2. **Gate = session state; presence lives in the agent-cache key, not a + cache veto.** `document_read` is built when the session has a READY + document-class upload (PDF, DOCX, TXT, MD, HTML — not tabular, decks or + images, which have other paths) or this turn attaches one. Positive + answers are memoized per process (monotonic in practice). Vetoing the + agent cache instead — the Memory-Space pattern — would have made every + attachment session rebuild its agent each turn and hit the strip on turn + 2 for the ~19% that keep a warm agent today: a quality regression PR-1 + alone would introduce. A `document_tools` element in `_create_cache_key` + flips at most once per session, on the attach turn, when restored history + has no document to lose; the resume path recomputes the same gate so a + paused agent's key is reproduced. +3. **Id recorded, never injected-filtered.** `DOCUMENT_TOOL_IDS` exists for + the record and is deliberately not in `INJECTED_TOOL_IDS`; the only control + is `DOCUMENT_READ_ENABLED` (default on, `=false` kills). +4. **`document_read` results are exempt from the tool-result offloader** + (`OFFLOAD_EXEMPT_TOOLS` in `core/tool_result_offload.py`, plus the + plugin's own `should_offload`). Offloading the slice the model just asked + for would undo the read. The tool's `max_pages` (default 8, hard cap 20) + is the bound instead. +5. **Slices persist in history.** A retrieved page range is a tool-result + document block with a unique Bedrock-safe name (`" p4-7 <6 hex>"`), + so it survives restore (the strip only touches top-level blocks) and never + collides. They are counted on the row (`documentSlices`); PR-4 should age + them like documents rather than let them re-write forever. +6. **Page identity.** The slice is a new PDF numbered 1..k; the payload's + `page_numbering` note and the tool description say "page k is original + page start+k−1 — cite original page numbers". The evaluation's + citation-page-identity family is the test of whether models follow it. +7. **No new dependency.** PDF slicing and text extraction use `pypdfium2` + (already pinned for thumbnails); DOCX text comes from a stdlib + `zipfile` + `ElementTree` extractor over `word/document.xml`. `page_range` + is PDF-only; DOCX and text use `pattern` or bounded `offset` reads + (the workspace read bound, 48 KB). +8. **Token numbers are heuristics.** Bedrock reports no per-block usage; + `documentTokens` is bytes/4 (documents) and a flat estimate per image, + the compaction estimator's own numbers. They are comparable across rows and + against the measured messages partition, which is all the decisions in + §6.1 need. +9. **Metrics share the compaction namespace** (`AgentCoreStack/Compaction`) + on purpose: the cut record, the tool-result offload record and the + document read are three answers to one question — what bounded this + session's prefix — and belong on one dashboard. +10. **`docs/specs/document-conversations-cost.md` does not exist in the + repo.** The numbers it would hold are §1 here and the validation report; + nothing in this spec depends on it. +11. **Validation nits carried in:** `INJECTED_TOOL_IDS` is in + `apis/shared/tools/injected.py` (line drifted); `WORKSPACE_READ_MAX_BYTES` + lives in `apis/shared/files/workspace.py`. Citations remain out of PR-1 + (§6 probe note). + +**PR-2 (2026-09-16)** + +12. **Outline is deterministic; only the abstract uses a model.** Page / + paragraph / line counts, headings (numbered, ALL CAPS, or title-cased + short lines; markdown `#`), table / figure mentions and a text sample + come from `pypdfium2` and the PR-1 DOCX extractor. The abstract is 3–5 + sentences from Nova Micro over the sample plus outline — the same cheap + text model the tool-batch summaries and the compaction summary run on, + and the extraction step never needs a frontier model. The §4A sketch said + Haiku over the document; a text model over extracted text is cheaper, + text-only is enough for an abstract, and the visual channel is preserved + by `document_read`, not by the digest. +13. **Built at `complete_upload` only, fire-and-forget.** The SPA upload flow + is where documents enter; agent-written files (`workspace_write`, Word / + Excel / PowerPoint tools) register `FileMetadata` directly and get no + digest. Uploads that predate PR-2 have none either. **PR-3 must generate + lazily on the restore path when `digest` is absent** (same builder, + `with_abstract` optional under a latency budget) rather than assume it. +14. **The digest is content-bearing.** `digest.abstract` and + `digest.sections` are denylisted; `digest.status` / `format` / `count` / + `tokens` are projected so the attachment profile can report coverage and + the per-file token cost the 1,500 ceiling is enforced against. +15. **Budget enforcement is in the renderer**, not the extractor: sections + are dropped from the end, then the abstract is truncated, and the opening + tag (the `document_read` handle) always survives. `digest.tokens` records + the rendered estimate, so "digest ≤1,500 tokens" is a stored fact per + file rather than a claim. + +**PR-3 (2026-09-16)** + +16. **Matching is by sanitized filename, not by id.** A document block + carries no upload id — only the name `PromptBuilder` gave it, which is + `FileSanitizer.sanitize_filename(filename)` (the extension's dot becomes + an underscore: `BBR Policy.pdf` → `BBR Policy_pdf`, duplicates get + `_2`/`_3`) — so restore matches blocks to the session's upload rows on + that name with the byte size as tiebreak, newest row first, each row + claimed once. Adding an upload id to the block itself would be cleaner + but changes the attach-turn bytes (a prefix change for every attachment + turn); left for a later PR that touches the prompt builder anyway. +17. **Lazy digests are outline-only and come from the restored bytes.** The + bytes are in `agent.messages` at the moment of the strip (§4E), so no S3 + read is needed, and the restore path is synchronous inside the agent + constructor, so no model call is made there. The digest is persisted so + the next restore renders identical bytes; the upload-path build (PR-2) + may later overwrite it with one that has an abstract, which changes the + rendered block once — one prefix re-write, visible as a jump in + `document_rehydrated.documentTokens`. +18. **Restore output is byte-stable** given the same upload rows: the + renderer is deterministic over `FileMetadata.filename`, `upload_id` and + the persisted digest, and unmatched blocks keep the pre-PR-3 placeholder + verbatim. Tested. The cache contract in CLAUDE.md is therefore honoured + on the restore path exactly as before. +19. **Never worse than before.** Every failure — lookup, matching, digest + build, persistence — falls back to the placeholder for that block and + is recorded as `document_stripped`; the lookup is attempted once per + restore, and only when the history holds a document block. +20. **The synchronous constraint is real.** `TurnBasedSessionManager.initialize` + runs under the Strands agent constructor with an event loop already + running, so it cannot await; the repository gained `_sync` bodies for the + session query and the digest write (boto3 was synchronous underneath all + along) rather than a thread-with-its-own-loop. + +**PR-4 (2026-09-16)** + +21. **Head-of-turn, not `update_after_turn`.** §4C said the trigger runs + post-turn. The compaction stack has since moved to "decide post-turn, + apply at the head of the next turn only when the re-write is free or + unavoidable" (`apply_pending_compaction`, thresholds spec §3.5), and + that predicate — cache expired, prefix changed, or the ceiling is + already breached — *is* this spec's cache-aware rule. So the offload + runs in the same slot, right after the parked cut is applied, and reads + the same facts (`updated_at`, `last_prefix_key`, `last_input_tokens`). + One decision point for every prefix mutation. +22. **The offload is the restore transformation, applied early.** A block is + replaced by exactly what PR-3 would render for it on a cold restore + (same matcher, same persisted digest, same renderer). That is what + makes it byte-stable: after an offload, the live prefix and the next + restore agree on that block. Unmatched blocks stay inline on the live + path (restore has to drop bytes; offload does not). +23. **Pinning is a pure function of the message list plus the incoming + prompt**, not Strands' `pin_message` metadata. Our slice is not + `SummarizingConversationManager`, and message metadata is persisted to + AgentCore Memory — writing pins there would alter stored bytes. Rules: + on the attach turn and the next (`DOCUMENT_OFFLOAD_PIN_TURNS=2`); the + incoming or previous prompt contains the document's name stem; a + `document_read` result for it sits in the recent turns. +24. **Slices are aged on both paths.** `document_read` page slices older + than `DOCUMENT_SLICE_MAX_TURNS` become a deterministic stub on the live + path (same gate) and on restore, so a warm agent and a cold restore + agree. The stub is a function of the block's unique name, so it is stable. +25. **Rollout is a bucket, not a boolean.** `DOCUMENT_OFFLOAD_ROLLOUT_PERCENT` + (default 100) over `crc32(session_id) % 100` gives the evaluation spec + its concurrent B/C arms; `DOCUMENT_OFFLOAD_ENABLED=false` is the kill + switch. `crc32`, not `hash()`, because the latter is salted per process. +26. **Ceiling without a policy snapshot.** When the compaction state has no + `policy.ceiling` yet, the reason falls back to + `CompactionPolicy.resolve(config, None)` (the fixed threshold) — a + conservative ceiling, so `over_ceiling` never fires early. + +**PR-5 (2026-09-16)** + +27. **Nothing to fix; something to prove.** The July finding was a real + defect in the pre-stack code (truncation ran at every `initialize()`, + and the bypass path rebuilt the agent every turn). The compaction stack + made the anchor a pure function of persisted state that moves only with + a cut or a cold cache, so the guard now holds by construction and by + test. PR-5 therefore adds the ledger event and metric that turn "holds by + construction" into a standing measurement, rather than re-implementing + the guard under a new name. The gap on the event is measured *before* the + advance saves state, because the save re-stamps `updated_at` and would + otherwise read as zero. + +--- + ## Sources - [Effective context engineering for AI agents — Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) diff --git a/docs/specs/load-test-assessment-2026-09.md b/docs/specs/load-test-assessment-2026-09.md new file mode 100644 index 000000000..f38e37510 --- /dev/null +++ b/docs/specs/load-test-assessment-2026-09.md @@ -0,0 +1,245 @@ +# Load-test report assessment — "Five performance fixes, proven by a controlled A/B" + +**Source:** `AgentCoreSolution-PerformanceTesting-ProductionReadyFixes.pdf`, AuraX platform team, +run 2026-09-15 on a production-mirror environment, Claude Haiku 4.5, 5M-token/min quota. +**Assessed against:** `develop` at `ca3e8d2c` (2026-09-18), strands-agents 1.55.0, +bedrock-agentcore 1.21.0. +**Status:** **COMPLETE — all five findings and one residual shipped.** Merged to `develop` 2026-09-19 +(#1154/#1155 verified in dev; the rest deploy on the next rollout): #1154 (bytecode + warm-up, bounded +CountTokens, session split memo, four cache families, CORS synth guard), #1155 +(`assistant_id` in the cache key → spreadsheet sessions cache their agent), +#1156 (system prompt counted against a probe message — Bedrock rejects an empty +conversation, verified live, so `systemTokens` had been chars/4 since it +shipped), #1157 (bounded memory retrieval, summary namespace off the per-message +path: 3 → 2 lookups), #1158 (Spreadsheet Analysis auto-enabled on attachment, +RBAC-gated). The response to AuraX is at +[`docs/one-pagers/load-test-response-to-aurax-2026-09.md`](../one-pagers/load-test-response-to-aurax-2026-09.md). +Still open: the Phase 3 A/B rerun (needs the provisioning gate), the +`RetrieveMemoryRecords` 30 → 300/s quota request, the AWS cold-start thread, and +the Phase 0 asks to AuraX. + +--- + +## 1. Verdict in one table + +| # | Report claim | Verdict on `develop` | Where it lives | +|---|---|---|---| +| 1 | The prompt is counted **three times per turn** because the agent is rebuilt every turn and re-splits its context attribution | **Confirmed for the majority cohort.** The split is memoised per `Agent` *instance*; only artifact-tool and no-injected-tool sessions reuse an instance. Every other injected-tool family rebuilds the agent each turn → 2 attribution counts + Strands' own 1 = 3 | `session/hooks/context_attribution.py:139-160`, `inference_api/chat/service.py:340`, `apis/shared/tools/injected.py:102` | +| 2 | The image ships **no compiled bytecode**, so each fresh micro-VM compiles the app + sympy on the first message | **Confirmed.** `uv sync` runs without `UV_COMPILE_BYTECODE`; `/app/src` is copied raw; the container runs as a non-root user, so nothing can be written back at runtime | `backend/Dockerfile.inference-api` | +| 3 | **One CountTokens call sits in front of every model stream** and gets throttled under load | **Confirmed.** `use_native_token_count=True` makes Strands `await _estimate_input_tokens()` before every model call; a throttle costs boto's retry/backoff on the reply path, then falls back to chars/4 | `core/model_config.py:434`, strands `event_loop/event_loop.py:494` | +| 4 | Browser uploads fail: the uploads bucket has **no CORS rule** | **Environment gap, not a mainline bug.** CDK attaches CORS to `UserFilesBucket` *only if* `CDK_DOMAIN_NAME` / `CDK_CORS_ORIGINS` is set; an unset value only prints a warning | `infrastructure/lib/constructs/data/file-upload-construct.ts:49-80`, `lib/config.ts:1477` | +| 5 | With the default tool set the assistant **refuses to read an attached CSV** and tells the user to enable a tool | **Confirmed.** CSV/XLSX are never sent inline at any size; the Spreadsheet Analysis tools are opt-in (`enabledByDefault: False`); the system prompt instructs the model to send the user to Customize → Tools | `inference_api/chat/routes.py:874-905, 1075-1090`, `core/system_prompt_builder.py:101-125` | +| R1 | Residual: **6 conversation-memory lookups per message** | **Confirmed mechanism, count depends on cohort.** A cold `initialize()` = 5 `ListEvents` (read_session ×2, read_agent ×2, list_messages ×1). *Every* user message = up to 3 `RetrieveMemoryRecords` (preferences, facts, summary namespaces) inside the SDK's `append_message`. Rebuilt-per-turn sessions pay all of it every turn | SDK `session_manager.py:320-333, 463-481, 783, 872`; `session/session_factory.py:205-235` | +| R2 | Residual: memory service quota **30 lookups/s**, crossed at ~5 msg/s | **Quota identified.** `RetrieveMemoryRecords` = 30 req/s/account (adjustable). `ListEvents` = 200/s/account (adjustable) + 20/s per actor-session (fixed). The report's "30" is the retrieval quota, not ListEvents | AWS quotas page (fetched 2026-09-18) | +| R3 | Residual: micro-VM cold start **~12 s** | **Plausible, unverified here.** No warm pool exists in the Runtime product; our own Aug-05 probe measured turn-1 at 7–8 s in dev | `docs/specs/agent-cache-extra-tools-bypass.md` §8.2 | + +Two supporting claims could not be verified: + +- **"One of the five is already contributed upstream."** No matching PR exists in this + repo, and a search of `strands-agents/sdk-python` for count_tokens / token-estimation + PRs found nothing resembling "background side-request + probe user message". Ask + AuraX for the link. +- **"The model rejects an empty conversation"** (the reason for their probe message). + Our attribution hook calls `count_tokens(messages=[])` for the system-prompt + partition. If Bedrock rejects that, Strands swallows the error and the + `systemTokens` partition has been chars/4 all along. Dev SSO was expired during + this assessment; the probe is one command (§5). + +--- + +## 2. Why the numbers are credible — and where they don't transfer + +**Mechanism matches the curve.** Bedrock throttles CountTokens on a *request-rate* +quota separate from the model's token quota. Three counts per turn at 35 msg/s is +~105 extra Bedrock requests/s on the critical path; removing them removes the +request-rate exhaustion while the *token* burn rises (their 53% → 72%), which is +exactly what "the fixes convert wasted request budget into productive output" +predicts. The 541 → 0 throttle result is consistent with our code. + +**The baseline is worse than `develop`, so the deltas are upper bounds.** Their branch +predates the "load-shedding subsystem" on mainline (our per-user quota runway and +cooldowns), and very likely predates #841 (2026-08-05, runtime session-id pinning). +Before #841 the agent cache *never* hit, so their baseline rebuilt the agent on +every turn for every session. On `develop` that is true only for non-artifact +injected-tool sessions — which was 76% of prod sessions when last measured +(2026-08-03), driven by `analyze_spreadsheet` / `list_spreadsheets`. Expect the +real-world gain on fix 1 to be large but smaller than 18.2 s → 6.6 s. + +**Load model arithmetic checks out.** 15 msg/user/day × 10% in the peak hour = +1.5 msg/user/hour, so 1 msg/s ≈ 2,400 daily users. 17 msg/s ≈ 40,800 is a fair +readiness target. + +**Cold-start cells are honestly footnoted.** The 59% / 88% success cells are ramp +first-steps, not load ceilings; the throttle curve and new-chat p95 carry the case. + +**Do not reuse their cost figure.** $0.0025/message at a 99% prompt-cache hit rate +reflects tiny, repetitive synthetic prompts. Prod runs at a write:read ratio near +1:2 with attachments at ~31% of spend; our own admin cost drill-down is the number. + +**Their harness is ours.** "2,000 synthetic accounts, SSE support, realistic +peak-hour task mix" describes `tests/load/` + `scripts/load-test/provision.sh`. That +is good news: the A/B is reproducible on dev without new tooling. + +--- + +## 3. Design notes on the fixes as described (read before re-implementing) + +1. **Backgrounding the count changes what compaction sees.** If + `_estimate_input_tokens` no longer awaits CountTokens, `projected_input_tokens` + is either `None` or chars/4 on that call. Our compaction policy and the + attribution hook both consume it. The prompt-cache contract makes compaction + timing a *cost* decision, so any change here must keep the authoritative count + on the turns that decide a checkpoint (cold start, post-tool batches) and may + only drop it where the warm-path delta is small. +2. **"Never mutates model configuration."** `CountTokensBedrockModel.count_tokens` + swaps `config["model_id"]` to the base id for the duration of the call. That is + safe only because count and stream never overlap on one instance. The moment + the count becomes a concurrent task, a `stream()` can read the base id and fail + on-demand invocation. Pass `modelId` explicitly instead of mutating config. +3. **Throttle latency is retry backoff, not the call.** Strands wraps the boto + client's default retry mode. A dedicated CountTokens client with + `retries={"total_max_attempts": 1}` and a ~2 s read timeout bounds the cost of a + throttle to one failed attempt before the heuristic fallback — a smaller change + than restructuring the event loop, and it composes with note 1. +4. **Auto-enabling spreadsheet tools must ride the cache key.** The existing + `document_tools` bit in `_create_cache_key` is the pattern: the gate answer and + the cache key must be computed by the same function or a paused agent gets + orphaned. RBAC still applies — injection is "enable for this turn", never + "grant". It changes `toolConfig` once on the attach turn (one prefix re-write, + same as `document_read` today). +5. **Bytecode: precompile, don't rely on runtime writes.** The image runs as + `bedrock_agentcore` (uid 1000) with `/app` chowned, so Python *could* write + `__pycache__`, but every micro-VM starts from the image layer, so it recompiles + every time regardless. `UV_COMPILE_BYTECODE=1` for the venv plus + `python -m compileall -q /app/src` in the final stage is the whole fix. + +--- + +## 4. Plan + +Ordered by (impact on new-chat p95 and throttles) ÷ (effort and risk). Each item +names its proof so we do not ship on the report's numbers alone. + +### Phase 0 — Get the artifacts (this week, no code) + +- Ask AuraX for: the branch/diff of the five fixes, the run ledgers, the + upstream PR link, and the exact `git merge-base` with our `develop`. A + contribution case is fastest to evaluate as a PR against `develop`; each fix + will need re-basing because the branch predates #839/#841 and the quota runway. +- Run the empty-messages CountTokens probe (§5) to settle whether our + `systemTokens` partition has ever been authoritative. + +### Phase 1 — No-regret changes (target: one PR each, all this sprint) + +**P1-A Precompile bytecode + warm the first turn** (fix 2) +- `Dockerfile.inference-api`: `ENV UV_COMPILE_BYTECODE=1` in the builder stage; + `RUN python -m compileall -q /app/src` after the source copy in the final stage. +- Inference-api lifespan: import `strands_tools.calculator` (sympy), build the + Bedrock / AgentCore Memory clients, and construct one throwaway + `CountTokensBedrockModel` at startup so the first message pays none of it. +- **Proof:** `docker run … python -X importtime -c "import apis.inference_api.main"` + before/after; then `agent_status` `thinking` cycle-0 timing and the Runtime + `InvocationLatency` on new sessions in dev. + +**P1-B Bound CountTokens on the reply path** (fix 3, notes 1–3) +- Dedicated `bedrock-runtime` client for counting with + `retries={"total_max_attempts": 1}`, 2 s read timeout; pass `modelId` explicitly + (remove the config swap). +- Keep the awaited authoritative count on cold start and on the first call after a + tool batch (checkpoint-deciding turns); on plain warm turns, where Strands counts + only the new messages, run it as a task and let compaction use the last + authoritative baseline + chars/4 for the delta. +- **Proof:** unit test injecting `ThrottlingException` on count asserts the model + call starts within the timeout; dev: `C#` rows show unchanged `cacheStatus` and + no compaction-timing drift over a 20-turn session. + +**P1-C Memoise the attribution split per session, and stop rebuilding the agent** +(fix 1, and half of R1) +- Short term: key the split by `(session_id, systemPromptHash, toolConfigHash)` in a + process-level dict instead of the `Agent` instance, so a rebuilt agent never + recounts. +- Real fix: extend `KEY_DESCRIBED_INJECTED_TOOL_IDS` to the spreadsheet-analysis, + Word, Excel, workspace and PowerPoint families (spec + `agent-cache-extra-tools-bypass.md` §6). Each family's builder must close over + only cache-key inputs; audit the closures first. +- **Proof:** `agent_cache outcome=hit` ratio per session in dev logs for a + spreadsheet-tool session goes from 0 to (turns−1)/turns; CountTokens call count + per turn (CloudTrail or a counter on the count client) drops to ≤1. + +**P1-D Make missing uploads-bucket CORS a synth-time error** (fix 4) +- In `validateConfig`, throw when file uploads are provisioned and + `buildCorsOrigins(config)` is empty, instead of `console.warn`. Add a jest case. +- Confirm dev/prod buckets carry the rule (`aws s3api get-bucket-cors`) — they + should, since uploads work there today. +- **Proof:** `npx jest` + a deliberate synth without `CDK_DOMAIN_NAME` fails. + +### Phase 2 — Behaviour and quota changes (next sprint) + +**P2-E Attached spreadsheets are readable on the default tool set** (fix 5) +- On a turn whose attachments include a tabular file, inject + `list_spreadsheets` + `analyze_spreadsheet` for that session when the caller's + effective RBAC grants them, mirroring `_document_tools_gate` (same function feeds + the builder and the cache key). Update the "HANDLING MISSING TOOLS" prompt clause + so the model stops sending users to the sidebar for a file they just attached. +- Consider a small-CSV inline path (e.g. ≤ 64 KB as `document_read` format `csv`) + so a tiny file never needs the sandbox at all. +- **Proof:** SPA e2e: attach a CSV with default tools, ask for a total, get a + number. `C#` row shows a single `toolConfigHash` change on the attach turn only. + +**P2-F Halve conversation-memory lookups per message; then raise the quota** (R1, R2) +- Long-term retrieval runs 3 `RetrieveMemoryRecords` on *every* user message. Move + preferences/facts retrieval to once per session (cache on the session manager, + refresh on a timer) and drop summary-namespace retrieval from `append_message` + entirely — checkpoint advance already calls `_retrieve_session_summaries` where + it is needed. Target ≤ 1 retrieval per message, ≤ 3 lookups per cold restore. +- P1-C removes the per-turn 5 `ListEvents` for the rebuilt-agent cohort. +- Only then file Service Quotas: `RetrieveMemoryRecords` 30 → 300/s (adjustable); + check `ListEvents` (200/s account, **20/s per actor-session is fixed** — a + single busy session cannot exceed it, so per-session lookups matter as much as + the account total). +- **Proof:** CloudWatch `Bedrock-AgentCore` throttle metrics by API during a + `locustfile_classroom.py` run; before/after lookups-per-message from CloudTrail. + +**P2-G Cold start** (R3) +- No code lever. Open the AWS support thread on Runtime warm capacity with our + measured turn-1 breakdown (P1-A gives the after-number). Track separately in + the `agent_status` cycle-0 timing so we can tell AWS boot from our import time. + +### Phase 3 — Reproduce the A/B on our harness and keep it + +- Run `tests/load/locustfile_classroom.py` against dev at 10 / 17 msg/s, 8-minute + holds, before and after Phase 1, with `scripts/load-test/watch-tpm.sh`. Record + new-chat first-token p95, Bedrock throttles, and success rate the way §2 of the + report does. +- Make the 17 msg/s run a release-gate check (manual, never CI — the provisioning + scripts mutate the shared user pool by design). + +--- + +## 5. Commands referenced + +Empty-messages CountTokens probe (needs a fresh `aws sso login --profile dev-ai`): + +```bash +AWS_PROFILE=dev-ai AWS_REGION=us-west-2 backend/.venv/bin/python - <<'EOF' +import boto3, botocore +c = boto3.client("bedrock-runtime", region_name="us-west-2", + config=botocore.config.Config(retries={"max_attempts": 1})) +mid = "anthropic.claude-haiku-4-5-20251001-v1:0" +sys_ = [{"text": "You are a helpful assistant."}] +for label, inp in [ + ("empty messages", {"converse": {"system": sys_, "messages": []}}), + ("probe user msg", {"converse": {"system": sys_, "messages": [{"role": "user", "content": [{"text": "hi"}]}]}}), +]: + try: + print(label, "OK", c.count_tokens(modelId=mid, input=inp)["inputTokens"]) + except Exception as e: + print(label, "ERR", type(e).__name__, str(e)[:140]) +EOF +``` + +Confirm a deployed uploads bucket has CORS: + +```bash +AWS_PROFILE=dev-ai aws s3api get-bucket-cors --bucket dev-boisestateai-v2-user-file-uploads-490617140655 +``` diff --git a/docs/specs/response-feedback.md b/docs/specs/response-feedback.md index 8768952fa..ebec4a142 100644 --- a/docs/specs/response-feedback.md +++ b/docs/specs/response-feedback.md @@ -1,7 +1,13 @@ # Response feedback -**Status:** PROPOSED — no code. Written 2026-09-04 from the "how would we -benefit?" conversation. +**Status:** PARTIALLY BUILT — capture and the read model shipped in PR #1142 +(2026-09-16, as document-context-offload PR-7) and the consequence +(retry-with-correction) in the PR stacked on it; §11 PR-1 is therefore +complete, and §11 PR-2 (implicit signals) followed for copy and continue. +Eval sampling (§11 PR-4) followed too, opt-in per environment, and §11 PR-3's +config-dimension attribution now has a fleet page. The author surfaces are not +built. See §13. Written 2026-09-04 from the "how would we benefit?" +conversation. **Refs:** `docs/specs/agentcore-evaluations-spike-findings.md` (the eval harness this feeds), `docs/specs/mid-turn-steering.md` (the injection path Phase 1 reuses), `docs/specs/agent-marketplace.md` D15 (the *other* feedback @@ -320,3 +326,117 @@ if PR-3 proves the signal is real. than copying whichever neighbour is read first. Agent Designer previews are exactly where an author would want to thumb their own work, but that data must never reach fleet aggregates. + +## 13. Built — PR #1142 (2026-09-16) + +Shipped as the outcome signal of `document-context-offload.md` §5 row 7, so +it was built against that spec's content-free rule first and reconciled with +this one after. What landed, and the decisions it took on this spec's open +points: + +- **Storage exactly per §5**: `F#{session_id}#{message_id}` on + `sessions-metadata`, `GSI_SK F#{message_id}`, idempotent upsert, overwrite + in place, never on the message object (`apis/shared/sessions/feedback.py`). + Rows carry `signal: "explicit"` from day one so §10's implicit rows can + join the family without a backfill; every thumb reader filters on it, so + the two are never summed (§10's rule, enforced). +- **Routes** `PUT`/`DELETE /sessions/{id}/messages/{message_id}/feedback` on + app-api (PUT rather than POST: the write is an upsert). Flag + `RESPONSE_FEEDBACK_ENABLED`, default on with a kill switch. +- **Reason set = §6's six buckets as codes** (`wrong`, `instructions`, + `length`, `tool_failed`, `outdated`, `other`). **Free text is not stored, + and this is a decision, not an omission**: the row sits beside the `C#` + cost row and is read by the content-free admin profile, whose test walks + the projection against the denylist. "Something else → free text" is + served by the §3 hand-off to the Agent report dialog, which already has + moderation and a scope. Revisit only together with §8 rule 1. +- **Read model**: the session profile (`GET /admin/costs/sessions/{id}/profile`) + joins thumbs to the call's document turn class with `n` per bucket + (§9's "every response carries its n"); `dataCoverage.feedback` says when + nothing is tracked. The other §7 attribution axes (model, compaction, + `agentSwitched`, skills) are further buckets in the same join loop. +- **Open question 4 settled**: preview sessions echo the thumb and persist + nothing, matching the `D#` write. Preview data never reaches aggregates. +- **Retry-with-correction (the consequence), second PR.** Open question 1 + settled: **a new turn, not `/steer`** — the steer path targets a *running* + turn through the lease row, and a finished turn is corrected by an + ordinary next message. The down-thumb reason row offers *Retry with that + in mind*, which prefills the composer (`ComposerDraftService`) with a + correction template for the reason code; the user edits and sends it as a + normal message, so it goes where messages go (AgentCore Memory) and never + touches the metadata table. When that message is added, the send path + links it to the thumb as `retryMessageId` on the `F#` row — an index, + never the text — and a later re-thumb keeps the link. This is open + question 2's answer for now: the pair is *recorded* without its content, + so Phase 6 can find it once the consent decision is made. The profile + reports `feedback.retried` and `reworkUsd` (§7 "rework cost": the thumbed + call rows plus the retry turn's consecutive assistant rows). +- **Implicit signals (§11 PR-2), third PR — copy and continue only.** + Same `F#` family under their own key, `F#{session}#{message}#{kind}`, so + they never collide with the thumb; `signal: "implicit"`, `kind`, and an + `ADD`ed `count`. `POST /sessions/{id}/messages/{message_id}/signals`, + fire-and-forget from the SPA (the Copy and Continue clicks in the actions + rail, once per message per kind per page load). The profile reports + `feedback.implicit` as *messages touched* per kind, on its own line — the + §10 rule that explicit and implicit are never summed is enforced in every + reader. Two of §10's four signals are deliberately not here: + **edit-and-resend** has no affordance in the SPA to hook, and + **abandonment** is deferred — a session that goes quiet after a good + answer is indistinguishable from one that goes quiet after a bad one, so + it needs its own design (or the §10 "dissatisfaction in the next message" + offline classifier) before it is worth a row. +- **Eval sampling (§11 PR-4), fourth PR — opt-in.** Down-thumbs carry + `GSI1PK = FEEDBACK#down` / `GSI1SK = updatedAt` on the existing + `UserTimestampIndex`, so the fleet's recent down-thumbs are one query with + no new index. `POST /admin/feedback/evaluations/run` (scope `admin.costs`) + judges up to N not-yet-judged ones in a background task through + `bedrock_agentcore.evaluation.EvaluationClient` over the runtime log + group, keyed by the runtime session id the chat proxy already pins; + `GET /admin/feedback/evaluations` is the queue with verdicts. Routing is + §6's table: `wrong` → Correctness + Faithfulness, `instructions` → + InstructionFollowing, `length` → Conciseness, `other`/none → Helpfulness; + `tool_failed` gets **no judge** and is corroborated against the call's + tool census on the `C#` row (ops, not model); `outdated` gets no judge yet + (the KB-freshness join is a follow-up). The verdict stored on the `F#` row + is per-evaluator value / rating / n / tokens — **the judge's `explanation` + is dropped at summarisation, refused at the storage write, and denylisted + in the content policy**, because it quotes the conversation. The profile + shows `feedback.evaluations` (judged, mean per evaluator, tool failures + corroborated). **Default OFF** (`FEEDBACK_EVAL_SAMPLING_ENABLED`, + `CDK_FEEDBACK_EVAL_SAMPLING_ENABLED=true` to enable): the managed judge + reads the sampled conversation's spans, which is the scoping decision the + evaluations spike (§2) says to make explicitly per environment, and §8 + rule 1 here. The IAM grant (Logs Insights on the runtime log group and + `aws/spans`; Evaluate / GetEvaluator) is wired but inert until an + environment opts in. Runs are admin-triggered; a schedule can follow once + a week of verdicts says the token spend is worth it. +- **Fleet attribution (§11 PR-3, §7 "config-dimension attribution"), fifth + PR.** `GET /admin/feedback/fleet?days=` aggregates a trailing window of + thumbs into arms on four dimensions: **model**, **agent switch**, + **distance from a compaction cut** (the compaction spec's §7.2 question, + bucketed in model calls: never / same call / 1-3 / 4+), and **document + turn class**. The window is a range read on the `FEEDBACK#down` / + `FEEDBACK#up` partitions, then one cost-row query per session to join; + both are capped (2,000 thumbs, 300 sessions) and exceeding either is + reported as `coverage.truncated` / `sessionsOmitted` rather than silently + under-counted. `/admin/feedback` renders it, under the `admin.costs` scope. + §9 is enforced rather than promised: **every arm carries its own `n`**, an + arm under the floor returns `downRate: null` so there is no number to + quote, and **no fleet-wide rate field exists** at any level — the page + states the gap between the two furthest-apart reportable arms instead of a + score. Open question 3 is answered: the floor is **20**, picked because + below it one extra thumb swings the rate by more than five points; it is + env-tunable with `FEEDBACK_ARM_MINIMUM_N`. + Two deviations from §11 PR-3 as written. It is **a page of its own, not a + panel beside the cost panels** — the cost drill-down is per conversation + and this question is fleet-shaped, so co-locating them would have meant a + panel that ignored the session in whose page it sat. And the **skills axis + is not built**: `enabledTools` lives on the session row, not the `C#` row, + so a per-call skills arm needs either a denormalisation at write time or a + second join, and neither is worth doing before the signal is proven. +- **Not built**: author/marketplace surfaces (PR-5), the report-dialog + escape hatch in the reason row, abandonment, the `outdated` → KB-freshness + join, a schedule for the sampler, the skills arm, and implicit signals in + the fleet view (their rows carry no `GSI1*` keys, so they are invisible to + the window query by construction — wiring them in is a follow-up to PR-2). + diff --git a/docs/specs/turn-latency-preamble.md b/docs/specs/turn-latency-preamble.md new file mode 100644 index 000000000..aac0de11e --- /dev/null +++ b/docs/specs/turn-latency-preamble.md @@ -0,0 +1,828 @@ +# Turn latency: inside the preamble + +**Status:** COMPLETE for the warm path. Five PRs shipped, merged, deployed and +validated on dev (#1184, #1191, #1193, #1198, #1201): + +| | Baseline | Now | +|---|---:|---:| +| Warm preamble | 455ms | **22-37ms** (-95%) | +| Warm pre-stream window | 591-656ms | **128-189ms** (-75%) | +| Cold `agent_build` | ~2950ms, opaque | 3286ms, **fully attributed** | + +The cold path is untouched by design and is where everything left lives. The +decomposition in PR-4 found its owner, and it is not what this spec assumed — +see **PR-5** for the one thing worth doing next. +**Follow-up to:** `docs/specs/agent-state-feedback.md` — its PR-3 measured the +pre-stream window into four stages and then declined to narrate three of them. +This spec opens the one that was left closed. +**Related:** CLAUDE.md § Token cost effectiveness (the "don't guess — verify" +tenet this spec is an application of), `agent-cache-extra-tools-bypass.md` §8 +(runtime session affinity, the other latency-not-cost fix). + +## Problem + +`agent-state-feedback.md` PR-3 instrumented the window between the user's click +and the first SSE byte, and recorded four turns on dev: + +| Turn | total | preamble | rag | tools | agent_build | +|------|------:|---------:|----:|------:|------------:| +| cold agent cache | 2542 | 802 | 0 | 261 | **1478** | +| warm | 641 | 494 | 0 | 146 | 0 | +| warm | 644 | 460 | 0 | 149 | 34 | +| warm | 678 | 484 | 0 | 154 | 38 | + +`agent_build` got a fix (PR-3 deferred it into the stream generator and narrates +it). `rag` and `tools` are small. **`preamble` was never opened.** On a warm +turn it is the single largest stage — 460–494ms of the 641–678ms handler total, +and the spec's own summary put its range at 450–900ms across the sample. + +That is now the largest avoidable wait in the product, and unlike `agent_build` +it is not a status problem: no label makes it shorter, and the loading indicator +already covers the window. It is a latency problem. + +## What `preamble` covers + +One `prelude.mark()` spanning ~500 lines of `inference_api/chat/routes.py`, from +handler entry to the quota round trip. The mark's own comment — "request +validation, model/settings resolution, file handling and the quota round trip" — +undersells it. Read from the code, a plain warm turn (non-resume, +non-continuation, no attachments) performs this sequence, entirely serially: + +| # | Call | DynamoDB work | +|---|------|---------------| +| 1 | `session_owned_by_other_user` | `SessionLookupIndex` Query | +| 2 | `_resolve_accessible_skill_ids` | RBAC (cached) + `SkillOwnerIndex` Query — only when `enabled_skills` is non-empty | +| 3 | `pop_pending_attachments` | `_get_session_by_gsi` | +| 4 | `ensure_session_metadata_exists` | `_get_session_by_gsi` | +| 5 | `clear_paused_turn` | `_get_session_by_gsi` | +| 6 | `clear_pending_interrupts` | `_get_session_by_gsi` | +| 7 | `clear_truncated_turn` | `_get_session_by_gsi` | +| 8 | `clear_interrupted_turn` | `_get_session_by_gsi` | +| 9 | `check_quota` → `_resolve_session_notice` → `get_session_metadata` | GSI Query (+ a lazy cost-aggregate backfill on legacy rows) | + +**Rows 1, 3, 4, 5, 6, 7, 8 and 9 all read the same item** — the session's `META` +row — eight times in one turn, each on its own round trip. Rows 5–8 then +short-circuit on the happy path (`if "pausedTurn" not in existing: return`), so +four of the eight reads exist only to discover there is nothing to clear. + +Three things multiply that. + +**They are synchronous boto3 calls inside `async def`.** `_get_session_by_gsi` +calls `table.query(...)` directly, with no `asyncio.to_thread`. Each one blocks +the runtime's event loop for its whole round trip. This is the same shape as the +starved-timer bug PR-3 already paid for — a synchronous `create_agent` occupying +the loop so `asyncio.wait` never fired its timeout. Here the consequence is +throughput as well as latency: the runtime-session affinity header deliberately +packs a conversation onto a warm microVM, so these blocking hops serialize +against *other users' turns on the same container*. + +**A fresh client per call.** `metadata.py` constructs `boto3.resource("dynamodb")` +in 28 separate function bodies; the repo has 89 such sites across `apis/` and +`agents/`, and no cached-client helper anywhere in `apis/shared/`. Measured +locally: ~327ms for the first construction in a process, **~1.5ms** each +thereafter. So this is ~12ms per turn — real, but not the story. Worth fixing +for the cold-container case and for tidiness, not for the warm number. + +**No VPC endpoints.** `network-construct.ts` provisions one NAT Gateway and zero +interface or gateway endpoints. This does *not* affect the preamble — +inference-api runs inside an AgentCore Runtime container on AWS-managed network, +not our VPC — but every app-api DynamoDB and S3 call egresses through NAT. A +DynamoDB gateway endpoint is free and reduces both latency and NAT +data-processing spend. Tracked here because it was found on the same sweep; it +belongs to app-api's latency, not the preamble's. + +### What is already fine + +Named so nobody re-audits them: RBAC permission resolution, the quota tier +resolver, and the user cost summary are all TTL-cached in-process. Feature flags +are pure `os.environ` reads with no IO. Skill resolution short-circuits to `[]` +when the turn selects no skills, so it costs nothing on a plain turn. Title +generation is already an `asyncio.create_task` off the critical path. + +## The honest status of the attribution + +**The eight-read count is read off the code. The attribution of 450–900ms to it +is a hypothesis.** The mechanism is strong and the arithmetic is plausible — 8 +serialized GSI queries at an in-region p50 of 10–15ms, p90 of 25–40ms, plus +event-loop blocking under concurrency — but `preamble` has no sub-marks, so +nothing here is measured. + +This is exactly the trap CLAUDE.md's cost-effectiveness tenet names for prompt +tokens, and it applies identically to milliseconds: *don't guess — verify*. The +repo has been burned by the inverse before (`agent_status`'s "and N more" was +dead code for a pinned sequential executor; PR-3's first two suppression attempts +each looked right and each failed on dev for a reason the unit tests could not +see). A refactor justified by an unmeasured hypothesis is how that happens again. + +So PR-1 measures, and nothing else. + +## PR-1 — sub-stage marks (BUILT) + +Decompose the single `preamble` mark into five ordered sub-stages, named with a +dotted prefix: + +| Stage | Covers | +|-------|--------| +| `preamble.ownership` | the cross-user session guard (read #1) | +| `preamble.skills` | effective-skill resolution (read #2, absent on a turn with no skills) | +| `preamble.files` | attachment recovery + upload-id resolution + inline partitioning (read #3, plus S3) | +| `preamble.session_state` | metadata pre-create and the four stale-marker clears (reads #4–#8) | +| `preamble.quota` | the quota round trip (read #9) | + +**`preamble` itself survives as a number.** `emit()` gains a `groups` field that +sums stages sharing a dotted prefix, so `groups.preamble` reproduces exactly what +the old single mark reported and the four-turn baseline above stays comparable. +Without that, decomposing the stage would have silently discarded the only +measurement anyone has. + +**Cost.** Four extra `perf_counter()` calls and one extra log field per turn. +Nothing reaches the model, the conversation, or the cacheable prefix — the same +standing `turn_timing.py` has had since it shipped. + +**Not gated behind a flag.** `TurnPrelude` is already best-effort in every +direction (`mark` swallows, `emit` swallows) and a flag would be a second thing +to get wrong for a measurement that cannot break a turn. The precedent is the +existing four marks, which shipped ungated for the same reason. + +**How to read it.** Unchanged from `turn_timing.py`'s own docstring — one +structured line per turn on the inference-api runtime log group, via +`filter-log-events --filter-pattern turn_prelude`. What PR-1 changes is that +`stages` now names *which* sub-stage owns the time instead of reporting one +opaque number. + +### What would falsify the hypothesis + +Stated in advance, so the measurement cannot be read to agree with whatever we +already believe: + +- **Hypothesis confirmed** if `preamble.session_state` dominates — it holds five + of the eight reads and does no other work. +- **Hypothesis wrong, and PR-2 must not be built as written**, if + `preamble.quota` or `preamble.files` dominates instead. Those are different + bugs with different fixes: quota is one cached GetItem plus a session read, and + files is S3 plus extraction, neither of which the single-read refactor touches. +- **Something unmodelled is going on** if the five sub-stages sum to visibly less + than `groups.preamble`. The stages are contiguous by construction, so a gap + means time is being spent between them — in code this spec has not accounted + for. + +## PR-1b — make the improvement provable (BUILT) + +PR-1 makes the preamble *attributable*. It does not make a fix *provable*, and +those are different problems: attribution needs one turn, proof needs a +population. + +### The field that would lie to us + +There is already a persisted field called `time_to_first_token` +(`apis/app_api/messages/models.py`, computed in `stream_coordinator.py`). It is +the obvious instrument and it is the wrong one. It measures +`first_token_time - stream_start_time`, and `stream_start_time` is set *inside* +the coordinator's generator — which, since `agent-state-feedback.md` PR-3 +deferred the build, runs after the preamble **and** after the agent build. So it +is structurally blind to: + +| Excluded from `time_to_first_token` | Size | +|---|---| +| app-api hop + auth + Runtime routing | ~478ms warm, ~1.5s cold | +| the whole preamble | 450–900ms | +| the agent build | 0–2728ms | + +**A successful PR-2 moves that field by exactly zero.** Anyone validating with +it concludes the work was pointless. This is the same trap this repo already +documented once for `latency.endToEndLatency` vs `turnDurationMs` — a field +whose name describes what you want and whose definition does not. Do not use it +here; it is a *model* TTFT, and it is correct at that job. + +### Server side: percentiles, not grepped lines + +`TurnPrelude.emit` now also writes one EMF record per turn into +`AgentCoreStack/TurnLatency`, through the existing +`apis/shared/observability/emf.py` helper — no new infrastructure, no new IAM. +Metric names are *derived* from the stage names (`preamble.session_state` → +`PreambleSessionStateMs`) rather than listed, so a new mark cannot silently go +unmeasured; the price is that a new mark quietly creates a metric stream +(~$0.30/month), which is the cheaper mistake. + +Dimension-less, like every other EMF caller here. `isResume`, `deferredBuild` +and `sessionId` ride as queryable log properties and the dashboard's Logs +Insights widgets do the slicing. That is not only metric-stream cost: a +dimension invites reading a p99 off a slice too thin to have one. + +`TurnLatencyObservabilityConstruct` graphs p50/p90/p99 per stage on **its own +dashboard** — the fourth, which costs $3/month. + +That cost was argued, not absorbed. `observability-platform-dashboard.test.ts` +pins the dashboard count precisely so crossing CloudWatch's free three stays a +deliberate trade; the first attempt therefore folded the widgets onto the +AgentCore Runtime board to stay inside it. That version worked and was still +worse: this board is read *while shipping a latency change*, side by side with +load-test output, and burying the stage breakdown under runtime-health widgets +answering an unrelated question made it harder to use for its one job. +$3/month against a 450–900ms wait on every turn is not a close call. **The next +dashboard should have to make the same argument** — the pinned count now reads +"three free + one bought" rather than being raised to whatever is convenient. + +The AgentCore Runtime board stays the natural companion: it graphs AWS's own +`Latency` p50/p90/p99 measured at the **data plane**, so the gap between it and +`PreludeTotalMs` is another read on the routing overhead no server-side stage +can see. Both dashboards' headers point at each other, and the platform health +dashboard links to all three drill-downs. + +Alongside the percentile graphs are two widgets that exist to catch our own +errors: a split by turn shape (a resume skips most of the preamble, so a +traffic-mix shift toward resumes would look exactly like a latency win) and an +**unaccounted-time** widget +(`PreludeTotalMs − (PreambleMs + RagMs + ToolsMs + AgentBuildMs)`), which is how +we find out the decomposition is incomplete. + +**No alarms.** A threshold needs a baseline and there is none yet. Inventing one +is the guessing this spec exists to prevent; add them once the dashboard has run +long enough to say what normal is. + +**Kill switch:** `TURN_LATENCY_METRICS_ENABLED` (default on). Note this differs +from PR-1's marks, which are deliberately ungated — a `perf_counter()` call +costs nothing, while an EMF line costs log ingestion and custom-metric charges. +Same reasoning that gave `PROMPT_CACHE_OBSERVABILITY_ENABLED` its switch. Like +every other flag of its kind here it is not threaded through CDK. + +### Client side: the part the server structurally cannot see + +No metric in `AgentCoreStack/TurnLatency` sees the app-api hop, auth or Runtime +routing, because they all start at handler entry. That is up to 20% of the turn, +and it is where a fix can be real on the server and invisible to the user. + +`tests/load` already covers it. The Locust harness measures **true client-side +TTFT** — dispatch of `POST /chat/stream` → first `content_block_delta` — through +CloudFront → ALB → app-api → AgentCore Runtime (`agentcore_load/users.py`). + +And the architecture hands us a clean decomposition for free. Because PR-3 +deferred the agent build into the stream generator, inference-api returns its +`StreamingResponse` **after the preamble but before the build**, so response +headers arrive at exactly the boundary between the two fixes: + +| Measure | Covers | Which fix it judges | +|---|---|---| +| **TTFB** (Locust's built-in `POST /chat/stream` time — headers) | fixed hop + **preamble** | PR-2 | +| **TTFT − TTFB** | **agent build** + model | the MCP pre-flight fix | + +**Verify this before trusting it.** It depends on headers flushing through the +AgentCore data plane ahead of the first body byte. The proxy's own half has a +regression test (`test_ttfb_under_200ms_with_x_accel_buffering`) but that test +mocks the upstream, so it proves the *proxy* does not buffer — not the data +plane. The check: run turns and compare Locust TTFB against `groups.preamble` +from the same turns. If TTFB tracks preamble, the channel is clean. If TTFB +tracks TTFT instead, the data plane is buffering and TTFB is useless as a +preamble proxy — fall back to comparing the EMF percentiles and accept that the +client-side half is only measurable end to end. + +### Design the comparison as A/B, not before/after + +Before/after across a deploy confounds with the two largest variance terms on +this path — container warmth and traffic mix — either of which can swamp the +effect being measured. The template already exists: +`scripts/probe_runtime_session_affinity.py` is a two-arm probe differing by +exactly one header, and its docstring makes the point that it deliberately +bypasses shared code rather than "building the fix to test the hypothesis." + +### Acceptance criteria, stated before the data + +| Claim | Passes if | Fails if | +|---|---|---| +| The eight reads are real cost | `PreambleSessionStateMs` p50 ≥ 100ms | < 40ms — the reads are cheap; drop PR-2 | +| PR-2 removes them | `PreambleMs` p50 falls ≥ 50ms | < 20ms — the reads were not the cost | +| It reaches the user | Locust TTFB p50 falls by the same amount | TTFB flat while `PreambleMs` falls — something downstream absorbs it | +| The decomposition is complete | unaccounted-time widget ≈ 0 | a visible gap — time is going somewhere unmodelled | + +### The arithmetic that does not close + +Stated plainly because it is the strongest argument against this spec's own +hypothesis. Eight serialized GSI queries at an in-region p50 of 10–15ms is +~100–150ms. The preamble is 460–494ms on a warm turn. **So even if the read +count is entirely right, PR-2 recovers roughly a quarter of the stage** — and +~300ms is unexplained by anything here. + +Two readings, and PR-1's sub-marks separate them: either DynamoDB latency from +an AgentCore Runtime container is much worse than in-region typical (in which +case `session_state` is larger than modelled and PR-2 is worth more), or a large +part of the preamble is something not yet identified (in which case PR-2 is a +rounding error and the real win is elsewhere). Either way the answer arrives +before the refactor is written, which is the entire point of sequencing it this +way. + +## THE MEASUREMENT (dev, 2026-09-19) — hypothesis confirmed, magnitude wrong + +Three turns on dev immediately after deploy: one cold, two warm. All times ms. + +| Turn | total | own | skills | files | **sess_state** | quota | **=preamble** | rag | tools | build | unacct | +|------|------:|----:|-------:|------:|---------------:|------:|--------------:|----:|------:|------:|-------:| +| cold | 4110 | 53 | 171 | 56 | **357** | 265 | **902** | 0 | 252 | 2951 | 5 | +| warm | 656 | 55 | 17 | 53 | **272** | 62 | **459** | 0 | 147 | 47 | 3 | +| warm | 591 | 53 | 5 | 53 | **278** | 62 | **451** | 0 | 136 | 1 | 3 | + +Every acceptance criterion stated in advance passes: + +| Claim | Threshold | Measured | Verdict | +|---|---|---|---| +| The eight reads are real cost | `session_state` p50 >= 100ms | **272-278ms** | **CONFIRMED** | +| The decomposition is complete | sub-stages ~= group | unaccounted **3ms** | **CONFIRMED** | +| The split preserved the baseline | ~= the old flat 494ms | **451-459ms** | **CONFIRMED** | + +`session_state` is ~60% of the warm preamble, exactly as predicted. + +### What was wrong: the per-read cost, by 4-5x + +This spec estimated an in-region GSI query at 10-15ms and concluded PR-2 would +"recover roughly a quarter of the stage". **That was wrong, conservatively.** + +`preamble.ownership` is exactly one GSI query and nothing else. It measures +**53-55ms**, dead stable across all three turns. `preamble.files` on a turn with +*no attachments* is also exactly one read (`pop_pending_attachments`) and also +measures **53ms**. So a DynamoDB GSI query from inside an AgentCore Runtime +container costs **~53ms**, not 10-15ms — 4-5x the in-region figure assumed here. + +That single number closes the arithmetic that previously did not: + +| Sub-stage | Reads of the META row | Measured | Implied per read | +|---|---:|---:|---:| +| `ownership` | 1 | 53 | 53 | +| `files` | 1 | 53 | 53 | +| `session_state` | 5 | 272-278 | ~55 | +| `quota` (session notice) | 1 | 62 | 62 | +| **total** | **8** | **~445 of the 451-459ms preamble** | | + +**The warm preamble is almost entirely DynamoDB round trips reading one item +eight times.** The "~300ms unexplained" this spec worried about does not exist; +it was the per-read cost being four times higher than assumed. The spec offered +two readings and said the sub-marks would decide between them — the first +("DynamoDB latency from an AgentCore Runtime container is much worse than +in-region typical, in which case `session_state` is larger than modelled and +PR-2 is worth more") is correct. + +### What this does to PR-2 + +Originally scoped as "collapse reads #3-#8". The measurement says go further: +**one read at the top of the handler can answer all four consumers**, because +`ownership`, `files`, `session_state` and the quota session-notice all read the +same item. + +| | Now | After | Saved | +|---|---:|---:|---:| +| `ownership` | 53 | ~53 (the one read) | 0 | +| `files` | 53 | ~0 | 53 | +| `session_state` | 275 | ~0 | 275 | +| `quota` notice | 62 | ~0 | 62 | +| `skills` | ~10 | ~10 | 0 | +| **preamble** | **~455** | **~65** | **~390** | + +A ~455ms stage becomes ~65ms, against an original estimate of 50-90ms saved. +**Still to be proven, not assumed** — the rule that produced this table applies +to the next one just as much. + +### Caveats on this measurement + +- **n=3, one session, one container.** Enough to size a 275ms effect against a + +/-5ms spread; not enough for a distribution. The EMF percentiles are what + make the post-PR-2 comparison sound, and they now have a baseline accruing. +- `skills` is 171ms cold vs 5-17ms warm — the RBAC/catalog caches working. Not + a PR-2 target. +- `agent_build` 2951ms cold, then 47ms and 1ms warm. The cache works; the cold + build remains the largest single number in the table and is the MCP pre-flight + item below, untouched by PR-2. +- `StreamSetupMs` appeared as a metric nobody listed, because metric names are + derived from marks rather than hand-kept. A listed table would have omitted it + silently. + +## PR-2 — read the session row once (SHIPPED #1191, VALIDATED on dev) + +**Result: the warm preamble fell 455ms -> 163ms (-64%), and the whole +pre-stream window 591-656ms -> 349-350ms (-46%).** Three turns on dev +2026-09-19, immediately after the image landed: + +| Sub-stage | Before | Predicted | **Measured** | +|---|---:|---:|---:| +| `ownership` | 53-55 | ~53 | **59-60** | +| `skills` | 5-17 | ~10 | **17** | +| `files` | 53 | ~0 | **4-5** | +| `session_state` | 272-278 | ~0 | **17-21** | +| `quota` | 62 | 62 (PR-2b) | **61-65** | +| **`groups.preamble`** | **451-459** | **~125** | **162-164** | +| **`totalMs`** | **591-656** | — | **349-350** | + +Regression checks, all passing: turns stream normally; a **brand-new session** +creates its row and generates its title, which is the `is_new_session=True` +path and therefore direct evidence that a snapshot with `row=None` is read as +"no row" rather than "nothing prefetched"; the conversation list resolves. + +### Where the remaining 163ms is, and the 38ms the prediction missed + +Predicted ~125ms, measured ~163ms. The gap is accounted for, not mysterious: + +- `session_state` is 17-21ms rather than ~0. Every helper still constructs + `boto3.resource("dynamodb")` + `.Table()` before reaching its snapshot + short-circuit — six of them, at the **~1.5ms per construction measured + independently earlier in this spec**. That is ~9ms, plus dict work. The + earlier client-construction measurement reproduces here, which is a small + confirmation that both numbers are real. +- `ownership` drifted 53-55 -> 59-60ms. Same single query, ordinary variance. + +So the warm preamble now decomposes as: one GSI read (~59) + the quota read +(~62) + skills (~17) + ~20ms of repeated boto3 client construction + change. + +**The floor is ~80ms**, reachable by PR-2b (removes the quota read) and a +cached-client helper (removes the ~20ms). The single remaining `ownership` +read is irreducible — it is the read. + +### A note on reading the dashboard across a deploy + +The 15-minute CloudWatch window covering this deploy reports +`PreambleMs` min 162 / avg 457 / max 934 — because it spans two populations, +the old code and the new. The minimum is the new code; the average is +meaningless here. This is exactly why the widgets are percentiles over a +period rather than a single number, and it is worth remembering before reading +the first window after any future deploy. + + + +**As built.** `load_session_meta()` performs one `SessionLookupIndex` query and +returns a `SessionMetaSnapshot` carrying two answers: the caller's own `META` +row, and whether the session belongs to someone else. Both were always in that +one response — `_get_session_by_gsi` simply discarded the half the ownership +probe needed, which is why the probe was a second query of the same index for +the same key. + +The route reads once and threads the snapshot into `pop_pending_attachments`, +`ensure_session_metadata_exists` and the four `clear_*` helpers. Each takes +`snapshot: Optional[SessionMetaSnapshot] = None`; `None` keeps the original +per-call read, which is what every non-preamble caller still gets. + +**Why the snapshot object and not the row dict.** `row is None` is a real +answer — "this user has no `META` row yet" — not "nothing was prefetched". Had +the helpers taken the row alone, a brand-new session would be indistinguishable +from an absent prefetch and would silently fall back to re-reading, so the first +turn of every conversation would keep the old cost. Pinned by +`test_a_snapshot_with_no_row_is_an_answer_not_a_cache_miss`. + +**Why explicit and not a per-request memo.** Memoising inside +`_get_session_by_gsi` would be a smaller diff and the wrong shape. CLAUDE.md's +"never cache session state; re-read per turn; never move backwards" rule exists +because this repo shipped that bug twice (#741 conversation history, #751 +compaction state). A snapshot a caller opts into cannot leak into a caller that +needs a fresh read; a request-scoped memo can, and the failure is silent. + +**Scope.** `ownership`, `files` and `session_state` — the three sub-stages that +read this item. The quota session-notice read is deliberately NOT included; see +PR-2b. + +Expected effect on the warm preamble, from the measured per-read cost: + +| Sub-stage | Before | After | +|---|---:|---:| +| `ownership` | 53 | ~53 (now the only read) | +| `files` | 53 | ~0 | +| `session_state` | 275 | ~0 | +| `quota` (PR-2b) | 62 | 62 | +| `skills` | ~10 | ~10 | +| **preamble** | **~455** | **~125** | + +Why it is safe: every one of those helpers is already best-effort and fail-open, +and the GSI is eventually consistent, so re-reading was never buying consistency +— two reads 50ms apart see the same stale-or-not row. The session's single-flight +lease already excludes a concurrent turn on the same session. The `SK` is static +(`S#{session_id}`), so the key the conditional writes target cannot drift between +the read and the write. And the atomic parts stay atomic: `pop_pending_attachments` +and `clear_interrupted_turn` still do their `ReturnValues=UPDATED_OLD` update, so +the snapshot only replaces the *gate* read, never the write. + +The ownership guard keeps its 404, now answered from the same snapshot instead +of its own query. + +**That gate is now passed.** `session_state` is where the time is (272-278ms of +a 451-459ms preamble), so the refactor of the marker helpers — load-bearing for +interrupt resume, the truncation "Continue" path and unconsumed-attachment +recovery — is paid for by a measured 275ms rather than by a guess. Scope widens +to include `ownership`, `files` and the quota session-notice, which read the +same item. + +## PR-2b — the quota session-notice read (SHIPPED #1193, VALIDATED on dev) + +**`preamble.quota` 61-65ms -> 4ms. Warm preamble 163ms -> 95ms; the whole +pre-stream window 349ms -> 263ms.** Measured on dev 2026-09-19: + +| Sub-stage | Baseline | After PR-2 | **After PR-2b** | +|---|---:|---:|---:| +| `ownership` | 53-55 | 59 | **54** | +| `skills` | 5-17 | 17 | **15** | +| `files` | 53 | 4 | **4** | +| `session_state` | 272-278 | 17-21 | **18** | +| `quota` | 62 | 61-65 | **4** | +| **`groups.preamble`** | **451-459** | **163** | **95** | +| **`totalMs`** | **591-656** | **349** | **263** | + +**The warm preamble is 79% below where this spec started**, and the whole +pre-stream window is down 57%. + +Regression: new session created, title generated, turns stream, recap footer +renders (6.8s then 2.2s). + +The COLD turn still shows `quota` at 237ms, and that is not a miss. On a fresh +container the quota **tier resolver** and the **user cost summary** are both +uncached, so their own reads dominate; only the session-notice read was ever in +scope here. On the warm path, where those caches are hot, it is the whole of +what remains. + + + +The eighth read, worth a measured **62ms**. Left out of PR-2 because it is not +the same shape as the others. + +`QuotaChecker._resolve_session_notice` calls `get_session_metadata`, which +returns a `SessionMetadata` model and performs a **lazy backfill** of the +cost aggregates when `totalCost` is missing from a legacy row. Handing it the +raw snapshot row would skip that backfill, and a legacy row without `totalCost` +would silently stop producing a session notice — turning a latency fix into a +quiet regression of the feature that exists to catch a conversation eating a +user's month. + +**As built.** `check_quota` and `_resolve_session_notice` take +`session_total_cost: Optional[float] = None`. The route supplies it **only when +the snapshot row actually carries `totalCost`**; `None` routes the checker back +through `get_session_metadata`, backfill and all. + +`None` means "not known" and never "zero" — the distinction is the whole safety +of this path, and it is pinned two ways: +`test_none_falls_back_to_the_read_and_its_lazy_backfill` and +`test_a_supplied_zero_is_honoured_as_zero_not_as_unknown`. The second exists +because a genuinely free session reports `0.0`, which is falsy: a truthiness +check would send it back to the very read this PR removes, and the bug would be +invisible — a correct answer, arrived at expensively. + +app-api's converse route calls `check_quota` without the kwarg and keeps its +read, pinned by `test_default_call_is_unchanged_for_callers_that_pass_nothing`. + +Expected: `preamble.quota` 62ms -> ~0, warm preamble ~163ms -> ~100ms. + +## PR-3 — cache the boto3 clients (SHIPPED #1198, VALIDATED — and it corrects this spec) + +**Warm preamble 95ms -> 22-37ms. Whole pre-stream window 263ms -> 128-189ms.** +Cumulatively the preamble is **down 95%** from the 455ms this spec opened with, +and the pre-stream window is down ~75%. + +| Stage | Baseline | PR-2 | PR-2b | **PR-3** | +|---|---:|---:|---:|---:| +| `ownership` | 53-55 | 59 | 54 | **6-7** | +| `skills` | 5-17 | 17 | 15 | 6-21 | +| `files` | 53 | 4 | 4 | **1** | +| `session_state` | 272-278 | 17-21 | 18 | **3-4** | +| `quota` | 62 | 61-65 | 4 | 5 | +| **`groups.preamble`** | **451-459** | **163** | **95** | **22-37** | +| **`totalMs`** | **591-656** | **349** | **263** | **128-189** | + +### THE CORRECTION: the 53ms was never the network + +This spec asserted, in bold, after PR-1: + +> a DynamoDB GSI query from inside an AgentCore Runtime container costs **~53ms**, +> not 10-15ms — 4-5x the in-region figure this spec assumed. + +**That was wrong.** `preamble.ownership` is exactly one GSI query and nothing +else. It measured 53-55ms across every run. With a cached client it measures +**6-7ms**. So the 53ms decomposes as: + +| | | +|---|---:| +| boto3 resource construction (throttled container CPU) | **~47ms** | +| the actual DynamoDB round trip | **~6ms** — entirely normal in-region | + +The local measurement recorded at the top of this spec — ~1.5ms per +construction — is a laptop number. On the container it is ~47ms, **thirty times +higher**, and this spec anchored on the laptop figure and then found a story +that fit it. The container numbers appeared to corroborate "DynamoDB is slow +here", and nothing challenged that until a fix aimed at something else moved +the wrong metric. + +**So PR-2 was the right fix for the wrong stated reason.** Collapsing eight +reads into one removed eight *client constructions*, not eight slow network +calls. The villain was always client construction — which is why PR-3, scoped +here as a modest ~15ms cleanup, produced a larger proportional win than the PR +written specifically to fix the problem. + +The reusable lesson is not about boto3. It is that **a number measured on a +laptop is not a measurement of production**, and that a hypothesis which keeps +fitting the data can still be wrong about mechanism while being right about +remedy. The sub-stage marks are what made the error visible at all: a single +`preamble` number would have shown the same total improvement and hidden the +fact that the explanation was wrong. + +### The new largest warm stage + +`tools` is now **98-102ms warm** (196-244ms cold) — larger than the entire +preamble. It covers system-prompt assembly, the single-flight lease, skill +resolution and every tool builder, and it has never been decomposed. On the +evidence above, the right move is to measure it before proposing a fix. + + + +PR-2 made this measurable. With the preamble's eight reads collapsed to one, +`preamble.session_state` still measured **17-21ms on dev while doing no IO at +all** — six helpers each constructing `boto3.resource("dynamodb")` before +reaching their snapshot short-circuit, at the ~1.5ms per construction measured +at the top of this spec. That residual is now ~25% of the remaining preamble. + +`apis/shared/aws_clients.py` caches resources and clients per +`(service, region)` and exposes `get_dynamodb_table()`. All **29** construction +sites in `sessions/metadata.py` are converted and 23 now-dead `import boto3` +lines removed. + +**Seven of those sites used single quotes** (`boto3.resource('dynamodb')`) and +the first pass silently missed them. Worth recording: a partial conversion +would have left the residual half-present and read on the dashboard as "the +cache underperforms" rather than "the cache was not applied", which is a much +harder thing to notice. + +### The moto trap + +`moto.mock_aws()` is entered **per test**. A client cached under one test's +mock keeps pointing at a backend that is torn down when that test ends, so the +next test silently talks to a dead backend — or to a live AWS endpoint. The +failure is order-dependent, which is the same shape as the static-memo leak +this repo already paid for across SPA spec files. + +So the cache is explicitly resettable, and all three `aws` fixtures +(`tests/shared`, `tests/lambdas`, `tests/fine_tuning`) reset it on entry **and** +on exit. `test_the_aws_fixture_leaves_no_client_behind` pins the teardown half, +because an entry-only reset would still let the last test in a file leak into a +file that never uses the fixture. + +Expected: `session_state` 17-21ms -> ~2-5ms, with smaller shavings on +`ownership` and `quota`. Combined with PR-2b the warm preamble should reach +**~80ms, from the 455ms this spec started at**. + +## PR-4 — decompose the agent build (SHIPPED #1201, VALIDATED on dev) + +With the warm preamble at 22-37ms, `agent_build` became the largest number in +the prelude by an order of magnitude: **~3300ms cold** against 1-49ms warm, and +nothing said which part of it that was. Seven sub-stages — `prompt`, +`registry`, `session_mgr`, `tools`, `hooks`, `plugins`, `finalize` — plus +`agent_build.rest`, grouped by the same mechanism PR-1 built so +`groups.agent_build` reproduces the pre-split series. + +### The answer, measured on a cold turn (2026-09-19) + +`agent_build` = 3286ms of a 4131ms prelude: + +| Sub-stage | ms | share | +|---|---:|---:| +| **`tools`** | **2039** | **62%** | +| `session_mgr` | 830 | 25% | +| `finalize` | 194 | 6% | +| `prompt` | 160 | 5% | +| `registry` | 50 | 2% | +| `plugins` | 13 | — | +| `hooks` | 0 | — | + +**`agent_build.tools` is the single largest number in the whole turn** — four +times what the preamble cost at its worst, and larger than everything this spec +has fixed so far, combined. + +`session_mgr` at 830ms is the other real number. That is +`SessionFactory.create_session_manager`, i.e. conversation restore from +AgentCore Memory, which had never been timed at all. + +### What it does and does NOT vindicate + +This spec's standing hypothesis was the serial MCP `tools/list` pre-flight, and +the pre-flight does live inside `agent_build.tools`. But the hypothesis cannot +be right about the **fix**, for a reason the measurement makes plain: + +**dev loads exactly one external MCP server, and this stage still costs +2039ms.** Parallelising across servers cannot help when there is one server. So +the proposed `asyncio.gather` was wrong twice over — wrong mechanism (see +below) *and* wrong target shape. + +`_build_filtered_tools()` also does registry filtering, gateway integration and +local tool assembly, so how the 2039ms splits between the MCP round trip and +everything else is **not yet known**. Given the correction recorded under PR-3, +that gap is not something to fill by reasoning. + +### The `gather` fix would have been a no-op + +Recorded because it nearly got built. `MCPClient.load_tools()` is `async def` +with **no await in its hot path**: `start()` blocks on +`_init_future.result()`, and `_list_all_tools_sync()` blocks on +`_invoke_on_background_thread(...).result()`. Every coroutine blocks the loop +before it yields, so `asyncio.gather` over the servers runs them strictly +sequentially. Same shape as the starved-timer bug `agent-state-feedback` PR-3 +already paid for. A real fix needs `asyncio.to_thread` per server *then* +gather — and the **merge** must stay deterministic even when the fetch is not, +because tool order reaches `toolConfig`, which is the prompt-cache prefix. + +### A contextvar, deliberately unlike PR-2 + +PR-2 threaded its snapshot explicitly and rejected an implicit memo; this does +the opposite. The asymmetry is the point: there, implicit state going wrong +meant stale session data in production (the bug CLAUDE.md names, shipped twice +here); here it means a timing number is mis-attributed — nothing the user sees, +nothing persisted, nothing reaching the model. Against that, the explicit route +costs a kwarg threaded through a type registry and three agent classes with +mismatched constructor signatures. See +`apis/shared/observability/build_stages.py`. + +Noted there too: contextvars do not cross into a `ThreadPoolExecutor`, and the +MCP load path crosses one, so a future caller marking from inside it would +silently record nothing. + +## PR-5 — split `agent_build.tools` (NOT STARTED — the one thing worth doing next) + +2039ms, 62% of the cold build, and undifferentiated. Split it into the MCP +pre-flight, gateway integration, and local tool assembly, exactly as PR-1 split +the preamble and PR-4 split the build. + +This is the third time the pattern would be applied and the first two both +overturned an assumption, so the ordering is not negotiable: **measure, then +decide whether a fix is worth building.** Specifically, do not assume the 2s is +the MCP network round trip — on a one-server session it may equally be a +synchronous SDK handshake, TLS setup, or tool-registry work nobody has looked +at. + +Second target after that: `agent_build.session_mgr` at 830ms (AgentCore Memory +restore, never timed). + +## Declined / overtaken — `asyncio.to_thread` for the DynamoDB calls + +Originally PR-3 in this spec, then PR-4, now **not recommended without new +evidence**. Two measurements removed its rationale: + +- PR-2 took seven of the eight blocking reads off the hot path, so the + remaining event-loop exposure is one query; and +- PR-3 showed the dominant cost was never IO wait at all — it was **CPU** + (boto3 client construction). Moving a 6ms query to a thread buys close to + nothing, and the thread hop is not free. + +It also remains unmeasurable without a load test, since its win is throughput +under concurrency and dev has effectively none. If it is revisited, it needs +the `tests/load` harness and its own justification, not this spec's. + +### The original argument, kept for the record + +Independent of PR-2 and independent of the measurement: whatever the reads cost, +doing them on the event loop is wrong under concurrency, and the codebase already +uses `asyncio.to_thread` for exactly this in ~20 places. The session-metadata +layer never got it. + +Worth its own PR rather than riding PR-2, because its win is throughput under +load and would be invisible in a single-user latency measurement — the two need +different evidence. + +## Beyond the preamble + +Found on the same sweep. Recorded so they are not re-derived; none is committed to. + +- **The agent build scales linearly with MCP servers.** `load_external_tools` + iterates enabled servers in a serial `for` loop, each doing a + `repository.get_tool` plus a blocking `list_tools_sync()` round trip to an + external MCP server — N cross-internet hops, one after another, inside the + 1478–2728ms cold build. Concurrent fetch plus a short-TTL cache of each + server's listing is the largest cold-turn win available. **Constraint:** the + merged tool order reaches `toolConfig`, which is the prompt-cache prefix, so + the *merge* must stay deterministic even when the fetch is not. See CLAUDE.md + § Prompt-cache stability. +- **A new `httpx.AsyncClient` per turn** in the BFF proxy — fresh DNS, TCP and + TLS to `bedrock-agentcore.` on every message, never pooled. Measured + 76–135ms from a laptop; in-region it is smaller but nonzero, and it sits inside + the ~478ms the recap already knows it cannot see. The catch is the lifecycle + comment already in `proxy_routes.py`: the client is deliberately closed in the + generator's `finally` because closing it via `async with` buffers the whole SSE + stream. A pooled module-level client has to outlive the request without + reintroducing that. +- **A shared cached-boto3-client module.** Promoted out of this list into + PR-3 below, once PR-2 made the cost visible. + +## The other two open items from `agent-state-feedback.md` + +Both were left open there. Neither is a bug, and neither needs a fix of its own: + +- **The recap's ~1.5s shortfall** *is* the pre-container window — app-api hop, + auth, Runtime routing. It is not a measurement error to correct but a real cost + to remove: the pooled-client item above and the cold-start path are what shrink + it. Measuring it instead would mean measuring on the client, which the + "always on / survives reload" trade already declined. +- **The missing footer on older conversations** is the deliberate "no number + beats a number nobody measured" rule, consistent with the tool-rail durations + and with `agent_status`'s non-persisted timings. Backfilling it would mean + inventing the number. + +## Non-goals + +- **Narrating the preamble.** `agent-state-feedback.md` PR-3 settled this: on a + warm turn no stage dominates and 41% of the wait is not even in the handler, so + there is no honest phase label. Making it shorter is the only lever. +- **Restructuring the handler wholesale.** The measurement is what decides how + much structure has to move; deciding that first is how PR-3's original plan + turned out to be unbuildable. +- **Deferring the preamble into the stream generator** (the structural version of + this fix, which would take TTFB to near zero). Genuinely attractive, and + explicitly out of scope until PR-1 says how much is left to win after the cheap + fixes. It would require converting the remaining HTTP-status guards to SSE + errors — which is the house rule anyway — but it is a much larger change than + anything above and should not ride a measurement PR. diff --git a/frontend/ai.client/README.md b/frontend/ai.client/README.md index 9053f1551..f8391d0bd 100644 --- a/frontend/ai.client/README.md +++ b/frontend/ai.client/README.md @@ -36,6 +36,42 @@ ng build This will compile your project and store the build artifacts in the `dist/` directory. By default, the production build optimizes your application for performance and speed. +### Initial-bundle budget + +The production build enforces an `initial` budget in `angular.json`: **1.6 MB warning, +2 MB error**, measured on _raw_ (pre-compression) bytes. Angular budgets have no +transfer-size mode, so raw size is the only lever — treat it as a proxy and sanity-check +the "Estimated transfer size" column when a change moves the number. + +Two things to know before you widen it: + +- **The `scripts` array is invisible to lazy loading.** Entries under + `architect.build.options.scripts` are emitted as a plain ` + + + diff --git a/infrastructure/assets/mcp-sandbox/live-view.js b/infrastructure/assets/mcp-sandbox/live-view.js new file mode 100644 index 000000000..1309ce4b3 --- /dev/null +++ b/infrastructure/assets/mcp-sandbox/live-view.js @@ -0,0 +1,289 @@ +/** + * Browser sign-in live view — DCV client for a browser takeover. + * + * `docs/specs/authenticated-web-assessment.md` D3. Plain ES2017, no framework, + * no build step, no third-party code: `window.dcv` comes from the Amazon DCV + * Web Client SDK, fetched and signature-verified at build time by + * `scripts/build/fetch-dcv-sdk.sh`. + * + * Trust model, which is the whole reason this file is not in the SPA + * ------------------------------------------------------------------ + * 1. **It never calls app-api.** The SPA mints the short-lived live-view URL + * and posts it in. app-api's CORS is a credentialed allowlist, so letting + * this page call it would mean allowlisting an origin that also serves + * untrusted MCP App HTML — handing every App a path to the user's session. + * 2. **It only accepts a URL from the origin that framed it.** Checked against + * `document.referrer`'s origin, so a sibling frame cannot feed it a stream. + * 3. **It stores nothing.** No storage, no cookies. The signed URL lives in a + * closure for as long as it is valid and is replaced when the SPA posts a + * fresh one. + * + * The SigV4 query parameters + * -------------------------- + * The live-view URL is SigV4 *query*-signed. The SDK forwards those parameters + * onto its own requests — including the WebSocket upgrade — through the + * `httpExtraSearchParams` callback. Without it the socket opens unsigned and + * the service rejects it. `connect` needs it just as much as `authenticate` + * does: the WS transport reads it directly when building the URI. + */ +(function () { + 'use strict'; + + var CONNECT_MESSAGE = 'browser-live-view/connect'; + var READY_MESSAGE = 'browser-live-view/ready'; + var DISPLAY_ID = 'display'; + + var statusEl = document.getElementById('status'); + var connection = null; + var currentUrl = null; + // Set the instant a connection attempt BEGINS, not when it completes. + // The parent posts the URL up to three times (after minting, on the + // iframe's `load`, and in reply to our `ready`) because it cannot know + // which arrives first. Guarding on `connection` alone is not enough: it is + // assigned only after `dcv.connect()` resolves, so two posts milliseconds + // apart both reach `dcv.authenticate` and the second socket's open closes + // the first — surfacing as `Close received after close` and auth code 10. + var starting = false; + // Set once `authenticate` has handed us a session. The SDK closes the auth + // WebSocket as soon as it is done, and reports that close through the + // `error` callback EVEN THOUGH authentication succeeded — measured on dev: + // `connect` is already running when `dcv.authenticate failed {code: 10}` + // arrives, and the stream then establishes normally. In Node the same close + // is a clean code 1000. Without this flag the error handler paints + // "Could not start the session" over a working stream, because #status is + // an absolutely-positioned overlay covering the whole display. + var authenticated = false; + + function setStatus(text) { + if (!statusEl) return; + if (text) { + statusEl.textContent = text; + statusEl.hidden = false; + } else { + statusEl.hidden = true; + } + } + + /** + * The origin allowed to drive this page: the document that framed us. + * + * A top-level visit has no parent to trust and renders nothing — that is a + * refusal, not a failure mode to work around. + */ + function parentOrigin() { + if (window.parent === window) return null; + try { + return document.referrer ? new URL(document.referrer).origin : null; + } catch (e) { + return null; + } + } + + function isConnectMessage(data) { + return ( + data && + typeof data === 'object' && + data.type === CONNECT_MESSAGE && + typeof data.url === 'string' && + data.url.length > 0 && + data.viewport && + typeof data.viewport.width === 'number' && + typeof data.viewport.height === 'number' && + data.viewport.width > 0 && + data.viewport.height > 0 + ); + } + + /** Forward the signed URL's query parameters onto the SDK's own requests. */ + function extraSearchParamsFor(signedUrl) { + return function () { + try { + return new URL(signedUrl).searchParams; + } catch (e) { + return new URLSearchParams(); + } + }; + } + + function start(signedUrl, viewport) { + // A duplicate post, or a re-mint for a still-live session: keep the + // fresher URL for any later use, but never open a second stream. Tearing + // a live one down mid-sign-in to reconnect with a newer signature would + // be the opposite of what re-minting is for. + if (starting || connection) { + currentUrl = signedUrl; + return; + } + currentUrl = signedUrl; + starting = true; + + if (!window.dcv) { + setStatus('The viewer failed to load.'); + return; + } + + setStatus('Connecting…'); + dcv.setLogLevel(dcv.LogLevel.WARN); + + var extras = extraSearchParamsFor(signedUrl); + + // Strip the query before handing the URL to the SDK. It APPENDS + // `httpExtraSearchParams` to whatever URL it is given, so passing the + // signed URL with its query still attached sends every SigV4 parameter + // twice and the service answers 403 — in the browser, a WebSocket that + // opens and immediately closes ("Close received after close"), surfacing + // as auth code 10. `connect` below has always stripped it; `authenticate` + // did not, which is why the viewer never streamed. + var endpoint; + try { + endpoint = new URL(signedUrl); + endpoint.search = ''; + endpoint = endpoint.toString(); + } catch (e) { + starting = false; + setStatus('The session address was not usable.'); + return; + } + + dcv.authenticate(endpoint, { + // AgentCore's presigned URL *is* the credential, so there is never an + // interactive prompt. Supplying a no-op is required — the SDK refuses a + // configuration with a missing auth callback. + promptCredentials: function () {}, + error: function (_auth, error) { + // A failure reported AFTER we already have a session is the auth + // socket closing behind a successful handshake, not a failure to + // reach the service. Surfacing it would hide a live stream. + if (authenticated) { + if (window.console) console.info('dcv auth socket closed', error); + return; + } + starting = false; + setStatus('Could not start the session. It may have ended.'); + if (window.console) console.error('dcv.authenticate failed', error); + }, + success: function (_auth, result) { + var first = (result && result[0]) || {}; + if (!first.sessionId || !first.authToken) { + starting = false; + setStatus('The session could not be opened.'); + return; + } + authenticated = true; + connect(first.sessionId, first.authToken, viewport, extras); + }, + httpExtraSearchParams: extras, + }); + } + +/** + * Pin the remote display to the browser session's real viewport. + * + * Mismatched dimensions are what crop or letterbox the stream, which is why + * the viewport travels on the event rather than being a constant in the + * frontend. + * + * Called from `firstFrame`, NOT from `connect().then()`. The display channel + * is not up when the connect promise resolves, so calling it there rejects + * with "Display channel is not available" — and because it returns a PROMISE, + * a try/catch around it never saw the failure; it surfaced as an unhandled + * rejection in the console instead. Measured on dev. + */ + function applyDisplayLayout(conn, viewport) { + if (!conn || typeof conn.requestDisplayLayout !== 'function') return; + try { + var result = conn.requestDisplayLayout([ + { + name: 'Main Display', + rect: { x: 0, y: 0, width: viewport.width, height: viewport.height }, + primary: true, + }, + ]); + // Not fatal either way: the stream renders at whatever the server chose. + if (result && typeof result.catch === 'function') { + result.catch(function (e) { + if (window.console) console.info('requestDisplayLayout declined', e); + }); + } + } catch (e) { + if (window.console) console.info('requestDisplayLayout threw', e); + } + } + + function connect(sessionId, authToken, viewport, extras) { + // The query is supplied through `httpExtraSearchParams`, so strip it here + // rather than sending it twice — the transport appends to whatever URL it + // is given. + var base; + try { + base = new URL(currentUrl); + base.search = ''; + } catch (e) { + starting = false; + setStatus('The session address was not usable.'); + return; + } + + dcv + .connect({ + url: base.toString(), + sessionId: sessionId, + authToken: authToken, + divId: DISPLAY_ID, + // `baseUrl` is deliberately NOT set: the SDK defaults it to `dcvjs` + // relative to this page, and the fetch script extracts to exactly that + // name so the default resolves. + // `httpExtraSearchParams` MUST live inside `observers` here. Passed + // at the top level it is silently ignored by `connect` — measured + // against the shipped bundle: top-level yields + // `/live-view/ws` with NO query at all, so the stream socket opens + // UNSIGNED and the service refuses it, while `observers` yields the + // signed URI. (`authenticate` is the opposite: it reads the callback + // from the top level. The two entry points differ.) This is also what + // AWS's own BrowserLiveView component does. + // + // Callback names are exactly these — no `on` prefix. The SDK invokes + // each with the connection as its FIRST argument, so the original + // arguments follow it. They go in `observers` too: it is observers or + // callbacks, never both. + observers: { + httpExtraSearchParams: extras, + firstFrame: function (conn) { + setStatus(''); + applyDisplayLayout(conn, viewport); + }, + disconnect: function (_conn, reason) { + connection = null; + starting = false; + authenticated = false; + setStatus('The sign-in session ended.'); + if (window.console) console.info('dcv disconnected', reason); + }, + }, + }) + .then(function (conn) { + connection = conn; + }) + .catch(function (error) { + starting = false; + setStatus('Could not connect to the sign-in session.'); + if (window.console) console.error('dcv.connect failed', error); + }); + } + + var allowed = parentOrigin(); + if (!allowed) { + setStatus('This viewer must be opened from the app.'); + return; + } + + window.addEventListener('message', function (event) { + if (event.origin !== allowed) return; + if (!isConnectMessage(event.data)) return; + start(event.data.url, event.data.viewport); + }); + + // The SPA also posts on the iframe's `load` event; this covers the race + // where that fires before this script has attached its listener. + window.parent.postMessage({ type: READY_MESSAGE }, allowed); +})(); diff --git a/infrastructure/gsi-inventory.json b/infrastructure/gsi-inventory.json index fa0c1ebea..97ef8c54b 100644 --- a/infrastructure/gsi-inventory.json +++ b/infrastructure/gsi-inventory.json @@ -1,5 +1,6 @@ { "tables": { + "agent-templates": [], "announcements": [], "api-keys": [ "KeyHashIndex" diff --git a/infrastructure/lib/config.ts b/infrastructure/lib/config.ts index f1b531fe0..bfed5a5b5 100644 --- a/infrastructure/lib/config.ts +++ b/infrastructure/lib/config.ts @@ -52,12 +52,14 @@ export interface AppConfig { managedKb: ManagedKbConfig; scheduledRuns: ScheduledRunsConfig; memorySpaces: MemorySpacesConfig; + feedbackEvalSampling: FeedbackEvalSamplingConfig; skills: SkillsConfig; agents: AgentsConfig; agentMarketplace: AgentMarketplaceConfig; fineTuning: FineTuningConfig; artifacts: ArtifactsConfig; mcpSandbox: McpSandboxConfig; + browser: BrowserConfig; mcpIdentity: McpIdentityConfig; gateway: GatewayConfig; /** @@ -93,6 +95,31 @@ export interface McpSandboxConfig { extraFrameAncestors: string[]; } +/** + * AgentCore Browser policy (docs/specs/authenticated-web-assessment.md D6). + */ +export interface BrowserConfig { + /** + * Hosts the browser must refuse to navigate to, as Chromium + * `URLBlocklist` entries. + * + * This is a **security control**, not a preference. A browser takeover hands + * a human a fully interactive Chromium, so the only thing that stops them + * navigating to the LMS and having an agent act as them is Chromium itself + * refusing — no check in our own code can, because it only ever sees the + * page the takeover started on. + * + * A blocklist rather than an allowlist because the list has to be + * maintainable: an allowlist of vendors under assessment would churn with + * every VPAT review, while institutional systems change about yearly. + * + * Lives here rather than as an S3 object edited in place, so a change to it + * is a reviewed deploy. Match the origin actually navigated to — a vanity + * CNAME that redirects is not what Chromium sees. + */ + urlBlocklist: string[]; +} + export interface ArtifactsConfig { // ACM certificate ARN for the artifact iframe origin (artifacts.{domainName}). // MUST be in us-east-1 — CloudFront requires its certs there. Validation @@ -274,6 +301,16 @@ export interface MemorySpacesConfig { enabled: boolean; } +/** + * Feedback eval sampling (response-feedback spec §11 PR-4): lets an admin + * send down-thumbed conversations to AgentCore Evaluations. **Opt-in** — + * the managed evaluator reads the conversation's spans (system prompt and + * user messages), so each environment turns it on deliberately. + */ +export interface FeedbackEvalSamplingConfig { + enabled: boolean; +} + /** * Skills feature flag (Skills v2). Default ON with a kill switch — the epic is * complete and dogfooded, so it ships enabled for every deployer (opt-out), @@ -872,6 +909,17 @@ export function loadConfig(scope: cdk.App): AppConfig { ? process.env.CDK_SCHEDULED_RUNS_ENABLED !== 'false' : scope.node.tryGetContext('scheduledRuns')?.enabled ?? true, }, + feedbackEvalSampling: { + // Default OFF, opt-in (the `fineTuning`-style deferred pattern inverted): + // only the literal "true" enables. Sending real conversations to an + // AWS-managed judge is the scoping decision the evaluations spike says to + // make explicitly per environment — a workflow's empty/unset variable must + // never make it. A `feedbackEvalSampling.enabled: true` cdk.json context + // also enables it. + enabled: process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED + ? process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED === 'true' + : scope.node.tryGetContext('feedbackEvalSampling')?.enabled ?? false, + }, memorySpaces: { // Default ON with a kill switch: Memory Spaces is a complete feature and // ships enabled for every deployer (opt-out, not opt-in — matches kbSync / @@ -951,6 +999,44 @@ export function loadConfig(scope: cdk.App): AppConfig { ? process.env.CDK_ARTIFACT_SHARE_INBOX_ENABLED !== 'false' : scope.node.tryGetContext('artifacts')?.shareInboxEnabled ?? true, }, + browser: { + // Hostnames Chromium refuses to navigate to during a browser session. + // This is a security control: it is what stops a human in a browser + // takeover navigating to a system the agent must not act inside. + // + // **Deliberately empty by default.** The contents are a per-deployment + // policy decision, not a property of this stack — the hosts that matter + // to one institution mean nothing to another — so this follows the + // `domainName` / `corsOrigins` convention: fork-neutral in the repo, + // supplied per environment by the `CDK_BROWSER_URL_BLOCKLIST` GitHub + // Actions variable that `platform.yml` forwards. Deployments that + // carried the old hardcoded seed MUST set that variable; a deploy whose + // list comes out empty while the browser tool is grantable says so in + // the deploy log (see the warning below). + // + // Comma-separated in the env var. Empty/unset falls through to context + // and then to `[]` — the house empty-string rule, because an unset + // GitHub Actions variable arrives as '' and must not be distinguishable + // from "not configured". "Block nothing" is the default, so opting out + // needs no sentinel; note it leaves the RBAC grant as the only control + // (spec Security 3). + // + // ⚠️ Chromium's URLBlocklist matches on HOST, not on the service behind + // it, so a site is only as blocked as its hostname list is complete. + // When adding an entry, enumerate the service's aliases first — vendor + // host, vanity CNAME, regional and mobile hostnames — and prefer the + // registrable domain (`instructure.com`) over one instance, so `.test.` + // and `.beta.` variants are covered rather than left as side doors. + // + // ⚠️ Blocking a vendor's *sign-in* host is usually wrong: an + // institutional login page is exactly what an accessibility or VPAT + // review needs to reach, which is the use case this feature exists for. + // Block where it LEADS, not the doorway. + urlBlocklist: + parseListEnv(process.env.CDK_BROWSER_URL_BLOCKLIST) + ?? scope.node.tryGetContext('browser')?.urlBlocklist + ?? [], + }, mcpSandbox: { certificateArn: process.env.CDK_MCP_SANDBOX_CERTIFICATE_ARN || scope.node.tryGetContext('mcpSandbox')?.certificateArn, extraFrameAncestors: process.env.CDK_MCP_SANDBOX_EXTRA_FRAME_ANCESTORS?.split(',') @@ -1162,12 +1248,51 @@ export function loadConfig(scope: cdk.App): AppConfig { + ` agentCoreAppLogs=${config.observability.agentCoreApplicationLogsEnabled}` ); + // Printed because this list is a security control supplied entirely from + // outside the repo: a deploy that ships an empty one has to say so, or a + // forgotten `CDK_BROWSER_URL_BLOCKLIST` variable is indistinguishable in + // the log from a deliberate "block nothing". + if (config.browser.urlBlocklist.length > 0) { + console.log( + ` Browser URL blocklist (${config.browser.urlBlocklist.length}): ` + + config.browser.urlBlocklist.join(', ') + ); + } else { + console.warn( + ' ⚠️ Browser URL blocklist is EMPTY — browser sessions can reach any' + + ' host. Set the CDK_BROWSER_URL_BLOCKLIST variable if this environment' + + ' is meant to block one. RBAC on browse_web / request_user_login is' + + ' then the only control.' + ); + } + // Validate configuration validateConfig(config); return config; } +/** + * Parse a comma-separated list environment variable. + * + * Returns undefined for a missing OR empty value so that nullish coalescing + * (??) falls through to context defaults — the house empty-string rule. An + * unset GitHub Actions variable is forwarded as '', and treating that as an + * explicit "empty list" would let a forgotten variable silently override a + * configured default. + * + * @param value The environment variable value to parse + * @returns Trimmed, non-empty entries, or undefined if unset/empty + */ +export function parseListEnv(value: string | undefined): string[] | undefined { + if (value === undefined || value.trim() === '') { + return undefined; + } + + const entries = value.split(',').map((s) => s.trim()).filter(Boolean); + return entries.length > 0 ? entries : undefined; +} + /** * Parse boolean environment variable with validation. * @@ -1451,11 +1576,25 @@ function validateConfig(config: AppConfig): void { }); } - // Validate top-level CORS origins. - if (!config.corsOrigins) { - console.warn( - 'Warning: no CORS origins configured. ' + - 'Set CDK_DOMAIN_NAME or CDK_CORS_ORIGINS to enable browser uploads.' + // Validate top-level CORS origins. Without at least one origin the uploads + // bucket (`FileUploadConstruct`) is created with NO CORS rule, and every + // browser upload fails at S3 with no server-side signal. A production-mirror + // load test shipped exactly that (docs/specs/load-test-assessment-2026-09.md + // §1 fix 4) because this used to be a console.warn lost in synth output. + // Fail synth instead; a deployment that genuinely has no browser front-end + // opts out explicitly. + // + // Gate on `buildCorsOrigins` -- the same filtered list FileUploadConstruct + // consumes -- not on the raw string. A value that is truthy but filters to + // nothing (`","` from a templated list's trailing comma, `" "` from a YAML + // value that quotes to a space, `"${UNSET_VAR},"` in CI) would otherwise + // pass the guard and still produce a bucket with no CORS rule. Pass no + // `additionalOrigins` here, for the same reason: the construct does not. + if (buildCorsOrigins(config).length === 0 && parseBooleanEnv(process.env.CDK_ALLOW_NO_CORS_ORIGINS) !== true) { + throw new Error( + 'No CORS origins configured: the uploads bucket would be created without a CORS rule ' + + 'and every browser upload would fail. Set CDK_DOMAIN_NAME (or CDK_CORS_ORIGINS), ' + + 'or set CDK_ALLOW_NO_CORS_ORIGINS=true for a deployment with no browser front-end.' ); } diff --git a/infrastructure/lib/constructs/agentcore/browser-policy-construct.ts b/infrastructure/lib/constructs/agentcore/browser-policy-construct.ts new file mode 100644 index 000000000..d542d9ea0 --- /dev/null +++ b/infrastructure/lib/constructs/agentcore/browser-policy-construct.ts @@ -0,0 +1,161 @@ +import * as cdk from 'aws-cdk-lib'; +import * as iam from 'aws-cdk-lib/aws-iam'; +import * as s3 from 'aws-cdk-lib/aws-s3'; +import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment'; +import { Construct } from 'constructs'; + +import { + AppConfig, + getAutoDeleteObjects, + getRemovalPolicy, + getResourceName, +} from '../../config'; + +/** + * The policy object's location, split so the deployment and the key the + * backend is told about are derived from ONE source. + * + * They were separate constants once, and the BucketDeployment's + * `destinationKeyPrefix` compounded with a source name that already carried + * the prefix — the object landed at `policies/policies/managed-policies.json` + * while `BROWSER_POLICY_S3` pointed at `policies/managed-policies.json`. The + * deploy looked clean and the policy silently resolved to a missing key, which + * is the worst shape for a security control to fail in. Keep these derived. + */ +const BROWSER_POLICY_PREFIX = 'policies'; +const BROWSER_POLICY_FILENAME = 'managed-policies.json'; + +/** Full object key, as both the deployment and the backend see it. */ +export const BROWSER_MANAGED_POLICY_KEY = + `${BROWSER_POLICY_PREFIX}/${BROWSER_POLICY_FILENAME}`; + +/** + * Render the Chromium managed-policy document. + * + * Exported so the JSON is unit-testable directly, and so there is exactly one + * place that decides what the browser is forbidden to reach. + * + * `URLBlocklist` is a flat list of Chromium URL-filter patterns. A bare host + * blocks that host on every scheme, port and path. See + * https://chromeenterprise.google/policies/#URLBlocklist + */ +export function buildBrowserManagedPolicy( + urlBlocklist: string[], +): Record { + return { URLBlocklist: [...urlBlocklist] }; +} + +export interface AgentCoreBrowserPolicyConstructProps { + config: AppConfig; + /** + * The browser's execution role. It is what reads the policy object from S3 + * when a session starts, so it needs `s3:GetObject` on the key. + */ + browserExecutionRole: iam.IRole; +} + +/** + * AgentCoreBrowserPolicyConstruct — the S3-hosted Chromium managed policy that + * every browser session is started with. + * + * Why this exists, and why it is not on the browser resource + * ---------------------------------------------------------- + * A takeover hands a human a fully interactive Chromium inside our AWS + * account. The only thing that can stop them navigating somewhere they should + * not — the LMS, say, to have an agent submit coursework — is Chromium itself + * refusing. A check in our own code cannot: it sees the page the takeover + * *started* on, and the user drives from there. + * + * The obvious home for that policy is the browser resource, via + * `CreateBrowser`'s `enterprisePolicies`. Three things rule it out: + * + * 1. There is no `UpdateBrowser` operation, and policy files are read from + * S3 "at the time of the API call" — so a policy attached at create time + * is frozen, and every edit replaces the browser resource. + * 2. `CfnBrowserCustom` (aws-cdk-lib 2.251.0) does not expose + * `enterprisePolicies` at all, so CFN cannot set it without a custom + * resource that would inherit problem 1. + * 3. `StartBrowserSession` accepts the same `enterprisePolicies` shape and + * reads it fresh on every session. + * + * So this construct owns only the *object and its IAM*; the backend passes the + * reference on every `start()` (`agents/builtin_tools/browser/session_pool.py`). + * + * ⚠️ **Session-level policies are RECOMMENDED-only.** The API's `type` enum + * accepts MANAGED, but the service rejects it — measured on dev 2026-09-19: + * "Invalid value for parameter 'type'. MANAGED is not supported for + * session-level policies." It does not degrade; `StartBrowserSession` fails and + * every browser session dies. + * + * Chromium treats recommended policies as user-overridable defaults, so what + * this deploys constrains the **agent** (which drives via CDP and never opens + * settings) but NOT a **human** holding the browser during a takeover. The + * un-overridable MANAGED policy needs `CreateBrowser`, which `CfnBrowserCustom` + * does not expose — so it needs a custom resource, and `request_user_login` + * must stay ungranted until that lands. See the spec's D6. + * + * The policy content is rendered from CDK config rather than being an object + * an admin edits in place, so changing what the browser may reach is a + * reviewed deploy. See `docs/specs/authenticated-web-assessment.md` D6. + */ +export class AgentCoreBrowserPolicyConstruct extends Construct { + public readonly bucket: s3.Bucket; + public readonly policyKey = BROWSER_MANAGED_POLICY_KEY; + + constructor( + scope: Construct, + id: string, + props: AgentCoreBrowserPolicyConstructProps, + ) { + super(scope, id); + + const { config, browserExecutionRole } = props; + + // Must be in the same region as the browser (service requirement). + this.bucket = new s3.Bucket(this, 'BrowserPolicyBucket', { + bucketName: getResourceName(config, 'browser-policy', config.awsAccount), + encryption: s3.BucketEncryption.S3_MANAGED, + blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL, + enforceSSL: true, + // Versioned on purpose: this is a security control, and being able to + // see what the browser was allowed to reach on a given date is worth + // more than the pennies of storage. + versioned: true, + removalPolicy: getRemovalPolicy(config), + autoDeleteObjects: getAutoDeleteObjects(config), + }); + + new s3deploy.BucketDeployment(this, 'BrowserPolicyDeployment', { + sources: [ + // Just the filename: `destinationKeyPrefix` below supplies the + // prefix. Passing the full key here is what doubled it. + s3deploy.Source.jsonData( + BROWSER_POLICY_FILENAME, + buildBrowserManagedPolicy(config.browser.urlBlocklist), + ), + ], + destinationBucket: this.bucket, + // Scoped prune: this deployment owns this prefix and nothing else. + destinationKeyPrefix: BROWSER_POLICY_PREFIX, + prune: true, + }); + + // The browser service reads the policy as the browser's execution role + // when a session starts. `GetObjectVersion` as well as `GetObject`, + // per the service's documented prerequisite, because the bucket is + // versioned and a pinned `versionId` would otherwise be unreadable. + browserExecutionRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'BrowserEnterprisePolicyS3Access', + effect: iam.Effect.ALLOW, + actions: ['s3:GetObject', 's3:GetObjectVersion'], + resources: [this.bucket.arnForObjects('policies/*')], + }), + ); + + new cdk.CfnOutput(this, 'BrowserPolicyBucketName', { + value: this.bucket.bucketName, + description: 'S3 bucket holding the browser Chromium managed policy', + }); + } +} diff --git a/infrastructure/lib/constructs/app-api/app-api-environment.ts b/infrastructure/lib/constructs/app-api/app-api-environment.ts index 3bfbabd7f..cdf1bc7d8 100644 --- a/infrastructure/lib/constructs/app-api/app-api-environment.ts +++ b/infrastructure/lib/constructs/app-api/app-api-environment.ts @@ -59,6 +59,8 @@ export interface AppApiSsmParams { announcementsTableArn: string; systemPromptsTableName: string; systemPromptsTableArn: string; + agentTemplatesTableName: string; + agentTemplatesTableArn: string; authProvidersTableName: string; authProvidersTableArn: string; authProviderSecretsArn: string; @@ -81,6 +83,8 @@ export interface AppApiSsmParams { voiceTicketSigningSecretArn: string; // Inference inferenceApiRuntimeEndpointUrl: string; + /** AgentCore Runtime CloudWatch log group (spans + content log records) for eval sampling. */ + agentCoreRuntimeLogGroupName: string; // File uploads userFilesBucketName: string; userFilesBucketArn: string; @@ -117,6 +121,8 @@ export interface AppApiBackendOverrides { memoryId: string; /** AgentCore Runtime endpoint URL (from InferenceAgentCoreConstruct.runtimeEndpointUrl). */ inferenceApiRuntimeEndpointUrl: string; + /** AgentCore Runtime log group name (from InferenceAgentCoreConstruct.runtimeLogGroupName). */ + agentCoreRuntimeLogGroupName: string; } /** Resolve every value the App API construct needs. @@ -178,6 +184,8 @@ export function resolveAppApiParams( announcementsTableArn: refs.announcementsTable.tableArn, systemPromptsTableName: refs.systemPromptsTable.tableName, systemPromptsTableArn: refs.systemPromptsTable.tableArn, + agentTemplatesTableName: refs.agentTemplatesTable.tableName, + agentTemplatesTableArn: refs.agentTemplatesTable.tableArn, authProvidersTableName: refs.authProvidersTable.tableName, authProvidersTableArn: refs.authProvidersTable.tableArn, authProviderSecretsArn: refs.authProviderSecretsSecret.secretArn, @@ -200,6 +208,7 @@ export function resolveAppApiParams( voiceTicketSigningSecretArn: refs.voiceTicketSigningSecret.secretArn, // Inference inferenceApiRuntimeEndpointUrl: overrides.inferenceApiRuntimeEndpointUrl, + agentCoreRuntimeLogGroupName: overrides.agentCoreRuntimeLogGroupName, // File uploads userFilesBucketName: refs.fileUploadBucket.bucketName, userFilesBucketArn: refs.fileUploadBucket.bucketArn, @@ -306,6 +315,7 @@ export function buildAppApiEnvironment( DYNAMODB_USER_MENU_LINKS_TABLE_NAME: params.userMenuLinksTableName, DYNAMODB_ANNOUNCEMENTS_TABLE_NAME: params.announcementsTableName, DYNAMODB_SYSTEM_PROMPTS_TABLE_NAME: params.systemPromptsTableName, + DYNAMODB_AGENT_TEMPLATES_TABLE_NAME: params.agentTemplatesTableName, COGNITO_USER_POOL_ID: params.cognitoUserPoolId, COGNITO_APP_CLIENT_ID: params.cognitoAppClientId, COGNITO_ISSUER_URL: params.cognitoIssuerUrl, @@ -338,6 +348,13 @@ export function buildAppApiEnvironment( // every read 502s (ResourceNotFoundException). inference-api already sets // the identical trio — app-api owns the CRUD surface, so it needs them too. MEMORY_SPACES_ENABLED: config.memorySpaces.enabled ? 'true' : 'false', + // Feedback eval sampling (response-feedback spec §11 PR-4): OPT-IN per + // environment — the admin batch sends down-thumbed conversations' spans to + // an AWS-managed evaluator. The runtime log group is where those spans and + // the content-bearing log records live (evaluations spike §1); it is wired + // regardless so turning the flag on is a one-variable change. + FEEDBACK_EVAL_SAMPLING_ENABLED: config.feedbackEvalSampling.enabled ? 'true' : 'false', + AGENTCORE_RUNTIME_LOG_GROUP: params.agentCoreRuntimeLogGroupName, DYNAMODB_MEMORY_SPACES_TABLE_NAME: params.memorySpacesTableName, S3_MEMORY_SPACES_BUCKET_NAME: params.memorySpacesBucketName, // Skills v2 (default ON with a kill switch per env). Skills live in the diff --git a/infrastructure/lib/constructs/app-api/app-api-iam-grants.ts b/infrastructure/lib/constructs/app-api/app-api-iam-grants.ts index 695cf6abe..4513ea120 100644 --- a/infrastructure/lib/constructs/app-api/app-api-iam-grants.ts +++ b/infrastructure/lib/constructs/app-api/app-api-iam-grants.ts @@ -48,6 +48,12 @@ export interface AppApiIamGrantsProps { * `refs` later if convenient. */ agentCoreMemoryArn: string; + /** + * AgentCore Runtime CloudWatch log group name. Feedback eval sampling runs + * Logs Insights queries against it (and `aws/spans`) to collect a + * conversation's spans for AgentCore Evaluations. + */ + agentCoreRuntimeLogGroupName: string; /** * SageMaker fine-tuning execution role ARN. Created by a sibling * construct in wireCompute() — passed in here. @@ -130,6 +136,21 @@ export function grantAppApiPermissions(props: AppApiIamGrantsProps): void { }), ); + // ── Agent templates (create-agent picker catalog) ── + // Admin-managed CRUD; per-user reads (the enabled catalog) go through the + // user-facing `/templates` endpoint, which uses the same table. + taskRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'AgentTemplatesTableAccess', + effect: iam.Effect.ALLOW, + actions: [ + 'dynamodb:GetItem', 'dynamodb:PutItem', 'dynamodb:UpdateItem', + 'dynamodb:DeleteItem', 'dynamodb:Query', 'dynamodb:Scan', + ], + resources: [props.refs.agentTemplatesTable.tableArn, `${props.refs.agentTemplatesTable.tableArn}/index/*`], + }), + ); + // ── RAG assistants table ── taskRole.addToPrincipalPolicy( new iam.PolicyStatement({ @@ -289,6 +310,63 @@ export function grantAppApiPermissions(props: AppApiIamGrantsProps): void { }), ); + // ── AgentCore Browser: Live View only ── + // app-api mints the short-lived Live View URL for a browser takeover + // (docs/specs/authenticated-web-assessment.md D2). The URL is SigV4 + // query-signed and lives at most 300 seconds, so it cannot be minted once + // by the agent and reused — app-api signs a fresh one per request, which + // is why these actions are needed here and not only on the Runtime role. + // + // Deliberately NARROWER than the Runtime's BrowserAccess statement: no + // Start/Stop, no ConnectBrowserAutomationStream. app-api never drives the + // browser and must not be able to — the agent owns the session lifecycle. + // UpdateBrowserStream is included because releasing a takeover from the + // API side is the next thing this route will need (an explicit "give the + // browser back" control), and GetBrowserSession so an ended session can be + // reported as such rather than surfacing a signing failure. + taskRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'BrowserLiveViewAccess', + effect: iam.Effect.ALLOW, + actions: [ + 'bedrock-agentcore:UpdateBrowserStream', + 'bedrock-agentcore:GetBrowserSession', + ], + resources: [props.refs.agentCoreBrowserArn], + }), + ); + + // ⚠️ `ConnectBrowserLiveViewStream` MUST be granted on `*`. AWS's own + // service reference lists NO resource types for it, while the two actions + // above list `browser` / `browser-custom`: + // + // GetBrowserSession -> ['browser', 'browser-custom'] + // UpdateBrowserStream -> ['browser', 'browser-custom'] + // ConnectBrowserLiveViewStream -> [] <- no resource types + // + // An action with no resource types NEVER matches a resource-scoped + // statement, so scoping it alongside the others was an implicit deny. It + // failed silently and late: `generate_live_view_url` only signs locally and + // makes no API call, so a URL was minted happily and the browser's + // WebSocket was closed by the service — surfacing as DCV auth code 10 + // ("Failed to communicate with server"), which reads like a service fault + // rather than a missing permission. Verified with + // `iam simulate-principal-policy`: allowed for the two above and + // implicitDeny for this one, from the SAME statement on the SAME ARN. + // + // `*` is as narrow as this action can be expressed; there is no + // browser-scoped form to fall back to. It is bounded by what the action + // itself permits — attaching to a live view stream — and app-api still + // cannot start, stop, or drive a browser. + taskRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'BrowserLiveViewConnect', + effect: iam.Effect.ALLOW, + actions: ['bedrock-agentcore:ConnectBrowserLiveViewStream'], + resources: ['*'], + }), + ); + // ── Cognito admin ops ── taskRole.addToPrincipalPolicy( new iam.PolicyStatement({ @@ -508,6 +586,44 @@ export function grantAppApiPermissions(props: AppApiIamGrantsProps): void { }), ); + // ── AgentCore Evaluations (feedback eval sampling, spec §11 PR-4) ── + // The admin batch judges down-thumbed conversations with the built-in + // evaluators. Two halves: the SDK's span collector runs Logs Insights + // queries over the runtime log group and `aws/spans` (StartQuery is + // resource-scoped; GetQueryResults/StopQuery are not), then calls the + // data-plane Evaluate with the spans and the control-plane GetEvaluator + // to learn each evaluator's level. Built-in evaluators are AWS-owned, so + // the bedrock-agentcore actions cannot be resource-scoped. The flag + // (FEEDBACK_EVAL_SAMPLING_ENABLED) defaults OFF; the grant is inert until + // an environment opts in. + taskRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'FeedbackEvalSpanQueries', + effect: iam.Effect.ALLOW, + actions: ['logs:StartQuery'], + resources: [ + `arn:aws:logs:${config.awsRegion}:${config.awsAccount}:log-group:${props.agentCoreRuntimeLogGroupName}:*`, + `arn:aws:logs:${config.awsRegion}:${config.awsAccount}:log-group:aws/spans:*`, + ], + }), + ); + taskRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'FeedbackEvalSpanQueryResults', + effect: iam.Effect.ALLOW, + actions: ['logs:GetQueryResults', 'logs:StopQuery'], + resources: ['*'], + }), + ); + taskRole.addToPrincipalPolicy( + new iam.PolicyStatement({ + sid: 'FeedbackEvalEvaluate', + effect: iam.Effect.ALLOW, + actions: ['bedrock-agentcore:Evaluate', 'bedrock-agentcore:GetEvaluator', 'bedrock-agentcore:ListEvaluators'], + resources: ['*'], + }), + ); + // ── Bedrock model invocation ── // Used by both title generation and the API-key `/chat/api-converse` // handler (apis/app_api/chat/converse_routes.py), which calls Bedrock diff --git a/infrastructure/lib/constructs/app-api/app-api-service-construct.ts b/infrastructure/lib/constructs/app-api/app-api-service-construct.ts index e681a25af..a317f034d 100644 --- a/infrastructure/lib/constructs/app-api/app-api-service-construct.ts +++ b/infrastructure/lib/constructs/app-api/app-api-service-construct.ts @@ -41,6 +41,12 @@ export interface AppApiServiceConstructProps { * `INFERENCE_API_URL` env var. */ inferenceApiRuntimeEndpointUrl: string; + /** + * AgentCore Runtime CloudWatch log group name (same-stack ref via + * InferenceAgentCoreConstruct.runtimeLogGroupName). The App API reads it + * for feedback eval sampling and is granted Logs Insights queries on it. + */ + agentCoreRuntimeLogGroupName: string; /** * Artifacts iframe origin URL (https://artifacts.{domain}). Same-stack * ref via ArtifactsDistributionConstruct; used as the App API @@ -95,6 +101,7 @@ export class AppApiServiceConstruct extends Construct { const params = resolveAppApiParams(props.refs, { memoryId: props.agentCoreMemoryId, inferenceApiRuntimeEndpointUrl: props.inferenceApiRuntimeEndpointUrl, + agentCoreRuntimeLogGroupName: props.agentCoreRuntimeLogGroupName, }); // ── Network resources (typed refs from PlatformStack) ── @@ -247,6 +254,7 @@ export class AppApiServiceConstruct extends Construct { taskRole: taskDefinition.taskRole, refs: props.refs, agentCoreMemoryArn: props.agentCoreMemoryArn, + agentCoreRuntimeLogGroupName: props.agentCoreRuntimeLogGroupName, sagemakerExecutionRoleArn: props.sagemakerExecutionRoleArn, }); diff --git a/infrastructure/lib/constructs/data/admin-tables-construct.ts b/infrastructure/lib/constructs/data/admin-tables-construct.ts index 0060f2168..b1d1cb02b 100644 --- a/infrastructure/lib/constructs/data/admin-tables-construct.ts +++ b/infrastructure/lib/constructs/data/admin-tables-construct.ts @@ -34,6 +34,7 @@ export class AdminTablesConstruct extends Construct { public readonly userMenuLinksTable: dynamodb.Table; public readonly announcementsTable: dynamodb.Table; public readonly systemPromptsTable: dynamodb.Table; + public readonly agentTemplatesTable: dynamodb.Table; constructor( scope: Construct, @@ -90,6 +91,20 @@ export class AdminTablesConstruct extends Construct { encryption: dynamodb.TableEncryption.AWS_MANAGED, }); + // Agent templates: admin-managed catalog of curated starting points for new + // agents. PK `TEMPLATE#`, SK `METADATA`. Read by the public + // `/templates` endpoint (create-agent picker) and admin CRUD. No GSI — the + // catalog is small, so listing is a Scan (mirrors SystemPromptsTable). + this.agentTemplatesTable = new dynamodb.Table(this, 'AgentTemplatesTable', { + tableName: getResourceName(config, 'agent-templates'), + partitionKey: { name: 'PK', type: dynamodb.AttributeType.STRING }, + sortKey: { name: 'SK', type: dynamodb.AttributeType.STRING }, + billingMode: dynamodb.BillingMode.PAY_PER_REQUEST, + pointInTimeRecoverySpecification: { pointInTimeRecoveryEnabled: true }, + removalPolicy: getRemovalPolicy(config), + encryption: dynamodb.TableEncryption.AWS_MANAGED, + }); + // ── SSM publications (consumed by restore tooling, app-api runtime) ── new ssm.StringParameter(this, 'UserSettingsTableNameParameter', { parameterName: `/${config.projectPrefix}/settings/user-settings-table-name`, @@ -130,5 +145,20 @@ export class AdminTablesConstruct extends Construct { tier: ssm.ParameterTier.STANDARD, }); + // Agent templates: name + arn published, mirroring system prompts. + new ssm.StringParameter(this, 'AgentTemplatesTableNameParameter', { + parameterName: `/${config.projectPrefix}/admin/agent-templates-table-name`, + stringValue: this.agentTemplatesTable.tableName, + description: 'Agent templates DynamoDB table name', + tier: ssm.ParameterTier.STANDARD, + }); + + new ssm.StringParameter(this, 'AgentTemplatesTableArnParameter', { + parameterName: `/${config.projectPrefix}/admin/agent-templates-table-arn`, + stringValue: this.agentTemplatesTable.tableArn, + description: 'Agent templates DynamoDB table ARN', + tier: ssm.ParameterTier.STANDARD, + }); + } } diff --git a/infrastructure/lib/constructs/data/file-upload-construct.ts b/infrastructure/lib/constructs/data/file-upload-construct.ts index 7e9179b6f..dc1b55686 100644 --- a/infrastructure/lib/constructs/data/file-upload-construct.ts +++ b/infrastructure/lib/constructs/data/file-upload-construct.ts @@ -103,6 +103,20 @@ export class FileUploadConstruct extends Construct { id: 'expire-objects', expiration: cdk.Duration.days(365), }, + { + // Compaction's tool-result offload (backend + // agents/main_agent/core/tool_result_offload.py) parks oversized + // tool results under `compaction-offload/{userId}/{sessionId}/` and + // keeps a retrieval reference in the conversation. The plugin never + // evicts from the model path (that would turn a retrieval into a + // miss mid-turn), so objects expire here instead — 90 days, matching + // AgentCore Memory's conversation retention. The shortest matching + // expiration wins, so this overrides the 365-day rule for the + // prefix only; user uploads are untouched. + id: 'expire-compaction-offload', + prefix: 'compaction-offload/', + expiration: cdk.Duration.days(90), + }, { id: 'abort-incomplete-multipart', abortIncompleteMultipartUploadAfter: cdk.Duration.days(1), diff --git a/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts b/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts index 73538dfaa..4de89907a 100644 --- a/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts +++ b/infrastructure/lib/constructs/inference-api/inference-agentcore-construct.ts @@ -50,6 +50,10 @@ export interface InferenceAgentCoreConstructProps { browserArn: string; /** AgentCore Browser ID — same provenance as browserArn. */ browserId: string; + /** S3 bucket holding the Chromium MANAGED policy (spec D6). */ + browserPolicyBucketName: string; + /** Object key of that policy file. */ + browserPolicyKey: string; alarmTopic?: sns.ITopic; } @@ -231,6 +235,27 @@ export class InferenceAgentCoreConstruct extends Construct { resources: [props.browserArn], })); + // The Chromium URL policy is passed on every StartBrowserSession, and the + // service reads the S3 object as **the caller** — this role — not as the + // browser's execution role. Granting only the browser role (which the + // service's own prerequisites document) produced: + // + // ValidationException ... Access denied to S3 object - bucket: ..., + // key: policies/managed-policies.json. Verify that the caller has + // permission to access this bucket and is the bucket owner. + // + // and that failure takes down EVERY browser session, not just the policy. + // Scoped to the one object rather than the prefix: this role only ever + // needs to read the policy it is passing. + runtimeExecutionRole.addToPolicy(new iam.PolicyStatement({ + sid: 'BrowserPolicyObjectRead', + effect: iam.Effect.ALLOW, + actions: ['s3:GetObject', 's3:GetObjectVersion'], + resources: [ + `arn:aws:s3:::${props.browserPolicyBucketName}/${props.browserPolicyKey}`, + ], + })); + // ============================================================ // Import Cognito SSM Parameters for JWT Authorizer // ============================================================ @@ -350,6 +375,14 @@ export class InferenceAgentCoreConstruct extends Construct { MEMORY_ARN: props.memoryArn, AGENTCORE_CODE_INTERPRETER_ID: props.codeInterpreterId, BROWSER_ID: props.browserId, + // The Chromium MANAGED policy passed on every StartBrowserSession. + // This is the control that stops a human in a takeover navigating to + // the LMS — no check in our code can, because it only ever sees the + // page the takeover started on. Spec D6. + // + // One variable, not a bucket/key pair, because the runtime's env-var + // budget is full (see the ceiling note below). + BROWSER_POLICY_S3: `s3://${props.browserPolicyBucketName}/${props.browserPolicyKey}`, // Gateway inbound auth mode. Sourced from the SAME config value that // builds the Gateway's authorizer, so the agent's data-plane auth and @@ -406,12 +439,22 @@ export class InferenceAgentCoreConstruct extends Construct { // bindings entirely (today's behavior). AGENTS_API_ENABLED: config.agents.enabled ? 'true' : 'false', + // ENABLE_QUOTA_ENFORCEMENT is deliberately NOT set. `quota.py` reads + // it with a 'true' default, and this was hardcoded to 'true' — so the + // entry only ever restated the default while consuming one of the 50 + // slots. Removing it leaves enforcement ON and frees a slot, which is + // exactly the remedy runtime-env-var-limit.test.ts recommends. If + // enforcement ever needs to be switchable, make it config-driven + // rather than re-adding a constant. + // Authentication - ENABLE_QUOTA_ENFORCEMENT: 'true', - // ⚠️ NO ROOM FOR NEW VARIABLES HERE — see the assertion in - // test/inference-agentcore-construct.test.ts. `AWS::BedrockAgentCore::Runtime` - // caps EnvironmentVariables at 50 and this construct is AT the cap. + // ⚠️ ALMOST NO ROOM HERE — see the assertion in + // test/runtime-env-var-limit.test.ts, which prints the live headroom. + // `AWS::BedrockAgentCore::Runtime` caps EnvironmentVariables at 50. + // This construct sat AT the cap until retiring the three dead + // directory variables above took it to 47/50; treat those 3 as a + // one-off reprieve, not permission to spend them casually. // Adding one more fails CloudFormation's *changeset validation* — after // synth, after tsc, after jest, after CI is green. It broke the dev // Platform Stack deploy on 2026-08-05 (`maximum size: [50], found: [51]`, @@ -425,10 +468,13 @@ export class InferenceAgentCoreConstruct extends Construct { // deployed environment requires an out-of-band Runtime update until a // slot is freed. - // Directories - UPLOAD_DIR: '/tmp/uploads', - OUTPUT_DIR: '/tmp/output', - GENERATED_IMAGES_DIR: '/tmp/generated_images', + // NOTE: UPLOAD_DIR / OUTPUT_DIR / GENERATED_IMAGES_DIR used to be set + // here to /tmp/*. The runtime never read them for anything but a log + // line — the directories actually resolved from __file__, inside the + // source tree — so they pointed operators at paths nothing used. The + // whole local-output mechanism has since been deleted (output goes to + // S3), so there is nothing left to configure. Retiring them freed three + // of the 50 slots called out below. // URLs FRONTEND_URL: config.domainName ? `https://${config.domainName}` : 'http://localhost:4200', diff --git a/infrastructure/lib/constructs/mcp-sandbox/mcp-sandbox-bucket-construct.ts b/infrastructure/lib/constructs/mcp-sandbox/mcp-sandbox-bucket-construct.ts index 4a278cf60..410064bec 100644 --- a/infrastructure/lib/constructs/mcp-sandbox/mcp-sandbox-bucket-construct.ts +++ b/infrastructure/lib/constructs/mcp-sandbox/mcp-sandbox-bucket-construct.ts @@ -1,6 +1,8 @@ +import * as cdk from 'aws-cdk-lib'; import * as s3 from 'aws-cdk-lib/aws-s3'; import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment'; import * as cloudfront from 'aws-cdk-lib/aws-cloudfront'; +import * as fs from 'fs'; import * as path from 'path'; import { Construct } from 'constructs'; @@ -70,11 +72,34 @@ export class McpSandboxBucketConstruct extends Construct { * locally and the aws-cdk-lib BucketDeployment Lambda uploads it. */ public deployShell(distribution: cloudfront.IDistribution): void { + const assetDir = path.resolve( + __dirname, '..', '..', '..', 'assets', 'mcp-sandbox', + ); + + // The browser sign-in viewer (`live-view.html`) needs the Amazon DCV Web + // Client SDK served beside it. The SDK is a EULA-licensed AWS download and + // this repo is public, so it is fetched at build time by + // `scripts/build/fetch-dcv-sdk.sh` and is gitignored — which means a synth + // that skipped that step would deploy a viewer with no client. + // + // A warning rather than an error on purpose: every CDK unit test synths + // this stack, and none of them should have to download a 1.6MB archive to + // do it. The viewer also fails visibly rather than silently ("The viewer + // failed to load"), so this is a nudge, not the only line of defence. + if ( + fs.existsSync(path.join(assetDir, 'live-view.html')) && + !fs.existsSync(path.join(assetDir, 'dcvjs', 'dcv.js')) + ) { + cdk.Annotations.of(this).addWarning( + 'Amazon DCV Web Client SDK not found at assets/mcp-sandbox/dcvjs/. ' + + 'The browser sign-in viewer will deploy without its client and fail ' + + 'to connect. Run scripts/build/fetch-dcv-sdk.sh before deploying.', + ); + } + new s3deploy.BucketDeployment(this, 'McpSandboxShellDeployment', { sources: [ - s3deploy.Source.asset( - path.resolve(__dirname, '..', '..', '..', 'assets', 'mcp-sandbox'), - ), + s3deploy.Source.asset(assetDir), ], destinationBucket: this.bucket, distribution, diff --git a/infrastructure/lib/constructs/observability/platform-dashboard-construct.ts b/infrastructure/lib/constructs/observability/platform-dashboard-construct.ts index 74cdf09a8..e012747ae 100644 --- a/infrastructure/lib/constructs/observability/platform-dashboard-construct.ts +++ b/infrastructure/lib/constructs/observability/platform-dashboard-construct.ts @@ -73,7 +73,8 @@ export class PlatformDashboardConstruct extends Construct { '', '**Drill-downs:** ' + `[AgentCore Runtime detail](/cloudwatch/home?region=${config.awsRegion}#dashboards:name=${getResourceName(config, 'agentcore-observability')}) · ` - + `[Prompt cache & token economics](/cloudwatch/home?region=${config.awsRegion}#dashboards:name=${getResourceName(config, 'prompt-cache-observability')})`, + + `[Prompt cache & token economics](/cloudwatch/home?region=${config.awsRegion}#dashboards:name=${getResourceName(config, 'prompt-cache-observability')}) · ` + + `[Turn latency — pre-stream stages](/cloudwatch/home?region=${config.awsRegion}#dashboards:name=${getResourceName(config, 'turn-latency-observability')})`, '', '_The chat path is SSE, so response times of tens of seconds are normal ' + 'and a sudden DROP in latency can mean turns are failing early._', diff --git a/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts b/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts new file mode 100644 index 000000000..02e4fd4f0 --- /dev/null +++ b/infrastructure/lib/constructs/observability/turn-latency-observability-construct.ts @@ -0,0 +1,269 @@ +import * as cdk from 'aws-cdk-lib'; +import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch'; +import { Construct } from 'constructs'; + +import { AppConfig, getResourceName } from '../../config'; + +export interface TurnLatencyObservabilityConstructProps { + config: AppConfig; + /** + * The log group the AgentCore Runtime actually writes to, from + * `InferenceAgentCoreConstruct.runtimeLogGroupName`. + * + * Must be passed, not derived — the group is service-created and named after + * the runtime *id*. The prompt-cache construct once guessed it, and its Logs + * Insights widgets silently returned nothing, which reads as "no traffic" + * rather than "wrong query". + */ + runtimeLogGroupName: string; +} + +/** + * TurnLatencyObservabilityConstruct — percentiles over the pre-stream stages + * emitted by `backend/src/apis/inference_api/chat/turn_timing.py`. + * + * WHY THIS EXISTS + * --------------- + * `docs/specs/turn-latency-preamble.md` proposes making the preamble faster. + * Nothing in the platform could previously tell whether such a change worked: + * the only instrument was a `turn_prelude` log line read by grep, and the + * recorded baseline is four turns. Against this path's real variance — a cold + * container is 6.7s against a warm 3.75s, an agent-cache miss 1478ms against a + * hit's 0ms — four samples cannot resolve a 100ms improvement. + * + * THE FOURTH DASHBOARD IS A DELIBERATE COST + * ----------------------------------------- + * CloudWatch's free tier is three dashboards and charges $3/month beyond it; + * `observability-platform-dashboard.test.ts` guards the count so the trade is + * always conscious. It was taken here on purpose: this board is read while + * shipping a latency change, side by side with the load-test output, and + * folding it into the AgentCore Runtime dashboard (the first attempt) buried + * the stage breakdown under runtime health widgets that answer an unrelated + * question. $3/month against a 450-900ms wait on every turn is not a close + * call. The AgentCore dashboard's AWS-reported `Latency` graph remains the + * natural companion — it measures at the data plane, so the gap between it and + * `PreludeTotalMs` is another read on the routing overhead no server-side stage + * can see — and the platform dashboard links to both. + * + * WHAT IT DELIBERATELY DOES NOT DO + * -------------------------------- + * These are percentile widgets, not sums. The metrics are dimension-less + * (matching every other EMF caller here); `isResume` / `deferredBuild` / + * `sessionId` ride as queryable log properties, so the Logs Insights widgets + * below do the slicing instead of dimension fan-out. That is not only about + * metric-stream cost: a dimension invites reading a p99 off a slice too thin to + * have one. + * + * **No alarms.** An alarm needs a threshold, there is no baseline yet, and + * inventing one is the exact guessing the spec this supports exists to prevent. + * Add them once the dashboard has run long enough to say what normal is. + * + * **These metrics start at handler entry.** They cannot see the app-api hop, + * auth, or Runtime routing — ~478ms warm, ~1.5s cold, up to 20% of the turn. + * Only a client-side measurement covers that, which is what `tests/load` is + * for. A dashboard that falls while the user's wait does not is a real + * possibility, and the header widget says so on the dashboard itself rather + * than only here. + */ +export class TurnLatencyObservabilityConstruct extends Construct { + public readonly dashboard: cloudwatch.Dashboard; + + constructor( + scope: Construct, + id: string, + props: TurnLatencyObservabilityConstructProps, + ) { + super(scope, id); + + const { config, runtimeLogGroupName } = props; + + // Must match TURN_LATENCY_EMF_NAMESPACE's default in turn_timing.py. + // Deliberately not set from CDK, for the same reason the prompt-cache + // namespace is not: dev and prod are separate AWS accounts, so an unscoped + // namespace cannot collide. The default IS the contract, and a drift here + // renders empty graphs — which look exactly like "no traffic". + const namespace = 'AgentCoreStack/TurnLatency'; + + const PERIOD = cdk.Duration.minutes(5); + + /** One stage at one statistic. Latency wants percentiles, never Sum. */ + const stage = (metricName: string, statistic: string, label?: string) => + new cloudwatch.Metric({ + namespace, + metricName, + statistic, + period: PERIOD, + label: label ?? `${metricName} ${statistic}`, + }); + + const percentiles = (metricName: string) => [ + stage(metricName, 'p50'), + stage(metricName, 'p90'), + stage(metricName, 'p99'), + ]; + + const preambleSubStages = (statistic: string) => [ + stage('PreambleOwnershipMs', statistic, 'ownership'), + stage('PreambleSkillsMs', statistic, 'skills'), + stage('PreambleFilesMs', statistic, 'files'), + stage('PreambleSessionStateMs', statistic, 'session_state'), + stage('PreambleQuotaMs', statistic, 'quota'), + ]; + + this.dashboard = new cloudwatch.Dashboard(this, 'TurnLatencyDashboard', { + dashboardName: getResourceName(config, 'turn-latency-observability'), + defaultInterval: cdk.Duration.hours(3), + }); + + this.dashboard.addWidgets( + new cloudwatch.TextWidget({ + markdown: [ + '# Turn Latency — pre-stream stages', + `**Project:** ${config.projectPrefix} | **Region:** ${config.awsRegion} | ` + + `**Namespace:** \`${namespace}\``, + '', + '_Every metric here starts at **inference-api handler entry**. None of them sees ' + + 'the app-api hop, auth, or AgentCore Runtime routing — ~478ms on a warm path and ' + + '~1.5s on a cold one. A stage falling here is necessary but **not sufficient** ' + + 'evidence that the user waits less; the client-side check is `tests/load` ' + + '(Locust TTFB + TTFT). See `docs/specs/turn-latency-preamble.md`._', + '', + '_`PreambleMs` is the sum of the five `Preamble*Ms` sub-stages, kept so the ' + + 'pre-split baseline stays comparable. AWS\'s own data-plane `Latency` for the ' + + `runtime is on the **${getResourceName(config, 'agentcore-observability')}** ` + + 'dashboard — the gap between it and `PreludeTotalMs` is the routing overhead._', + ].join('\n'), + width: 24, + height: 5, + }), + ); + + // Row 1 — the two numbers that decide whether a change worked. + this.dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Prelude total (handler entry → agent ready)', + left: percentiles('PreludeTotalMs'), + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Preamble (all five sub-stages) — PR-2 moves this or it did not work', + left: percentiles('PreambleMs'), + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + ); + + // Row 2 — which sub-stage owns the preamble. Split by statistic rather than + // stacking fifteen lines on one axis, which is a picture nobody reads. + this.dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Preamble breakdown (p50) — which sub-stage owns it', + left: preambleSubStages('p50'), + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Preamble breakdown (p90) — where the tail lives', + left: preambleSubStages('p90'), + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + ); + + // Row 3 — the stages after the preamble. `AgentBuildMs` is bimodal by + // construction (cache hit ≈ 0, miss ≈ 1.5-2.7s), so its p50 and p99 + // describe two different populations and the gap between them is the + // agent-cache hit rate showing through. + this.dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Agent build — the p50/p99 spread is the cache hit/miss split', + left: percentiles('AgentBuildMs'), + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'RAG + tool assembly (p90)', + left: [stage('RagMs', 'p90', 'rag'), stage('ToolsMs', 'p90', 'tools')], + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + ); + + // Row 3b — inside the agent build. `agent_build` is the largest number + // left in the prelude (~2950ms cold against 1-47ms warm) and nothing said + // which part of it that was; these are the same decomposition move that + // opened the preamble. `AgentBuildMs` above is their sum, so the pre-split + // series stays comparable. + // + // p90 rather than p50: a warm build is ~0 across the board, so p50 would + // be a row of flat lines. The cold builds — the ones worth fixing — live + // in the upper percentiles by construction. + this.dashboard.addWidgets( + new cloudwatch.GraphWidget({ + title: 'Agent build breakdown (p90) — which part of a cold build is slow', + left: [ + stage('AgentBuildPromptMs', 'p90', 'system prompt'), + stage('AgentBuildRegistryMs', 'p90', 'tool registry'), + stage('AgentBuildSessionMgrMs', 'p90', 'session manager (memory restore)'), + stage('AgentBuildToolsMs', 'p90', 'tools (incl. MCP pre-flight)'), + ], + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + new cloudwatch.GraphWidget({ + title: 'Agent build breakdown, part 2 (p90)', + left: [ + stage('AgentBuildHooksMs', 'p90', 'hooks'), + stage('AgentBuildPluginsMs', 'p90', 'plugins'), + stage('AgentBuildFinalizeMs', 'p90', 'finalize'), + stage('AgentBuildRestMs', 'p90', 'remainder'), + ], + leftYAxis: { min: 0 }, + width: 12, + height: 6, + }), + ); + + // Row 4 — the slicing that dimensions would have done, done in Logs + // Insights instead. A resume skips most of the preamble, so mixing the two + // populations is what would make a traffic-mix shift look like a latency + // win. + this.dashboard.addWidgets( + new cloudwatch.LogQueryWidget({ + title: 'Preamble by turn shape (resume turns skip most of it)', + logGroupNames: [runtimeLogGroupName], + queryLines: [ + 'filter ispresent(PreambleMs)', + 'stats count(*) as turns, pct(PreambleMs, 50) as p50, ' + + 'pct(PreambleMs, 90) as p90 by isResume, deferredBuild', + 'sort turns desc', + ], + width: 12, + height: 6, + }), + // The honesty check on the decomposition itself. If the stages do not + // account for the total, time is being spent between them, in code the + // spec has not modelled — and that gap is the next thing to chase. + new cloudwatch.LogQueryWidget({ + title: 'Unaccounted prelude time (total − preamble − rag − tools − agent_build)', + logGroupNames: [runtimeLogGroupName], + queryLines: [ + 'filter ispresent(PreludeTotalMs)', + 'fields PreludeTotalMs - (PreambleMs + RagMs + ToolsMs + AgentBuildMs) as unaccountedMs', + 'stats count(*) as turns, avg(unaccountedMs) as avgUnaccounted, ' + + 'pct(unaccountedMs, 90) as p90Unaccounted', + ], + width: 12, + height: 6, + }), + ); + } +} diff --git a/infrastructure/lib/constructs/platform-compute-refs.ts b/infrastructure/lib/constructs/platform-compute-refs.ts index bc0f23b11..76f946c9c 100644 --- a/infrastructure/lib/constructs/platform-compute-refs.ts +++ b/infrastructure/lib/constructs/platform-compute-refs.ts @@ -86,6 +86,7 @@ export interface PlatformComputeRefs { userMenuLinksTable: dynamodb.ITable; announcementsTable: dynamodb.ITable; systemPromptsTable: dynamodb.ITable; + agentTemplatesTable: dynamodb.ITable; sharedConversationsTable: dynamodb.ITable; sharedConversationsBucket: s3.IBucket; fileUploadBucket: s3.IBucket; @@ -130,6 +131,15 @@ export interface PlatformComputeRefs { agentCoreCodeInterpreterId: string; agentCoreBrowserArn: string; agentCoreBrowserId: string; + /** + * S3 location of the Chromium MANAGED policy every browser session is + * started with (docs/specs/authenticated-web-assessment.md D6). Passed to + * `StartBrowserSession` rather than attached to the browser resource: the + * resource is immutable (no UpdateBrowser) and CfnBrowserCustom does not + * expose enterprisePolicies. + */ + browserPolicyBucketName: string; + browserPolicyKey: string; // ── MCP sandbox edge mcpSandboxProxyOrigin: string; diff --git a/infrastructure/lib/constructs/spa/spa-distribution-construct.ts b/infrastructure/lib/constructs/spa/spa-distribution-construct.ts index 7007a520d..33486ad80 100644 --- a/infrastructure/lib/constructs/spa/spa-distribution-construct.ts +++ b/infrastructure/lib/constructs/spa/spa-distribution-construct.ts @@ -49,6 +49,18 @@ export interface SpaDistributionConstructProps { * ALL_VIEWER_EXCEPT_HOST_HEADER pass cookies + CSRF + auth headers * untouched. compress=false to preserve `text/event-stream`. * + * compress=false does NOT mean `/api/*` responses travel uncompressed: + * it means CloudFront doesn't compress *for* us. app-api gzips its own + * JSON (`apis/shared/middleware/compression.py`), where the response's + * content type is known rather than guessed from a path pattern, and + * CloudFront passes an origin's `Content-Encoding` straight through. + * Accept-Encoding reaches the origin because CACHING_DISABLED leaves + * EnableAcceptEncodingGzip/Brotli off — with both off, CloudFront + * treats Accept-Encoding as an ordinary header, and + * ALL_VIEWER_EXCEPT_HOST_HEADER forwards it verbatim. Turning + * compress=true on would put the edge back in front of the SSE stream + * and gain nothing the origin isn't already doing. + * * Security headers: * - X-Content-Type-Options, X-Frame-Options=DENY (default-deny iframe * embedding), Referrer-Policy=strict-origin-when-cross-origin, HSTS diff --git a/infrastructure/lib/platform-stack.ts b/infrastructure/lib/platform-stack.ts index 042db7383..27d6e68e6 100644 --- a/infrastructure/lib/platform-stack.ts +++ b/infrastructure/lib/platform-stack.ts @@ -70,6 +70,7 @@ import { MemorySpacesConstruct } from './constructs/memory/memory-spaces-constru import { AgentCoreMemoryConstruct } from './constructs/agentcore/memory-construct'; import { AgentCoreCodeInterpreterConstruct } from './constructs/agentcore/code-interpreter-construct'; import { AgentCoreBrowserConstruct } from './constructs/agentcore/browser-construct'; +import { AgentCoreBrowserPolicyConstruct } from './constructs/agentcore/browser-policy-construct'; import { AgentCoreGatewayConstruct } from './constructs/gateway/agentcore-gateway-construct'; // MCP sandbox (S3 + CloudFront — Platform edge surface) @@ -87,6 +88,7 @@ import { PlatformDashboardConstruct } from './constructs/observability/platform- import { LambdaAlarmsConstruct } from './constructs/observability/lambda-alarms-construct'; import { EcsServiceAlarmsConstruct } from './constructs/observability/ecs-service-alarms-construct'; import { PromptCacheObservabilityConstruct } from './constructs/observability/prompt-cache-observability-construct'; +import { TurnLatencyObservabilityConstruct } from './constructs/observability/turn-latency-observability-construct'; // Fine-tuning (data half lives in Platform) import { FineTuningDataConstruct } from './constructs/fine-tuning/fine-tuning-data-construct'; @@ -185,6 +187,7 @@ export class PlatformStack extends cdk.Stack { public readonly userMenuLinksTable: dynamodb.ITable; public readonly announcementsTable: dynamodb.ITable; public readonly systemPromptsTable: dynamodb.ITable; + public readonly agentTemplatesTable: dynamodb.ITable; public readonly sharedConversationsTable: dynamodb.ITable; public readonly sharedConversationsBucket: s3.IBucket; public readonly fileUploadBucket: s3.IBucket; @@ -256,6 +259,8 @@ export class PlatformStack extends cdk.Stack { public readonly agentCoreCodeInterpreterId: string; public readonly agentCoreBrowser: bedrock.CfnBrowserCustom; public readonly agentCoreBrowserArn: string; + public readonly browserPolicyBucketName: string; + public readonly browserPolicyKey: string; public readonly agentCoreBrowserId: string; // ── Internal handles for the two-step wiring methods @@ -426,6 +431,7 @@ export class PlatformStack extends cdk.Stack { this.userMenuLinksTable = adminTables.userMenuLinksTable; this.announcementsTable = adminTables.announcementsTable; this.systemPromptsTable = adminTables.systemPromptsTable; + this.agentTemplatesTable = adminTables.agentTemplatesTable; const fileUpload = new FileUploadConstruct(this, 'FileUpload', { config }); this.fileUploadBucket = fileUpload.bucket; @@ -683,6 +689,20 @@ export class PlatformStack extends cdk.Stack { this.agentCoreBrowserArn = agentCoreBrowserConstruct.browserArn; this.agentCoreBrowserId = agentCoreBrowserConstruct.browserId; + // The Chromium managed policy every browser session is started with. + // Deliberately NOT attached to the browser resource: there is no + // UpdateBrowser, policies are read from S3 at API-call time and frozen + // thereafter, and CfnBrowserCustom does not expose enterprisePolicies at + // all. inference-api passes the reference on each StartBrowserSession. + // See docs/specs/authenticated-web-assessment.md D6. + const browserPolicy = new AgentCoreBrowserPolicyConstruct( + this, + 'AgentCoreBrowserPolicy', + { config, browserExecutionRole: agentCoreBrowserConstruct.executionRole }, + ); + this.browserPolicyBucketName = browserPolicy.bucket.bucketName; + this.browserPolicyKey = browserPolicy.policyKey; + // AgentCore Gateway — config-only (MCP protocol, AWS_IAM // authorizer, IAM execution role with invoke rights against the // /^${prefix}-mcp-/ Lambda naming convention used by the @@ -830,6 +850,7 @@ export class PlatformStack extends cdk.Stack { userMenuLinksTable: this.userMenuLinksTable, announcementsTable: this.announcementsTable, systemPromptsTable: this.systemPromptsTable, + agentTemplatesTable: this.agentTemplatesTable, sharedConversationsTable: this.sharedConversationsTable, sharedConversationsBucket: this.sharedConversationsBucket, fileUploadBucket: this.fileUploadBucket, @@ -856,6 +877,8 @@ export class PlatformStack extends cdk.Stack { agentCoreCodeInterpreterId: this.agentCoreCodeInterpreterId, agentCoreBrowserArn: this.agentCoreBrowserArn, agentCoreBrowserId: this.agentCoreBrowserId, + browserPolicyBucketName: this.browserPolicyBucketName, + browserPolicyKey: this.browserPolicyKey, mcpSandboxProxyOrigin: this.mcpSandboxProxyOrigin, }; @@ -869,6 +892,8 @@ export class PlatformStack extends cdk.Stack { codeInterpreterId: this.agentCoreCodeInterpreterId, browserArn: this.agentCoreBrowserArn, browserId: this.agentCoreBrowserId, + browserPolicyBucketName: this.browserPolicyBucketName, + browserPolicyKey: this.browserPolicyKey, }); // Cross-service dashboard + alarms over the EMF metrics both APIs emit @@ -881,6 +906,17 @@ export class PlatformStack extends cdk.Stack { runtimeLogGroupName: inferenceApi.runtimeLogGroupName, }); + // Percentiles over the pre-stream stages `turn_timing.py` emits. Sits + // beside the prompt-cache dashboard and for the same reason — it needs the + // runtime's service-created log group name — but in its own namespace and + // its own dashboard: that one answers "what did this cost", this one + // answers "how long did the user wait". The fourth dashboard is $3/month + // beyond CloudWatch's free three, taken deliberately; see the construct. + new TurnLatencyObservabilityConstruct(this, 'TurnLatencyObservability', { + config: this._config, + runtimeLogGroupName: inferenceApi.runtimeLogGroupName, + }); + const sagemakerPrivateSubnetIds = this.vpc.privateSubnets .map((s) => s.subnetId) .join(','); @@ -898,6 +934,7 @@ export class PlatformStack extends cdk.Stack { agentCoreMemoryArn: this.agentCoreMemoryArn, agentCoreMemoryId: this.agentCoreMemoryId, inferenceApiRuntimeEndpointUrl: inferenceApi.runtimeEndpointUrl, + agentCoreRuntimeLogGroupName: inferenceApi.runtimeLogGroupName, artifactsOrigin: this.artifactsOriginUrl, sagemakerExecutionRoleArn: sagemaker.executionRole.roleArn, sagemakerSecurityGroupId: sagemaker.securityGroup.securityGroupId, @@ -1005,6 +1042,7 @@ export class PlatformStack extends cdk.Stack { { name: 'user-menu-links', table: this.userMenuLinksTable }, { name: 'announcements', table: this.announcementsTable }, { name: 'system-prompts', table: this.systemPromptsTable }, + { name: 'agent-templates', table: this.agentTemplatesTable }, { name: 'shared-conversations', table: this.sharedConversationsTable }, { name: 'user-file-uploads', table: this.fileUploadTable }, { name: 'rag-assistants', table: this.ragAssistantsTable }, diff --git a/infrastructure/package-lock.json b/infrastructure/package-lock.json index 5f9634d68..5588c2640 100644 --- a/infrastructure/package-lock.json +++ b/infrastructure/package-lock.json @@ -1,12 +1,12 @@ { "name": "infrastructure", - "version": "1.22.0", + "version": "1.23.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "infrastructure", - "version": "1.22.0", + "version": "1.23.0", "dependencies": { "aws-cdk-lib": "2.265.0", "constructs": "10.6.0" diff --git a/infrastructure/package.json b/infrastructure/package.json index 4503d8f14..1889efd94 100644 --- a/infrastructure/package.json +++ b/infrastructure/package.json @@ -1,6 +1,6 @@ { "name": "infrastructure", - "version": "1.22.0", + "version": "1.23.0", "bin": { "infrastructure": "bin/infrastructure.js" }, diff --git a/infrastructure/test/api-security-headers.test.ts b/infrastructure/test/api-security-headers.test.ts index da48fa539..f9c876130 100644 --- a/infrastructure/test/api-security-headers.test.ts +++ b/infrastructure/test/api-security-headers.test.ts @@ -27,6 +27,7 @@ function seedRequiredContext(app: cdk.App): void { app.node.setContext('awsRegion', MOCK_REGION); app.node.setContext('awsAccount', MOCK_ACCOUNT); app.node.setContext('vpcCidr', '10.0.0.0/16'); + app.node.setContext('corsOrigins', 'http://localhost:4200'); app.node.setContext('production', false); app.node.setContext('retainDataOnDelete', false); app.node.setContext('frontend', { cloudFrontPriceClass: 'PriceClass_100' }); diff --git a/infrastructure/test/app-api-environment.test.ts b/infrastructure/test/app-api-environment.test.ts index 7a69d93d3..4ebac20b6 100644 --- a/infrastructure/test/app-api-environment.test.ts +++ b/infrastructure/test/app-api-environment.test.ts @@ -54,6 +54,21 @@ describe('buildAppApiEnvironment — Memory Spaces', () => { it('wires the tabl }); }); +/** + * Agent Templates: the admin CRUD routes and the public `/templates` picker + * feed read the table name from `DYNAMODB_AGENT_TEMPLATES_TABLE_NAME`. Mirrors + * the system-prompts wiring. Read by app-api only — inference-api never sets it. + */ +describe('buildAppApiEnvironment — Agent Templates', () => { + it('wires DYNAMODB_AGENT_TEMPLATES_TABLE_NAME from the resolved table name', () => { + const env = buildAppApiEnvironment( + createMockConfig(), + stubParams({ agentTemplatesTableName: 'test-project-agent-templates' }), + ); + expect(env.DYNAMODB_AGENT_TEMPLATES_TABLE_NAME).toBe('test-project-agent-templates'); + }); +}); + /** * Guards the owner-facing upgrade offer's flag wiring. * diff --git a/infrastructure/test/app-api-sizing-config.test.ts b/infrastructure/test/app-api-sizing-config.test.ts index 4bc99cc27..98dc705ae 100644 --- a/infrastructure/test/app-api-sizing-config.test.ts +++ b/infrastructure/test/app-api-sizing-config.test.ts @@ -13,6 +13,7 @@ const BASE: Record = { awsRegion: 'us-west-2', awsAccount: '123456789012', vpcCidr: '10.0.0.0/16', + corsOrigins: 'http://localhost:4200', production: false, retainDataOnDelete: false, frontend: { cloudFrontPriceClass: 'PriceClass_100' }, diff --git a/infrastructure/test/browser-live-view-grants.test.ts b/infrastructure/test/browser-live-view-grants.test.ts new file mode 100644 index 000000000..e7a3a7edc --- /dev/null +++ b/infrastructure/test/browser-live-view-grants.test.ts @@ -0,0 +1,136 @@ +/** + * app-api's AgentCore Browser grant for the Live View route. + * + * A Live View URL is SigV4 query-signed and lives at most 300 seconds, so it + * cannot be minted once by the agent and reused — app-api signs a fresh one + * per request (`docs/specs/authenticated-web-assessment.md` D2), which is why + * the task role needs browser permissions at all. + * + * The thing these tests exist to hold down is the *narrowness* of that grant. + * The obvious lazy fix — copy the Runtime's `BrowserAccess` statement onto the + * task role — would let app-api start, stop and **drive** browser sessions. + * app-api is the surface reachable with a user's session cookie; the agent + * owns the browser lifecycle, and app-api must only ever be able to look. + */ +import * as cdk from 'aws-cdk-lib'; +import { Template } from 'aws-cdk-lib/assertions'; +import { PlatformStack } from '../lib/platform-stack'; +import { createMockConfig, mockSsmContext, MOCK_ACCOUNT, MOCK_REGION } from './helpers/mock-config'; + +interface PolicyStatement { + Effect?: string; + Action?: string | string[]; + Resource?: string | string[]; + Sid?: string; +} + +function asArray(v: string | string[] | undefined): string[] { + if (v === undefined) return []; + return Array.isArray(v) ? v : [v]; +} + +describe('app-api browser Live View grant', () => { + let statementsBySid: Map; + + beforeAll(() => { + const config = createMockConfig(); + const app = new cdk.App(); + mockSsmContext(app, config); + const stack = new PlatformStack(app, 'TestPlatformStack', { + config, + env: { account: MOCK_ACCOUNT, region: MOCK_REGION }, + }); + stack.wireCompute(); + const template = Template.fromStack(stack); + + // CDK splits an oversized inline policy into managed overflow policies + // attached to the same role, so both resource types must be scanned. + statementsBySid = new Map(); + for (const type of ['AWS::IAM::Policy', 'AWS::IAM::ManagedPolicy']) { + for (const [, r] of Object.entries(template.findResources(type))) { + const stmts = + (r.Properties as { PolicyDocument?: { Statement?: PolicyStatement[] } }) + ?.PolicyDocument?.Statement ?? []; + for (const s of stmts) { + if (!s.Sid) continue; + statementsBySid.set(s.Sid, [...(statementsBySid.get(s.Sid) ?? []), s]); + } + } + } + }); + + it('grants app-api exactly what minting a live view needs', () => { + const matches = statementsBySid.get('BrowserLiveViewAccess') ?? []; + expect(matches).toHaveLength(1); + + const [statement] = matches; + expect(statement.Effect).toBe('Allow'); + expect(asArray(statement.Action).sort()).toEqual([ + 'bedrock-agentcore:GetBrowserSession', + 'bedrock-agentcore:UpdateBrowserStream', + ]); + }); + + it('grants ConnectBrowserLiveViewStream on *, because it has no resource type', () => { + // AWS's service reference lists NO resource types for this action, so a + // resource-scoped statement never matches it and it is an implicit deny. + // That is not a theoretical concern: it shipped that way, and because + // `generate_live_view_url` signs locally without calling AWS, the URL + // minted fine and the failure surfaced only as the browser's WebSocket + // being closed — DCV auth code 10, which looks like a service fault. + const matches = statementsBySid.get('BrowserLiveViewConnect') ?? []; + expect(matches).toHaveLength(1); + + const [statement] = matches; + expect(statement.Effect).toBe('Allow'); + expect(asArray(statement.Resource)).toEqual(['*']); + + // `*` is unavoidable here, so the statement must carry NOTHING else. + expect(asArray(statement.Action)).toEqual([ + 'bedrock-agentcore:ConnectBrowserLiveViewStream', + ]); + }); + + it('does NOT let app-api start, stop or drive a browser', () => { + // Both statements together: splitting the grant must not have opened a + // side door in the one that had to widen to `*`. + const actions = [ + ...asArray(statementsBySid.get('BrowserLiveViewAccess')?.[0]?.Action), + ...asArray(statementsBySid.get('BrowserLiveViewConnect')?.[0]?.Action), + ]; + + // Each of these is on the Runtime's own BrowserAccess statement and must + // stay off app-api's: the agent owns the session lifecycle, and + // ConnectBrowserAutomationStream is the one that would let a route + // reachable with a session cookie actually drive the browser. + for (const forbidden of [ + 'bedrock-agentcore:StartBrowserSession', + 'bedrock-agentcore:StopBrowserSession', + 'bedrock-agentcore:ConnectBrowserAutomationStream', + ]) { + expect(actions).not.toContain(forbidden); + } + expect(actions).not.toContain('bedrock-agentcore:*'); + }); + + it('keeps the resource-typed actions scoped to the browser, not to *', () => { + const [statement] = statementsBySid.get('BrowserLiveViewAccess') ?? []; + + const resources = asArray(statement?.Resource); + expect(resources).toHaveLength(1); + expect(resources[0]).not.toBe('*'); + }); + + it('leaves the Runtime role its own, wider grant', () => { + // The narrowing above must not have been achieved by editing the shared + // statement the agent depends on to browse at all. + const [runtime] = statementsBySid.get('BrowserAccess') ?? []; + + expect(asArray(runtime?.Action)).toEqual( + expect.arrayContaining([ + 'bedrock-agentcore:StartBrowserSession', + 'bedrock-agentcore:ConnectBrowserAutomationStream', + ]), + ); + }); +}); diff --git a/infrastructure/test/browser-live-view-viewer.test.ts b/infrastructure/test/browser-live-view-viewer.test.ts new file mode 100644 index 000000000..d3319b493 --- /dev/null +++ b/infrastructure/test/browser-live-view-viewer.test.ts @@ -0,0 +1,269 @@ +/** + * The live-view viewer (`assets/mcp-sandbox/live-view.js`) runs the real file + * in a `vm` context against a minimal DOM stub. No jsdom: the script touches + * only `document.getElementById`, `document.referrer`, `window.parent` and + * `addEventListener`, so a stub is both sufficient and honest about what the + * page depends on. + * + * What this guards is a defect measured on deployed dev: the SPA posts the + * minted URL up to THREE times (after minting, on the iframe's `load`, and in + * reply to the viewer's `ready`) because it cannot know which fires first. + * The viewer originally guarded on an established `connection`, which is + * assigned only after `dcv.connect()` resolves — so two posts milliseconds + * apart both called `dcv.authenticate`, the second socket's open closed the + * first, and the stream died with `Close received after close` / auth code 10. + */ +import * as fs from 'fs'; +import * as path from 'path'; +import * as vm from 'vm'; + +const SOURCE = fs.readFileSync( + path.join(__dirname, '..', 'assets', 'mcp-sandbox', 'live-view.js'), + 'utf8', +); + +const PARENT = 'https://app.example.test'; +const SIGNED = 'https://bedrock-agentcore.us-west-2.amazonaws.com/live-view?X-Amz-Signature=abc'; + +interface Harness { + authCalls: unknown[]; + connectCalls: any[]; + post(url?: string): void; + postFrom(origin: string, url?: string): void; + fireAuthError(): void; + extraSearchParams(): URLSearchParams; + status(): string; + failAuth(): void; + succeedAuth(): void; +} + +function load(): Harness { + const listeners: ((e: unknown) => void)[] = []; + const authCalls: unknown[] = []; + const connectCalls: any[] = []; + const statusEl = { textContent: '', hidden: false }; + let authConfig: any = null; + + const sandbox: any = { + URL, + URLSearchParams, + console: { error() {}, info() {} }, + document: { + referrer: `${PARENT}/thread`, + getElementById: (id: string) => (id === 'status' ? statusEl : { }), + }, + }; + sandbox.window = sandbox; + sandbox.window.parent = { postMessage() {} }; + sandbox.addEventListener = (type: string, fn: (e: unknown) => void) => { + if (type === 'message') listeners.push(fn); + }; + sandbox.dcv = { + LogLevel: { WARN: 'warn' }, + setLogLevel() {}, + authenticate(url: string, cfg: any) { + authCalls.push(url); + authConfig = cfg; + }, + connect(cfg: any) { + connectCalls.push(cfg); + return new Promise(() => {}); // never settles: mid-connect is the window under test + }, + }; + + vm.createContext(sandbox); + vm.runInContext(SOURCE, sandbox); + + return { + authCalls, + connectCalls, + post(url = SIGNED) { + this.postFrom(PARENT, url); + }, + postFrom(origin: string, url = SIGNED) { + const event = { + origin, + data: { type: 'browser-live-view/connect', url, viewport: { width: 1280, height: 800 } }, + }; + listeners.forEach((fn) => fn(event)); + }, + status: () => (statusEl.hidden ? '' : statusEl.textContent), + fireAuthError: () => authConfig.error({}, { code: 10 }), + extraSearchParams: () => authConfig.httpExtraSearchParams() as URLSearchParams, + failAuth: () => authConfig.error({}, { code: 10 }), + succeedAuth: () => authConfig.success({}, [{ sessionId: 's', authToken: 't' }]), + }; +} + +describe('browser live-view viewer', () => { + it('authenticates once even when the parent posts three times', () => { + const h = load(); + + h.post(); + h.post(); + h.post(); + + // Three posts, one stream. A second `authenticate` here is the exact + // defect that killed the session on dev. + expect(h.authCalls).toHaveLength(1); + }); + + it('still only authenticates once while the first connect is in flight', () => { + const h = load(); + + h.post(); + h.succeedAuth(); // connect() called, promise deliberately never resolves + h.post(); + + expect(h.authCalls).toHaveLength(1); + }); + + it('lets the user retry after a failed attempt rather than latching shut', () => { + const h = load(); + + h.post(); + h.failAuth(); + h.post(); + + // The latch must release on every terminal outcome, or one bad mint makes + // the viewer permanently dead for that session. + expect(h.authCalls).toHaveLength(2); + }); + + it('hands the SDK a URL with NO query string', () => { + const h = load(); + + h.post(); + + // The SDK APPENDS `httpExtraSearchParams` to whatever URL it is given. + // Passing the signed URL with its query still attached sends every SigV4 + // parameter twice, which the service answers with 403 — measured. That is + // why the viewer never streamed on dev. + expect(h.authCalls).toHaveLength(1); + expect(String(h.authCalls[0])).not.toContain('?'); + expect(String(h.authCalls[0])).not.toContain('X-Amz-Signature'); + }); + + it('still forwards the signature through httpExtraSearchParams', () => { + const h = load(); + + h.post(); + + // Stripping the query must not mean dropping it: the transport reads it + // from this callback when it builds the socket URI. + const params = h.extraSearchParams(); + expect(params.get('X-Amz-Signature')).toBe('abc'); + }); + + it('puts httpExtraSearchParams in observers, where connect actually reads it', () => { + const h = load(); + h.post(); + h.succeedAuth(); + + // Measured against the shipped bundle: at the TOP LEVEL this callback is + // silently ignored by `connect`, and the stream socket opens with no query + // at all — unsigned, and refused by the service. Inside `observers` it is + // honoured. (`authenticate` is the opposite and reads it top-level.) + const cfg = h.connectCalls[0]; + expect(typeof cfg.observers?.httpExtraSearchParams).toBe('function'); + expect(cfg.httpExtraSearchParams).toBeUndefined(); + + // Observers or callbacks, never both — the SDK ignores one if both appear. + expect(cfg.callbacks).toBeUndefined(); + expect(typeof cfg.observers.firstFrame).toBe('function'); + expect(typeof cfg.observers.disconnect).toBe('function'); + }); + + it('sizes the display only once the first frame has arrived', () => { + const h = load(); + h.post(); + h.succeedAuth(); + + const cfg = h.connectCalls[0]; + const conn = { + requestDisplayLayout: jest.fn((_layout: any[]) => Promise.resolve()), + }; + + // Nothing may be requested before the display channel exists: calling it + // when the connect promise resolves rejects with "Display channel is not + // available", and since it returns a PROMISE a try/catch never sees it — + // it lands as an unhandled rejection. Measured on dev. + expect(conn.requestDisplayLayout).not.toHaveBeenCalled(); + + cfg.observers.firstFrame(conn); + + expect(conn.requestDisplayLayout).toHaveBeenCalledTimes(1); + const layout = conn.requestDisplayLayout.mock.calls[0][0]; + expect(layout[0].rect).toEqual({ x: 0, y: 0, width: 1280, height: 800 }); + }); + + it('swallows a rejected display-layout request rather than leaking it', async () => { + const h = load(); + h.post(); + h.succeedAuth(); + + const conn = { + requestDisplayLayout: () => Promise.reject(new Error('Display channel is not available')), + }; + + // The stream is still usable at whatever size the server chose, so this + // must not become an unhandled rejection. + expect(() => h.connectCalls[0].observers.firstFrame(conn)).not.toThrow(); + await new Promise((r) => setTimeout(r, 0)); + }); + + it('does not report failure for an auth error that lands AFTER success', () => { + const h = load(); + h.post(); + h.succeedAuth(); + h.connectCalls[0].observers.firstFrame({}); + + // The SDK closes the auth socket once it is done and reports that close + // through `error` even though authentication SUCCEEDED — measured on dev, + // where `connect` is already running when code 10 arrives and the stream + // then establishes normally. #status is an absolutely-positioned overlay, + // so surfacing this would paint an error across a working stream. + h.fireAuthError(); + + expect(h.status()).toBe(''); + }); + + it('still reports failure when auth fails before any session exists', () => { + const h = load(); + h.post(); + + h.fireAuthError(); + + expect(h.status()).toContain('Could not start the session'); + }); + + it('ignores a connect message from any other origin', () => { + const h = load(); + + // A sibling frame must not be able to drive this viewer onto a stream of + // its choosing — this is the page where a password gets typed. + h.postFrom('https://evil.example'); + + expect(h.authCalls).toHaveLength(0); + }); + + it('refuses to render when it was not framed', () => { + const listeners: ((e: unknown) => void)[] = []; + const statusEl = { textContent: '', hidden: false }; + const sandbox: any = { + URL, URLSearchParams, console: { error() {}, info() {} }, + document: { referrer: '', getElementById: () => statusEl }, + }; + sandbox.window = sandbox; + sandbox.window.parent = sandbox; // top-level: parent === window + sandbox.addEventListener = (t: string, fn: (e: unknown) => void) => { + if (t === 'message') listeners.push(fn); + }; + sandbox.dcv = { LogLevel: { WARN: 'w' }, setLogLevel() {}, authenticate() { throw new Error('must not authenticate'); } }; + vm.createContext(sandbox); + vm.runInContext(SOURCE, sandbox); + + expect(statusEl.textContent).toContain('must be opened from the app'); + expect(listeners).toHaveLength(0); + }); +}); diff --git a/infrastructure/test/browser-url-policy.test.ts b/infrastructure/test/browser-url-policy.test.ts new file mode 100644 index 000000000..a07ee7bfc --- /dev/null +++ b/infrastructure/test/browser-url-policy.test.ts @@ -0,0 +1,156 @@ +/** + * The Chromium URL policy the browser is started with (spec D6). + * + * ⚠️ Applied at session level it is RECOMMENDED-only — the service rejects + * MANAGED — so it constrains the agent but not a human in a takeover. The + * un-overridable version needs `CreateBrowser`. + * + * These tests care about two things regardless: that the document says what we + * think it says, and that the browser can actually read it. + */ +import * as cdk from 'aws-cdk-lib'; +import { Template } from 'aws-cdk-lib/assertions'; +import { PlatformStack } from '../lib/platform-stack'; +import { + buildBrowserManagedPolicy, + BROWSER_MANAGED_POLICY_KEY, +} from '../lib/constructs/agentcore/browser-policy-construct'; +import { createMockConfig, mockSsmContext, MOCK_ACCOUNT, MOCK_REGION } from './helpers/mock-config'; + +describe('buildBrowserManagedPolicy', () => { + it('renders a Chromium URLBlocklist document', () => { + expect(buildBrowserManagedPolicy(['a.example.com', 'b.example.com'])).toEqual({ + URLBlocklist: ['a.example.com', 'b.example.com'], + }); + }); + + it('is a blocklist, never an allowlist', () => { + // An allowlist of assessment targets would need a new entry per VPAT + // review, and the browser resource is immutable — that design was + // rejected in D6 for exactly that reason. + const policy = buildBrowserManagedPolicy(['x.example.com']); + + expect(policy).not.toHaveProperty('URLAllowlist'); + }); + + it('copies the list rather than aliasing the config array', () => { + const list = ['a.example.com']; + const policy = buildBrowserManagedPolicy(list) as { URLBlocklist: string[] }; + list.push('mutated.example.com'); + + expect(policy.URLBlocklist).toEqual(['a.example.com']); + }); + + it('renders an empty blocklist rather than omitting the key', () => { + // An absent URLBlocklist and an empty one mean the same thing to Chromium, + // but the explicit key makes "this environment blocks nothing" visible in + // the deployed object instead of looking like a failed render. + expect(buildBrowserManagedPolicy([])).toEqual({ URLBlocklist: [] }); + }); +}); + +describe('the policy object key', () => { + it('is not double-prefixed', () => { + // Regression: the BucketDeployment's `destinationKeyPrefix` compounded + // with a source name that already carried the prefix, so the object landed + // at `policies/policies/managed-policies.json` while BROWSER_POLICY_S3 + // pointed one level up. The deploy looked clean and the policy silently + // resolved to a missing key. + expect(BROWSER_MANAGED_POLICY_KEY).toBe('policies/managed-policies.json'); + expect(BROWSER_MANAGED_POLICY_KEY).not.toContain('policies/policies'); + }); +}); + +describe('browser policy in the stack', () => { + let template: Template; + + beforeAll(() => { + const config = createMockConfig(); + const app = new cdk.App(); + mockSsmContext(app, config); + const stack = new PlatformStack(app, 'TestPlatformStack', { + config, + env: { account: MOCK_ACCOUNT, region: MOCK_REGION }, + }); + stack.wireCompute(); + template = Template.fromStack(stack); + }); + + it('versions the policy bucket so past states are auditable', () => { + // This is a security control; what the browser was allowed to reach on a + // given date is worth more than the storage. + const buckets = Object.values(template.findResources('AWS::S3::Bucket')); + const policyBucket = buckets.find((b) => + JSON.stringify((b.Properties as { BucketName?: unknown })?.BucketName ?? '').includes( + 'browser-policy', + ), + ); + + expect(policyBucket).toBeDefined(); + expect((policyBucket!.Properties as { VersioningConfiguration?: { Status?: string } }) + .VersioningConfiguration?.Status).toBe('Enabled'); + }); + + it('lets the browser execution role read the policy, and nothing else', () => { + const policies = Object.values(template.findResources('AWS::IAM::Policy')); + const statements = policies.flatMap( + (p) => + ((p.Properties as { PolicyDocument?: { Statement?: { Sid?: string; Action?: string | string[] }[] } }) + ?.PolicyDocument?.Statement) ?? [], + ); + const grant = statements.find((s) => s.Sid === 'BrowserEnterprisePolicyS3Access'); + + expect(grant).toBeDefined(); + const actions = Array.isArray(grant!.Action) ? grant!.Action : [grant!.Action]; + expect(actions.sort()).toEqual(['s3:GetObject', 's3:GetObjectVersion']); + // Read-only: the browser must never be able to rewrite its own policy. + expect(actions.join(',')).not.toMatch(/PutObject|DeleteObject/); + }); + + it('lets the CALLER of StartBrowserSession read the policy object', () => { + // The service reads the policy as the caller — the runtime role — not as + // the browser's execution role, despite the documented prerequisite + // naming the latter. Granting only the browser role produced + // "Access denied to S3 object ... Verify that the caller has permission", + // and that failure takes down EVERY browser session, not just the policy. + // CDK splits an oversized inline policy into managed overflow policies on + // the same role, so both resource types must be scanned. + const statements = ['AWS::IAM::Policy', 'AWS::IAM::ManagedPolicy'].flatMap((type) => + Object.values(template.findResources(type)).flatMap( + (p) => + ((p.Properties as { + PolicyDocument?: { Statement?: { Sid?: string; Action?: string | string[] }[] }; + })?.PolicyDocument?.Statement) ?? [], + ), + ); + + const caller = statements.find((s) => s.Sid === 'BrowserPolicyObjectRead'); + expect(caller).toBeDefined(); + const actions = Array.isArray(caller!.Action) ? caller!.Action : [caller!.Action]; + expect(actions.sort()).toEqual(['s3:GetObject', 's3:GetObjectVersion']); + + // The browser execution role keeps its own grant — the service documents + // it as a prerequisite, and it costs one read-only object. + expect( + statements.find((s) => s.Sid === 'BrowserEnterprisePolicyS3Access'), + ).toBeDefined(); + }); + + it('passes the policy location to the runtime as a single S3 URI', () => { + // Single variable on purpose — the runtime's 50-env-var ceiling is full. + const runtimes = template.findResources('AWS::BedrockAgentCore::Runtime'); + const [runtime] = Object.values(runtimes); + const env = (runtime.Properties as { EnvironmentVariables?: Record }) + .EnvironmentVariables ?? {}; + + expect(env).toHaveProperty('BROWSER_POLICY_S3'); + expect(env).not.toHaveProperty('BROWSER_POLICY_BUCKET'); + + // The URI the backend is told must name the key the deployment writes. + // A template test cannot see inside the asset zip, so assert the two are + // derived from the same constant rather than two that drifted. + expect(JSON.stringify(env.BROWSER_POLICY_S3)).toContain( + BROWSER_MANAGED_POLICY_KEY, + ); + }); +}); diff --git a/infrastructure/test/config.test.ts b/infrastructure/test/config.test.ts index 423f0534c..b3c856581 100644 --- a/infrastructure/test/config.test.ts +++ b/infrastructure/test/config.test.ts @@ -1,5 +1,5 @@ import * as cdk from 'aws-cdk-lib'; -import { loadConfig, AppConfig, +import { loadConfig, buildCorsOrigins, AppConfig, OBSERVABILITY_DEFAULT_AGENTCORE_ACTIVE_SESSION_THRESHOLD, OBSERVABILITY_DEFAULT_AGENTCORE_ERROR_THRESHOLD, OBSERVABILITY_DEFAULT_ALB_TARGET_5XX_THRESHOLD, @@ -194,6 +194,65 @@ describe('RAG Ingestion Configuration', () => { // Environment Variable Loading Tests // ============================================================ + describe('Browser URL blocklist (spec D6)', () => { + // This list is a security control: it is what stops a human in a browser + // takeover navigating to a system the agent must not act inside. It is + // supplied entirely from outside the repo (the stack ships fork-neutral), + // so the cases below pin the three behaviours that matter — empty by + // default, an override populates it, and an unset GitHub Actions variable + // (which arrives as '') is NOT mistaken for a deliberate empty list. + const BLOCKLIST_KEY = 'CDK_BROWSER_URL_BLOCKLIST'; + + afterEach(() => { + delete process.env[BLOCKLIST_KEY]; + }); + + test('is empty when unset — no institution-specific hosts are baked in', () => { + delete process.env[BLOCKLIST_KEY]; + + // The old build seeded `instructure.com` here. That is BSU's LMS policy, + // not a property of this stack, and a fork inherited it with no + // breadcrumb. It now lives in the per-environment variable. + expect(loadConfig(app).browser.urlBlocklist).toEqual([]); + }); + + test('an override populates it and is trimmed', () => { + process.env[BLOCKLIST_KEY] = 'one.example.com, two.example.com'; + + expect(loadConfig(app).browser.urlBlocklist).toEqual([ + 'one.example.com', + 'two.example.com', + ]); + }); + + test('an empty variable falls through to context, not to an empty list', () => { + // An unset GitHub Actions variable is forwarded as ''. If that were read + // as an explicit "block nothing" it would silently override a configured + // context default — a forgotten variable would disable the control and + // look identical in the log to a deliberate opt-out. + app.node.setContext('browser', { + urlBlocklist: ['from.context.example.com'], + }); + process.env[BLOCKLIST_KEY] = ''; + + expect(loadConfig(app).browser.urlBlocklist).toEqual([ + 'from.context.example.com', + ]); + }); + + test('a variable of only separators and spaces is treated as unset', () => { + app.node.setContext('browser', { + urlBlocklist: ['from.context.example.com'], + }); + process.env[BLOCKLIST_KEY] = ' , , '; + + expect(loadConfig(app).browser.urlBlocklist).toEqual([ + 'from.context.example.com', + ]); + }); + }); + + describe('Environment Variable Loading', () => { test('loads CORS origins from CDK_RAG_CORS_ORIGINS environment variable', () => { process.env.CDK_RAG_CORS_ORIGINS = 'https://example.com,https://test.com'; @@ -426,6 +485,31 @@ describe('RAG Ingestion Configuration', () => { // (same ternary as kbSync; empty workflow var must not disable) // ============================================================ + describe('Feedback eval sampling flag (opt-in)', () => { + test('defaults to DISABLED when CDK_FEEDBACK_EVAL_SAMPLING_ENABLED is unset', () => { + delete process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED; + expect(loadConfig(app).feedbackEvalSampling.enabled).toBe(false); + }); + + test('an empty string (unset workflow variable) stays disabled', () => { + process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED = ''; + expect(loadConfig(app).feedbackEvalSampling.enabled).toBe(false); + }); + + test('only the literal "true" enables it', () => { + process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED = 'true'; + expect(loadConfig(app).feedbackEvalSampling.enabled).toBe(true); + process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED = 'yes'; + expect(loadConfig(app).feedbackEvalSampling.enabled).toBe(false); + }); + + test('cdk.json context feedbackEvalSampling.enabled=true enables when env is unset', () => { + delete process.env.CDK_FEEDBACK_EVAL_SAMPLING_ENABLED; + app.node.setContext('feedbackEvalSampling', { enabled: true }); + expect(loadConfig(app).feedbackEvalSampling.enabled).toBe(true); + }); + }); + describe('Scheduled Runs feature flag', () => { test('defaults to enabled when CDK_SCHEDULED_RUNS_ENABLED is unset', () => { delete process.env.CDK_SCHEDULED_RUNS_ENABLED; @@ -1190,6 +1274,109 @@ describe('RAG Ingestion Configuration', () => { }); }); + // ============================================================ + // Uploads-bucket CORS guard + // ============================================================ + + describe('Top-level CORS origins guard', () => { + /** + * Without an origin the uploads bucket is created with no CORS rule and + * every browser upload fails at S3. This used to be a console.warn; a + * production-mirror environment shipped the bug that way + * (docs/specs/load-test-assessment-2026-09.md §1 fix 4). + */ + beforeEach(() => { + // The shared beforeEach seeds domainName; remove it so the top-level + // origin list is genuinely empty. + app = new cdk.App(); + app.node.setContext('projectPrefix', 'test-project'); + app.node.setContext('awsRegion', 'us-east-1'); + app.node.setContext('awsAccount', '123456789012'); + app.node.setContext('vpcCidr', '10.0.0.0/16'); + app.node.setContext('frontend', { cloudFrontPriceClass: 'PriceClass_100' }); + app.node.setContext('appApi', { cpu: 256, memory: 512, desiredCount: 1, maxCapacity: 2 }); + app.node.setContext('inferenceApi', {}); + app.node.setContext('fineTuning', {}); + app.node.setContext('artifacts', { retentionDays: 90, extraFrameAncestors: [] }); + app.node.setContext('mcpSandbox', { extraFrameAncestors: [] }); + app.node.setContext('ragIngestion', { + additionalCorsOrigins: '', + lambdaMemorySize: 10240, + lambdaTimeout: 900, + embeddingModel: 'amazon.titan-embed-text-v2', + vectorDimension: 1024, + vectorDistanceMetric: 'cosine', + }); + delete process.env.CDK_DOMAIN_NAME; + delete process.env.CDK_CORS_ORIGINS; + delete process.env.CDK_ALLOW_NO_CORS_ORIGINS; + }); + + afterEach(() => { + delete process.env.CDK_ALLOW_NO_CORS_ORIGINS; + delete process.env.CDK_CORS_ORIGINS; + delete process.env.CDK_DOMAIN_NAME; + }); + + test('fails synth when neither a domain nor CORS origins is configured', () => { + expect(() => loadConfig(app)).toThrow(/uploads bucket would be created without a CORS rule/); + }); + + test('passes when CDK_DOMAIN_NAME supplies the origin', () => { + process.env.CDK_DOMAIN_NAME = 'ai.example.edu'; + + const config = loadConfig(app); + + expect(config.corsOrigins).toBe('https://ai.example.edu'); + }); + + test('passes when CDK_CORS_ORIGINS supplies an origin without a domain', () => { + process.env.CDK_CORS_ORIGINS = 'http://localhost:4200'; + + expect(loadConfig(app).corsOrigins).toBe('http://localhost:4200'); + }); + + test('CDK_ALLOW_NO_CORS_ORIGINS=true is the explicit opt-out', () => { + process.env.CDK_ALLOW_NO_CORS_ORIGINS = 'true'; + + expect(loadConfig(app).corsOrigins).toBe(''); + }); + + test('a non-true opt-out value does not disable the guard', () => { + process.env.CDK_ALLOW_NO_CORS_ORIGINS = 'false'; + + expect(() => loadConfig(app)).toThrow(/CDK_ALLOW_NO_CORS_ORIGINS=true/); + }); + + /** + * The guard must gate on the same value the uploads bucket consumes: + * FileUploadConstruct attaches CORS only when buildCorsOrigins(config) + * is non-empty, and that filters out blank entries. A raw string that is + * truthy but filters to nothing (a trailing comma from a templated list, + * a YAML value that quotes to a single space, an unset CI variable) would + * otherwise pass the guard and still ship a bucket with no CORS rule -- + * the exact bug the guard exists to prevent. + */ + test.each([ + ['a lone separator', ','], + ['whitespace only', ' '], + ['separators only', ',,'], + ['padded separators', ' , '], + ])('fails synth when CDK_CORS_ORIGINS is %s and filters to no origins', (_label, value) => { + process.env.CDK_CORS_ORIGINS = value; + + expect(() => loadConfig(app)).toThrow(/uploads bucket would be created without a CORS rule/); + }); + + test('a blank entry alongside a real origin still passes and yields the real origin', () => { + process.env.CDK_CORS_ORIGINS = ' , https://ok.edu'; + + const config = loadConfig(app); + + expect(buildCorsOrigins(config)).toEqual(['https://ok.edu']); + }); + }); + // ============================================================ // Precedence Tests // ============================================================ @@ -1560,6 +1747,7 @@ describe('Observability Configuration', () => { a.node.setContext('awsRegion', 'us-east-1'); a.node.setContext('awsAccount', '123456789012'); a.node.setContext('vpcCidr', '10.0.0.0/16'); + a.node.setContext('corsOrigins', 'http://localhost:4200'); a.node.setContext('frontend', { cloudFrontPriceClass: 'PriceClass_100' }); a.node.setContext('appApi', { cpu: 256, memory: 512, desiredCount: 1, maxCapacity: 4, diff --git a/infrastructure/test/constructs.test.ts b/infrastructure/test/constructs.test.ts index 9f1856a5a..98e675e89 100644 --- a/infrastructure/test/constructs.test.ts +++ b/infrastructure/test/constructs.test.ts @@ -211,12 +211,33 @@ describe('CostTrackingTablesConstruct', () => { }); describe('AdminTablesConstruct', () => { - it('creates 4 DDB tables', () => { + it('creates 5 DDB tables', () => { const stack = testStack(); new AdminTablesConstruct(stack, 'Admin', { config: createMockConfig() }); const t = Template.fromStack(stack); - // user-settings, user-menu-links, announcements, system-prompts. - t.resourceCountIs('AWS::DynamoDB::Table', 4); + // user-settings, user-menu-links, announcements, system-prompts, + // agent-templates. + t.resourceCountIs('AWS::DynamoDB::Table', 5); + }); + + it('creates the agent-templates table with PK/SK and no GSI', () => { + // Mirrors SystemPromptsTable exactly: PK/SK string keys, AWS-managed + // encryption, no secondary index (the repository lists via Scan). + const stack = testStack(); + new AdminTablesConstruct(stack, 'Admin', { config: createMockConfig() }); + const t = Template.fromStack(stack); + const agentTemplates = Object.values(t.findResources('AWS::DynamoDB::Table')) + .filter((table: any) => + (table.Properties.TableName as string)?.includes('agent-templates'), + ); + expect(agentTemplates).toHaveLength(1); + const props = (agentTemplates[0] as any).Properties; + expect(props.KeySchema).toEqual([ + { AttributeName: 'PK', KeyType: 'HASH' }, + { AttributeName: 'SK', KeyType: 'RANGE' }, + ]); + expect(props.GlobalSecondaryIndexes).toBeUndefined(); + expect(props.SSESpecification).toEqual({ SSEEnabled: true }); // AWS_MANAGED (aws/dynamodb key) }); it('gives only the announcements table a TTL attribute', () => { diff --git a/infrastructure/test/helpers/mock-config.ts b/infrastructure/test/helpers/mock-config.ts index 947317a20..3fe7ed5cd 100644 --- a/infrastructure/test/helpers/mock-config.ts +++ b/infrastructure/test/helpers/mock-config.ts @@ -134,6 +134,7 @@ export function createMockConfig(overrides: Partial = {}): AppConfig memorySpaces: { enabled: false, }, + feedbackEvalSampling: { enabled: false }, skills: { enabled: false, }, @@ -155,6 +156,9 @@ export function createMockConfig(overrides: Partial = {}): AppConfig mcpSandbox: { extraFrameAncestors: [], }, + browser: { + urlBlocklist: ['blocked.example.com'], + }, mcpIdentity: { tokenEnrichment: { enabled: false, diff --git a/infrastructure/test/mcp-sandbox-csp-function.test.ts b/infrastructure/test/mcp-sandbox-csp-function.test.ts index 2ee3620e0..feacbe461 100644 --- a/infrastructure/test/mcp-sandbox-csp-function.test.ts +++ b/infrastructure/test/mcp-sandbox-csp-function.test.ts @@ -102,7 +102,7 @@ describe('buildCspHeader — default (no _meta.ui.csp)', () => { expect(csp).toContain("img-src 'self' data: blob:"); expect(csp).toContain("font-src 'self' data: blob:"); expect(csp).toContain("media-src 'self' data: blob:"); - expect(csp).toContain("connect-src 'self'"); + expect(csp).toContain("connect-src 'self' data: blob:"); expect(csp).toContain("worker-src 'self' blob:"); }); @@ -141,7 +141,7 @@ describe('buildCspHeader — declared domains', () => { expect(csp).toContain("img-src 'self' data: blob: https://esm.sh"); expect(csp).toContain("media-src 'self' data: blob: https://esm.sh"); expect(csp).toContain("worker-src 'self' blob: https://esm.sh"); - expect(csp).toContain('connect-src \'self\' https://esm.sh'); + expect(csp).toContain('connect-src \'self\' data: blob: https://esm.sh'); }); test('CesiumJS map-server: multiple domains on connect-src and resource-* directives', () => { @@ -162,7 +162,7 @@ describe('buildCspHeader — declared domains', () => { FRAME_ANCESTORS, ); expect(csp).toContain( - 'connect-src \'self\' https://*.openstreetmap.org https://cesium.com https://*.cesium.com', + 'connect-src \'self\' data: blob: https://*.openstreetmap.org https://cesium.com https://*.cesium.com', ); expect(csp).toContain( "script-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data: https://*.openstreetmap.org https://cesium.com https://*.cesium.com", @@ -199,7 +199,7 @@ describe('buildCspHeader — declared domains', () => { { connectDomains: ['https://api.example.com'] }, FRAME_ANCESTORS, ); - expect(csp).toContain('connect-src \'self\' https://api.example.com'); + expect(csp).toContain('connect-src \'self\' data: blob: https://api.example.com'); expect(csp).toContain("script-src 'self' 'unsafe-inline' 'unsafe-eval' blob: data:"); expect(csp).not.toMatch(/script-src[^;]*https:\/\/api\.example\.com/); }); @@ -216,7 +216,7 @@ describe('buildCspHeader — declared domains', () => { }, FRAME_ANCESTORS, ); - expect(csp).toContain('connect-src \'self\' https://good.com'); + expect(csp).toContain('connect-src \'self\' data: blob: https://good.com'); expect(csp).not.toContain('evil.com'); expect(csp).not.toContain('X-Injected'); // And the directive separator structure is intact. @@ -289,11 +289,11 @@ describe('handler', () => { const result = handler(event); expect(result.headers['content-security-policy']).toBeDefined(); expect(result.headers['content-security-policy'].value).toContain( - "connect-src 'self'", + "connect-src 'self' data: blob:", ); // Default → no resource domains beyond keywords/blob/data expect(result.headers['content-security-policy'].value).not.toMatch( - /connect-src 'self' \S+/, + /connect-src 'self' data: blob: \S+/, ); }); @@ -313,7 +313,7 @@ describe('handler', () => { }; const result = handler(event); expect(result.headers['content-security-policy'].value).toContain( - 'connect-src \'self\' https://esm.sh', + 'connect-src \'self\' data: blob: https://esm.sh', ); expect(result.headers['content-security-policy'].value).not.toBe( "default-src 'self'", @@ -323,10 +323,14 @@ describe('handler', () => { test('with malformed ?csp=, falls back to default without throwing', () => { const event = makeEvent({ csp: { value: 'not-json' } }); const result = handler(event); - expect(result.headers['content-security-policy'].value).toContain("connect-src 'self'"); - expect(result.headers['content-security-policy'].value).not.toMatch( - /connect-src 'self' \S/, - ); + // The point is that malformed input injects NO domains. `data:` and + // `blob:` are unconditional constants (the DCV SDK fetches its WASM from + // a data: URI), so pin the directive exactly rather than asserting that + // nothing follows 'self'. + const connectSrc = result.headers['content-security-policy'].value + .split('; ') + .find((d: string) => d.startsWith('connect-src')); + expect(connectSrc).toBe("connect-src 'self' data: blob:"); }); test('always emits frame-ancestors so framing control is not lost on the dynamic path', () => { @@ -355,3 +359,30 @@ describe('handler', () => { expect(result.headers['content-security-policy']).toBeDefined(); }); }); + +describe('connect-src must allow the DCV decoder', () => { + test("data: and blob: are present even with no declared connectDomains", () => { + // The browser sign-in live view loads the Amazon DCV Web Client SDK, which + // fetches its WebAssembly decoder from an inline `data:` URI. Measured on + // dev: without this the browser blocks it and DCV fails with + // "Display channel is not available", so the viewer shows a black frame. + // Every sibling directive already allows data:/blob:; connect-src did not. + const csp = buildCspHeader({}, "'none'"); + const connectSrc = csp.split('; ').find((d: string) => d.startsWith('connect-src')); + + expect(connectSrc).toContain('data:'); + expect(connectSrc).toContain('blob:'); + }); + + test('declared connect domains are still appended, not replaced', () => { + const csp = buildCspHeader( + { connectDomains: ['https://bedrock-agentcore.us-west-2.amazonaws.com'] }, + "'none'", + ); + const connectSrc = csp.split('; ').find((d: string) => d.startsWith('connect-src')); + + expect(connectSrc).toBe( + "connect-src 'self' data: blob: https://bedrock-agentcore.us-west-2.amazonaws.com", + ); + }); +}); diff --git a/infrastructure/test/observability-dynamodb-alarms.test.ts b/infrastructure/test/observability-dynamodb-alarms.test.ts index 1bb1fbb51..b49c50c60 100644 --- a/infrastructure/test/observability-dynamodb-alarms.test.ts +++ b/infrastructure/test/observability-dynamodb-alarms.test.ts @@ -41,8 +41,8 @@ describe('DynamoDB per-table alarms', () => { // Tied to the real table count, so a new table without an alarm fails here. it('covers every table in the stack — one throttle alarm each', () => { - expect(tableCount).toBe(27); - // 27 per-table throttle alarms + 1 account-level UserErrors alarm. + expect(tableCount).toBe(28); + // 28 per-table throttle alarms + 1 account-level UserErrors alarm. expect(ddbAlarmNames()).toHaveLength(tableCount + 1); }); diff --git a/infrastructure/test/observability-platform-dashboard.test.ts b/infrastructure/test/observability-platform-dashboard.test.ts index 9711bb126..264c57e24 100644 --- a/infrastructure/test/observability-platform-dashboard.test.ts +++ b/infrastructure/test/observability-platform-dashboard.test.ts @@ -47,14 +47,25 @@ describe('Unified platform dashboard', () => { }); }); - // CloudWatch charges $3/month beyond three. - it('keeps the stack at exactly three dashboards (the CloudWatch free ceiling)', () => { - template.resourceCountIs('AWS::CloudWatch::Dashboard', 3); + // CloudWatch's free tier is three dashboards and charges $3/month beyond it. + // The count is pinned so that crossing the tier stays a deliberate trade + // rather than a side effect of someone adding widgets. + // + // It was crossed once, on purpose: the fourth is + // `turn-latency-observability` (docs/specs/turn-latency-preamble.md), which + // is read while shipping a latency change, side by side with load-test + // output. Folding it into the AgentCore Runtime board was tried first and + // buried the stage breakdown under runtime-health widgets answering an + // unrelated question. $3/month against a 450-900ms wait on every turn is not + // a close call — but the next one should be argued the same way. + it('keeps the stack at exactly four dashboards (three free + one bought)', () => { + template.resourceCountIs('AWS::CloudWatch::Dashboard', 4); }); - it('links to the two drill-down dashboards instead of duplicating them', () => { + it('links to the three drill-down dashboards instead of duplicating them', () => { expect(dashboardBody).toContain(`${MOCK_PREFIX}-agentcore-observability`); expect(dashboardBody).toContain(`${MOCK_PREFIX}-prompt-cache-observability`); + expect(dashboardBody).toContain(`${MOCK_PREFIX}-turn-latency-observability`); }); it('names the SNS topic alarms route to, so an operator can find it', () => { diff --git a/infrastructure/test/platform-stack.test.ts b/infrastructure/test/platform-stack.test.ts index aff99fbf7..ac0e2390f 100644 --- a/infrastructure/test/platform-stack.test.ts +++ b/infrastructure/test/platform-stack.test.ts @@ -152,18 +152,20 @@ describe('PlatformStack', () => { describe('DynamoDB tables', () => { it('creates all shared tables', () => { - // 27 tables. Was 26 — the announcements table was added for the - // feature-announcement system (admin-authored notices + per-user - // acknowledgement rows). Prior note: the audit-log table was added - // for the administrative audit trail (delegated admin scopes, PR-5); - // before that the memory-spaces table was added for the Memory Spaces - // feature; before that the system-prompts table was added for - // admin-managed Conversation Modes (custom system prompt catalog); - // previously 24 before the standalone "assistants" table was - // decommissioned (the python app uses rag-assistants for both - // assistant config and document metadata via + // 28 tables. Was 27 — the agent-templates table was added for the + // admin-managed Agent Templates catalog (mirrors system-prompts, + // read by app_api only). Before that: the announcements table was + // added for the feature-announcement system (admin-authored notices + // + per-user acknowledgement rows). Prior note: the audit-log table + // was added for the administrative audit trail (delegated admin + // scopes, PR-5); before that the memory-spaces table was added for + // the Memory Spaces feature; before that the system-prompts table + // was added for admin-managed Conversation Modes (custom system + // prompt catalog); previously 24 before the standalone "assistants" + // table was decommissioned (the python app uses rag-assistants for + // both assistant config and document metadata via // DYNAMODB_ASSISTANTS_TABLE_NAME). - template.resourceCountIs('AWS::DynamoDB::Table', 27); + template.resourceCountIs('AWS::DynamoDB::Table', 28); }); }); @@ -173,8 +175,10 @@ describe('PlatformStack', () => { // artifacts-content, skill-resources (admin-managed Skills reference files), // memory-spaces (Memory Spaces feature content bucket), // shared-conversations (share snapshot-body offload), - // alb-access-logs (who terminated a connection — SSE disconnect attribution) - template.resourceCountIs('AWS::S3::Bucket', 10); + // alb-access-logs (who terminated a connection — SSE disconnect attribution), + // browser-policy (the Chromium MANAGED policy every browser session + // starts with — spec D6) + template.resourceCountIs('AWS::S3::Bucket', 11); }); }); @@ -228,9 +232,13 @@ describe('PlatformStack', () => { // CDK generates their physical names, which means the backend // workflow's `update-function-code` step has no way to find them // except through SSM — the deploy-time-discovery bucket above. + // + // Raised 49 → 51 for the agent-templates table name + ARN publishes + // (mirrors the system-prompts name+arn pair; consumed by restore + // tooling and ad-hoc IAM scoping). const params = template.findResources('AWS::SSM::Parameter'); expect(Object.keys(params).length).toBeGreaterThanOrEqual(30); - expect(Object.keys(params).length).toBeLessThanOrEqual(49); + expect(Object.keys(params).length).toBeLessThanOrEqual(51); }); }); diff --git a/infrastructure/test/runtime-env-var-limit.test.ts b/infrastructure/test/runtime-env-var-limit.test.ts index d1af2f7a0..c323c29a1 100644 --- a/infrastructure/test/runtime-env-var-limit.test.ts +++ b/infrastructure/test/runtime-env-var-limit.test.ts @@ -41,7 +41,7 @@ describe('AgentCore Runtime environment variable ceiling', () => { // minimal config under-counts and the guard would report headroom that no // real environment has. `tokenExchange` alone adds three // (TOKEN_EXCHANGE_URL / _CLIENT_ID / _SECRET_ID) — without it this test - // counts 47 while prod and dev deploy 50. Any future conditional block + // counts 44 while prod and dev deploy 47. Any future conditional block // must be enabled here too, or this guard quietly stops guarding. const config = createMockConfig({ domainName: 'example.com', diff --git a/infrastructure/test/security-policy.test.ts b/infrastructure/test/security-policy.test.ts index 045659469..f9eaf1d2d 100644 --- a/infrastructure/test/security-policy.test.ts +++ b/infrastructure/test/security-policy.test.ts @@ -277,6 +277,30 @@ describe('Security policy hardening', () => { }); }); + describe('App-api agent-templates grant', () => { + // The admin CRUD routes and the public /templates picker feed both run on + // app-api and read/write this table. Mirrors SystemPromptsTableAccess and + // must stay scoped — never Action:* / Resource:*. + it('app-api role has scoped read/write on the agent-templates table', () => { + const matches = statementsWithSid('AgentTemplatesTableAccess'); + if (matches.length === 0) { + throw new Error( + "Could not locate the app-api agent-templates grant. " + + "Looked for Sid 'AgentTemplatesTableAccess'. If the Sid was renamed, update this test.", + ); + } + for (const s of matches) { + const actions = asArray(s.Action); + for (const a of ['dynamodb:GetItem', 'dynamodb:PutItem', 'dynamodb:UpdateItem', + 'dynamodb:DeleteItem', 'dynamodb:Query', 'dynamodb:Scan']) { + expect(actions).toContain(a); + } + const resources = asArray(s.Resource); + expect(resources).not.toContain('*'); + } + }); + }); + describe('AgentCore runtime user-settings grant', () => { // Regression guard: inference-agentcore-construct.ts injects // DYNAMODB_USER_SETTINGS_TABLE_NAME, which makes UserSettingsRepository diff --git a/infrastructure/test/turn-latency-observability.test.ts b/infrastructure/test/turn-latency-observability.test.ts new file mode 100644 index 000000000..f25eb4ac3 --- /dev/null +++ b/infrastructure/test/turn-latency-observability.test.ts @@ -0,0 +1,117 @@ +import * as cdk from 'aws-cdk-lib'; +import { Template } from 'aws-cdk-lib/assertions'; + +import { TurnLatencyObservabilityConstruct } from '../lib/constructs/observability/turn-latency-observability-construct'; +import { createMockConfig, MOCK_ACCOUNT, MOCK_PREFIX, MOCK_REGION } from './helpers/mock-config'; + +// Runtime *id* with its AWS-assigned suffix + endpoint qualifier, not the +// project prefix — a prefix-named group exists in no account and returns zero +// rows silently, which reads as "no traffic". Same fixture shape as the +// prompt-cache suite, guarding the same mistake. +const MOCK_RUNTIME_LOG_GROUP = + `/aws/bedrock-agentcore/runtimes/${MOCK_PREFIX}_agentcore_runtime-AbC123XyZ0-DEFAULT`; + +function synth(): Template { + const app = new cdk.App(); + const stack = new cdk.Stack(app, 'Test', { + env: { account: MOCK_ACCOUNT, region: MOCK_REGION }, + }); + new TurnLatencyObservabilityConstruct(stack, 'TurnLatencyObservability', { + config: createMockConfig(), + runtimeLogGroupName: MOCK_RUNTIME_LOG_GROUP, + }); + return Template.fromStack(stack); +} + +describe('TurnLatencyObservabilityConstruct', () => { + let t: Template; + let body: string; + + beforeAll(() => { + t = synth(); + const dashboards = t.findResources('AWS::CloudWatch::Dashboard'); + body = JSON.stringify(Object.values(dashboards)[0].Properties.DashboardBody); + }); + + it('creates the dashboard with the conventional name', () => { + t.hasResourceProperties('AWS::CloudWatch::Dashboard', { + DashboardName: `${MOCK_PREFIX}-turn-latency-observability`, + }); + }); + + it('graphs every stage turn_timing.py emits', () => { + // A name that drifts from `_metric_name()` renders an EMPTY graph rather + // than erroring — indistinguishable from "that stage never ran". + for (const metric of [ + 'PreludeTotalMs', + 'PreambleMs', + 'PreambleOwnershipMs', + 'PreambleSkillsMs', + 'PreambleFilesMs', + 'PreambleSessionStateMs', + 'PreambleQuotaMs', + 'RagMs', + 'ToolsMs', + 'AgentBuildMs', + // Sub-stages of the build (docs/specs/turn-latency-preamble.md PR-4). + 'AgentBuildPromptMs', + 'AgentBuildRegistryMs', + 'AgentBuildSessionMgrMs', + 'AgentBuildToolsMs', + 'AgentBuildHooksMs', + 'AgentBuildPluginsMs', + 'AgentBuildFinalizeMs', + 'AgentBuildRestMs', + ]) { + expect(body).toContain(metric); + } + }); + + it('uses the namespace turn_timing.py defaults to', () => { + // CDK does not set TURN_LATENCY_EMF_NAMESPACE (dev and prod are separate + // accounts), so its default IS the contract between the two sides. + expect(body).toContain('AgentCoreStack/TurnLatency'); + }); + + it('plots percentiles, never Sum — a summed latency is meaningless', () => { + expect(body).toContain('p50'); + expect(body).toContain('p90'); + expect(body).toContain('p99'); + expect(body).not.toContain('"stat":"Sum"'); + }); + + it('queries the runtime log group for the slices dimensions would have made', () => { + expect(body).toContain(MOCK_RUNTIME_LOG_GROUP); + // isResume / deferredBuild ride as EMF log properties rather than metric + // dimensions: dimensions multiply metric streams and invite reading a p99 + // off a slice too thin to have one. + expect(body).toContain('isResume'); + expect(body).toContain('deferredBuild'); + }); + + it('keeps an unaccounted-time widget, so an incomplete decomposition shows', () => { + expect(body).toContain('unaccountedMs'); + }); + + it('carries the "this cannot see the app-api hop" caveat on the dashboard itself', () => { + // The caveat has to live where the number is read. A reader who sees + // PreambleMs fall and concludes the user waits less is wrong by up to + // ~1.5s of routing this namespace is structurally blind to, and a comment + // in the CDK source does not reach them. + expect(body).toContain('handler entry'); + expect(body).toContain('tests/load'); + }); + + it('points at the AgentCore board for the AWS-measured comparison', () => { + // PreludeTotalMs vs AWS's data-plane Latency is how the routing overhead + // becomes visible; a reader who does not know the companion exists cannot + // make that comparison. + expect(body).toContain(`${MOCK_PREFIX}-agentcore-observability`); + }); + + it('creates no alarms — there is no baseline to threshold against yet', () => { + // Deliberate. Inventing a threshold before the dashboard has run is the + // guessing docs/specs/turn-latency-preamble.md exists to prevent. + t.resourceCountIs('AWS::CloudWatch::Alarm', 0); + }); +}); diff --git a/scripts/build/build-one.sh b/scripts/build/build-one.sh index 075bd3361..efad30a82 100755 --- a/scripts/build/build-one.sh +++ b/scripts/build/build-one.sh @@ -145,6 +145,7 @@ case "$SERVICE" in "backend/src/apis/shared/timestamps.py" "backend/src/apis/shared/dynamo_errors.py" "backend/src/apis/shared/feature_flags.py" + "backend/src/apis/shared/aws_clients.py" ) # Both kb-sync Lambdas are arm64 (see the kb-sync CDK construct). PLATFORM="linux/arm64" @@ -192,6 +193,7 @@ case "$SERVICE" in "backend/src/apis/shared/scheduled_prompts" "backend/src/apis/shared/sessions_bff" "backend/src/apis/shared/sessions" + "backend/src/apis/shared/browser_takeover" "backend/src/apis/shared/storage" "backend/src/apis/shared/observability" ) diff --git a/scripts/build/fetch-dcv-sdk.sh b/scripts/build/fetch-dcv-sdk.sh new file mode 100755 index 000000000..1a805ecab --- /dev/null +++ b/scripts/build/fetch-dcv-sdk.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# +# Fetch and verify the Amazon DCV Web Client SDK for the browser-takeover +# live view (docs/specs/authenticated-web-assessment.md D3). +# +# WHY THIS IS A BUILD STEP AND NOT A VENDORED DIRECTORY +# ----------------------------------------------------- +# The SDK is a EULA-licensed AWS download, not an open-source package, and this +# repository is PUBLIC. Committing the archive would be redistribution. AWS's +# own instruction is to "place the extracted directory on your web server", so +# we fetch it at build time and deploy it to the sandbox origin. Nothing +# licensed lands in git, and the signature is checked on every build rather +# than trusted once at commit time. +# +# It is deliberately NOT on npm: the only `dcv` package on the public registry +# is an unrelated third-party Vue component library, and `dcv-ui` does not +# exist at all. `bedrock-agentcore`'s BrowserLiveView React component imports +# both as undeclared dependencies, which is why we do not use it. +# +# WHAT ACTUALLY MAKES THIS SAFE +# ------------------------------ +# Not "it's from AWS" — vendors ship compromised artifacts too. Three pins do: +# +# 1. VERSION — an exact archive, so the bytes cannot change under us. +# 2. ARCHIVE_SHA256 — pinned HERE, deliberately not read from the published +# .sha256sum: whoever could swap the archive on the CDN could swap that +# file too. Changing it requires a reviewed diff. +# 3. GPG_FINGERPRINT — pinned for the same reason. Importing the key from +# the network and trusting whatever it serves *today* is trust-on-first- +# use on every build, which is verification theater. We import into a +# throwaway keyring and assert the fingerprint before verifying anything. +# +# ⚠️ The fingerprint below was captured from AWS's published key and pinned at +# review time; AWS does not appear to publish it out-of-band, so it is +# trust-on-first-use ONCE, caught by code review thereafter. If you can obtain +# it through a second channel, do, and note it here. +# +# Residual risk worth stating: dcv.js runs in the viewer page and forwards the +# user's keystrokes to the remote browser, so a compromised SDK could capture a +# password. That is exactly why the page is served from the sandbox origin, +# which holds no session cookie, rather than from the SPA. +# +set -euo pipefail + +VERSION="1.13.2-1074" +ARCHIVE="nice-dcv-web-client-sdk-${VERSION}.zip" +BASE_URL="https://d1uj6qtbmh3dt5.cloudfront.net" +ARCHIVE_URL="${BASE_URL}/webclientsdk/${ARCHIVE}" +SIGNATURE_URL="${ARCHIVE_URL}.sign" +GPG_KEY_URL="${BASE_URL}/NICE-GPG-KEY" + +ARCHIVE_SHA256="31933c6b692983cfeb7fb2ca47d3a76433e61eaee84feb710aecca17919a4dee" +GPG_FINGERPRINT="5B9EEBC864449701F6CE566A11B5C70A170C6114" + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +# The UMD build: a plain